fix(webapp,run-engine,scripts): harden v2 cross-table tests and tighten the recovery query

- updateMetadata cross-table test: wrap the body in try/finally so stopFlushing
  always runs and the flush loop cannot bleed into later tests on a failure path.
- cancelling cross-table cancel-cascade test: poll for the child CANCELED status
  with a deadline instead of a fixed 1s sleep, to de-flake it under slow CI.
- recover-stuck-runs: constrain each UNION branch by id = ANY(runIds) so the
  recovery query scans only candidate rows instead of unioning both full tables
  before the join.

.changeset/agent-skills-bundled-in-sdk.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/sdk": patch
+"trigger.dev": patch
+---
+
+`@trigger.dev/sdk` now bundles the Trigger.dev agent skills and a curated snapshot of the docs those skills reference. The skills that `trigger skills` installs into your coding agent read this content from node_modules, so the guidance your AI assistant follows is pinned to the SDK version installed in your project and stays current across upgrades instead of going stale until the next reinstall.

.changeset/ai-sdk-7-support.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Adds AI SDK 7 support. The `ai` peer range now includes v7, and the `chat.agent` / chat surfaces work against v7's ESM-only build. On v7, install `@ai-sdk/otel` alongside `ai` and the SDK registers it for you so `experimental_telemetry` spans keep flowing into your run traces (v7 stopped emitting them from `ai` core). v5 and v6 keep working unchanged.

.changeset/backpressure-scale-up-freeze.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Add optional `shouldPauseScaling` to the supervisor consumer pool scaling options to freeze scale-up while it returns true (scale-down stays allowed).

.changeset/chat-agent-hardening.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/sdk": patch
+"@trigger.dev/core": patch
+---
+
+Reliability fixes for `chat.agent`. A user message sent while the agent is streaming is no longer delivered twice (which could run a duplicate turn), input appends now carry an idempotency key so a retried send can't duplicate a message, stopping a generation clears the streaming state so a page reload doesn't replay the stopped turn, and runs can now carry the full set of dashboard tags instead of being silently truncated. `onTurnComplete` now fires on errored turns (with the thrown error attached) and the failed turn's user message is persisted so it isn't lost on the next run. Custom agents and manual `chat.writeTurnComplete` callers now trim the output stream, sending a custom action no longer leaves a second stream reader running, and a long-lived `watch` subscription no longer grows its dedupe set without bound.

.changeset/chat-agent-tools.md

@@ -0,0 +1,15 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Add a `tools` option to `chat.agent`. Declaring your tools here threads them into the SDK's internal `convertToModelMessages`, so each tool's `toModelOutput` is re-applied when prior-turn history is re-converted.
+
+```ts
+chat.agent({
+  tools: { readFile, search },
+  run: async ({ messages, tools, signal }) =>
+    streamText({ model, messages, tools, abortSignal: signal }),
+});
+```
+
+Also exports `InferChatUIMessageFromTools<typeof tools>` to derive the chat `UIMessage` type (typed tool parts) directly from a tool set.

.changeset/chat-boot-cursor.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/sdk": patch
+"@trigger.dev/core": patch
+---
+
+Continuation chat boots no longer stall for around 10 seconds before the first turn. The `session.in` resume cursor is now found with a non-blocking records read instead of draining an SSE long-poll (which always waited out its full 5 second inactivity window, twice per boot), the boot reads run concurrently, and chat snapshots carry the cursor so subsequent boots skip the scan entirely.

.changeset/chat-headstart-custom-backends.md

@@ -0,0 +1,26 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+`chat.headStart` now works with the `chat.customAgent` and `chat.createSession` backends, not only `chat.agent`. The warm step-1 response hands over to your loop the same way it does for a managed agent.
+
+In a `chat.customAgent` loop, consume the handover on turn 0:
+
+```ts
+const conversation = new chat.MessageAccumulator();
+const { isFinal, skipped } = await conversation.consumeHandover({ payload });
+if (skipped) return; // warm handler aborted, so exit without a turn
+if (isFinal) {
+  await chat.writeTurnComplete(); // step 1 is the response, no streamText
+} else {
+  const result = streamText({ model, messages: conversation.modelMessages, tools });
+  // Pass originalMessages so the handed-over tool round merges into the
+  // step-1 assistant instead of starting a new message.
+  const response = await chat.pipeAndCapture(result, {
+    originalMessages: conversation.uiMessages,
+  });
+  if (response) await conversation.addResponse(response);
+}
+```
+
+With `chat.createSession`, the iterator surfaces it as `turn.handover`; call `turn.complete()` with no argument on a final handover. The lower-level `chat.waitForHandover()` and `accumulator.applyHandover()` are also exported for hand-rolled loops.

.changeset/chat-headstart-hydrate.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Fix `chat.headStart` when `hydrateMessages` is registered. The warm route's step-1 partial now reaches the agent's accumulator on the hydrate path, so `onTurnComplete` carries the full first turn (the head-start user message included), tool-call handovers resume from step 2 instead of re-running step 1, and the assistant `messageId` stays stable across the handover.

.changeset/chat-headstart-reasoning.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Preserve reasoning parts across the `chat.headStart` handover. Extended-thinking models' step-1 reasoning now lands in the durable session history (and `onTurnComplete`) under the same assistant `messageId`, with provider metadata intact so Anthropic thinking signatures survive replays.

.changeset/chat-headstart-trigger-config.md

@@ -0,0 +1,15 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Add `triggerConfig` support to `chat.headStart()` and `chat.openSession()`, so the auto-triggered handover-prepare run inherits tags, queue, machine, and other session trigger options the same way `chat.createStartSessionAction()` does. The `chat:{chatId}` tag is prepended automatically.
+
+```ts
+export const POST = chat.headStart({
+  agentId: "my-agent",
+  triggerConfig: { tags: ["org:acme"], queue: "chat" },
+  run: async ({ chat }) => streamText({ ...chat.toStreamTextOptions(), model }),
+});
+```
+
+Because the session is created once on the first head-start turn and is idempotent on the chat id, this is the only place to set those options for a head-start chat's lifetime. `chat.createStartSessionAction()` now also forwards `maxDuration`, `region`, and `lockToVersion` so both session entry points stay consistent.

.changeset/chat-slim-wire-merge.md

@@ -0,0 +1,31 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two changes that work together:
+
+- The per-turn merge now overlays the wire copy's tool-part state advancement onto the agent's existing chain — `state` + the matching resolution field (`output` / `errorText` / `approval`) come from the wire, everything else (text, reasoning, tool `input`, provider metadata) stays whatever the snapshot or `hydrateMessages` returned. Previously a full-message replace overwrote those fields with whatever the client shipped, so a slimmed wire copy landed a tool call with no `arguments` on the next LLM call. Covers `output-available` / `output-error` (HITL `addToolOutput`) and `approval-responded` / `output-denied` (approval flow).
+- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim assistant messages that carry advanced tool parts. The wire payload is just `{ id, role, parts: [<state + resolution field>] }` for `submit-message` continuations; everything else passes through. Reasoning blobs and full tool inputs no longer ride the wire on every `addToolOutput` / `addToolApproveResponse`, so continuation payloads stay well under the `.in/append` cap on long agent loops.
+
+Note: `onValidateMessages` receives the slim wire on HITL turns. If you call `validateUIMessages` from `ai` against the full `messages` array it will reject the slim assistant; filter to user messages (or skip on HITL turns) — see the updated docstring on `onValidateMessages` for the recommended pattern.
+
+For `hydrateMessages` hooks that persist the chain, this release also adds a small helper to the `@trigger.dev/sdk/ai` surface:
+
+```ts
+import { chat, upsertIncomingMessage } from "@trigger.dev/sdk/ai";
+
+chat.agent({
+  hydrateMessages: async ({ chatId, trigger, incomingMessages }) => {
+    const record = await db.chat.findUnique({ where: { id: chatId } });
+    const stored = record?.messages ?? [];
+    if (upsertIncomingMessage(stored, { trigger, incomingMessages })) {
+      await db.chat.update({ where: { id: chatId }, data: { messages: stored } });
+    }
+    return stored;
+  },
+});
+```
+
+It pushes fresh user messages by id, no-ops on HITL continuations (the incoming shares an id with the existing assistant — the runtime overlays the new tool-state advance), and skips on non-`submit-message` triggers. Returns `true` if it mutated `stored` so the caller knows whether to persist.
+
+Net effect: `chat.addToolOutput(...)` / `chat.addToolApproveResponse(...)` on multi-step reasoning agents (OpenAI Responses with `store: false`, Anthropic extended thinking, etc.) no longer blows the cap and no longer corrupts the LLM input.

.changeset/chat-system-prompt-caching.md

@@ -0,0 +1,25 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Cache your chat agent's system prompt with Anthropic prompt caching. `chat.toStreamTextOptions()` now emits the system prompt as a cacheable message when you opt in, so a large, stable system block is billed at cache-read rates on every turn instead of full price.
+
+```ts
+// at the streamText call site (Anthropic sugar)
+streamText({
+  ...chat.toStreamTextOptions({ cacheControl: { type: "ephemeral" } }),
+  messages,
+});
+
+// provider-agnostic equivalent
+chat.toStreamTextOptions({
+  systemProviderOptions: { anthropic: { cacheControl: { type: "ephemeral" } } },
+});
+
+// or where the prompt is defined
+chat.prompt.set(SYSTEM_PROMPT, {
+  providerOptions: { anthropic: { cacheControl: { type: "ephemeral" } } },
+});
+```
+
+Without an option, `system` stays a plain string. Pairs with a `prepareMessages` cache breakpoint to cache the conversation prefix across turns too.

.changeset/chat-transport-recreate-missing-session.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+`useTriggerChatTransport` now recovers when restored session state points at a session that no longer exists in the current environment

.changeset/cli-dev-without-project.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+Running a CLI command like `dev`, `deploy`, `preview`, or `update` before initializing a project no longer crashes with a raw `Cannot find matching package.json` stack trace. The CLI now detects the missing project and points you to `npx trigger.dev@latest init` instead.

.changeset/cli-init-ai-tooling.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+`trigger init` now sets up your AI coding assistant as part of project setup: pick the MCP server, the agent skills, or both, then scaffold with the CLI or hand off to your assistant. Adds a new `getting-started` agent skill that teaches assistants how to bootstrap Trigger.dev (install the SDK, write `trigger.config.ts`, create a first task, run `trigger dev`), so the AI-driven setup path works end to end. It ships in the CLI alongside the existing skills, version-matched to your SDK.

.changeset/coerce-concurrency-key-to-string.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Coerce numeric `concurrencyKey` values to string at the API boundary across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming batch endpoint.

.changeset/create-session-stop-continuation.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Fix two `chat.createSession()` bugs: stopping a generation no longer wedges the run (the turn loop raced a `totalUsage` promise that never settles after a stop-abort), and continuation runs now wait for the next message instead of invoking the model with an empty prompt.

.changeset/custom-agent-loop-fixes.md

@@ -0,0 +1,9 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Three fixes for custom agent loops (`chat.customAgent`, `chat.createSession`, and hand-rolled `MessageAccumulator` loops):
+
+- Continuation runs no longer replay already-answered user messages into the first turn. The `.in` resume cursor is now seeded before any listener attaches (the same boot logic `chat.agent` uses), so a chat that continues after a cancel, crash, or upgrade only sees genuinely new messages.
+- Steering a hand-rolled loop mid-stream no longer wipes the in-flight assistant response. `chat.pipeAndCapture` now stamps a server-generated message id on the stream, so a `prepareStep` injection keeps the partial text instead of replacing the message.
+- Task-backed tools (`ai.toolExecute`) now work from custom agent loops: the parent's session is threaded to the child run, so child tasks can stream progress into the chat with `chat.stream.writer({ target: "root" })` instead of failing with "session handle is not initialized".

.changeset/dequeue-latency-histogram.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Record client-side dequeue API latency in the supervisor consumer pool as a Prometheus histogram (`queue_consumer_pool_dequeue_duration_seconds`, labelled by `outcome`: success/empty/error).

.changeset/duplicate-task-ids.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/core": patch
+"trigger.dev": patch
+---
+
+`dev` and `deploy` now fail with a clear error when two tasks are defined with the same id, including across different task types (e.g. a scheduled task and a regular task sharing an id). Previously the second definition silently overwrote the first, so one of the tasks would vanish with no warning. Task ids are detected as duplicates during indexing (naming each offending id and the files it was found in), and the same rule is enforced server-side when the background worker is registered.

.changeset/env-vars-tracing-forceflush-typecheck.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Fix `@trigger.dev/core` build: cast the underlying log record exporter when calling `forceFlush` so it typechecks against the updated OpenTelemetry `LogRecordExporter` type (which no longer declares `forceFlush`).

.changeset/envvars-import-is-secret.md

@@ -0,0 +1,12 @@
+---
+"@trigger.dev/core": patch
+---
+
+`envvars.upload` now accepts an optional `isSecret` flag, letting you create the imported variables as secret (redacted) environment variables. When omitted, variables default to non-secret.
+
+```ts
+await envvars.upload("proj_1234", "prod", {
+  variables: { STRIPE_SECRET_KEY: "sk_live_..." },
+  isSecret: true,
+});
+```

.changeset/errors-api-schemas.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Add request and response schemas for the new Errors API (error groups). These back the env-scoped HTTP endpoints for listing error groups, retrieving a single group, and changing its state (resolve, ignore, unresolve), plus a `filter[error]` option on the runs list to fetch the runs behind a group. Exported from `@trigger.dev/core/v3` so the SDK can reuse them.

.changeset/httpserver-skip-body-parsing.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Add an optional `skipBodyParsing` flag to the internal HTTP server route definition, letting a route respond without reading or parsing the request body.

.changeset/large-trigger-payload-offload.md

@@ -0,0 +1,8 @@
+---
+"@trigger.dev/core": patch
+"@trigger.dev/sdk": patch
+---
+
+Offload large trigger payloads to object storage before sending the trigger API request. The SDK uploads packets at or above the existing 128KB limit and sends an `application/store` pointer instead of embedding large JSON in the request body. `TriggerTaskRequestBody` now validates that `application/store` payloads are non-empty storage paths.
+
+Payload uploads use the same resolved `ApiClient` as the trigger call (including `requestOptions.clientConfig`), not only the global `apiClientManager.client` — so custom `baseURL`, access token, and preview branch apply to both presign and trigger.

.changeset/mcp-trigger-task-no-default-wait.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+The MCP server no longer tells the AI agent to wait for a run to complete after every `trigger_task` call. Waiting is now opt-in: the agent only waits when you ask it to (for example "trigger and then wait for it to finish"). This avoids burning tokens polling runs you didn't need to block on and keeps responses clearer.

.changeset/mint-token-command.md

@@ -0,0 +1,9 @@
+---
+"trigger.dev": patch
+---
+
+Adds `trigger.dev mint-token`, which mints a short-lived delegated token from your stored personal access token. The token authenticates against the API as you, can be narrowed with `--cap` and given a lifetime with `--ttl`, and prints to stdout so it can be captured.
+
+```bash
+UAT=$(trigger.dev mint-token --ttl 3600 --cap read:runs)
+```

.changeset/mollifier-buffer-pipeline-list-entries.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/redis-worker": patch
+---
+
+Pipeline the per-entry `HGETALL` fetches in `MollifierBuffer.listEntriesForEnv`. The previous serial implementation issued one Redis round-trip per runId returned by `LRANGE`, which dominated stale-sweep wall-time at any meaningful backlog (at the sweep's default maxCount=1000, this is ~1000 RTTs per env per pass). Behaviour is unchanged — entries are still skipped when the entry hash has been torn down by a concurrent drainer ack/fail between the LRANGE and the HGETALL.

.changeset/mollifier-configurable-constants.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/redis-worker": patch
+---
+
+Make mollifier buffer and drainer internals configurable. `MollifierBuffer` now accepts `ackGraceTtlSeconds`, `maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options, and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All default to their previous hardcoded values, so existing behaviour is unchanged.

.changeset/mollifier-drain-batch-size.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/redis-worker": patch
+---
+
+`MollifierDrainer` accepts a `drainBatchSize` option (default 1) that controls how many entries are popped per env per tick — in-flight handlers remain capped by the global `concurrency`. `MollifierBuffer` also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new `mollifier:draining` ZSET maintained atomically with pop/ack/fail/requeue (observability-only).

.changeset/mollifier-tag-cap.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/redis-worker": patch
+---
+
+Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation.

.changeset/otel-suite-0218.md

@@ -0,0 +1,7 @@
+---
+"@trigger.dev/core": patch
+"trigger.dev": patch
+"@trigger.dev/sdk": patch
+---
+
+Update the bundled OpenTelemetry packages to their latest releases (`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1, `@opentelemetry/host-metrics` 0.38.3).

.changeset/pre.json

@@ -19,26 +19,64 @@
     "@trigger.dev/sdk": "4.4.6"
   },
   "changesets": [
+    "agent-skills-bundled-in-sdk",
     "agent-skills",
     "ai-prompts",
+    "ai-sdk-7-support",
     "ai-tool-helpers",
+    "backpressure-scale-up-freeze",
     "bundle-skills-single-pass",
     "cap-idempotency-key-length",
+    "chat-agent-hardening",
     "chat-agent-on-boot-hook",
+    "chat-agent-tools",
     "chat-agent",
+    "chat-boot-cursor",
+    "chat-headstart-custom-backends",
+    "chat-headstart-hydrate",
+    "chat-headstart-reasoning",
+    "chat-headstart-trigger-config",
     "chat-history-read-primitives",
     "chat-session-attributes",
+    "chat-slim-wire-merge",
     "chat-start-session-action-typed-client-data",
+    "chat-system-prompt-caching",
+    "chat-transport-recreate-missing-session",
     "cli-deploy-skip-rewrite-timestamp",
+    "cli-dev-without-project",
+    "cli-init-ai-tooling",
+    "coerce-concurrency-key-to-string",
+    "create-session-stop-continuation",
+    "custom-agent-loop-fixes",
+    "dequeue-latency-histogram",
+    "duplicate-task-ids",
+    "env-vars-tracing-forceflush-typecheck",
+    "envvars-import-is-secret",
+    "large-trigger-payload-offload",
     "locals-key-dual-package-fix",
     "mcp-agent-chat-sessions",
     "mcp-list-runs-region",
+    "mcp-trigger-task-no-default-wait",
     "mock-chat-agent-test-harness",
+    "mollifier-buffer-pipeline-list-entries",
+    "mollifier-configurable-constants",
+    "mollifier-drain-batch-size",
     "mollifier-redis-worker-primitives",
+    "mollifier-tag-cap",
+    "otel-suite-0218",
     "plugin-auth-path",
+    "project-environments-endpoint",
     "resource-catalog-runtime-registration",
+    "retry-middleware-errors",
     "retry-sigsegv",
     "runs-list-region-filter",
-    "sessions-primitive"
+    "s2-batch-transform-linger-fix",
+    "sessions-primitive",
+    "span-api-cached-cost",
+    "trigger-client",
+    "trigger-skill-namespace-and-docs",
+    "trigger-skills-installer",
+    "unflatten-attributes-conflict",
+    "warm-start-external-trace-context-leak"
   ]
 }

.changeset/project-environments-endpoint.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Add `GetProjectEnvironmentsResponseBody` and `ProjectEnvironment` schemas for the new `GET /api/v1/projects/{projectRef}/environments` endpoint, which lists the parent environments (dev, staging, preview, prod) a personal access token can access for a project. Dev is scoped to the token owner and branch (preview child) environments are excluded.

.changeset/retry-middleware-errors.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of failing the run on the first attempt. The error was already classified as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did not include it, so the retry flow fell through to `fail_run`. Fixes #3231.

.changeset/runner-send-debug-logs-gate.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+Runner debug logs are now disabled by default. Set `SEND_RUN_DEBUG_LOGS=true` on the supervisor to re-enable them.

.changeset/s2-batch-transform-linger-fix.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/core": patch
+"trigger.dev": patch
+---
+
+Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream.

.changeset/span-api-cached-cost.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+The run span API response now includes `cachedCost` and `cacheCreationCost` on the `ai` object, alongside the existing `inputCost` / `outputCost` / `totalCost`. `inputCost` reflects only the non-cached input, so these fields let you reconstruct the full cost breakdown for prompt-cached calls.

.changeset/trigger-client.md

@@ -0,0 +1,18 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Add `TriggerClient` for running multiple SDK clients side-by-side, each with its own auth, preview branch, and baseURL. Useful when a single process needs to trigger tasks or read runs across multiple projects, environments, or preview branches without mutating shared global state.
+
+```ts
+import { TriggerClient } from "@trigger.dev/sdk";
+
+const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
+const preview = new TriggerClient({
+  accessToken: process.env.TRIGGER_PREVIEW_KEY,
+  previewBranch: "signup-flow",
+});
+
+await prod.tasks.trigger("send-email", payload);
+await preview.runs.list({ status: ["COMPLETED"] });
+```

.changeset/trigger-skill-namespace-and-docs.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/sdk": patch
+"trigger.dev": patch
+---
+
+The agent skills installed by `trigger skills` are now namespaced with a `trigger-` prefix (e.g. `trigger-authoring-tasks`, `trigger-getting-started`) so they don't collide with unrelated skills in your coding agent's skills directory. Adds a `trigger-cost-savings` skill for auditing and reducing compute spend (right-sizing machines, `maxDuration`, batching, debounce), and `@trigger.dev/sdk` now bundles the full Trigger.dev documentation so your agent can read the complete, version-pinned reference directly from node_modules.

.changeset/trigger-skills-installer.md

@@ -0,0 +1,11 @@
+---
+"trigger.dev": patch
+---
+
+`trigger skills` installs Trigger.dev agent skills into your coding agent so it knows how to write tasks, schedules, realtime, and chat.agent code. The skills ship with the CLI and are copied into each tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and Codex / AGENTS.md), and `trigger dev` offers to install them on first run.
+
+```bash
+trigger skills --target claude-code
+```
+
+Replaces the previous `install-rules` command, which stays as an alias.

.changeset/unflatten-attributes-conflict.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Fix `TypeError` in `unflattenAttributes` when the input attribute map contains conflicting dotted key paths (e.g. both `a.b` set to a scalar and `a.b.c` set to a value). The path-walk loop now applies last-write-wins when a prior key wrote a primitive, null, or array at an intermediate slot, matching the existing precedent in `AttributeFlattener.addAttribute`. Callers no longer crash when handed malformed external attribute inputs.

.changeset/warm-start-external-trace-context-leak.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/core": patch
+---
+
+Fix external trace context leaking across runs on warm-started workers with `processKeepAlive` enabled. Every subsequent run's attempt span was being exported with the first run's `traceId` and `parentSpanId`, breaking causal-chain navigation in external APM tools. Runs without an external trace context are unaffected.

.claude/rules/sdk-packages.md

@@ -9,4 +9,4 @@ paths:
 - Default to **patch**. Get maintainer approval for minor. Never select major without explicit approval.
 - `@trigger.dev/core`: **Never import the root**. Always use subpath imports (e.g., `@trigger.dev/core/v3`).
 - Do NOT update `rules/` or `.claude/skills/trigger-dev-tasks/` unless explicitly asked. These are maintained in separate dedicated passes.
-- Test changes using `references/hello-world` reference project.
+- Test changes using the `hello-world` project in the [`triggerdotdev/references`](https://github.com/triggerdotdev/references) repo.

.claude/skills/errors-api-e2e/SKILL.md

@@ -0,0 +1,199 @@
+---
+name: errors-api-e2e
+description: End-to-end smoke test for the public Errors HTTP API (error groups). Seeds failed runs into ClickHouse so the error materialized views populate, then drives the real endpoints against the running webapp — list (with filters + pagination), retrieve, resolve/ignore/unresolve, the `filter[error]` runs filter, user attribution via the `trigger.dev mint-token` -> JWT exchange, and the 401/403/404 negatives. Use for "smoke test the errors API", "test the errors API e2e", "prove the errors endpoints work", or to re-verify after changes.
+allowed-tools: Read, Bash
+---
+
+# Errors API — end-to-end smoke test
+
+Proves the public Errors API against the **running** webapp with real HTTP. No
+mocks. The error data plane is ClickHouse (`errors_v1` + `error_occurrences_v1`,
+both materialized-view-fed from `task_runs_v2`) plus Postgres `ErrorGroupState`
+for lifecycle status; this skill seeds straight into `task_runs_v2` and lets the
+MVs do the rest.
+
+Code under test:
+- `apps/webapp/app/routes/api.v1.errors.ts` — `GET /api/v1/errors` (list).
+- `apps/webapp/app/routes/api.v1.errors.$errorId.ts` — `GET /api/v1/errors/:errorId` (detail).
+- `apps/webapp/app/routes/api.v1.errors.$errorId.{resolve,ignore,unresolve}.ts` — state actions.
+- `apps/webapp/app/presenters/v3/ApiErrorListPresenter.server.ts` / `ApiErrorGroupPresenter.server.ts`.
+- `apps/webapp/app/presenters/v3/ApiRunListPresenter.server.ts` — the `filter[error]` addition on `GET /api/v1/runs`.
+- `apps/webapp/app/v3/services/errorGroupActions.server.ts` — resolve/ignore/unresolve (nullable `userId`).
+- Attribution: `api.v1.projects.$projectRef.$env.jwt.ts` stamps `act:{sub}` for PAT **and** UAT exchanges; `@trigger.dev/rbac` surfaces `act.sub` through bearer auth; the action handlers read `authentication.actor?.sub`.
+
+`errorId` is `error_<fingerprint>` (round-trips via `ErrorId` in `@trigger.dev/core/v3/isomorphic`).
+
+## Prerequisites
+
+- Webapp running on http://localhost:3030 (`pnpm run dev --filter webapp`). Confirm `curl -s http://localhost:3030/healthcheck`.
+- DB seeded (`pnpm run db:seed`), and a local ClickHouse reachable at `CLICKHOUSE_URL` (the `pnpm run docker` stack).
+- The CLI built + logged in to localhost:3030 (`pnpm run build --filter trigger.dev`; profile `default` points at localhost:3030). Needed only for the attribution leg.
+
+> Important wiring facts the seed relies on (verified):
+> - The MVs read the error type/message from `error.data.*`, so the seeded
+>   `error` JSON column **must** be wrapped: `{"data": {"type": ..., "message": ..., "stack": ...}}`.
+> - The MVs only fire for failed statuses: `SYSTEM_FAILURE | CRASHED | INTERRUPTED | COMPLETED_WITH_ERRORS | TIMED_OUT`, and require a non-empty `error_fingerprint`.
+> - `GET /api/v1/runs` lists run **ids** from ClickHouse but **hydrates from Postgres** `TaskRun`. So the error-list/detail/action legs work from a ClickHouse-only seed, but the `filter[error]` leg needs a **paired** Postgres `TaskRun` row whose `id` equals the ClickHouse `run_id`.
+
+Run everything from the repo root in one shell. Invoke the built CLI via a
+function (a `CLI="node …"` variable won't word-split under zsh):
+```bash
+cli() { node packages/cli-v3/dist/esm/index.js "$@"; }
+PROFILE=default
+```
+
+## Setup — resolve a dev environment + connection strings
+
+```bash
+cd apps/webapp
+CHURL=$(grep -E "^CLICKHOUSE_URL=" .env | head -1 | cut -d= -f2- | tr -d '"')
+DBURL=$(grep -E "^DATABASE_URL=" .env | head -1 | cut -d= -f2- | tr -d '"' | tr -d "'" | sed 's/?.*//')
+
+# Pick the seeded hello-world dev env (proj_rrkpdguyagvsoktglnod). Adjust the
+# WHERE if you want a different project.
+read ENV ORG PROJ REF < <(psql "$DBURL" -t -A -F' ' -c "
+  SELECT re.id, re.\"organizationId\", re.\"projectId\", p.\"externalRef\"
+  FROM \"RuntimeEnvironment\" re
+  JOIN \"Project\" p ON p.id = re.\"projectId\"
+  WHERE re.slug='dev' AND p.\"externalRef\"='proj_rrkpdguyagvsoktglnod' LIMIT 1;")
+APIKEY=$(psql "$DBURL" -t -A -c "SELECT \"apiKey\" FROM \"RuntimeEnvironment\" WHERE id='$ENV';")
+cd ..
+H="Authorization: Bearer $APIKEY"
+B="http://localhost:3030"
+```
+
+## Steps
+
+### 1. Seed two error groups (ClickHouse, MV-fed)
+
+```bash
+RUN=$(node -e 'console.log(Date.now().toString(36))')
+TASK="errors-api-e2e-$RUN"; FP_A="fpA${RUN}"; FP_B="fpB${RUN}"
+ERRID_A="error_$FP_A"; ERRID_B="error_$FP_B"
+NOW_CH=$(node -e 'console.log(new Date().toISOString().replace("T"," ").replace("Z","").slice(0,23))')
+NOW_MS=$(node -e 'console.log(Date.now())')
+Q=$(python3 -c "import urllib.parse;print(urllib.parse.quote('INSERT INTO trigger_dev.task_runs_v2 FORMAT JSONEachRow'))")
+
+mkrow() { # status fingerprint errorType message runId
+  echo "{\"environment_id\":\"$ENV\",\"organization_id\":\"$ORG\",\"project_id\":\"$PROJ\",\"run_id\":\"$5\",\"friendly_id\":\"run_$5\",\"status\":\"$1\",\"environment_type\":\"DEVELOPMENT\",\"engine\":\"V2\",\"task_identifier\":\"$TASK\",\"created_at\":\"$NOW_CH\",\"updated_at\":\"$NOW_CH\",\"error\":{\"data\":{\"type\":\"$3\",\"message\":\"$4\",\"stack\":\"at x (a.ts:1:1)\"}},\"error_fingerprint\":\"$2\",\"task_version\":\"20240101.1\",\"_version\":\"$NOW_MS\",\"_is_deleted\":0}"
+}
+ROWS="$(mkrow COMPLETED_WITH_ERRORS $FP_A AlphaBoom 'alpha boom happened' r_a1_$RUN)
+$(mkrow COMPLETED_WITH_ERRORS $FP_A AlphaBoom 'alpha boom happened' r_a2_$RUN)
+$(mkrow CRASHED $FP_B BetaCrash 'beta crash happened' r_b1_$RUN)"
+printf '%s' "$ROWS" | curl -s "$CHURL/?query=$Q" --data-binary @-
+
+# Poll until both fingerprints appear in errors_v1 (the MV is near-instant locally).
+for i in $(seq 1 10); do
+  N=$(curl -s "$CHURL" --data-binary "SELECT count() FROM (SELECT 1 FROM trigger_dev.errors_v1 WHERE environment_id='$ENV' AND error_fingerprint IN ('$FP_A','$FP_B') GROUP BY error_fingerprint)")
+  [ "$N" = "2" ] && break; sleep 1
+done
+echo "seeded fingerprints in errors_v1: $N (want 2)"
+```
+PASS: `N = 2`. Alpha has 2 occurrences, beta 1.
+
+### 2. List + filters + pagination
+
+```bash
+curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bperiod%5D=1d" -H "$H" \
+  | python3 -c "import sys,json;d=json.load(sys.stdin);print('count',len(d['data']),[(e['id'],e['status'],e['count']) for e in d['data']])"
+```
+PASS: 2 groups, both `status=unresolved`, alpha `count=2`, beta `count=1`, ids `error_<fp>`.
+
+Assert each filter narrows correctly (each should return the noted shape):
+```bash
+curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bstatus%5D=unresolved&filter%5Bperiod%5D=1d" -H "$H" | python3 -c "import sys,json;print('unresolved:',len(json.load(sys.stdin)['data']))"   # 2
+curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bsearch%5D=AlphaBoom&filter%5Bperiod%5D=1d" -H "$H" | python3 -c "import sys,json;print('search:',[e['errorType'] for e in json.load(sys.stdin)['data']])"   # ['AlphaBoom']
+curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bperiod%5D=1d&page%5Bsize%5D=1" -H "$H" | python3 -c "import sys,json;d=json.load(sys.stdin);print('page size 1:',len(d['data']),'next?',bool(d['pagination'].get('next')))"   # 1 / True
+```
+PASS: `unresolved: 2`, `search: ['AlphaBoom']`, `page size 1: 1 / next? True`.
+
+### 3. Retrieve detail
+
+```bash
+curl -s "$B/api/v1/errors/$ERRID_A" -H "$H" \
+  | python3 -c "import sys,json;d=json.load(sys.stdin);print(d['id'],d['errorType'],d['status'],d['count'],d['affectedVersions'],d['resolvedBy'])"
+```
+PASS: `error_<fpA> AlphaBoom unresolved 2 ['20240101.1'] None`.
+
+### 4. Resolve / ignore / unresolve (env API key — `resolvedBy` null)
+
+```bash
+st(){ python3 -c "import sys,json;d=json.load(sys.stdin);print('status',d['status'],'| resolvedInVersion',d['resolvedInVersion'],'| resolvedBy',d['resolvedBy'],'| ignoredUntil',bool(d['ignoredUntil']),'| reason',d['ignoredReason'])"; }
+
+curl -s -X POST "$B/api/v1/errors/$ERRID_A/resolve" -H "$H" -H 'Content-Type: application/json' -d '{"resolvedInVersion":"20240101.1"}' >/dev/null
+curl -s "$B/api/v1/errors/$ERRID_A" -H "$H" | st   # status resolved | resolvedInVersion 20240101.1 | resolvedBy None
+
+curl -s -X POST "$B/api/v1/errors/$ERRID_B/ignore" -H "$H" -H 'Content-Type: application/json' -d '{"duration":3600000,"reason":"known flake"}' >/dev/null
+curl -s "$B/api/v1/errors/$ERRID_B" -H "$H" | st   # status ignored | ignoredUntil True | reason known flake
+
+curl -s -X POST "$B/api/v1/errors/$ERRID_A/unresolve" -H "$H" >/dev/null
+curl -s "$B/api/v1/errors/$ERRID_A" -H "$H" | st   # status unresolved
+```
+PASS: each transition reflected; `filter[status]=ignored` returns only beta:
+```bash
+curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bstatus%5D=ignored&filter%5Bperiod%5D=1d" -H "$H" | python3 -c "import sys,json;print([e['id'] for e in json.load(sys.stdin)['data']])"   # [error_<fpB>]
+```
+
+### 5. `filter[error]` on the runs list (paired PG + CH seed)
+
+The runs list hydrates from Postgres, so seed a matching `TaskRun` row + a CH row
+that share `run_id`/`id` and carry a fingerprint:
+```bash
+RID="re2e${RUN}"; FRID="run_${RID}"; FP_R="fpR${RUN}"
+psql "$DBURL" -v ON_ERROR_STOP=1 -c "
+  INSERT INTO \"TaskRun\" (id, \"friendlyId\", \"taskIdentifier\", payload, \"traceId\", \"spanId\", \"runtimeEnvironmentId\", \"projectId\", queue, status, \"createdAt\", \"updatedAt\")
+  VALUES ('$RID','$FRID','$TASK','{}','trace_$RID','span_$RID','$ENV','$PROJ','task/$TASK','COMPLETED_WITH_ERRORS', now(), now())
+  ON CONFLICT (id) DO NOTHING;" >/dev/null
+ROW="{\"environment_id\":\"$ENV\",\"organization_id\":\"$ORG\",\"project_id\":\"$PROJ\",\"run_id\":\"$RID\",\"friendly_id\":\"$FRID\",\"status\":\"COMPLETED_WITH_ERRORS\",\"environment_type\":\"DEVELOPMENT\",\"engine\":\"V2\",\"task_identifier\":\"$TASK\",\"created_at\":\"$NOW_CH\",\"updated_at\":\"$NOW_CH\",\"error\":{\"data\":{\"type\":\"RunsFilterErr\",\"message\":\"for runs filter\",\"stack\":\"at x\"}},\"error_fingerprint\":\"$FP_R\",\"task_version\":\"20240101.1\",\"_version\":\"$NOW_MS\",\"_is_deleted\":0}"
+printf '%s' "$ROW" | curl -s "$CHURL/?query=$Q" --data-binary @-
+sleep 1
+curl -s "$B/api/v1/runs?filter%5Berror%5D=error_$FP_R" -H "$H" | python3 -c "import sys,json;d=json.load(sys.stdin);print('runs:',[r['id'] for r in d['data']])"
+```
+PASS: one run, `run_<RID>` (status maps to `FAILED`). Proves `filter[error]` -> fingerprint -> CH -> PG hydration.
+
+### 6. Attribution — `mint-token` -> JWT exchange records the acting user
+
+```bash
+TOKEN=$(cli mint-token --profile $PROFILE --client errors-api-e2e 2>/dev/null)            # UAT
+ENVJWT=$(curl -sS -X POST "$B/api/v1/projects/$REF/dev/jwt" -H "Authorization: Bearer $TOKEN" \
+  -H 'Content-Type: application/json' -d '{"claims":{"scopes":["read:errors","write:errors"]}}' \
+  | python3 -c "import sys,json;print(json.load(sys.stdin)['token'])")
+# Decoded env JWT carries act.sub = the user id.
+node -e 'const p=JSON.parse(Buffer.from(process.argv[1].split(".")[1],"base64url").toString());console.log("act:",JSON.stringify(p.act))' "$ENVJWT"
+
+curl -s -X POST "$B/api/v1/errors/$ERRID_A/resolve" -H "Authorization: Bearer $ENVJWT" \
+  -H 'Content-Type: application/json' -d '{"resolvedInVersion":"20240101.2"}' >/dev/null
+curl -s "$B/api/v1/errors/$ERRID_A" -H "$H" | python3 -c "import sys,json;d=json.load(sys.stdin);print('resolvedBy:',d['resolvedBy'])"
+```
+PASS: `act.sub` is the user id (matches `cli whoami`), and `detail.resolvedBy` equals that user id (not null). A plain env key leaves it null (step 4). A **PAT** exchanged the same way also stamps `act` — repeat with the stored PAT to confirm `ignoredByUserId` attribution.
+
+### 7. Negatives
+
+```bash
+curl -s -o /dev/null -w 'unknown id: %{http_code} (404)\n'        "$B/api/v1/errors/error_doesnotexist0000" -H "$H"
+curl -s -o /dev/null -w 'no auth list: %{http_code} (401)\n'      "$B/api/v1/errors"
+curl -s -o /dev/null -w 'no auth resolve: %{http_code} (401)\n'   -X POST "$B/api/v1/errors/$ERRID_B/resolve" -H 'Content-Type: application/json' -d '{}'
+
+# read-only JWT must be denied on write, allowed on read
+READJWT=$(curl -sS -X POST "$B/api/v1/projects/$REF/dev/jwt" -H "Authorization: Bearer $TOKEN" \
+  -H 'Content-Type: application/json' -d '{"claims":{"scopes":["read:errors"]}}' | python3 -c "import sys,json;print(json.load(sys.stdin)['token'])")
+curl -s -o /dev/null -w 'read JWT write: %{http_code} (403)\n'    -X POST "$B/api/v1/errors/$ERRID_B/resolve" -H "Authorization: Bearer $READJWT" -H 'Content-Type: application/json' -d '{}'
+curl -s -o /dev/null -w 'read JWT read: %{http_code} (200)\n'     "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK" -H "Authorization: Bearer $READJWT"
+```
+PASS: `404`, `401`, `401`, `403`, `200` respectively.
+
+## Result
+
+Report PASS only if: step 1 lands 2 groups in `errors_v1`; step 2's filters and
+pagination narrow correctly; step 3 returns the detail; step 4's resolve/ignore/
+unresolve flip status (and `filter[status]` follows); step 5's `filter[error]`
+returns the paired run; step 6 records `resolvedBy` = the acting user via the
+JWT exchange (null with a plain env key); and step 7 returns 404/401/401/403/200.
+A red leg is a bug or a missing prereq — report the exact status + body and file
+a Linear issue, don't tune around it.
+
+## Notes / gotchas
+
+- Run files use a unique `$RUN` suffix per invocation, so reruns don't collide and seeded rows stay isolated by their unique task identifier. They are local-dev test rows (90-day ClickHouse TTL); no cleanup required.
+- After **adding** the route files, the classic Remix dev compiler may not register them until a dev-server restart (a stale manifest returns Remix's HTML 404 on the new paths). If `POST …/resolve` returns a 404 HTML page rather than 401/200, restart `pnpm run dev --filter webapp`.
+- The rbac `act` extraction lives in `@trigger.dev/rbac` (a built dep). After editing it, `pnpm run build --filter @trigger.dev/rbac` and restart the webapp so the attribution leg (step 6) reflects the change.

.cursor/mcp.json

@@ -1,3 +1,7 @@
 {
-  "mcpServers": {}
+  "mcpServers": {
+    "linear": {
+      "url": "https://mcp.linear.app/mcp"
+    }
+  }
 }

.env.example

@@ -17,6 +17,11 @@ NODE_ENV=development
 CLICKHOUSE_URL=http://default:password@localhost:8123
 RUN_REPLICATION_CLICKHOUSE_URL=http://default:password@localhost:8123
 RUN_REPLICATION_ENABLED=1
+# Store task run spans/traces in ClickHouse so the dashboard trace view is
+# populated in local dev. The local stack is ClickHouse-backed (see above), so
+# leaving this unset falls back to the "postgres" store and dev run traces show
+# up empty even though the run itself appears.
+EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2
 
 # Set this to UTC because Node.js uses the system timezone
 TZ="UTC"

.github/VOUCHED.td

@@ -21,4 +21,6 @@ chengzp
 bharathkumar39293
 bhekanik
 jrossi
-ThullyoCunha
+ThullyoCunha
+ConProgramming
+saasjesus

.github/workflows/changesets-pr.yml

@@ -36,7 +36,7 @@ jobs:
       - name: Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           cache: "pnpm"
 
       - name: Install dependencies
@@ -44,7 +44,7 @@ jobs:
 
       - name: Create release PR
         id: changesets
-        uses: changesets/action@6a0a831ff30acef54f2c6aa1cbbc1096b066edaf # v1.7.0
+        uses: changesets/action@63a615b9cd06ba9a3e6d13796c7fbcb080a60a0b # v1.8.0
         with:
           version: pnpm run changeset:version
           commit: "chore: release"

.github/workflows/check-review-md.yml

@@ -7,7 +7,6 @@ on:
       - "docs/**"
       - ".changeset/**"
       - ".server-changes/**"
-      - "references/**"
 
 concurrency:
   group: review-md-drift-${{ github.event.pull_request.number }}
@@ -15,7 +14,10 @@ concurrency:
 
 jobs:
   audit:
+    # Set the ENABLE_CLAUDE_CODE repository variable to 'false' to turn off Claude
+    # jobs; leave it unset (the default) to keep them enabled.
     if: >-
+      vars.ENABLE_CLAUDE_CODE != 'false' &&
       github.event.pull_request.draft == false &&
       github.event.pull_request.head.repo.full_name == github.repository
     runs-on: ubuntu-latest
@@ -32,7 +34,7 @@ jobs:
 
       - name: Run Claude Code
         id: claude
-        uses: anthropics/claude-code-action@fefa07e9c665b7320f08c3b525980457f22f58aa # v1.0.111
+        uses: anthropics/claude-code-action@787c5a0ce96a9a6cfb050ea0c8f4c05f2447c251 # v1.0.133
         with:
           anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
           use_sticky_comment: true

.github/workflows/claude-md-audit.yml

@@ -8,7 +8,6 @@ on:
       - ".changeset/**"
       - ".server-changes/**"
       - "**/*.md"
-      - "references/**"
 
 concurrency:
   group: claude-md-audit-${{ github.event.pull_request.number }}
@@ -16,7 +15,10 @@ concurrency:
 
 jobs:
   audit:
+    # Set the ENABLE_CLAUDE_CODE repository variable to 'false' to turn off Claude
+    # jobs; leave it unset (the default) to keep them enabled.
     if: >-
+      vars.ENABLE_CLAUDE_CODE != 'false' &&
       github.event.pull_request.draft == false &&
       github.event.pull_request.head.repo.full_name == github.repository
     runs-on: ubuntu-latest
@@ -34,14 +36,15 @@ jobs:
 
       - name: Run Claude Code
         id: claude
-        uses: anthropics/claude-code-action@fefa07e9c665b7320f08c3b525980457f22f58aa # v1.0.111
+        uses: anthropics/claude-code-action@787c5a0ce96a9a6cfb050ea0c8f4c05f2447c251 # v1.0.133
         with:
           anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
           use_sticky_comment: true
           allowed_bots: "devin-ai-integration[bot]"
 
           claude_args: |
-            --max-turns 15
+            --max-turns 25
+            --model claude-opus-4-8
             --allowedTools "Read,Glob,Grep,Bash(git diff:*)"
 
           prompt: |

.github/workflows/claude.yml

@@ -12,11 +12,16 @@ on:
 
 jobs:
   claude:
+    # Set the ENABLE_CLAUDE_CODE repository variable to 'false' to turn off Claude
+    # jobs; leave it unset (the default) to keep them enabled.
     if: |
-      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
-      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
-      (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
-      (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
+      vars.ENABLE_CLAUDE_CODE != 'false' &&
+      (
+        (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
+        (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
+        (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
+        (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
+      )
     runs-on: ubuntu-latest
     permissions:
       contents: write
@@ -39,7 +44,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           cache: "pnpm"
 
       - name: 📥 Download deps
@@ -50,7 +55,7 @@ jobs:
 
       - name: Run Claude Code
         id: claude
-        uses: anthropics/claude-code-action@fefa07e9c665b7320f08c3b525980457f22f58aa # v1.0.111
+        uses: anthropics/claude-code-action@787c5a0ce96a9a6cfb050ea0c8f4c05f2447c251 # v1.0.133
         with:
           anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
 

.github/workflows/dependabot-critical-alerts.yml

@@ -0,0 +1,87 @@
+name: Dependabot Critical Alerts
+
+on:
+  schedule:
+    - cron: "0 8 * * *"  # Daily 08:00 UTC
+  workflow_dispatch:
+    inputs:
+      severity:
+        description: "Severity to alert on"
+        type: choice
+        options:
+          - critical
+          - high
+          - medium
+          - low
+        default: critical
+
+concurrency:
+  group: ${{ github.workflow }}
+  cancel-in-progress: false
+
+permissions:
+  contents: read
+
+jobs:
+  alert:
+    name: Post critical alerts
+    # Set the ENABLE_DEPENDABOT_ALERTS repository variable to 'false' to turn off
+    # the Dependabot alert/summary notifiers — e.g. forks/mirrors that lack the
+    # DEPENDABOT_ALERTS_TOKEN / SLACK_BOT_TOKEN secrets. Defaults to enabled.
+    if: ${{ vars.ENABLE_DEPENDABOT_ALERTS != 'false' }}
+    runs-on: ubuntu-latest
+    environment: dependabot-summary
+    env:
+      SEVERITY: ${{ inputs.severity || 'critical' }}
+    steps:
+      - name: Fetch alerts
+        id: alerts
+        env:
+          GH_TOKEN: ${{ secrets.DEPENDABOT_ALERTS_TOKEN }}
+          REPO: ${{ github.repository }}
+        run: |
+          set -euo pipefail
+          gh api -X GET "/repos/$REPO/dependabot/alerts" \
+            -F state=open -F severity="$SEVERITY" --paginate > pages.json
+          jq -s 'add' pages.json > alerts.json
+          TOTAL=$(jq 'length' alerts.json)
+          echo "total=$TOTAL" >> "$GITHUB_OUTPUT"
+          if [ "$TOTAL" = "0" ]; then
+            exit 0
+          fi
+          LIST=$(jq -r '
+            map("• <\(.html_url)|#\(.number)> *\(.dependency.package.name)* - \(.security_advisory.summary)")
+            | join("\n")
+          ' alerts.json)
+          {
+            echo "list<<EOF"
+            echo "$LIST"
+            echo "EOF"
+          } >> "$GITHUB_OUTPUT"
+
+      - name: Build Slack payload
+        if: steps.alerts.outputs.total != '0'
+        env:
+          REPO: ${{ github.repository }}
+          CHANNEL: ${{ vars.SLACK_CHANNEL_ID }}
+          TOTAL: ${{ steps.alerts.outputs.total }}
+          LIST: ${{ steps.alerts.outputs.list }}
+        run: |
+          jq -n \
+            --arg channel "$CHANNEL" \
+            --arg repo "$REPO" \
+            --arg total "$TOTAL" \
+            --arg list "$LIST" \
+            --arg severity "$SEVERITY" \
+            '{
+              channel: $channel,
+              text: ":bufo-alarma: `\($repo)` - *\($total) open \($severity) alert(s)*\n\($list)\n\n<https://github.com/\($repo)/security/dependabot?q=is%3Aopen+severity%3A\($severity)|View \($severity) alerts>"
+            }' > payload.json
+
+      - name: Post Slack alert
+        if: steps.alerts.outputs.total != '0'
+        uses: slackapi/slack-github-action@45a88b9581bfab2566dc881e2cd66d334e621e2c # v3.0.3
+        with:
+          method: chat.postMessage
+          token: ${{ secrets.SLACK_BOT_TOKEN }}
+          payload-file-path: payload.json

.github/workflows/dependabot-weekly-summary.yml

@@ -19,6 +19,10 @@ permissions:
 jobs:
   summary:
     name: Post weekly Dependabot summary
+    # Set the ENABLE_DEPENDABOT_ALERTS repository variable to 'false' to turn off
+    # the Dependabot alert/summary notifiers — e.g. forks/mirrors that lack the
+    # DEPENDABOT_ALERTS_TOKEN / SLACK_BOT_TOKEN secrets. Defaults to enabled.
+    if: ${{ vars.ENABLE_DEPENDABOT_ALERTS != 'false' }}
     runs-on: ubuntu-latest
     environment: dependabot-summary
     env:

.github/workflows/e2e-webapp-auth-full.yml

@@ -85,12 +85,12 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           cache: "pnpm"
 
       - name: 🐳 Login to DockerHub
         if: ${{ env.DOCKERHUB_USERNAME }}
-        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}

.github/workflows/e2e-webapp.yml

@@ -59,13 +59,13 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           cache: "pnpm"
 
       # ..to avoid rate limits when pulling images
       - name: 🐳 Login to DockerHub
         if: ${{ env.DOCKERHUB_USERNAME }}
-        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}

.github/workflows/e2e.yml

@@ -37,7 +37,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
 
       - name: 📥 Download deps
         run: pnpm install --frozen-lockfile --filter trigger.dev...

.github/workflows/helm-prerelease.yml

@@ -68,10 +68,15 @@ jobs:
 
   prerelease:
     needs: lint-and-test
+    # Set the ENABLE_HELM_PRERELEASE repository variable to 'false' to turn off
+    # publishing the chart to GHCR — e.g. forks/mirrors that lack write_package
+    # on the owner's charts namespace. Defaults to enabled; the lint-and-test
+    # job above always runs regardless.
     if: |
-      (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository) ||
+      vars.ENABLE_HELM_PRERELEASE != 'false' &&
+      ((github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository) ||
       github.event_name == 'push' ||
-      github.event_name == 'workflow_dispatch'
+      github.event_name == 'workflow_dispatch')
     runs-on: ubuntu-latest
     permissions:
       contents: read
@@ -97,7 +102,7 @@ jobs:
           for file in ./charts/*.tgz; do echo "Extracting $file"; tar -xzf "$file" -C ./charts; done
 
       - name: Log in to Container Registry
-        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
         with:
           registry: ${{ env.REGISTRY }}
           username: ${{ github.actor }}

.github/workflows/pr_checks.yml

@@ -39,7 +39,6 @@ jobs:
               - '!.changeset/**'
               - '!hosting/**'
               - '!.github/**'
-              - '!references/**'
               - '!**/*.md'
               - '!**/.env.example'
       - uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1

.github/workflows/preview-dispatch.yml

@@ -0,0 +1,76 @@
+name: 🌱 Preview environment dispatch
+
+# Opt-in per-PR preview environments
+
+on:
+  pull_request:
+    types: [opened, reopened, synchronize, closed, labeled, unlabeled]
+
+# Serialize a PR's events so dispatches arrive in order. Cloud-side concurrency
+# collapses by branch but can't fix out-of-order arrival — e.g. a push racing a
+# close could cancel the in-flight destroy and leak the preview. One short API
+# call, so queuing is cheap; cancel-in-progress: false lets an in-flight
+# dispatch finish (GitHub keeps only the latest pending, the desired behavior).
+concurrency:
+  group: preview-dispatch-${{ github.event.pull_request.number }}
+  cancel-in-progress: false
+
+permissions: {}
+
+jobs:
+  dispatch:
+    name: Dispatch preview-deploy to cloud
+    runs-on: ubuntu-latest
+    #   label added             -> create
+    #   new commit while labeled -> update
+    #   label removed / PR closed -> destroy
+    if: >-
+      github.event.pull_request.head.repo.full_name == github.repository &&
+      (
+        (github.event.action == 'labeled' && github.event.label.name == 'preview') ||
+        (github.event.action == 'unlabeled' && github.event.label.name == 'preview') ||
+        (
+          contains(github.event.pull_request.labels.*.name, 'preview') &&
+          contains(fromJSON('["opened","reopened","synchronize","closed"]'), github.event.action)
+        )
+      )
+    steps:
+      - name: Build dispatch payload
+        id: payload
+        env:
+          ACTION: ${{ github.event.action }}
+          BRANCH: ${{ github.event.pull_request.head.ref }}
+          COMMIT: ${{ github.event.pull_request.head.sha }}
+        run: |
+          set -euo pipefail
+          # Map the GitHub PR action to the cloud pipeline's lifecycle event.
+          case "$ACTION" in
+            labeled | opened | reopened) EVENT=opened ;;
+            synchronize) EVENT=synchronize ;;
+            unlabeled | closed) EVENT=closed ;;
+            *) echo "unexpected action: $ACTION" >&2; exit 1 ;;
+          esac
+          # jq --arg JSON-escapes every value, so a branch name containing
+          # quotes/braces can't break or inject into the client payload.
+          payload=$(jq -nc \
+            --arg b "$BRANCH" \
+            --arg c "$COMMIT" \
+            --arg e "$EVENT" \
+            '{branch_name: $b, commit: $c, pull_request_event: $e}')
+          {
+            echo "client_payload=$payload"
+            echo "summary=$EVENT for $BRANCH @ ${COMMIT:0:7}"
+          } >> "$GITHUB_OUTPUT"
+
+      - name: Log dispatch
+        env:
+          SUMMARY: ${{ steps.payload.outputs.summary }}
+        run: echo "Dispatching preview-deploy event ($SUMMARY)"
+
+      - name: Send repository_dispatch
+        uses: peter-evans/repository-dispatch@28959ce8df70de7be546dd1250a005dd32156697 # v4.0.1
+        with:
+          token: ${{ secrets.CROSS_REPO_PAT }}
+          repository: triggerdotdev/cloud
+          event-type: preview-deploy
+          client-payload: ${{ steps.payload.outputs.client_payload }}

.github/workflows/preview-packages.yml

@@ -0,0 +1,83 @@
+name: 📦 Preview packages (pkg.pr.new)
+
+# Publishes installable preview builds of the public @trigger.dev/* packages
+# for every push to a branch, via https://pkg.pr.new. These are NOT published
+# to npm — pkg.pr.new serves them by commit SHA and drops install instructions
+# in a comment on the associated PR, e.g.
+#   npm i https://pkg.pr.new/@trigger.dev/sdk@<sha>
+#
+# Prerequisites:
+#   - The pkg.pr.new GitHub App must be installed on triggerdotdev/trigger.dev
+#     (https://github.com/apps/pkg-pr-new). Publishing fails until it is.
+#
+# Fork note: pkg.pr.new authenticates with a GitHub Actions OIDC token, which
+# GitHub does not issue to pull_request workflows from forks. This `push`
+# trigger therefore covers branches pushed to this repo (the core team), not
+# external fork PRs. Adding fork coverage would require a workflow_run two-stage
+# setup.
+
+on:
+  push:
+    branches-ignore:
+      - main
+      - changeset-release/main
+    paths:
+      - "package.json"
+      - "packages/**"
+      - "pnpm-lock.yaml"
+      - "pnpm-workspace.yaml"
+      - "turbo.json"
+      - ".github/workflows/preview-packages.yml"
+      - "scripts/stamp-preview-version.mjs"
+      - "scripts/updateVersion.ts"
+
+concurrency:
+  group: preview-packages-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+  id-token: write # OIDC token used by pkg.pr.new to authenticate the publish
+
+jobs:
+  publish:
+    name: Build and publish previews
+    runs-on: ubuntu-latest
+    if: github.repository == 'triggerdotdev/trigger.dev'
+    steps:
+      - name: ⬇️ Checkout repo
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+
+      - name: ⎔ Setup pnpm
+        uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
+        with:
+          version: 10.33.2
+
+      - name: ⎔ Setup node
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        with:
+          node-version: 20.20.0
+          cache: "pnpm"
+
+      - name: 📥 Install dependencies
+        run: pnpm install --frozen-lockfile
+
+      - name: 📀 Generate Prisma client
+        run: pnpm run generate
+
+      # Stamp a unique 0.0.0-preview-<sha> version before building so it can't
+      # collide with real npm versions and so updateVersion.ts bakes it into the
+      # runtime VERSION constant. See scripts/stamp-preview-version.mjs.
+      - name: 🏷️ Stamp preview version
+        run: node scripts/stamp-preview-version.mjs
+        env:
+          GITHUB_SHA: ${{ github.sha }}
+
+      - name: 🔨 Build packages
+        run: pnpm run build --filter "@trigger.dev/*" --filter "trigger.dev"
+
+      - name: 🚀 Publish previews to pkg.pr.new
+        run: pnpm exec pkg-pr-new publish --pnpm --compact --commentWithSha './packages/*'

.github/workflows/publish-docs.yml

@@ -0,0 +1,36 @@
+name: 📚 Publish docs
+
+on:
+  push:
+    tags:
+      - "docs-release-*"
+
+# Only needs to move the docs-live ref; Mintlify's GitHub app deploys from it.
+permissions:
+  contents: write
+
+concurrency:
+  group: publish-docs
+  cancel-in-progress: false
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    steps:
+      - name: 📥 Checkout tagged commit
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - name: 🔗 Check for broken links
+        working-directory: ./docs
+        run: npx mintlify@4.0.393 broken-links
+
+      - name: 🚀 Fast-forward docs-live to the tagged commit
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          gh api -X PATCH \
+            "repos/${{ github.repository }}/git/refs/heads/docs-live" \
+            -f sha="${{ github.sha }}" \
+            -F force=false

.github/workflows/publish-webapp.yml

@@ -14,6 +14,24 @@ on:
         type: string
         required: false
         default: ""
+      image_registry:
+        description: The registry namespace to publish under (e.g. ghcr.io/<owner>)
+        type: string
+        required: false
+        default: ""
+    outputs:
+      version:
+        description: The published image tag
+        value: ${{ jobs.publish.outputs.version }}
+      short_sha:
+        description: Short commit SHA of the published build
+        value: ${{ jobs.publish.outputs.short_sha }}
+      image_repo:
+        description: The image repository the build was published to (without tag)
+        value: ${{ jobs.publish.outputs.image_repo }}
+      digest:
+        description: Multi-arch index digest (sha256:...) of the published image
+        value: ${{ jobs.publish.outputs.digest }}
     secrets:
       SENTRY_AUTH_TOKEN:
         required: false
@@ -26,6 +44,8 @@ jobs:
     outputs:
       version: ${{ steps.get_tag.outputs.tag }}
       short_sha: ${{ steps.get_commit.outputs.sha_short }}
+      image_repo: ${{ steps.set_tags.outputs.image_repo }}
+      digest: ${{ steps.build_push.outputs.digest }}
     steps:
       - name: 🏭 Setup Depot CLI
         uses: depot/setup-action@15c09a5f77a0840ad4bce955686522a257853461 # v1.7.1
@@ -50,17 +70,22 @@ jobs:
       - name: 📛 Set the tags
         id: set_tags
         run: |
-          ref_without_tag=ghcr.io/triggerdotdev/trigger.dev
-          image_tags=$ref_without_tag:${STEPS_GET_TAG_OUTPUTS_TAG}
+          # The registry namespace is resolved by the caller (defaulting to
+          # ghcr.io/<owner>, overridable via the IMAGE_REGISTRY repository
+          # variable); the webapp image lives at <registry>/<repo-name>. A fork
+          # therefore publishes to its own package automatically.
+          image_tags=$REF_WITHOUT_TAG:${STEPS_GET_TAG_OUTPUTS_TAG}
 
           # when pushing the mutable main tag, also push an immutable-by-convention
           # full-commit-sha tag so a commit can be resolved to a specific digest
           if [[ "${STEPS_GET_TAG_OUTPUTS_TAG}" == "main" ]]; then
-            image_tags=$image_tags,$ref_without_tag:${GITHUB_SHA}
+            image_tags=$image_tags,$REF_WITHOUT_TAG:${GITHUB_SHA}
           fi
 
           echo "image_tags=${image_tags}" >> "$GITHUB_OUTPUT"
+          echo "image_repo=${REF_WITHOUT_TAG}" >> "$GITHUB_OUTPUT"
         env:
+          REF_WITHOUT_TAG: ${{ format('{0}/{1}', inputs.image_registry || vars.IMAGE_REGISTRY || format('ghcr.io/{0}', github.repository_owner), github.event.repository.name) }}
           STEPS_GET_TAG_OUTPUTS_TAG: ${{ steps.get_tag.outputs.tag }}
           STEPS_GET_TAG_OUTPUTS_IS_SEMVER: ${{ steps.get_tag.outputs.is_semver }}
 
@@ -82,7 +107,7 @@ jobs:
           STEPS_GET_TAG_OUTPUTS_IS_SEMVER: ${{ steps.get_tag.outputs.is_semver }}
 
       - name: 🐙 Login to GitHub Container Registry
-        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
         with:
           registry: ghcr.io
           username: ${{ github.repository_owner }}
@@ -90,7 +115,7 @@ jobs:
 
       - name: 🐳 Build image and push to GitHub Container Registry
         id: build_push
-        uses: depot/build-push-action@5f3b3c2e5a00f0093de47f657aeaefcedff27d18 # v1.17.0
+        uses: depot/build-push-action@98e78adca7817480b8185f474a400b451d74e287 # v1.18.0
         with:
           file: ./docker/Dockerfile
           platforms: linux/amd64,linux/arm64
@@ -115,6 +140,6 @@ jobs:
         continue-on-error: true
         uses: actions/attest-build-provenance@a2bbfa25375fe432b6a289bc6b6cd05ecd0c4c32 # v4.1.0
         with:
-          subject-name: ghcr.io/triggerdotdev/trigger.dev
+          subject-name: ${{ steps.set_tags.outputs.image_repo }}
           subject-digest: ${{ steps.build_push.outputs.digest }}
           push-to-registry: true

.github/workflows/publish-worker-v4.yml

@@ -8,6 +8,11 @@ on:
         type: string
         required: false
         default: ""
+      image_registry:
+        description: The registry namespace to publish under (e.g. ghcr.io/<owner>)
+        type: string
+        required: false
+        default: ""
   push:
     tags:
       - "re2-test-*"
@@ -65,24 +70,28 @@ jobs:
       - name: 📛 Set tags to push
         id: set_tags
         run: |
-          ref_without_tag=ghcr.io/triggerdotdev/${STEPS_GET_REPOSITORY_OUTPUTS_REPO}
+          # Resolved by the caller when invoked from publish.yml; falls back to the
+          # IMAGE_REGISTRY repository variable (or ghcr.io/<owner>) for the direct
+          # push triggers above, so a fork publishes to its own namespace.
+          ref_without_tag=${IMAGE_REGISTRY}/${STEPS_GET_REPOSITORY_OUTPUTS_REPO}
           image_tags=$ref_without_tag:${STEPS_GET_TAG_OUTPUTS_TAG}
 
           echo "image_tags=${image_tags}" >> "$GITHUB_OUTPUT"
         env:
+          IMAGE_REGISTRY: ${{ inputs.image_registry || vars.IMAGE_REGISTRY || format('ghcr.io/{0}', github.repository_owner) }}
           STEPS_GET_REPOSITORY_OUTPUTS_REPO: ${{ steps.get_repository.outputs.repo }}
           STEPS_GET_TAG_OUTPUTS_TAG: ${{ steps.get_tag.outputs.tag }}
           STEPS_GET_TAG_OUTPUTS_IS_SEMVER: ${{ steps.get_tag.outputs.is_semver }}
 
       - name: 🐙 Login to GitHub Container Registry
-        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
         with:
           registry: ghcr.io
           username: ${{ github.repository_owner }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
       - name: 🐳 Build image and push to GitHub Container Registry
-        uses: depot/build-push-action@5f3b3c2e5a00f0093de47f657aeaefcedff27d18 # v1.17.0
+        uses: depot/build-push-action@98e78adca7817480b8185f474a400b451d74e287 # v1.18.0
         with:
           file: ./apps/${{ matrix.package }}/Containerfile
           platforms: linux/amd64,linux/arm64

.github/workflows/publish-worker.yml

@@ -8,6 +8,11 @@ on:
         type: string
         required: false
         default: ""
+      image_registry:
+        description: The registry namespace to publish under (e.g. ghcr.io/<owner>)
+        type: string
+        required: false
+        default: ""
     secrets:
       DOCKERHUB_USERNAME:
         required: false
@@ -56,12 +61,12 @@ jobs:
           tag: ${{ inputs.image_tag }}
 
       - name: 🐋 Set up Docker Buildx
-        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
+        uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
 
       # ..to avoid rate limits when pulling images
       - name: 🐳 Login to DockerHub
         if: ${{ env.DOCKERHUB_USERNAME }}
-        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
@@ -72,7 +77,7 @@ jobs:
 
       # ..to push image
       - name: 🐙 Login to GitHub Container Registry
-        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
         with:
           registry: ghcr.io
           username: ${{ github.repository_owner }}
@@ -83,7 +88,10 @@ jobs:
           docker tag infra_image "$REGISTRY/$REPOSITORY:$IMAGE_TAG"
           docker push "$REGISTRY/$REPOSITORY:$IMAGE_TAG"
         env:
-          REGISTRY: ghcr.io/triggerdotdev
+          # Resolved by the caller when invoked from publish.yml; falls back to the
+          # IMAGE_REGISTRY repository variable (or ghcr.io/<owner>) for the direct
+          # push triggers above, so a fork publishes to its own namespace.
+          REGISTRY: ${{ inputs.image_registry || vars.IMAGE_REGISTRY || format('ghcr.io/{0}', github.repository_owner) }}
           REPOSITORY: ${{ steps.get_repository.outputs.repo }}
           IMAGE_TAG: ${{ steps.get_tag.outputs.tag }}
 

.github/workflows/publish.yml

@@ -15,6 +15,8 @@ on:
         required: false
       SENTRY_AUTH_TOKEN:
         required: false
+      CROSS_REPO_PAT:
+        required: false
   push:
     branches:
       - main
@@ -74,6 +76,9 @@ jobs:
       SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
     with:
       image_tag: ${{ inputs.image_tag }}
+      # Target registry namespace. Defaults to ghcr.io/<owner> so a fork publishes
+      # to its own namespace; set the IMAGE_REGISTRY repository variable to override.
+      image_registry: ${{ vars.IMAGE_REGISTRY || format('ghcr.io/{0}', github.repository_owner) }}
 
   publish-worker:
     needs: [typecheck]
@@ -86,6 +91,7 @@ jobs:
       DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
     with:
       image_tag: ${{ inputs.image_tag }}
+      image_registry: ${{ vars.IMAGE_REGISTRY || format('ghcr.io/{0}', github.repository_owner) }}
 
   publish-worker-v4:
     needs: [typecheck]
@@ -96,3 +102,62 @@ jobs:
     uses: ./.github/workflows/publish-worker-v4.yml
     with:
       image_tag: ${{ inputs.image_tag }}
+      image_registry: ${{ vars.IMAGE_REGISTRY || format('ghcr.io/{0}', github.repository_owner) }}
+
+  # OS-level CVE scan of the image just published above. Report-only (writes to
+  # the run summary); runs alongside the worker publishes and never blocks them.
+  scan-webapp:
+    needs: [publish-webapp]
+    permissions:
+      contents: read
+      packages: read # pull the just-published image from GHCR
+    uses: ./.github/workflows/trivy-image-webapp.yml
+    with:
+      image-ref: ${{ needs.publish-webapp.outputs.image_repo }}:${{ needs.publish-webapp.outputs.version }}
+
+  # Announce the freshly published mutable `main` webapp image to subscriber
+  # repos via repository_dispatch, handing them a digest-pinned ref to build or
+  # deploy from. The repo, ref prefix, and dispatch target all default to the
+  # canonical values and can be overridden by repository variables.
+  #
+  # `push` only: release builds reach publish.yml via workflow_call (from
+  # release.yml) with an explicit image_tag while github.ref_name is still
+  # `main`, so gate on the event to avoid dispatching — and failing on the
+  # absent CROSS_REPO_PAT — during a release.
+  dispatch-main-image:
+    name: 📣 Dispatch main image
+    needs: [publish-webapp]
+    if: github.repository == (vars.MAIN_IMAGE_DISPATCH_REPO || 'triggerdotdev/trigger.dev') && github.event_name == 'push' && startsWith(github.ref_name, vars.MAIN_IMAGE_DISPATCH_REF_PREFIX || 'main')
+    runs-on: ubuntu-latest
+    permissions: {}
+    steps:
+      - name: Build dispatch payload
+        id: payload
+        env:
+          IMAGE_REPO: ${{ needs.publish-webapp.outputs.image_repo }}
+          DIGEST: ${{ needs.publish-webapp.outputs.digest }}
+          COMMIT: ${{ github.sha }}
+        run: |
+          set -euo pipefail
+          # Pin to the exact multi-arch index just pushed so subscribers resolve a
+          # single immutable artifact rather than chasing the moving `main` tag.
+          if [[ -z "${DIGEST}" ]]; then
+            echo "::error::publish-webapp produced no image digest; refusing to dispatch"
+            exit 1
+          fi
+          image="${IMAGE_REPO}@${DIGEST}"
+          # jq --arg JSON-escapes every value, so the ref/commit can't break out of
+          # or inject into the client payload.
+          payload=$(jq -nc \
+            --arg img "$image" \
+            --arg c "$COMMIT" \
+            '{image: $img, commit: $c}')
+          echo "client_payload=$payload" >> "$GITHUB_OUTPUT"
+
+      - name: Send repository_dispatch
+        uses: peter-evans/repository-dispatch@28959ce8df70de7be546dd1250a005dd32156697 # v4.0.1
+        with:
+          token: ${{ secrets.CROSS_REPO_PAT }}
+          repository: ${{ vars.MAIN_IMAGE_DISPATCH_TARGET || 'triggerdotdev/cloud' }}
+          event-type: main-image-published
+          client-payload: ${{ steps.payload.outputs.client_payload }}

.github/workflows/release-helm.yml

@@ -63,6 +63,11 @@ jobs:
 
   release:
     needs: lint-and-test
+    # Set the ENABLE_HELM_PRERELEASE repository variable to 'false' to turn off
+    # publishing the chart to GHCR — e.g. forks/mirrors that lack write_package
+    # on the owner's charts namespace. Defaults to enabled; the lint-and-test
+    # job above always runs regardless.
+    if: ${{ vars.ENABLE_HELM_PRERELEASE != 'false' }}
     runs-on: ubuntu-latest
     permissions:
       contents: write # for gh-release
@@ -87,7 +92,7 @@ jobs:
           for file in ./charts/*.tgz; do echo "Extracting $file"; tar -xzf "$file" -C ./charts; done
 
       - name: Log in to Container Registry
-        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
         with:
           registry: ${{ env.REGISTRY }}
           username: ${{ github.actor }}

.github/workflows/release.yml

@@ -90,7 +90,7 @@ jobs:
       - name: Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           cache: "pnpm"
 
       # npm v11.5.1 or newer is required for OIDC support
@@ -112,7 +112,7 @@ jobs:
 
       - name: Publish
         id: changesets
-        uses: changesets/action@6a0a831ff30acef54f2c6aa1cbbc1096b066edaf # v1.7.0
+        uses: changesets/action@63a615b9cd06ba9a3e6d13796c7fbcb080a60a0b # v1.8.0
         with:
           publish: pnpm run changeset:release
           createGithubReleases: false
@@ -288,7 +288,7 @@ jobs:
       - name: Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           cache: "pnpm"
 
       # npm v11.5.1 or newer is required for OIDC support

.github/workflows/sdk-compat.yml

@@ -70,7 +70,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           cache: "pnpm"
 
       - name: 🥟 Setup Bun
@@ -112,7 +112,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           cache: "pnpm"
 
       - name: 🦕 Setup Deno
@@ -158,7 +158,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           cache: "pnpm"
 
       - name: 📥 Download deps

.github/workflows/trivy-image-webapp.yml

@@ -0,0 +1,75 @@
+name: Trivy Image Scan (webapp)
+
+# OS-level CVE scan of a published webapp image. Called by the publish pipeline
+# (publish.yml) to scan each build right after it's pushed to GHCR — so every
+# main build and every release is scanned, not rebuilt. Also runnable ad-hoc
+# via workflow_dispatch against any image ref.
+#
+# Report-only: writes a table to the run summary. No SARIF upload, no gate.
+# Library/dependency CVEs are covered by Dependabot, so this is restricted to
+# OS packages (`vuln-type: os`) to avoid double-reporting.
+
+on:
+  workflow_call:
+    inputs:
+      image-ref:
+        description: "Full image ref to scan (e.g. ghcr.io/triggerdotdev/trigger.dev:main)"
+        type: string
+        required: true
+  workflow_dispatch:
+    inputs:
+      image-ref:
+        description: "Full image ref to scan"
+        type: string
+        required: false
+        default: "ghcr.io/triggerdotdev/trigger.dev:main"
+
+permissions: {}
+
+concurrency:
+  group: trivy-image-webapp-${{ inputs.image-ref }}
+  cancel-in-progress: true
+
+jobs:
+  scan:
+    name: Scan
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: read # pull the image from GHCR
+    steps:
+      # Authenticate to GHCR so the scan also works for private images
+      # (GITHUB_TOKEN isn't forwarded to Docker automatically). Harmless for
+      # public images. Pairs with the packages: read permission above.
+      - name: Log in to GitHub Container Registry
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Run Trivy image scan
+        uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
+        with:
+          scan-type: image
+          image-ref: ${{ inputs.image-ref }}
+          # vuln-type maps to --pkg-types: OS packages only (library deps are
+          # Dependabot's job). ignore-unfixed drops vulns with no patch yet.
+          vuln-type: os
+          ignore-unfixed: true
+          severity: HIGH,CRITICAL
+          format: table
+          output: trivy-image-webapp.txt
+
+      - name: Job summary
+        if: always()
+        env:
+          IMAGE_REF: ${{ inputs.image-ref }}
+        run: |
+          {
+            echo "## Trivy Image Scan (webapp) — \`${IMAGE_REF}\`"
+            echo '```'
+            # GitHub step summary is capped at 1 MiB; truncate large reports.
+            head -c 900000 trivy-image-webapp.txt 2>/dev/null || echo "(no report produced)"
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"

.github/workflows/typecheck.yml

@@ -25,7 +25,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           cache: "pnpm"
 
       - name: 📥 Download deps

.github/workflows/unit-tests-internal.yml

@@ -16,9 +16,11 @@ jobs:
     name: "🧪 Unit Tests: Internal"
     runs-on: ubuntu-latest
     strategy:
+      # one flaky shard shouldn't cancel its siblings - lets us re-run only the failed shard
+      fail-fast: false
       matrix:
-        shardIndex: [1, 2, 3, 4, 5, 6, 7, 8]
-        shardTotal: [8]
+        shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
+        shardTotal: [12]
     env:
       DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
       SHARD_INDEX: ${{ matrix.shardIndex }}
@@ -53,7 +55,7 @@ jobs:
       - name: ⬇️ Checkout repo
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
-          fetch-depth: 0
+          fetch-depth: 1
           persist-credentials: false
 
       - name: ⎔ Setup pnpm
@@ -64,13 +66,13 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           cache: "pnpm"
 
       # ..to avoid rate limits when pulling images
       - name: 🐳 Login to DockerHub
         if: ${{ env.DOCKERHUB_USERNAME }}
-        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
@@ -81,12 +83,22 @@ jobs:
       - name: 🐳 Pre-pull testcontainer images
         if: ${{ env.DOCKERHUB_USERNAME }}
         run: |
+          # Retry each pull - DockerHub registry timeouts are a recurring transient CI flake.
+          pull() {
+            for attempt in 1 2 3; do
+              docker pull "$1" && return 0
+              echo "::warning::docker pull $1 failed (attempt ${attempt}/3); retrying in 10s"
+              sleep 10
+            done
+            echo "::error::docker pull $1 failed after 3 attempts"
+            return 1
+          }
           echo "Pre-pulling Docker images with authenticated session..."
-          docker pull postgres:14
-          docker pull clickhouse/clickhouse-server:25.4-alpine
-          docker pull redis:7-alpine
-          docker pull testcontainers/ryuk:0.11.0
-          docker pull electricsql/electric:1.2.4
+          pull postgres:14
+          pull clickhouse/clickhouse-server:26.2.19.43-alpine@sha256:c6ad6a7eb2fb5999df3adfb8b69a0c7222c68fa9b8f6b04a088564ebbc959251
+          pull redis:7.2
+          pull testcontainers/ryuk:0.14.0
+          pull electricsql/electric:1.2.4
           echo "Image pre-pull complete"
 
       - name: 📥 Download deps
@@ -96,7 +108,7 @@ jobs:
         run: pnpm run generate
 
       - name: 🧪 Run Internal Unit Tests
-        run: pnpm run test:internal --reporter=default --reporter=blob --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
+        run: pnpm run test:internal --reporter=default --reporter=blob --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --passWithNoTests
 
       - name: Gather all reports
         if: ${{ !cancelled() }}
@@ -123,7 +135,7 @@ jobs:
       - name: ⬇️ Checkout repo
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
-          fetch-depth: 0
+          fetch-depth: 1
           persist-credentials: false
 
       - name: ⎔ Setup pnpm
@@ -134,7 +146,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           # no cache enabled, we're not installing deps
 
       - name: Download blob reports from GitHub Actions Artifacts
@@ -145,4 +157,4 @@ jobs:
           merge-multiple: true
 
       - name: Merge reports
-        run: pnpm dlx vitest@3.1.4 run --merge-reports --pass-with-no-tests
+        run: pnpm dlx vitest@4.1.7 run --merge-reports --pass-with-no-tests

.github/workflows/unit-tests-packages.yml

@@ -16,9 +16,11 @@ jobs:
     name: "🧪 Unit Tests: Packages"
     runs-on: ubuntu-latest
     strategy:
+      # one flaky shard shouldn't cancel its siblings - lets us re-run only the failed shard
+      fail-fast: false
       matrix:
-        shardIndex: [1]
-        shardTotal: [1]
+        shardIndex: [1, 2, 3]
+        shardTotal: [3]
     env:
       DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
       SHARD_INDEX: ${{ matrix.shardIndex }}
@@ -53,7 +55,7 @@ jobs:
       - name: ⬇️ Checkout repo
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
-          fetch-depth: 0
+          fetch-depth: 1
           persist-credentials: false
 
       - name: ⎔ Setup pnpm
@@ -64,13 +66,13 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           cache: "pnpm"
 
       # ..to avoid rate limits when pulling images
       - name: 🐳 Login to DockerHub
         if: ${{ env.DOCKERHUB_USERNAME }}
-        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
@@ -81,12 +83,22 @@ jobs:
       - name: 🐳 Pre-pull testcontainer images
         if: ${{ env.DOCKERHUB_USERNAME }}
         run: |
+          # Retry each pull - DockerHub registry timeouts are a recurring transient CI flake.
+          pull() {
+            for attempt in 1 2 3; do
+              docker pull "$1" && return 0
+              echo "::warning::docker pull $1 failed (attempt ${attempt}/3); retrying in 10s"
+              sleep 10
+            done
+            echo "::error::docker pull $1 failed after 3 attempts"
+            return 1
+          }
           echo "Pre-pulling Docker images with authenticated session..."
-          docker pull postgres:14
-          docker pull clickhouse/clickhouse-server:25.4-alpine
-          docker pull redis:7-alpine
-          docker pull testcontainers/ryuk:0.11.0
-          docker pull electricsql/electric:1.2.4
+          pull postgres:14
+          pull clickhouse/clickhouse-server:26.2.19.43-alpine@sha256:c6ad6a7eb2fb5999df3adfb8b69a0c7222c68fa9b8f6b04a088564ebbc959251
+          pull redis:7.2
+          pull testcontainers/ryuk:0.14.0
+          pull electricsql/electric:1.2.4
           echo "Image pre-pull complete"
 
       - name: 📥 Download deps
@@ -96,7 +108,7 @@ jobs:
         run: pnpm run generate
 
       - name: 🧪 Run Package Unit Tests
-        run: pnpm run test:packages --reporter=default --reporter=blob --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
+        run: pnpm run test:packages --reporter=default --reporter=blob --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --passWithNoTests
 
       - name: Gather all reports
         if: ${{ !cancelled() }}
@@ -123,7 +135,7 @@ jobs:
       - name: ⬇️ Checkout repo
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
-          fetch-depth: 0
+          fetch-depth: 1
           persist-credentials: false
 
       - name: ⎔ Setup pnpm
@@ -134,7 +146,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           # no cache enabled, we're not installing deps
 
       - name: Download blob reports from GitHub Actions Artifacts
@@ -145,4 +157,4 @@ jobs:
           merge-multiple: true
 
       - name: Merge reports
-        run: pnpm dlx vitest@3.1.4 run --merge-reports --pass-with-no-tests
+        run: pnpm dlx vitest@4.1.7 run --merge-reports --pass-with-no-tests

.github/workflows/unit-tests-webapp.yml

@@ -16,9 +16,11 @@ jobs:
     name: "🧪 Unit Tests: Webapp"
     runs-on: ubuntu-latest
     strategy:
+      # one flaky shard shouldn't cancel its siblings - lets us re-run only the failed shard
+      fail-fast: false
       matrix:
-        shardIndex: [1, 2, 3, 4, 5, 6, 7, 8]
-        shardTotal: [8]
+        shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
+        shardTotal: [10]
     env:
       DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
       SHARD_INDEX: ${{ matrix.shardIndex }}
@@ -53,7 +55,7 @@ jobs:
       - name: ⬇️ Checkout repo
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
-          fetch-depth: 0
+          fetch-depth: 1
           persist-credentials: false
 
       - name: ⎔ Setup pnpm
@@ -64,13 +66,13 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           cache: "pnpm"
 
       # ..to avoid rate limits when pulling images
       - name: 🐳 Login to DockerHub
         if: ${{ env.DOCKERHUB_USERNAME }}
-        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
@@ -81,12 +83,23 @@ jobs:
       - name: 🐳 Pre-pull testcontainer images
         if: ${{ env.DOCKERHUB_USERNAME }}
         run: |
+          # Retry each pull - DockerHub registry timeouts are a recurring transient CI flake.
+          pull() {
+            for attempt in 1 2 3; do
+              docker pull "$1" && return 0
+              echo "::warning::docker pull $1 failed (attempt ${attempt}/3); retrying in 10s"
+              sleep 10
+            done
+            echo "::error::docker pull $1 failed after 3 attempts"
+            return 1
+          }
           echo "Pre-pulling Docker images with authenticated session..."
-          docker pull postgres:14
-          docker pull clickhouse/clickhouse-server:25.4-alpine
-          docker pull redis:7-alpine
-          docker pull testcontainers/ryuk:0.11.0
-          docker pull electricsql/electric:1.2.4
+          pull postgres:14
+          pull clickhouse/clickhouse-server:26.2.19.43-alpine@sha256:c6ad6a7eb2fb5999df3adfb8b69a0c7222c68fa9b8f6b04a088564ebbc959251
+          pull redis:7.2
+          pull testcontainers/ryuk:0.14.0
+          pull electricsql/electric:1.2.4
+          pull minio/minio:latest
           echo "Image pre-pull complete"
 
       - name: 📥 Download deps
@@ -96,7 +109,7 @@ jobs:
         run: pnpm run generate
 
       - name: 🧪 Run Webapp Unit Tests
-        run: pnpm run test:webapp --reporter=default --reporter=blob --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
+        run: pnpm run test:webapp --reporter=default --reporter=blob --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --passWithNoTests
         env:
           DATABASE_URL: postgresql://postgres:postgres@localhost:5432/postgres
           DIRECT_URL: postgresql://postgres:postgres@localhost:5432/postgres
@@ -131,7 +144,7 @@ jobs:
       - name: ⬇️ Checkout repo
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
-          fetch-depth: 0
+          fetch-depth: 1
           persist-credentials: false
 
       - name: ⎔ Setup pnpm
@@ -142,7 +155,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 20.20.0
+          node-version: 20.20.2
           # no cache enabled, we're not installing deps
 
       - name: Download blob reports from GitHub Actions Artifacts
@@ -153,4 +166,4 @@ jobs:
           merge-multiple: true
 
       - name: Merge reports
-        run: pnpm dlx vitest@3.1.4 run --merge-reports --pass-with-no-tests
+        run: pnpm dlx vitest@4.1.7 run --merge-reports --pass-with-no-tests

.github/workflows/workflow-checks.yml

@@ -36,6 +36,11 @@ jobs:
 
   zizmor:
     name: Zizmor
+    # Uploads SARIF to the Security tab, which requires GitHub code scanning to be
+    # enabled on the repository. Set the ENABLE_WORKFLOW_SECURITY_SCAN repository
+    # variable to 'false' to skip this job where code scanning isn't available;
+    # leave it unset (the default) to run the scan.
+    if: ${{ vars.ENABLE_WORKFLOW_SECURITY_SCAN != 'false' }}
     runs-on: ubuntu-latest
     permissions:
       security-events: write # Upload SARIF to GitHub Security tab
@@ -48,4 +53,4 @@ jobs:
           persist-credentials: false
 
       - name: Run zizmor
-        uses: zizmorcore/zizmor-action@b1d7e1fb5de872772f31590499237e7cce841e8e # v0.5.3
+        uses: zizmorcore/zizmor-action@5f14fd08f7cf1cb1609c1e344975f152c7ee938d # v0.5.6

.gitignore

@@ -72,4 +72,6 @@ apps/**/public/build
 .mcp.log
 .mcp.json
 .cursor/debug.log
-ailogger-output.log
+ailogger-output.log
+# per-package vitest timing capture (transient; merged into root test-timings.json)
+.vitest-timing.json

.nvmrc

@@ -1 +1 @@
-v20.20.0
+v20.20.2

.server-changes/README.md

@@ -38,6 +38,14 @@ Speed up batch queue processing by removing stalls and fixing retry race
 
 The body text (below the frontmatter) is a one-line description of the change. Keep it concise — it will appear in release notes.
 
+### Writing guidance
+
+These entries are public-facing - they ship verbatim in user-visible release notes. A few rules to keep them clean:
+
+- **One sentence is usually enough.** The body is the bullet in the changelog. If you need a paragraph, you're probably describing the implementation rather than the change.
+- **Describe behavior, not implementation.** Skip internal scopes, middleware names, library specifics, framework internals. Users care about what's different for them, not how it's wired.
+- **Never name internal tools or infra.** Observability stacks, internal services, infra components, monitoring backends, CI surfaces, AWS specifics - none of these belong in user-facing notes.
+
 ## Lifecycle
 
 1. Engineer adds a `.server-changes/` file in their PR

.server-changes/llm-spend-currency-label.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Add a currency unit to the agent dashboard "LLM spend" chart label, so it now reads "LLM spend ($)".

.server-changes/logs-search-memory-and-pagination.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: fix
+---
+
+Keep logs search within bounded ClickHouse memory when browsing long time ranges, and fix pagination that could skip or duplicate entries sharing a timestamp.

.server-changes/prisma-infrastructure-error-capture.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: fix
+---
+
+Log Prisma infrastructure errors (P1xxx) centrally and obfuscate their messages (which carry the DB hostname) on API responses that previously returned the raw message, without changing status codes or headers.

.server-changes/rbac-permission-enforcement.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: feature
+---
+
+Enforce role-based permissions across the dashboard and API. New permission boundaries cover: runs (cancel, replay, bulk actions), deployments (rollback, promote, cancel), prompt versions, organization members (invite, resend, revoke), billing and seat purchases, integrations (GitHub and Vercel), and environment variables and API keys (restricted by environment tier). Roles without access can no longer read or change these, gated controls are disabled with a tooltip, and gated pages show a permission-denied panel instead of redirecting away. Behaviour is unchanged in the default configuration, where permissions stay permissive.

.server-changes/route-taskrun-reads-through-run-store.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Route Postgres task run reads through the run store so they can be retargeted to a different backing store without changing call sites.

.server-changes/sso.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: feature
+---
+
+SAML/OIDC single sign-on: SSO login with optional per-domain enforcement, JIT provisioning, and periodic re-validation against the IdP.

.server-changes/task-run-plan-type-clickhouse.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Store the run's plan type on the runs analytics table so reporting can group runs by plan.

.server-changes/task-type-filter-segmented-control.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Replace the Task type filter on the Tasks page with a segmented control: "All" plus icon-only Agent, Standard, and Scheduled segments (each with a tooltip showing its label and number-key shortcut). Filtering is now single-select (one task type at a time) instead of multi-select. Shortcut keys 0–3 select each segment.

.vouch.yml

@@ -1,3 +1,4 @@
 vouch:
   - github: edosrecki
   - github: GautamBytes
+  - github: ConProgramming

.vscode/launch.json

@@ -47,78 +47,6 @@
       "url": "http://localhost:3030",
       "webRoot": "${workspaceFolder}/apps/webapp/app"
     },
-    {
-      "type": "node-terminal",
-      "request": "launch",
-      "name": "Debug V3 init CLI",
-      "command": "pnpm exec trigger init",
-      "cwd": "${workspaceFolder}/references/init-shell",
-      "sourceMaps": true
-    },
-    {
-      "type": "node-terminal",
-      "request": "launch",
-      "name": "Debug V3 init dev CLI",
-      "command": "pnpm exec trigger dev",
-      "cwd": "${workspaceFolder}/references/init-shell",
-      "sourceMaps": true
-    },
-    {
-      "type": "node-terminal",
-      "request": "launch",
-      "name": "Debug V3 Dev CLI",
-      "command": "pnpm exec trigger dev",
-      "cwd": "${workspaceFolder}/references/hello-world",
-      "sourceMaps": true
-    },
-    {
-      "type": "node-terminal",
-      "request": "launch",
-      "name": "Debug Dev Next.js Realtime",
-      "command": "pnpm exec trigger dev",
-      "cwd": "${workspaceFolder}/references/nextjs-realtime",
-      "sourceMaps": true
-    },
-    {
-      "type": "node-terminal",
-      "request": "launch",
-      "name": "Debug prisma-catalog deploy CLI",
-      "command": "pnpm exec trigger deploy --self-hosted --load-image",
-      "cwd": "${workspaceFolder}/references/prisma-catalog",
-      "sourceMaps": true
-    },
-    {
-      "type": "node-terminal",
-      "request": "launch",
-      "name": "Debug V3 Deploy CLI",
-      "command": "pnpm exec trigger deploy --self-hosted --load-image",
-      "cwd": "${workspaceFolder}/references/hello-world",
-      "sourceMaps": true
-    },
-    {
-      "type": "node-terminal",
-      "request": "launch",
-      "name": "Debug V3 list-profiles CLI",
-      "command": "pnpm exec trigger list-profiles --log-level debug",
-      "cwd": "${workspaceFolder}/references/hello-world",
-      "sourceMaps": true
-    },
-    {
-      "type": "node-terminal",
-      "request": "launch",
-      "name": "Debug V3 update CLI",
-      "command": "pnpm exec trigger update",
-      "cwd": "${workspaceFolder}/references/hello-world",
-      "sourceMaps": true
-    },
-    {
-      "type": "node-terminal",
-      "request": "launch",
-      "name": "Debug V3 Management",
-      "command": "pnpm run management",
-      "cwd": "${workspaceFolder}/references/hello-world",
-      "sourceMaps": true
-    },
     {
       "type": "node",
       "request": "attach",
@@ -135,14 +63,6 @@
       "cwd": "${workspaceFolder}/packages/cli-v3",
       "sourceMaps": true
     },
-    {
-      "type": "node-terminal",
-      "request": "launch",
-      "name": "debug v3 hello-world dev",
-      "command": "pnpm exec trigger dev",
-      "cwd": "${workspaceFolder}/references/hello-world",
-      "sourceMaps": true
-    },
     {
       "type": "node-terminal",
       "request": "launch",
@@ -158,14 +78,6 @@
       "command": "pnpm run test ./src/run-queue/index.test.ts --run",
       "cwd": "${workspaceFolder}/internal-packages/run-engine",
       "sourceMaps": true
-    },
-    {
-      "type": "node-terminal",
-      "request": "launch",
-      "name": "Debug d3-demo",
-      "command": "pnpm exec trigger dev",
-      "cwd": "${workspaceFolder}/references/d3-demo",
-      "sourceMaps": true
     }
   ]
 }

.vscode/settings.json

@@ -1,5 +1,5 @@
 {
-  "deno.enablePaths": ["references/deno-reference", "runtime_tests/tests/deno"],
+  "deno.enablePaths": ["runtime_tests/tests/deno"],
   "debug.toolBarLocation": "commandCenter",
   "typescript.tsdk": "node_modules/typescript/lib",
   "search.exclude": {

AGENTS.md

@@ -7,13 +7,13 @@ This repository is a pnpm monorepo managed with Turbo. It contains multiple apps
 - `apps/supervisor` – Node application for executing built tasks.
 - `packages/*` – Published packages such as `@trigger.dev/sdk`, the CLI (`trigger.dev`), and shared libraries.
 - `internal-packages/*` – Internal-only packages used by the webapp and other apps.
-- `references/*` – Example projects for manual testing and development of new features.
+- Example/reference projects for manual testing live in a separate repo: [`triggerdotdev/references`](https://github.com/triggerdotdev/references).
 - `ai/references` – Contains additional documentation including an overview (`repo.md`) and testing guidelines (`tests.md`).
 
 See `ai/references/repo.md` for a more complete explanation of the workspaces.
 
 ## Development setup
-1. Install dependencies with `pnpm i` (pnpm `10.33.2` and Node.js `20.20.0` are required).
+1. Install dependencies with `pnpm i` (pnpm `10.33.2` and Node.js `20.20.2` are required).
 2. Copy `.env.example` to `.env` and generate a random 16 byte hex string for `ENCRYPTION_KEY` (`openssl rand -hex 16`). Update other secrets if needed.
 3. Start the local services with Docker:
    ```bash
@@ -65,5 +65,5 @@ Refer to `ai/references/tests.md` for details on writing tests. Tests should avo
   ```bash
   pnpm run dev --filter docs
   ```
-- `references/README.md` explains how to create new reference projects for manual testing.
+- The [`triggerdotdev/references`](https://github.com/triggerdotdev/references) repo's README explains how to create new reference projects for manual testing.
 

CLAUDE.md

@@ -69,6 +69,17 @@ containerTest("should use both", async ({ prisma, redisOptions }) => {
 });
 ```
 
+## Code Style
+
+### Imports
+
+**Prefer static imports over dynamic imports.** Only use dynamic `import()` when:
+- Circular dependencies cannot be resolved otherwise
+- Code splitting is genuinely needed for performance
+- The module must be loaded conditionally at runtime
+
+Dynamic imports add unnecessary overhead in hot paths and make code harder to analyze. If you find yourself using `await import()`, ask if a regular `import` statement would work instead.
+
 ## Changesets and Server Changes
 
 When modifying any public package (`packages/*` or `integrations/*`), add a changeset:
@@ -127,7 +138,7 @@ Docs live in `docs/` as a Mintlify site (MDX format). See `docs/CLAUDE.md` for c
 
 ### Reference Projects
 
-The `references/` directory contains test workspaces for testing SDK and platform features. Use `references/hello-world` to manually test changes before submitting PRs.
+Reference/example projects for testing SDK and platform features live in a separate repo: [`triggerdotdev/references`](https://github.com/triggerdotdev/references). Clone it alongside this repo and use its `projects/hello-world` to manually test changes before submitting PRs. See that repo's README for setup and linking to a local monorepo build.
 
 ## Docker Image Guidelines
 
@@ -156,15 +167,17 @@ export const myTask = task({
 
 The `rules/` directory contains versioned SDK documentation distributed via the SDK installer. Current version: `rules/manifest.json`. Do NOT update `rules/` or `.claude/skills/trigger-dev-tasks/` unless explicitly asked - these are maintained in separate dedicated passes.
 
-## Testing with hello-world Reference Project
+## Testing with the hello-world Reference Project
+
+The reference projects live in the separate [`triggerdotdev/references`](https://github.com/triggerdotdev/references) repo - clone it alongside this repo.
 
 First-time setup:
 
-1. `pnpm run db:seed` to seed the database
-2. Build CLI: `pnpm run build --filter trigger.dev && pnpm i`
-3. Authorize: `cd references/hello-world && pnpm exec trigger login -a http://localhost:3030`
+1. `pnpm run db:seed` to seed the database (creates the References org + hello-world project)
+2. Build the CLI/packages you want to test: `pnpm run build --filter trigger.dev`
+3. In your `references` clone, follow its README to link to your local monorepo build, then authorize: `cd projects/hello-world && pnpm exec trigger login -a http://localhost:3030`
 
-Running: `cd references/hello-world && pnpm exec trigger dev`
+Running (from your `references` clone): `cd projects/hello-world && pnpm exec trigger dev`
 
 ## Local Task Testing Workflow
 
@@ -179,7 +192,8 @@ curl -s http://localhost:3030/healthcheck  # Verify running
 ### Step 2: Start Trigger Dev in Background
 
 ```bash
-cd references/hello-world && pnpm exec trigger dev
+# in your triggerdotdev/references clone
+cd projects/hello-world && pnpm exec trigger dev
 # Wait for "Local worker ready [node]"
 ```
 

CONTRIBUTING.md

@@ -29,7 +29,7 @@ branch are tagged into a release periodically.
 
 ### Prerequisites
 
-- [Node.js](https://nodejs.org/en) version 20.20.0
+- [Node.js](https://nodejs.org/en) version 20.20.2
 - [pnpm package manager](https://pnpm.io/installation) version 10.33.2
 - [Docker](https://www.docker.com/get-started/)
 - [protobuf](https://github.com/protocolbuffers/protobuf)
@@ -49,7 +49,7 @@ branch are tagged into a release periodically.
    ```
    cd trigger.dev
    ```
-3. Ensure you are on the correct version of Node.js (20.20.0). If you are using `nvm`, there is an `.nvmrc` file that will automatically select the correct version of Node.js when you navigate to the repository.
+3. Ensure you are on the correct version of Node.js (20.20.2). If you are using `nvm`, there is an `.nvmrc` file that will automatically select the correct version of Node.js when you navigate to the repository.
 
 4. Run `corepack enable` to use the correct version of pnpm (`10.33.2`) as specified in the root `package.json` file.
 
@@ -107,24 +107,32 @@ branch are tagged into a release periodically.
 
 ## Manual testing using hello-world
 
-We use the `<root>/references/hello-world` subdirectory as a staging ground for testing changes to the SDK (`@trigger.dev/sdk` at `<root>/packages/trigger-sdk`), the Core package (`@trigger.dev/core` at `<root>packages/core`), the CLI (`trigger.dev` at `<root>/packages/cli-v3`) and the platform (The remix app at `<root>/apps/webapp`). The instructions below will get you started on using the `hello-world` for local development of Trigger.dev.
+The `hello-world` reference project (and the others) live in a separate repo:
+[`triggerdotdev/references`](https://github.com/triggerdotdev/references). Clone it
+alongside this repo. It's the staging ground for testing changes to the SDK
+(`@trigger.dev/sdk` at `<root>/packages/trigger-sdk`), the Core package
+(`@trigger.dev/core` at `<root>/packages/core`), the CLI (`trigger.dev` at
+`<root>/packages/cli-v3`) and the platform (the Remix app at `<root>/apps/webapp`).
+To exercise your local monorepo changes, the reference project links to your local
+build — see the references repo's README for the `pnpm run link` flow.
+
+> Paths below such as `projects/hello-world` are relative to your `references`
+> clone, not this repo.
 
 ### First-time setup
 
 First, make sure you are running the webapp according to the instructions above. The seed step from setup already created a `hello-world` project under the `References` org with the stable ref `proj_rrkpdguyagvsoktglnod` — log in at http://localhost:3030 with any email to access it. Then:
 
-1. Build the CLI (skip if you already ran the build step in setup)
+1. Build the CLI and packages (skip if you already ran the build step in setup)
 
 ```sh
-pnpm run build --filter trigger.dev
-# Make it accessible to `pnpm exec`
-pnpm i
+pnpm run build --filter trigger.dev --filter "@trigger.dev/*"
 ```
 
-2. Change into the `<root>/references/hello-world` directory and authorize the CLI to the local server:
+2. In your `references` clone, link to your local monorepo build (see its README), then change into `projects/hello-world` and authorize the CLI to the local server:
 
 ```sh
-cd references/hello-world
+cd projects/hello-world
 cp .env.example .env
 pnpm exec trigger login -a http://localhost:3030
 ```
@@ -134,7 +142,7 @@ This will open a new browser window and authorize the CLI against your local use
 You can optionally pass a `--profile` flag to the `login` command, which will allow you to use the CLI with separate accounts/servers. We suggest using a profile called `local` for your local development:
 
 ```sh
-cd references/hello-world
+cd projects/hello-world
 pnpm exec trigger login -a http://localhost:3030 --profile local
 # later when you run the dev or deploy command:
 pnpm exec trigger dev --profile local
@@ -147,35 +155,35 @@ The following steps should be followed any time you start working on a new featu
 
 1. Make sure the webapp is running on localhost:3030
 
-2. Open a terminal window and build the CLI and packages and watch for changes
+2. In this repo, open a terminal window and build the CLI and packages and watch for changes (the reference project links against this build)
 
 ```sh
 pnpm run dev --filter trigger.dev --filter "@trigger.dev/*"
 ```
 
-3. Open another terminal window, and change into the `<root>/references/hello-world` directory.
+3. Open another terminal window, and change into `projects/hello-world` in your `references` clone.
 
 4. Run the `dev` command, which will register all the local tasks with the platform and allow you to start testing task execution:
 
 ```sh
-# in <root>/references/hello-world
+# in <references-clone>/projects/hello-world
 pnpm exec trigger dev
 ```
 
 If you want additional debug logging, you can use the `--log-level debug` flag:
 
 ```sh
-# in <root>/references/hello-world
+# in <references-clone>/projects/hello-world
 pnpm exec trigger dev --log-level debug
 ```
 
-5. If you make any changes in the CLI/Core/SDK, you'll need to `CTRL+C` to exit the `dev` command and restart it to pickup changes. Any changes to the files inside of the `hello-world/src/trigger` dir will automatically be rebuilt by the `dev` command.
+5. If you make any changes in the CLI/Core/SDK, you'll need to `CTRL+C` to exit the `dev` command and restart it to pickup changes. Any changes to the files inside the reference project's `src/trigger` dir will automatically be rebuilt by the `dev` command.
 
 6. Navigate to the `hello-world` project in your local dashboard at localhost:3030 and you should see the list of tasks.
 
-7. Go to the "Test" page in the sidebar and select a task. Then enter a payload and click "Run test". You can tell what the payloads should be by looking at the relevant task file inside the `/references/hello-world/src/trigger` folder. Many of them accept an empty payload.
+7. Go to the "Test" page in the sidebar and select a task. Then enter a payload and click "Run test". You can tell what the payloads should be by looking at the relevant task file inside the reference project's `src/trigger` folder. Many of them accept an empty payload.
 
-8. Feel free to add additional files in `hello-world/src/trigger` to test out specific aspects of the system, or add in edge cases.
+8. Feel free to add additional files in the reference project's `src/trigger` dir to test out specific aspects of the system, or add in edge cases.
 
 ## Adding and running migrations
 

apps/supervisor/package.json

@@ -18,6 +18,7 @@
     "@kubernetes/client-node": "^1.0.0",
     "@trigger.dev/core": "workspace:*",
     "dockerode": "^4.0.6",
+    "ioredis": "~5.6.0",
     "p-limit": "^6.2.0",
     "prom-client": "^15.1.0",
     "socket.io": "4.7.4",
@@ -25,6 +26,7 @@
     "zod": "3.25.76"
   },
   "devDependencies": {
+    "@internal/testcontainers": "workspace:*",
     "@types/dockerode": "^3.3.33"
   }
 }

apps/supervisor/src/backpressure/backpressureMetrics.ts

@@ -0,0 +1,34 @@
+import { Counter, Gauge, type Registry } from "prom-client";
+
+/** Prometheus metrics for dequeue backpressure. */
+export class BackpressureMetrics {
+  /** 1 while backpressure is engaged (computed signal, set even in dry-run). */
+  readonly engaged: Gauge<string>;
+  /** 1 when running in dry-run (gates inert). */
+  readonly dryRun: Gauge<string>;
+  /** Dequeue attempts the gate skipped - or would have, in dry-run (labelled). */
+  readonly skipsTotal: Counter<string>;
+
+  constructor(opts: { register: Registry; prefix?: string }) {
+    const prefix = opts.prefix ?? "supervisor_backpressure";
+
+    this.engaged = new Gauge({
+      name: `${prefix}_engaged`,
+      help: "1 while dequeue backpressure is engaged (computed signal, regardless of dry-run)",
+      registers: [opts.register],
+    });
+
+    this.dryRun = new Gauge({
+      name: `${prefix}_dry_run`,
+      help: "1 when dequeue backpressure is in dry-run mode (gates inert)",
+      registers: [opts.register],
+    });
+
+    this.skipsTotal = new Counter({
+      name: `${prefix}_skipped_dequeues_total`,
+      help: "Dequeue attempts skipped by backpressure (or would be, in dry-run)",
+      labelNames: ["dry_run"],
+      registers: [opts.register],
+    });
+  }
+}

apps/supervisor/src/backpressure/backpressureMonitor.test.ts

@@ -0,0 +1,353 @@
+import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
+import { Registry } from "prom-client";
+import { BackpressureMonitor, type BackpressureSignalSource } from "./backpressureMonitor.js";
+import { BackpressureMetrics } from "./backpressureMetrics.js";
+
+function countingSource(verdict: { engaged: boolean } | null): {
+  source: BackpressureSignalSource;
+  reads: () => number;
+} {
+  let reads = 0;
+  return {
+    source: {
+      read: async () => {
+        reads++;
+        return verdict;
+      },
+    },
+    reads: () => reads,
+  };
+}
+
+describe("BackpressureMonitor", () => {
+  beforeEach(() => {
+    vi.useFakeTimers();
+  });
+
+  afterEach(() => {
+    vi.useRealTimers();
+  });
+
+  it("when disabled, never skips dequeue and never reads the signal source", () => {
+    // Even though the source would report "engaged", a disabled monitor must be
+    // a complete no-op: this is the backwards-compatibility guarantee.
+    const { source, reads } = countingSource({ engaged: true });
+    const monitor = new BackpressureMonitor({ enabled: false, source });
+
+    monitor.start();
+
+    expect(monitor.shouldSkipDequeue()).toBe(false);
+    expect(reads()).toBe(0);
+
+    monitor.stop();
+  });
+
+  it("when enabled and the source reports engaged, skips dequeue after a refresh", async () => {
+    const { source } = countingSource({ engaged: true });
+    const monitor = new BackpressureMonitor({ enabled: true, source, refreshIntervalMs: 1000 });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0); // flush the initial async read
+
+    expect(monitor.shouldSkipDequeue()).toBe(true);
+
+    monitor.stop();
+  });
+
+  it("when enabled and the source reports clear, does not skip dequeue", async () => {
+    const { source } = countingSource({ engaged: false });
+    const monitor = new BackpressureMonitor({ enabled: true, source, refreshIntervalMs: 1000 });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+
+    expect(monitor.shouldSkipDequeue()).toBe(false);
+
+    monitor.stop();
+  });
+
+  it("fails open (stops skipping) when the source throws", async () => {
+    let call = 0;
+    const source: BackpressureSignalSource = {
+      read: async () => {
+        call++;
+        if (call === 1) {
+          return { engaged: true };
+        }
+        throw new Error("signal source unreachable");
+      },
+    };
+    const monitor = new BackpressureMonitor({ enabled: true, source, refreshIntervalMs: 1000 });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+    expect(monitor.shouldSkipDequeue()).toBe(true); // engaged from the first read
+
+    await vi.advanceTimersByTimeAsync(1000); // next refresh throws
+    expect(monitor.shouldSkipDequeue()).toBe(false); // fail-open: a dead source must not pin the brake
+
+    monitor.stop();
+  });
+
+  it("fails open when the source reports unknown (null)", async () => {
+    const { source } = countingSource(null);
+    const monitor = new BackpressureMonitor({ enabled: true, source, refreshIntervalMs: 1000 });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+
+    expect(monitor.shouldSkipDequeue()).toBe(false);
+
+    monitor.stop();
+  });
+
+  it("fails open when the cached verdict goes stale (older than max age)", async () => {
+    // Source stops updating (e.g. hangs) after the first read; the verdict ages out.
+    const source: BackpressureSignalSource = {
+      read: async () => ({ engaged: true, ts: Date.now() }),
+    };
+    const monitor = new BackpressureMonitor({
+      enabled: true,
+      source,
+      refreshIntervalMs: 1_000_000, // effectively only the initial read fires
+      maxVerdictAgeMs: 15_000,
+    });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+    expect(monitor.shouldSkipDequeue()).toBe(true);
+
+    await vi.advanceTimersByTimeAsync(15_001); // verdict now older than max age
+    expect(monitor.shouldSkipDequeue()).toBe(false);
+
+    monitor.stop();
+  });
+
+  it("does not read the source on the hot path (reads are driven by the refresh tick)", async () => {
+    const { source, reads } = countingSource({ engaged: true });
+    const monitor = new BackpressureMonitor({ enabled: true, source, refreshIntervalMs: 1000 });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+    expect(reads()).toBe(1); // just the initial refresh
+
+    for (let i = 0; i < 1000; i++) {
+      monitor.shouldSkipDequeue();
+    }
+
+    expect(reads()).toBe(1); // hot-path calls performed zero I/O
+
+    monitor.stop();
+  });
+
+  it("does not start an overlapping refresh while one is in flight", async () => {
+    let reads = 0;
+    const source: BackpressureSignalSource = {
+      // Never resolves - simulates a hung read.
+      read: () => {
+        reads++;
+        return new Promise<{ engaged: boolean } | null>(() => {});
+      },
+    };
+    const monitor = new BackpressureMonitor({ enabled: true, source, refreshIntervalMs: 1000 });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(3000); // several intervals while the first read hangs
+
+    expect(reads).toBe(1); // in-flight guard prevents stacking
+
+    monitor.stop();
+  });
+
+  it("stops refreshing after stop()", async () => {
+    const { source, reads } = countingSource({ engaged: true });
+    const monitor = new BackpressureMonitor({ enabled: true, source, refreshIntervalMs: 1000 });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+    const readsAtStop = reads();
+
+    monitor.stop();
+    await vi.advanceTimersByTimeAsync(5000);
+
+    expect(reads()).toBe(readsAtStop);
+  });
+
+  it("isEngaged reflects the hard engaged state (the signal for freezing scale-up)", async () => {
+    const { source } = countingSource({ engaged: true });
+    const monitor = new BackpressureMonitor({ enabled: true, source, refreshIntervalMs: 1000 });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+
+    expect(monitor.isEngaged()).toBe(true);
+
+    monitor.stop();
+  });
+
+  it("isEngaged is false when clear and when stale", async () => {
+    const source: BackpressureSignalSource = {
+      read: async () => ({ engaged: true, ts: Date.now() }),
+    };
+    const monitor = new BackpressureMonitor({
+      enabled: true,
+      source,
+      refreshIntervalMs: 1_000_000,
+      maxVerdictAgeMs: 15_000,
+    });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+    expect(monitor.isEngaged()).toBe(true);
+
+    await vi.advanceTimersByTimeAsync(15_001); // stale → fail-open
+    expect(monitor.isEngaged()).toBe(false);
+
+    monitor.stop();
+  });
+
+  it("ramps the dequeue gate after release instead of resuming instantly", async () => {
+    let engaged = true;
+    let rnd = 0.5;
+    const source: BackpressureSignalSource = { read: async () => ({ engaged }) };
+    const monitor = new BackpressureMonitor({
+      enabled: true,
+      source,
+      refreshIntervalMs: 1000,
+      rampMs: 10_000,
+      random: () => rnd,
+    });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+    expect(monitor.shouldSkipDequeue()).toBe(true); // hard engaged
+
+    // Release: the next refresh observes the clear verdict and starts the ramp.
+    engaged = false;
+    await vi.advanceTimersByTimeAsync(1000);
+    expect(monitor.isEngaged()).toBe(false);
+
+    // Just after release (progress ~0): skip probability ~1, so skip regardless.
+    rnd = 0.99;
+    expect(monitor.shouldSkipDequeue()).toBe(true);
+
+    // Halfway through the ramp (progress 0.5): skip probability 0.5.
+    await vi.advanceTimersByTimeAsync(5000);
+    rnd = 0.4;
+    expect(monitor.shouldSkipDequeue()).toBe(true); // 0.4 < 0.5 → skip
+    rnd = 0.6;
+    expect(monitor.shouldSkipDequeue()).toBe(false); // 0.6 ≥ 0.5 → allow
+
+    // Past the ramp window: never skip.
+    await vi.advanceTimersByTimeAsync(5000);
+    rnd = 0.0;
+    expect(monitor.shouldSkipDequeue()).toBe(false);
+
+    monitor.stop();
+  });
+
+  it("fails open on an engaged verdict with no timestamp when staleness is enforced", async () => {
+    // A verdict claiming engaged but carrying no ts can't be checked for freshness;
+    // when maxVerdictAgeMs is set we must not trust it (else a dead producer could
+    // pin the brake forever).
+    const { source } = countingSource({ engaged: true }); // no ts
+    const monitor = new BackpressureMonitor({
+      enabled: true,
+      source,
+      refreshIntervalMs: 1000,
+      maxVerdictAgeMs: 15_000,
+    });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+
+    expect(monitor.computeEngaged()).toBe(false);
+    expect(monitor.shouldSkipDequeue()).toBe(false);
+
+    monitor.stop();
+  });
+
+  it("in dry-run, the gates are inert but computeEngaged still reflects the real signal", async () => {
+    const { source } = countingSource({ engaged: true });
+    const monitor = new BackpressureMonitor({
+      enabled: true,
+      source,
+      refreshIntervalMs: 1000,
+      dryRun: true,
+    });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+
+    expect(monitor.computeEngaged()).toBe(true); // real signal, for observability/metrics
+    expect(monitor.isEngaged()).toBe(false); // inert: no scale-up freeze
+    expect(monitor.shouldSkipDequeue()).toBe(false); // inert: no dequeue skip
+
+    monitor.stop();
+  });
+
+  it("logs on verdict transitions", async () => {
+    let engaged = true;
+    const source: BackpressureSignalSource = { read: async () => ({ engaged }) };
+    const logs: Array<{ message: string; meta?: Record<string, unknown> }> = [];
+    const logger = {
+      info: (message: string, meta?: Record<string, unknown>) => logs.push({ message, meta }),
+    };
+    const monitor = new BackpressureMonitor({
+      enabled: true,
+      source,
+      refreshIntervalMs: 1000,
+      logger,
+    });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+    expect(logs.some((l) => l.meta?.engaged === true)).toBe(true);
+
+    engaged = false;
+    await vi.advanceTimersByTimeAsync(1000);
+    expect(logs.some((l) => l.meta?.engaged === false)).toBe(true);
+
+    monitor.stop();
+  });
+
+  it("records prometheus metrics", async () => {
+    const { source } = countingSource({ engaged: true });
+    const register = new Registry();
+    const metrics = new BackpressureMetrics({ register });
+    const monitor = new BackpressureMonitor({
+      enabled: true,
+      source,
+      refreshIntervalMs: 1000,
+      metrics,
+    });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+
+    expect(await register.metrics()).toContain("supervisor_backpressure_engaged 1");
+
+    monitor.shouldSkipDequeue();
+    expect(await register.metrics()).toMatch(
+      /supervisor_backpressure_skipped_dequeues_total\{dry_run="false"\} [1-9]/
+    );
+
+    monitor.stop();
+  });
+
+  it("resumes instantly when no ramp is configured", async () => {
+    let engaged = true;
+    const source: BackpressureSignalSource = { read: async () => ({ engaged }) };
+    const monitor = new BackpressureMonitor({ enabled: true, source, refreshIntervalMs: 1000 });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+    expect(monitor.shouldSkipDequeue()).toBe(true);
+
+    engaged = false;
+    await vi.advanceTimersByTimeAsync(1000);
+    expect(monitor.shouldSkipDequeue()).toBe(false); // no ramp → instant resume
+
+    monitor.stop();
+  });
+});

apps/supervisor/src/backpressure/backpressureMonitor.ts

@@ -0,0 +1,179 @@
+import type { BackpressureMetrics } from "./backpressureMetrics.js";
+
+export interface BackpressureLogger {
+  info(message: string, meta?: Record<string, unknown>): void;
+}
+
+export type BackpressureVerdict = {
+  engaged: boolean;
+  /** Epoch ms the verdict was produced. Used for consumer-side staleness fail-open. */
+  ts?: number;
+};
+
+/**
+ * Source of the current backpressure verdict. `read()` returns `null` when the
+ * verdict is unknown (missing/unreadable) - the monitor treats unknown as
+ * "not engaged" (fail-open).
+ */
+export interface BackpressureSignalSource {
+  read(): Promise<BackpressureVerdict | null>;
+}
+
+export type BackpressureMonitorOptions = {
+  enabled: boolean;
+  source: BackpressureSignalSource;
+  refreshIntervalMs?: number;
+  /**
+   * If set, a cached verdict older than this is treated as unknown (fail-open).
+   * Guards against the source silently going stale (e.g. hanging reads).
+   */
+  maxVerdictAgeMs?: number;
+  /**
+   * If set, after backpressure releases the dequeue gate stays partially engaged
+   * for this long, skipping a linearly-decaying fraction of attempts so the
+   * aggregate dequeue rate ramps from ~0 to full instead of snapping to full and
+   * re-flooding a freshly-recovered cluster. 0/unset = instant resume.
+   */
+  rampMs?: number;
+  /** Injectable RNG for the resume ramp; defaults to Math.random. */
+  random?: () => number;
+  /**
+   * When true, the gates are inert (never skip dequeues, never freeze scale-up).
+   * computeEngaged() still reflects the real signal so it can be observed.
+   */
+  dryRun?: boolean;
+  logger?: BackpressureLogger;
+  metrics?: BackpressureMetrics;
+};
+
+const DEFAULT_REFRESH_INTERVAL_MS = 1000;
+
+export class BackpressureMonitor {
+  private verdict: BackpressureVerdict | null = null;
+  private timer?: ReturnType<typeof setInterval>;
+  private refreshInFlight = false;
+  private wasEngaged = false;
+  private releasedAt?: number;
+
+  constructor(private readonly opts: BackpressureMonitorOptions) {
+    this.opts.metrics?.dryRun.set(this.opts.dryRun ? 1 : 0);
+  }
+
+  start(): void {
+    if (!this.opts.enabled) {
+      return;
+    }
+
+    void this.refreshTick();
+    this.timer = setInterval(
+      () => void this.refreshTick(),
+      this.opts.refreshIntervalMs ?? DEFAULT_REFRESH_INTERVAL_MS
+    );
+  }
+
+  /** Skip a tick if the previous refresh is still in flight, so slow/hung reads can't stack. */
+  private async refreshTick(): Promise<void> {
+    if (this.refreshInFlight) {
+      return;
+    }
+    this.refreshInFlight = true;
+    try {
+      await this.refresh();
+    } finally {
+      this.refreshInFlight = false;
+    }
+  }
+
+  stop(): void {
+    if (this.timer) {
+      clearInterval(this.timer);
+      this.timer = undefined;
+    }
+  }
+
+  /**
+   * Raw hard backpressure state: true while the (fresh) verdict says engaged,
+   * ignoring dry-run. Used for observability/metrics so the real signal is
+   * visible even when the gates are inert.
+   */
+  computeEngaged(): boolean {
+    const verdict = this.verdict;
+    if (verdict?.engaged !== true) {
+      return false;
+    }
+
+    // When staleness enforcement is on, an engaged verdict must carry a fresh
+    // timestamp. A missing or stale ts can't be trusted (a dead producer could
+    // otherwise pin the brake forever), so fail open.
+    const maxAge = this.opts.maxVerdictAgeMs;
+    if (maxAge !== undefined) {
+      if (verdict.ts === undefined || Date.now() - verdict.ts > maxAge) {
+        return false;
+      }
+    }
+
+    return true;
+  }
+
+  /**
+   * Effective hard state: the signal for freezing consumer-pool scale-up. Inert
+   * (false) in dry-run. Hot-path read, no I/O.
+   */
+  isEngaged(): boolean {
+    return this.opts.dryRun ? false : this.computeEngaged();
+  }
+
+  /** Hot-path read: synchronous, never performs I/O. Inert (false) in dry-run. */
+  shouldSkipDequeue(): boolean {
+    const wouldSkip = this.computeShouldSkip();
+    if (wouldSkip) {
+      this.opts.metrics?.skipsTotal.inc({ dry_run: this.opts.dryRun ? "true" : "false" });
+    }
+    return this.opts.dryRun ? false : wouldSkip;
+  }
+
+  private computeShouldSkip(): boolean {
+    if (this.computeEngaged()) {
+      return true;
+    }
+
+    // Post-release ramp: skip a linearly-decaying fraction of attempts so the
+    // aggregate dequeue rate climbs back to full over rampMs rather than snapping.
+    const rampMs = this.opts.rampMs;
+    if (rampMs && this.releasedAt !== undefined) {
+      const elapsed = Date.now() - this.releasedAt;
+      if (elapsed < rampMs) {
+        const skipProbability = 1 - elapsed / rampMs;
+        return (this.opts.random ?? Math.random)() < skipProbability;
+      }
+    }
+
+    return false;
+  }
+
+  private async refresh(): Promise<void> {
+    try {
+      this.verdict = await this.opts.source.read();
+    } catch {
+      // Fail-open: a dead/unreachable source must never pin the brake. Treat as
+      // unknown (no verdict) so dequeue resumes as if backpressure were off.
+      this.verdict = null;
+    }
+
+    // Track the engaged→released transition to anchor the resume ramp. Use the
+    // staleness-aware state so a stale verdict doesn't pin wasEngaged / the gauge.
+    const nowEngaged = this.computeEngaged();
+    this.opts.metrics?.engaged.set(nowEngaged ? 1 : 0);
+
+    if (nowEngaged !== this.wasEngaged) {
+      this.opts.logger?.info("backpressure verdict changed", {
+        engaged: nowEngaged,
+        dryRun: !!this.opts.dryRun,
+      });
+    }
+    if (this.wasEngaged && !nowEngaged) {
+      this.releasedAt = Date.now();
+    }
+    this.wasEngaged = nowEngaged;
+  }
+}

apps/supervisor/src/backpressure/redisBackpressureSignalSource.test.ts

@@ -0,0 +1,62 @@
+import { redisTest } from "@internal/testcontainers";
+import { Redis } from "ioredis";
+import { describe, expect } from "vitest";
+import { RedisBackpressureSignalSource } from "./redisBackpressureSignalSource.js";
+
+const KEY = "backpressure:test";
+
+describe("RedisBackpressureSignalSource", () => {
+  redisTest("returns null when the key is absent", async ({ redisOptions }) => {
+    const redis = new Redis(redisOptions);
+    try {
+      const source = new RedisBackpressureSignalSource(redis, KEY);
+      expect(await source.read()).toBeNull();
+    } finally {
+      await redis.quit();
+    }
+  });
+
+  redisTest("parses a valid engaged verdict", async ({ redisOptions }) => {
+    const redis = new Redis(redisOptions);
+    try {
+      await redis.set(KEY, JSON.stringify({ engaged: true, ts: 1_700_000_000_000 }));
+      const source = new RedisBackpressureSignalSource(redis, KEY);
+      expect(await source.read()).toEqual({ engaged: true, ts: 1_700_000_000_000 });
+    } finally {
+      await redis.quit();
+    }
+  });
+
+  redisTest("parses a clear verdict", async ({ redisOptions }) => {
+    const redis = new Redis(redisOptions);
+    try {
+      await redis.set(KEY, JSON.stringify({ engaged: false }));
+      const source = new RedisBackpressureSignalSource(redis, KEY);
+      expect(await source.read()).toEqual({ engaged: false });
+    } finally {
+      await redis.quit();
+    }
+  });
+
+  redisTest("returns null for malformed JSON (fail-open)", async ({ redisOptions }) => {
+    const redis = new Redis(redisOptions);
+    try {
+      await redis.set(KEY, "not json {");
+      const source = new RedisBackpressureSignalSource(redis, KEY);
+      expect(await source.read()).toBeNull();
+    } finally {
+      await redis.quit();
+    }
+  });
+
+  redisTest("returns null for valid JSON of the wrong shape (fail-open)", async ({ redisOptions }) => {
+    const redis = new Redis(redisOptions);
+    try {
+      await redis.set(KEY, JSON.stringify({ foo: "bar" }));
+      const source = new RedisBackpressureSignalSource(redis, KEY);
+      expect(await source.read()).toBeNull();
+    } finally {
+      await redis.quit();
+    }
+  });
+});

apps/supervisor/src/backpressure/redisBackpressureSignalSource.ts

@@ -0,0 +1,35 @@
+import type { Redis } from "ioredis";
+import { z } from "zod";
+import type { BackpressureSignalSource, BackpressureVerdict } from "./backpressureMonitor.js";
+
+const VerdictSchema = z.object({
+  engaged: z.boolean(),
+  ts: z.number().optional(),
+});
+
+/** Reads the backpressure verdict from a Redis key written by the cluster-side aggregator. */
+export class RedisBackpressureSignalSource implements BackpressureSignalSource {
+  constructor(
+    private readonly redis: Redis,
+    private readonly key: string
+  ) {}
+
+  async read(): Promise<BackpressureVerdict | null> {
+    const raw = await this.redis.get(this.key);
+    if (raw === null) {
+      return null;
+    }
+
+    // A malformed or wrong-shaped value is treated as unknown (null) so the
+    // monitor fails open rather than acting on garbage.
+    let json: unknown;
+    try {
+      json = JSON.parse(raw);
+    } catch {
+      return null;
+    }
+
+    const parsed = VerdictSchema.safeParse(json);
+    return parsed.success ? parsed.data : null;
+  }
+}

apps/supervisor/src/env.ts

@@ -34,6 +34,10 @@ const Env = z
 
     // Dequeue settings (provider mode)
     TRIGGER_DEQUEUE_ENABLED: BoolEnv.default(true),
+    // Which worker-queue class this supervisor fleet serves. "default" pulls the
+    // region queue (standard/agent runs); "scheduled" pulls the dedicated
+    // scheduled-lineage queue. Run a separate fleet per class for isolation.
+    TRIGGER_WORKER_QUEUE_CLASS: z.enum(["default", "scheduled"]).default("default"),
     TRIGGER_DEQUEUE_INTERVAL_MS: z.coerce.number().int().default(250),
     TRIGGER_DEQUEUE_IDLE_INTERVAL_MS: z.coerce.number().int().default(1000),
     TRIGGER_DEQUEUE_MAX_RUN_COUNT: z.coerce.number().int().default(1),
@@ -47,11 +51,44 @@ const Env = z
     TRIGGER_DEQUEUE_SCALING_BATCH_WINDOW_MS: z.coerce.number().int().positive().default(1000), // Batch window for metrics processing (ms)
     TRIGGER_DEQUEUE_SCALING_DAMPING_FACTOR: z.coerce.number().min(0).max(1).default(0.7), // Smooths consumer count changes after EWMA (0=no scaling, 1=immediate)
 
+    // Dequeue backpressure - off by default. When enabled, the supervisor reads a
+    // verdict from Redis (written by the cluster-side aggregator) and pauses dequeues
+    // while the worker cluster can't schedule pods. Disabled = total no-op: no Redis
+    // client is created, no reads happen, and the dequeue loop is unaffected.
+    TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED: BoolEnv.default(false),
+    // Safety default: even when enabled, backpressure only logs what it would do.
+    // Set to false to actually skip dequeues / freeze scale-up.
+    TRIGGER_DEQUEUE_BACKPRESSURE_DRY_RUN: BoolEnv.default(true),
+    TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_KEY: z.string().default("engine:dequeue:backpressure"),
+    TRIGGER_DEQUEUE_BACKPRESSURE_REFRESH_MS: z.coerce.number().int().positive().default(1000),
+    TRIGGER_DEQUEUE_BACKPRESSURE_RAMP_MS: z.coerce.number().int().min(0).default(30_000), // Resume ramp window after release; 0 = instant resume
+
+    TRIGGER_DEQUEUE_BACKPRESSURE_MAX_VERDICT_AGE_MS: z.coerce
+      .number()
+      .int()
+      .positive()
+      .default(15_000), // Stale verdict → fail-open (treat as not engaged)
+    TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_HOST: z.string().optional(),
+    TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_PORT: z.coerce.number().int().optional(),
+    TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_USERNAME: z.string().optional(),
+    TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_PASSWORD: z.string().optional(),
+    TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_TLS_DISABLED: BoolEnv.default(false),
+
     // Optional services
     TRIGGER_WARM_START_URL: z.string().optional(),
     TRIGGER_CHECKPOINT_URL: z.string().optional(),
     TRIGGER_METADATA_URL: z.string().optional(),
 
+    // Warm-start delivery verification: after a warm-start hit, probe the
+    // platform and cold-start the run if no runner acted on the dispatch
+    TRIGGER_WARM_START_VERIFY_ENABLED: BoolEnv.default(false),
+    TRIGGER_WARM_START_VERIFY_DELAY_MS: z.coerce
+      .number()
+      .int()
+      .min(1_000)
+      .max(60_000)
+      .default(10_000),
+
     // Used by the resource monitor
     RESOURCE_MONITOR_ENABLED: BoolEnv.default(false),
     RESOURCE_MONITOR_OVERRIDE_CPU_TOTAL: z.coerce.number().optional(),
@@ -87,6 +124,14 @@ const Env = z
     COMPUTE_TRACE_OTLP_ENDPOINT: z.string().url().optional(), // Override for span export (derived from TRIGGER_API_URL if unset)
     COMPUTE_SNAPSHOT_DELAY_MS: z.coerce.number().int().min(0).max(60_000).default(5_000),
     COMPUTE_SNAPSHOT_DISPATCH_LIMIT: z.coerce.number().int().min(1).max(100).default(10),
+    // Instance create retries for transient placement failures (1 = no retries)
+    COMPUTE_INSTANCE_CREATE_MAX_ATTEMPTS: z.coerce.number().int().min(1).max(10).default(3),
+    COMPUTE_INSTANCE_CREATE_RETRY_BASE_DELAY_MS: z.coerce
+      .number()
+      .int()
+      .min(0)
+      .max(10_000)
+      .default(250),
 
     // Kubernetes settings
     KUBERNETES_FORCE_ENABLED: BoolEnv.default(false),
@@ -256,6 +301,15 @@ const Env = z
     // Debug
     DEBUG: BoolEnv.default(false),
     SEND_RUN_DEBUG_LOGS: BoolEnv.default(false),
+
+    // Wide-event observability - off by default. Emits one flat-keyed JSON
+    // line per natural unit of work (dequeue iteration, HTTP request, socket
+    // lifecycle). High-QPS hotpath, so the kill switch must be honoured.
+    TRIGGER_WIDE_EVENTS_ENABLED: BoolEnv.default(false),
+    // When true, also emit wide events for high-frequency HTTP routes
+    // (heartbeat, snapshots-since, logs/debug). Off in prod to keep event
+    // volume manageable; on in test environments for full-fidelity debugging.
+    TRIGGER_WIDE_EVENTS_NOISY_ROUTES: BoolEnv.default(false),
   })
   .superRefine((data, ctx) => {
     if (data.COMPUTE_SNAPSHOTS_ENABLED && !data.TRIGGER_METADATA_URL) {
@@ -272,6 +326,14 @@ const Env = z
         path: ["TRIGGER_WORKLOAD_API_DOMAIN"],
       });
     }
+    if (data.TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED && !data.TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_HOST) {
+      ctx.addIssue({
+        code: z.ZodIssueCode.custom,
+        message:
+          "TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_HOST is required when TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED is true",
+        path: ["TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_HOST"],
+      });
+    }
   })
   .transform((data) => ({
     ...data,

apps/supervisor/src/index.ts

@@ -21,40 +21,79 @@ import {
   isKubernetesEnvironment,
 } from "@trigger.dev/core/v3/serverOnly";
 import { createK8sApi } from "./clients/kubernetes.js";
-import { collectDefaultMetrics } from "prom-client";
+import { collectDefaultMetrics, Histogram } from "prom-client";
 import { register } from "./metrics.js";
 import { PodCleaner } from "./services/podCleaner.js";
 import { FailedPodHandler } from "./services/failedPodHandler.js";
 import { getWorkerToken } from "./workerToken.js";
 import { OtlpTraceService } from "./services/otlpTraceService.js";
+import {
+  WarmStartVerificationService,
+  type WarmStartTimings,
+} from "./services/warmStartVerificationService.js";
 import { extractTraceparent, getRestoreRunnerId } from "./util.js";
+import { Redis } from "ioredis";
+import { BackpressureMonitor } from "./backpressure/backpressureMonitor.js";
+import { RedisBackpressureSignalSource } from "./backpressure/redisBackpressureSignalSource.js";
+import { BackpressureMetrics } from "./backpressure/backpressureMetrics.js";
+import {
+  fromContext,
+  recordPhaseSince,
+  runWideEvent,
+  setExtra,
+  setMeta,
+  type WideEventOptions,
+} from "./wideEvents/index.js";
 
 if (env.METRICS_COLLECT_DEFAULTS) {
   collectDefaultMetrics({ register });
 }
 
+const workloadCreateDuration = new Histogram({
+  name: "workload_create_duration_seconds",
+  help: "Duration of workload manager create calls. A create may include backend-internal retries, so one observation can span multiple attempts.",
+  labelNames: ["backend", "outcome"],
+  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60],
+  registers: [register],
+});
+
 class ManagedSupervisor {
   private readonly workerSession: SupervisorSession;
   private readonly metricsServer?: HttpServer;
   private readonly workloadServer: WorkloadServer;
   private readonly workloadManager: WorkloadManager;
+  private readonly workloadManagerBackend: "compute" | "kubernetes" | "docker";
   private readonly computeManager?: ComputeWorkloadManager;
   private readonly logger = new SimpleStructuredLogger("managed-supervisor");
   private readonly resourceMonitor: ResourceMonitor;
   private readonly checkpointClient?: CheckpointClient;
+  private readonly warmStartVerifier?: WarmStartVerificationService;
 
   private readonly podCleaner?: PodCleaner;
   private readonly failedPodHandler?: FailedPodHandler;
   private readonly tracing?: OtlpTraceService;
+  private readonly backpressureMonitor?: BackpressureMonitor;
+  private readonly backpressureRedis?: Redis;
 
   private readonly isKubernetes = isKubernetesEnvironment(env.KUBERNETES_FORCE_ENABLED);
   private readonly warmStartUrl = env.TRIGGER_WARM_START_URL;
 
+  private readonly wideEventOpts: WideEventOptions = {
+    service: "supervisor",
+    env: { nodeId: env.TRIGGER_WORKER_INSTANCE_NAME },
+    enabled: env.TRIGGER_WIDE_EVENTS_ENABLED,
+  };
+  private readonly wideEventsNoisyRoutes = env.TRIGGER_WIDE_EVENTS_NOISY_ROUTES;
+
   constructor() {
+    // Strip secret-like env vars before debug-logging the rest. Add any new
+    // secret env var here so it never lands in the DEBUG "Starting up" log.
     const {
       TRIGGER_WORKER_TOKEN,
       MANAGED_WORKER_SECRET,
       COMPUTE_GATEWAY_AUTH_TOKEN,
+      DOCKER_REGISTRY_PASSWORD,
+      TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_PASSWORD,
       ...envWithoutSecrets
     } = env;
 
@@ -118,14 +157,22 @@ class ManagedSupervisor {
           instanceName: env.TRIGGER_WORKER_INSTANCE_NAME,
           otelEndpoint: env.OTEL_EXPORTER_OTLP_ENDPOINT,
           prettyLogs: env.RUNNER_PRETTY_LOGS,
+          sendRunDebugLogs: env.SEND_RUN_DEBUG_LOGS,
+        },
+        createRetry: {
+          maxAttempts: env.COMPUTE_INSTANCE_CREATE_MAX_ATTEMPTS,
+          baseDelayMs: env.COMPUTE_INSTANCE_CREATE_RETRY_BASE_DELAY_MS,
         },
       });
       this.computeManager = computeManager;
       this.workloadManager = computeManager;
+      this.workloadManagerBackend = "compute";
+    } else if (this.isKubernetes) {
+      this.workloadManager = new KubernetesWorkloadManager(workloadManagerOptions);
+      this.workloadManagerBackend = "kubernetes";
     } else {
-      this.workloadManager = this.isKubernetes
-        ? new KubernetesWorkloadManager(workloadManagerOptions)
-        : new DockerWorkloadManager(workloadManagerOptions);
+      this.workloadManager = new DockerWorkloadManager(workloadManagerOptions);
+      this.workloadManagerBackend = "docker";
     }
 
     if (this.isKubernetes) {
@@ -166,6 +213,42 @@ class ManagedSupervisor {
       );
     }
 
+    if (env.TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED) {
+      this.backpressureRedis = new Redis({
+        host: env.TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_HOST,
+        port: env.TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_PORT,
+        username: env.TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_USERNAME,
+        password: env.TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_PASSWORD,
+        ...(env.TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_TLS_DISABLED ? {} : { tls: {} }),
+        maxRetriesPerRequest: null,
+      });
+      this.backpressureRedis.on("error", (error) =>
+        this.logger.error("Backpressure redis error", { error: error.message })
+      );
+
+      this.backpressureMonitor = new BackpressureMonitor({
+        enabled: true,
+        source: new RedisBackpressureSignalSource(
+          this.backpressureRedis,
+          env.TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_KEY
+        ),
+        refreshIntervalMs: env.TRIGGER_DEQUEUE_BACKPRESSURE_REFRESH_MS,
+        maxVerdictAgeMs: env.TRIGGER_DEQUEUE_BACKPRESSURE_MAX_VERDICT_AGE_MS,
+        rampMs: env.TRIGGER_DEQUEUE_BACKPRESSURE_RAMP_MS,
+        dryRun: env.TRIGGER_DEQUEUE_BACKPRESSURE_DRY_RUN,
+        logger: this.logger,
+        metrics: new BackpressureMetrics({ register }),
+      });
+
+      this.logger.log("🛑 Dequeue backpressure enabled", {
+        key: env.TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_KEY,
+        refreshIntervalMs: env.TRIGGER_DEQUEUE_BACKPRESSURE_REFRESH_MS,
+        maxVerdictAgeMs: env.TRIGGER_DEQUEUE_BACKPRESSURE_MAX_VERDICT_AGE_MS,
+        rampMs: env.TRIGGER_DEQUEUE_BACKPRESSURE_RAMP_MS,
+        dryRun: env.TRIGGER_DEQUEUE_BACKPRESSURE_DRY_RUN,
+      });
+    }
+
     this.workerSession = new SupervisorSession({
       workerToken: getWorkerToken(),
       apiUrl: env.TRIGGER_API_URL,
@@ -175,6 +258,7 @@ class ManagedSupervisor {
       dequeueIdleIntervalMs: env.TRIGGER_DEQUEUE_IDLE_INTERVAL_MS,
       queueConsumerEnabled: env.TRIGGER_DEQUEUE_ENABLED,
       maxRunCount: env.TRIGGER_DEQUEUE_MAX_RUN_COUNT,
+      queueClass: env.TRIGGER_WORKER_QUEUE_CLASS,
       metricsRegistry: register,
       scaling: {
         strategy: env.TRIGGER_DEQUEUE_SCALING_STRATEGY,
@@ -186,18 +270,20 @@ class ManagedSupervisor {
         ewmaAlpha: env.TRIGGER_DEQUEUE_SCALING_EWMA_ALPHA,
         batchWindowMs: env.TRIGGER_DEQUEUE_SCALING_BATCH_WINDOW_MS,
         dampingFactor: env.TRIGGER_DEQUEUE_SCALING_DAMPING_FACTOR,
+        // Freeze scale-up while backpressure is hard-engaged (not during the resume
+        // ramp). Undefined when backpressure is disabled → no effect on scaling.
+        shouldPauseScaling: () => this.backpressureMonitor?.isEngaged() ?? false,
       },
       runNotificationsEnabled: env.TRIGGER_WORKLOAD_API_ENABLED,
       heartbeatIntervalSeconds: env.TRIGGER_WORKER_HEARTBEAT_INTERVAL_SECONDS,
       sendRunDebugLogs: env.SEND_RUN_DEBUG_LOGS,
       preDequeue: async () => {
-        if (!env.RESOURCE_MONITOR_ENABLED) {
-          return {};
-        }
+        // Synchronous, hot-path-safe cached read; undefined when backpressure is disabled.
+        const skipForBackpressure = this.backpressureMonitor?.shouldSkipDequeue() ?? false;
 
-        if (this.isKubernetes) {
-          // Not used in k8s for now
-          return {};
+        if (!env.RESOURCE_MONITOR_ENABLED || this.isKubernetes) {
+          // Resource monitor is not used in k8s; backpressure is the only gate there.
+          return { skipDequeue: skipForBackpressure };
         }
 
         const resources = await this.resourceMonitor.getNodeResources();
@@ -207,7 +293,10 @@ class ManagedSupervisor {
             cpu: resources.cpuAvailable,
             memory: resources.memoryAvailable,
           },
-          skipDequeue: resources.cpuAvailable < 0.25 || resources.memoryAvailable < 0.25,
+          skipDequeue:
+            skipForBackpressure ||
+            resources.cpuAvailable < 0.25 ||
+            resources.memoryAvailable < 0.25,
         };
       },
       preSkip: async () => {
@@ -228,6 +317,19 @@ class ManagedSupervisor {
       });
     }
 
+    if (env.TRIGGER_WARM_START_VERIFY_ENABLED && this.warmStartUrl) {
+      this.logger.log("Warm-start delivery verification enabled", {
+        delayMs: env.TRIGGER_WARM_START_VERIFY_DELAY_MS,
+      });
+
+      this.warmStartVerifier = new WarmStartVerificationService({
+        workerClient: this.workerSession.httpClient,
+        delayMs: env.TRIGGER_WARM_START_VERIFY_DELAY_MS,
+        createWorkload: (message, timings) => this.createWorkload(message, timings),
+        wideEventOpts: this.wideEventOpts,
+      });
+    }
+
     this.workerSession.on("runNotification", async ({ time, run }) => {
       this.logger.verbose("runNotification", { time, run });
 
@@ -239,149 +341,171 @@ class ManagedSupervisor {
       async ({ time, message, dequeueResponseMs, pollingIntervalMs }) => {
         this.logger.verbose(`Received message with timestamp ${time.toLocaleString()}`, message);
 
-        if (message.completedWaitpoints.length > 0) {
-          this.logger.debug("Run has completed waitpoints", {
-            runId: message.run.id,
-            completedWaitpoints: message.completedWaitpoints.length,
-          });
-        }
+        const traceparent = extractTraceparent(message.run.traceContext);
 
-        if (!message.image) {
-          this.logger.error("Run has no image", { runId: message.run.id });
-          return;
-        }
+        await runWideEvent(
+          {
+            ...this.wideEventOpts,
+            op: "dequeue",
+            kind: "inbound",
+            traceparent,
+            setup: (state) => {
+              setMeta(state, "run_id", message.run.friendlyId);
+              setMeta(state, "env_id", message.environment.id);
+              setMeta(state, "org_id", message.organization.id);
+              setMeta(state, "project_id", message.project.id);
+              if (message.deployment.friendlyId) {
+                setMeta(state, "deployment_id", message.deployment.friendlyId);
+              }
+              setMeta(state, "machine_preset", message.run.machine.name);
+              state.extras.iteration = "dequeue";
+              state.extras.dequeue_response_ms = dequeueResponseMs;
+              state.extras.polling_interval_ms = pollingIntervalMs;
+              state.extras.completed_waitpoints = message.completedWaitpoints.length;
+            },
+          },
+          async () => {
+            if (message.completedWaitpoints.length > 0) {
+              this.logger.debug("Run has completed waitpoints", {
+                runId: message.run.id,
+                completedWaitpoints: message.completedWaitpoints.length,
+              });
+            }
 
-        const { checkpoint, ...rest } = message;
+            if (!message.image) {
+              setExtra(fromContext(), "path_taken", "skipped_no_image");
+              this.logger.error("Run has no image", { runId: message.run.id });
+              return;
+            }
 
-        // Register trace context early so snapshot spans work for all paths
-        // (cold create, restore, warm start). Re-registration on restore is safe
-        // since dequeue always provides fresh context.
-        if (this.computeManager?.traceSpansEnabled) {
-          const traceparent = extractTraceparent(message.run.traceContext);
+            const { checkpoint, ...rest } = message;
 
-          if (traceparent) {
-            this.workloadServer.registerRunTraceContext(message.run.friendlyId, {
-              traceparent,
-              envId: message.environment.id,
-              orgId: message.organization.id,
-              projectId: message.project.id,
-            });
-          }
-        }
-
-        if (checkpoint) {
-          this.logger.debug("Restoring run", { runId: message.run.id });
-
-          if (this.computeManager) {
-            try {
-              const runnerId = getRestoreRunnerId(message.run.friendlyId, checkpoint.id);
-
-              const didRestore = await this.computeManager.restore({
-                snapshotId: checkpoint.location,
-                runnerId,
-                runFriendlyId: message.run.friendlyId,
-                snapshotFriendlyId: message.snapshot.friendlyId,
-                machine: message.run.machine,
-                traceContext: message.run.traceContext,
+            // Register trace context early so snapshot spans work for all paths
+            // (cold create, restore, warm start). Re-registration on restore is safe
+            // since dequeue always provides fresh context.
+            if (this.computeManager?.traceSpansEnabled && traceparent) {
+              this.workloadServer.registerRunTraceContext(message.run.friendlyId, {
+                traceparent,
                 envId: message.environment.id,
                 orgId: message.organization.id,
                 projectId: message.project.id,
-                dequeuedAt: message.dequeuedAt,
               });
+            }
 
-              if (didRestore) {
-                this.logger.debug("Compute restore successful", {
-                  runId: message.run.id,
-                  runnerId,
-                });
-              } else {
-                this.logger.error("Compute restore failed", { runId: message.run.id, runnerId });
+            if (checkpoint) {
+              setExtra(fromContext(), "path_taken", "restore");
+              this.logger.debug("Restoring run", { runId: message.run.id });
+
+              if (this.computeManager) {
+                const restoreStart = performance.now();
+                try {
+                  const runnerId = getRestoreRunnerId(message.run.friendlyId, checkpoint.id);
+
+                  const didRestore = await this.computeManager.restore({
+                    snapshotId: checkpoint.location,
+                    runnerId,
+                    runFriendlyId: message.run.friendlyId,
+                    snapshotFriendlyId: message.snapshot.friendlyId,
+                    machine: message.run.machine,
+                    traceContext: message.run.traceContext,
+                    envId: message.environment.id,
+                    orgId: message.organization.id,
+                    projectId: message.project.id,
+                    hasPrivateLink: message.organization.hasPrivateLink,
+                    dequeuedAt: message.dequeuedAt,
+                  });
+                  recordPhaseSince("restore", restoreStart, undefined);
+                  setExtra(fromContext(), "did_restore", didRestore);
+
+                  if (didRestore) {
+                    this.logger.debug("Compute restore successful", {
+                      runId: message.run.id,
+                      runnerId,
+                    });
+                  } else {
+                    this.logger.error("Compute restore failed", {
+                      runId: message.run.id,
+                      runnerId,
+                    });
+                  }
+                } catch (error) {
+                  recordPhaseSince(
+                    "restore",
+                    restoreStart,
+                    error instanceof Error ? error : new Error(String(error))
+                  );
+                  this.logger.error("Failed to restore run (compute)", { error });
+                }
+
+                return;
               }
-            } catch (error) {
-              this.logger.error("Failed to restore run (compute)", { error });
+
+              if (!this.checkpointClient) {
+                this.logger.error("No checkpoint client", { runId: message.run.id });
+                return;
+              }
+
+              const restoreStart = performance.now();
+              try {
+                const didRestore = await this.checkpointClient.restoreRun({
+                  runFriendlyId: message.run.friendlyId,
+                  snapshotFriendlyId: message.snapshot.friendlyId,
+                  body: {
+                    ...rest,
+                    checkpoint,
+                  },
+                });
+                recordPhaseSince("restore", restoreStart, undefined);
+                setExtra(fromContext(), "did_restore", didRestore);
+
+                if (didRestore) {
+                  this.logger.debug("Restore successful", { runId: message.run.id });
+                } else {
+                  this.logger.error("Restore failed", { runId: message.run.id });
+                }
+              } catch (error) {
+                recordPhaseSince(
+                  "restore",
+                  restoreStart,
+                  error instanceof Error ? error : new Error(String(error))
+                );
+                this.logger.error("Failed to restore run", { error });
+              }
+
+              return;
             }
 
-            return;
-          }
+            this.logger.debug("Scheduling run", { runId: message.run.id });
 
-          if (!this.checkpointClient) {
-            this.logger.error("No checkpoint client", { runId: message.run.id });
-            return;
-          }
+            const warmStartStart = performance.now();
+            const didWarmStart = await this.tryWarmStart(message, traceparent);
+            const warmStartCheckMs = Math.round(performance.now() - warmStartStart);
+            recordPhaseSince("warm_start", warmStartStart, undefined);
+            setExtra(fromContext(), "did_warm_start", didWarmStart);
 
-          try {
-            const didRestore = await this.checkpointClient.restoreRun({
-              runFriendlyId: message.run.friendlyId,
-              snapshotFriendlyId: message.snapshot.friendlyId,
-              body: {
-                ...rest,
-                checkpoint,
-              },
+            if (didWarmStart) {
+              setExtra(fromContext(), "path_taken", "warm_start");
+              this.logger.debug("Warm start successful", { runId: message.run.id });
+              // A hit only means the response was written to the long-poll
+              // socket, not that the runner received it. Schedule a delivery
+              // verification that cold-starts the run if nobody acts on it.
+              this.warmStartVerifier?.schedule(message, {
+                dequeueResponseMs,
+                pollingIntervalMs,
+                warmStartCheckMs,
+              });
+              return;
+            }
+
+            setExtra(fromContext(), "path_taken", "cold_create");
+
+            await this.createWorkload(message, {
+              dequeueResponseMs,
+              pollingIntervalMs,
+              warmStartCheckMs,
             });
-
-            if (didRestore) {
-              this.logger.debug("Restore successful", { runId: message.run.id });
-            } else {
-              this.logger.error("Restore failed", { runId: message.run.id });
-            }
-          } catch (error) {
-            this.logger.error("Failed to restore run", { error });
           }
-
-          return;
-        }
-
-        this.logger.debug("Scheduling run", { runId: message.run.id });
-
-        const warmStartStart = performance.now();
-        const didWarmStart = await this.tryWarmStart(message);
-        const warmStartCheckMs = Math.round(performance.now() - warmStartStart);
-
-        if (didWarmStart) {
-          this.logger.debug("Warm start successful", { runId: message.run.id });
-          return;
-        }
-
-        try {
-          if (!message.deployment.friendlyId) {
-            // mostly a type guard, deployments always exists for deployed environments
-            // a proper fix would be to use a discriminated union schema to differentiate between dequeued runs in dev and in deployed environments.
-            throw new Error("Deployment is missing");
-          }
-
-          await this.workloadManager.create({
-            dequeuedAt: message.dequeuedAt,
-            dequeueResponseMs,
-            pollingIntervalMs,
-            warmStartCheckMs,
-            envId: message.environment.id,
-            envType: message.environment.type,
-            image: message.image,
-            machine: message.run.machine,
-            orgId: message.organization.id,
-            projectId: message.project.id,
-            deploymentFriendlyId: message.deployment.friendlyId,
-            deploymentVersion: message.backgroundWorker.version,
-            runId: message.run.id,
-            runFriendlyId: message.run.friendlyId,
-            version: message.version,
-            nextAttemptNumber: message.run.attemptNumber,
-            snapshotId: message.snapshot.id,
-            snapshotFriendlyId: message.snapshot.friendlyId,
-            placementTags: message.placementTags,
-            traceContext: message.run.traceContext,
-            annotations: message.run.annotations,
-            hasPrivateLink: message.organization.hasPrivateLink,
-          });
-
-          // Disabled for now
-          // this.resourceMonitor.blockResources({
-          //   cpu: message.run.machine.cpu,
-          //   memory: message.run.machine.memory,
-          // });
-        } catch (error) {
-          this.logger.error("Failed to create workload", { error });
-        }
+        );
       }
     );
 
@@ -404,6 +528,8 @@ class ManagedSupervisor {
       checkpointClient: this.checkpointClient,
       computeManager: this.computeManager,
       tracing: this.tracing,
+      wideEventOpts: this.wideEventOpts,
+      wideEventsNoisyRoutes: this.wideEventsNoisyRoutes,
     });
 
     this.workloadServer.on("runConnected", this.onRunConnected.bind(this));
@@ -412,6 +538,8 @@ class ManagedSupervisor {
 
   async onRunConnected({ run }: { run: { friendlyId: string } }) {
     this.logger.debug("Run connected", { run });
+    // The dispatched run reached a runner on this node - no fallback needed.
+    this.warmStartVerifier?.cancel(run.friendlyId);
     this.workerSession.subscribeToRunNotifications([run.friendlyId]);
   }
 
@@ -420,19 +548,97 @@ class ManagedSupervisor {
     this.workerSession.unsubscribeFromRunNotifications([run.friendlyId]);
   }
 
-  private async tryWarmStart(dequeuedMessage: DequeuedMessage): Promise<boolean> {
+  private async createWorkload(message: DequeuedMessage, timings: WarmStartTimings) {
+    const createStart = performance.now();
+    try {
+      if (!message.deployment.friendlyId) {
+        // mostly a type guard, deployments always exists for deployed environments
+        // a proper fix would be to use a discriminated union schema to differentiate between dequeued runs in dev and in deployed environments.
+        throw new Error("Deployment is missing");
+      }
+
+      if (!message.image) {
+        // same type-guard situation as deployment above
+        throw new Error("Image is missing");
+      }
+
+      await this.workloadManager.create({
+        dequeuedAt: message.dequeuedAt,
+        dequeueResponseMs: timings.dequeueResponseMs,
+        pollingIntervalMs: timings.pollingIntervalMs,
+        warmStartCheckMs: timings.warmStartCheckMs,
+        envId: message.environment.id,
+        envType: message.environment.type,
+        image: message.image,
+        machine: message.run.machine,
+        orgId: message.organization.id,
+        projectId: message.project.id,
+        deploymentFriendlyId: message.deployment.friendlyId,
+        deploymentVersion: message.backgroundWorker.version,
+        runId: message.run.id,
+        runFriendlyId: message.run.friendlyId,
+        version: message.version,
+        nextAttemptNumber: message.run.attemptNumber,
+        snapshotId: message.snapshot.id,
+        snapshotFriendlyId: message.snapshot.friendlyId,
+        placementTags: message.placementTags,
+        traceContext: message.run.traceContext,
+        annotations: message.run.annotations,
+        hasPrivateLink: message.organization.hasPrivateLink,
+      });
+      recordPhaseSince("workload_create", createStart, undefined);
+      workloadCreateDuration.observe(
+        { backend: this.workloadManagerBackend, outcome: "success" },
+        (performance.now() - createStart) / 1000
+      );
+
+      // Disabled for now
+      // this.resourceMonitor.blockResources({
+      //   cpu: message.run.machine.cpu,
+      //   memory: message.run.machine.memory,
+      // });
+    } catch (error) {
+      recordPhaseSince(
+        "workload_create",
+        createStart,
+        error instanceof Error ? error : new Error(String(error))
+      );
+      workloadCreateDuration.observe(
+        { backend: this.workloadManagerBackend, outcome: "error" },
+        (performance.now() - createStart) / 1000
+      );
+      this.logger.error("Failed to create workload", {
+        runId: message.run.friendlyId,
+        error,
+      });
+    }
+  }
+
+  private async tryWarmStart(
+    dequeuedMessage: DequeuedMessage,
+    traceparent: string | undefined
+  ): Promise<boolean> {
     if (!this.warmStartUrl) {
       return false;
     }
 
     const warmStartUrlWithPath = new URL("/warm-start", this.warmStartUrl);
 
+    const headers: Record<string, string> = {
+      "Content-Type": "application/json",
+    };
+    // Propagate the inbound W3C traceparent so the upstream warm-start
+    // receiver continues the same trace instead of minting a new one. Gated
+    // by the same kill switch as the wide-event emission so the whole PR is
+    // a no-op on the wire when disabled.
+    if (this.wideEventOpts.enabled && traceparent) {
+      headers.traceparent = traceparent;
+    }
+
     try {
       const res = await fetch(warmStartUrlWithPath.href, {
         method: "POST",
-        headers: {
-          "Content-Type": "application/json",
-        },
+        headers,
         body: JSON.stringify({ dequeuedMessage }),
       });
 
@@ -468,6 +674,7 @@ class ManagedSupervisor {
     this.logger.log("Starting up");
 
     // Optional services
+    this.backpressureMonitor?.start();
     await this.podCleaner?.start();
     await this.failedPodHandler?.start();
     await this.metricsServer?.start();
@@ -488,10 +695,15 @@ class ManagedSupervisor {
 
   async stop() {
     this.logger.log("Shutting down");
+    // Stop the verifier first: its timer can otherwise fire mid-shutdown and
+    // cold-create a workload on a node that is going down.
+    this.warmStartVerifier?.stop();
     await this.workloadServer.stop();
     await this.workerSession.stop();
 
     // Optional services
+    this.backpressureMonitor?.stop();
+    await this.backpressureRedis?.quit();
     await this.podCleaner?.stop();
     await this.failedPodHandler?.stop();
     await this.metricsServer?.stop();