336 Commits

Author	SHA1	Message	Date
Saadi Myftija	38e78f8c7e	feat(webapp): deployment lifecycle telemetry events (#4778) ... Deployments currently leave little analytical trace. This PR makes every deployment emit two analytics events to enable useful queries. It also enables comparing deployments across build paths, CLI versions, runtimes, and orgs. ### Where the events come from ``` trigger deploy │ ▼ initialize ─────────────────────────────▶ ✨ deployment.initialized │ createdAt ▼ PENDING waiting for a build slot ┐ │ startedAt │ queue time ▼ ┘ INSTALLING build server installs deps ┐ │ installedAt (native paths only) │ install time ▼ ┘ BUILDING the image is built ┐ │ builtAt │ building time ▼ ┘ DEPLOYING indexing + registry push ┐ │ deployedAt / failedAt / canceledAt │ deploying time ▼ ┘ DEPLOYED · FAILED · TIMED_OUT · CANCELED │ └───────────────────────────────────▶ ✨ deployment.finished ``` `deployment.finished` fires exactly once, whichever way the deployment ends, and is backdated to cover the deployment's real lifetime. Not every path visits every state (Depot deploys skip PENDING/INSTALLING, for example) — a phase duration is simply omitted when its state was never entered. ### What each event carries - **Which path built it**: `depot`, `native`, or `native_local_bundle` - **How it ended**: status, plus an error class and message when it failed - **How long each phase took**: queue, install, building, deploying, and total — derived from the timestamps above - **Who and with what**: org, project, environment, runtime, CLI version, and how the deploy was triggered (CLI, GitHub, Vercel) With that, one query gives failure rate per build path, duration percentiles per phase, adoption per CLI version, or a per-org health table. ### Fixes that ride along - The old `deployment.outcome` span was silently dropped ~95% of the time (it was subject to trace sampling). The new events opt out of sampling explicitly, so every deployment is counted. - The fail/timeout/finalize transitions were racy: a late timeout could overwrite a successful deployment. They now use guarded writes, so exactly one caller wins the terminal transition — and exactly one event is emitted. - Canceled deployments previously recorded nothing; they do now. - The deployment's CLI version is now stored at initialization (new nullable column), so even deploys that fail early are attributable to a CLI release. - Telemetry is flushed on shutdown (the last batch used to be lost on every webapp deploy), and an optional second exporter can mirror just these events into a dedicated dataset.	2026-08-26 12:57:46 +02:00
Daniel Sutton	00e3c151d4	feat(webapp): RUN_OPS_SHARDS config, topology and N-way store wiring (#4764) ... Part of the RunOps N-way sharding work. This lets the webapp hold N run-ops stores, configured by a single `RUN_OPS_SHARDS` JSON descriptor, and routes to them through the existing keyed router. **Inert with `RUN_OPS_SHARDS` unset** — the topology, the wiring and `ROUTING_ENABLED` are byte-identical to today. ## What's here - **`RUN_OPS_SHARDS`** — a zod-validated JSON array of shard descriptors (`key`, `region`, `url`, `replicaUrl`, `directUrl`, `replication`, `knobs`, `aliasOf`), validated at boot in the `parseMachinePresetCsv` style. Unset or `[]` → no shards. - **One run-ops client factory** — `buildRunOpsWriterClient`/`buildRunOpsReplicaClient` collapse into one `buildRunOpsClient` parameterized by role and resolved pool knobs. The control-plane builders (`buildWriterClient`/`buildReplicaClient`) are a separate path and stay untouched; every resolved value matches the former builders. - **Shard loop in `selectRunOpsTopology`** — one client pair per descriptor; an `aliasOf: "new"` descriptor reuses the new store's clients by reference and opens no pool. - **N-way `buildRunStore`** — builds N dedicated stores + the keyed router via a new `RoutingRunStore.fromShards`, keeping the two-store compat router when no shards are configured. - **`UnknownShardKey`** — raised when an id resolves to an unconfigured key; never falls back to another store. `fromShards` injects `resolveShard` so a gen-2 id routes to its own shard. - **Per-shard transaction resilience** — each shard gets its own retry budget. - **Mint bound** — `computeMintShard` intersects the active mint list with the configured descriptor keys, so a key with no descriptor is never minted into. - **Boot table** — logs `key`, address fingerprint (host:port/db, no credentials), and role, only when shards are configured. ## Ordering constraint Do **not** configure a `RUN_OPS_SHARDS` descriptor in any environment until the routing-semantics change (TRI-13427) lands — three fan-out sites still truncate at N>2. Merging this PR alone is safe (inert with the var unset); configuring a descriptor is what must wait. ## Testing - Run-store corpus: green with zero test-file diffs (the bit-identical proof for the compat router). - `runOpsDbTopology.test.ts` 17/17, `runStore.server.test.ts` 4/4, `runOpsMigration` family 149/149. - New unit suites: descriptor validation, pool-knob value tables, `fromShards` routing + `UnknownShardKey`, boot-table formatter, mint bound. - typecheck (webapp + run-store), knip, lint, format: pass. ## Changelog Internal run-ops sharding infrastructure. No changeset or `.server-changes`: the change is inert with `RUN_OPS_SHARDS` unset and has no user-visible behaviour. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-26 09:19:12 +01:00
Saadi Myftija	1eda438a41	feat(webapp): put the admin dashboard behind an env var flag (#4774) ... Adds an `ADMIN_DASHBOARD_ENABLED` env var (default: enabled) that turns the admin dashboard and user impersonation off for an entire instance. When disabled: - every admin dashboard page redirects away, and the admin navigation isn't rendered - existing impersonation cookies are ignored, and any lingering session is actively terminated with an audit record - every flow that could start an impersonation responds 404, and no impersonation tokens are minted Stopping an impersonation always works regardless of the flag, so nothing gets stuck. Machine-to-machine admin API endpoints are not affected. The variable is documented for self-hosters; instances that don't set it are unaffected.	2026-08-25 15:37:43 +02:00
Eric Allam	11e1cd8174	feat(webapp): isolate the runs list ClickHouse read pool (#4763) ... ## Summary Improves the performance and reliability of the runs list and the `runs.list` API, especially for large projects and filtered views. ## What changed - **Filtered runs-list queries use `PREWHERE`.** Immutable and additive-only filters (tags, task identifier, version, queue, region, machine, and the rest) are applied in `PREWHERE` on the `task_runs_v2 FINAL` scan, so ClickHouse filters, and uses the tags skip index, before it reconciles versions and materialises the wide columns. Same results, far less memory per query. `status` stays in `WHERE`: it changes across a run's versions, so filtering it before `FINAL` could return stale rows. - **The runs-list ClickHouse pool gets per-query guardrails**, all env-configurable: a `max_execution_time` paired with the client request timeout, a per-query `max_memory_usage`, a `max_threads` cap, and `readonly`. Each bounds a single query to itself, so a heavy query can't affect other queries, and they are safe as pool-level settings only because this pool is read-only. - **Billing and bulk count reads move to the read pool**, off the write pool. Defaults are conservative for self-hosters; production values are set via env.	2026-08-25 09:19:48 +01:00
Saadi Myftija	f866210388	feat(cli): experimental --local-bundle deploy mode (#4331) ... Adds an experimental `--local-bundle` flag to native build deployments: the project is installed and bundled on the local machine (exactly like in the depot path) and only the resulting build context is uploaded. The remote build then runs just the container image build. ### Design - The uploaded artifact is the same build context classic deploys produce: bundled output, a synthesized package.json with the resolved externals, build.json, and the generated Containerfile. The bundle is secret-free: build.json is deliberately scrubbed because it is copied into the image, and build-arg values never enter the bundle at all. - Build-arg values are sent with the deployment initialization request instead, stored encrypted (aes-256-gcm) in a new `WorkerDeployment.buildEnvVars` column, and cleared on every terminal status transition. They exist at rest only for the active build window, always encrypted. - A dedicated `GET /api/v1/deployments/:id/build-env-vars` endpoint returns the decrypted values to the same principals that can already read the environment's variables. It answers with an empty record for deployments without stored values or in a terminal state, keeping secret access to a single auditable route. - Size limits are enforced server side and pre-checked client side. If the server does not acknowledge storing the values, the CLI fails fast instead of letting the remote build run without them. - A `--from-bundle <dir>` mode builds a deployment image straight from such a bundle directory, skipping config loading and bundling entirely. In attach mode it fetches the stored build-arg values through the new endpoint. - Env var syncing (the `syncEnvVars` extension) happens client side, before the deployment initializes, since the remote side never sees the unscrubbed manifest. - Bundle artifacts use a distinct type and storage prefix so the server can always distinguish them from source uploads.	2026-08-25 09:50:49 +02:00
Eric Allam	60d71da90e	perf(webapp,run-engine): cut CPU on the engine-facing worker-action routes (#4746) ... Cuts CPU on the `engine/v1/worker-actions/*` routes a managed supervisor calls, and adds the benchmark harness the numbers come from. Measured on a local stack: **on-CPU per completed run 9.07ms → 6.59ms (−27%)**, busy fraction 45.6% → 33.8%, with every worker-action p50 down 23–27%. Load was 5,000 runs / 24 virtual supervisors / 90s window / 30,120 requests / 0 errors. Query-count work from the same investigation is deliberately **not** here — it will follow as a separate PR. ## The three changes **1. Split the event-loop monitor in two (~14% of on-CPU, plus ~5pp of GC).** `eventLoopMonitor.server.ts` installs a global `async_hooks` hook: `init` writes a `Map` entry for *every* async resource the process creates, `before` calls `process.hrtime()` and `context.active()` on every one. Enabling any async hook also puts V8 on the slow path for promise instrumentation process-wide. `EVENT_LOOP_MONITOR_ENABLED` defaulted to `"1"`, so this was the shipping configuration. The blocked-loop detector is now opt-in (`EVENT_LOOP_MONITOR_ENABLED`, default `0`). The event-loop *utilization* gauge — a single interval timer with no per-request cost — moves to its own flag (`EVENT_LOOP_UTILIZATION_MONITOR_ENABLED`, default `1`) and stays on, so the useful half survives without the expensive half. A/B under identical load: | | monitor on | monitor off | change | |---|---|---|---| | on-CPU per run | 9.08ms | 7.25ms | −20% | | GC self time | 9.80% | 5.05% | −4.75pp | | dequeue p50 | 76.6ms | 62.8ms | −18% | | attempts/start p50 | 56.3ms | 43.5ms | −23% | **2. Bucket route matching by first static path segment (10.4% → 3.9% of on-CPU).** `patches/@remix-run__router@1.23.3.patch` already memoized flattened branches and compiled path regexes. What remained was the linear scan: `matchRouteBranch` walked the ranked branch list calling `matchPath` per branch across 521 route files, so every worker-action request paid a scan proportional to the whole route table. Branches are now indexed by their lowercased leading segment, with one always-considered list for branches whose leading segment is dynamic, splat or optional (and for root/pathless paths). A request walks only its own bucket merged with that list. Route-matching self time dropped 64% (3.6s → 1.3s over a 90s window). Ordering is preserved exactly: both lists hold indexes into the already rank-sorted branch array and are walked in ascending-index order, so the first match found is the same branch the full scan would have found. Bucketing lowercases on both sides, so case-insensitive matching still resolves and `caseSensitive: true` routes are still rejected by `matchPath` itself. A pathname whose own leading segment can't be bucketed falls back to the full scan. Verified equivalent to the unpatched matcher over 20,050 pathnames (literal, dynamic, splat, optional, case variants, basenames, percent-encoded) with zero mismatches. `apps/webapp/test/routeMatchingPatch.test.ts` pins the matching semantics rather than the optimisation, so it still passes without the patch. **3. Demote per-heartbeat and per-dequeue `info` logs to `debug`.** These are the two highest-rate engine calls and each wrote a synchronous structured log line on every request. Synchronous `console` writes can block the loop when stdout backs up, which costs more than the ~1.3% CPU share suggests. ## The harness Two benchmarks, neither in the default suite (they run for minutes, attach the V8 profiler, and report numbers rather than assert on them). See `apps/webapp/test/bench/README.md`. - `apps/webapp/test/bench/engineHttp.bench.test.ts` — spawns a real webapp against throwaway Postgres/Redis containers, seeds a production environment with a promoted managed deployment, and drives a closed-loop supervisor pool through the full lifecycle. Profiling runs over CDP rather than `--cpu-prof` so it covers only the measured window instead of being swamped by boot, and `performance.eventLoopUtilization()` is sampled *inside* the webapp process. - `internal-packages/run-engine/src/engine/bench/runEngineLifecycle.bench.test.ts` — drives `RunEngine` directly, profiling enqueue and lifecycle separately so engine cost isn't mixed with request-stack overhead. - `apps/webapp/test/bench/analyzeProfile.ts` — dependency-free `.cpuprofile` analyzer that symbolicates through the build's source maps and ranks CPU by package, self time and total time. Percentages are shares of on-CPU time (V8's `(idle)`/`(program)` excluded). `startWebapp` gains `overrideEnv`, applied after the worker-disable defaults, so the HTTP bench can re-enable the run engine worker that drains the master queue into the worker queues a supervisor dequeues from. The local OTel collector gains a traces pipeline. It only defined a metrics pipeline, so pointing `INTERNAL_OTEL_TRACE_EXPORTER_URL` at it locally failed and the webapp silently fell back to the console span logger. ## Configuration For operators upgrading: - `EVENT_LOOP_MONITOR_ENABLED` (now defaults to `0`) — the per-async-resource blocked-loop detector. Set to `1` to restore the previous behaviour and keep emitting `event-loop-blocked` spans. - `EVENT_LOOP_UTILIZATION_MONITOR_ENABLED` (new, defaults to `1`) — the `nodejs.event_loop.utilization` gauge. Unchanged in behaviour; it just has its own flag now so it survives turning the detector off. ## Notes for review - `pnpm-lock.yaml` changes only because the router patch content changed, which changes its patch hash. - One thing the profile ruled out: with a real OTLP collector receiving spans, tracing costs ~1.7% of on-CPU at 100% sampling and ~0.8% at the production rate. Span shipping is not a hidden cost, so nothing here touches it. - Caveats on the numbers: a laptop, not production hardware, so DB and Redis *latency* are unrepresentative (client-side CPU is what's ranked); single webapp process; throughput varies ~5% run to run, which is why the claims rest on on-CPU per run rather than req/s. ## Verification - 20,050-pathname router equivalence check vs the unpatched matcher, zero mismatches - `apps/webapp/test/routeMatchingPatch.test.ts` (12 cases) passes - webapp e2e smoke suite (68 tests) passes through the patched router - run-engine suites covering the snapshot/attempt paths pass - `typecheck`, `format`, `lint`, `knip` clean	2026-08-21 11:53:16 +01:00
Eric Allam	56f875680c	fix(webapp): let the dashboard agent use a configurable base URL (#4738) ... ## Summary Lets the dashboard agent point at a specific Trigger instance instead of assuming it runs on the same instance as the webapp. Adds an optional `DASHBOARD_AGENT_BASE_URL`; when unset it falls back to the SDK default. ## Root cause The agent's session start, token mint, head start, in-proxy and the client transport all built the agent's base URL from the webapp's own origin (`API_ORIGIN ?? APP_ORIGIN`). That only holds when the agent project runs on the same instance as the webapp. When it runs elsewhere, `DASHBOARD_AGENT_SECRET_KEY` belongs to that other instance, so the webapp's own API rejects it with an "Invalid API key" and the chat can't start. ## Fix `dashboardAgentApiOrigin()` now returns `DASHBOARD_AGENT_BASE_URL` or the SDK default, never the webapp origin. A concrete default (rather than an unset value) keeps it independent of `TRIGGER_API_URL`, which a webapp may point at a different host. Every server call site already routes through that helper; the client transport reads the value from the root loader via a new `useDashboardAgentBaseUrl` hook.	2026-08-20 14:14:14 +01:00
Oskar Otwinowski	8b0385c429	feat(run-engine): trigger tasks pinned to an external deployment id (#4664) ... The SDK discovers an external deployment id at runtime (explicit TRIGGER_EXTERNAL_DEPLOYMENT_ID always; platform commit-SHA variables and generic fallbacks when TRIGGER_AUTOMATIC_SKEW_VERSION_PROTECTION=1) and sends it alongside lockToVersion; the server resolves precedence (version > external id > current). An id held by a deployed deployment pins the run to that worker; an in-flight or unknown id parks the run in PENDING_VERSION with the id in TaskRun.annotations, wakes it pinned when a deployment carrying the id finalizes (ClickHouse candidates, Postgres authoritative), and expires it after a deadline that re-checks Postgres before acting. Parking outranks delaying and preserves delayUntil. The id is projected to ClickHouse task_runs_v2.external_deployment_id during replication. Redis cache for id-to-worker resolution, guarded version-aware writes. Ids are not unique. Several deployments can hold one id - a --force rebuild is the ordinary way to get there - so resolution always picks the highest version among the candidates, never the newest by timestamp. The rule is applied identically on both paths that can bind a run to a worker: resolveExternalDeployment at trigger time, and PendingVersionSystem when a landing deployment wakes a parked run. Version comparison is numeric on the counter half, so 20260807.10 outranks 20260807.9. A run whose id never lands expires at the deadline with EXTERNAL_DEPLOYMENT_NOT_FOUND and an error naming the id it waited for, which is what a failed build or a typo looks like from the caller. Default deadline is one hour (EXTERNAL_DEPLOYMENT_PARK_DEADLINE_MS). Debounce registration happens in both the parked and the delayed branch through one helper, so a debounced run that parks still binds its debounce key; without it every later trigger for the same key created another parked run, and all of them executed when the deployment landed. The two DELAYED-only status checks in DebounceSystem also accept PENDING_VERSION, without which the lock-contention fallback would rethrow a 5xx the SDK retries and amplifies, and the fast path would push every trigger on a parked key through the redlock. Resolution is skipped in development. A dev environment cannot hold a WorkerDeployment - trigger dev registers a BackgroundWorker with nothing behind it, and deploy --env refuses dev - so an external deployment id there could only ever park, and the parked run then expired against the dev TTL while a connected dev worker sat idle. The id is still annotated so the dashboard shows what the app sent (TRI-13000).	2026-08-19 17:43:53 +02:00
Chris Arderne	b4313c8199	feat: logs search v2 (#4615)	2026-08-18 14:59:46 +01:00
Katia Bulatova	e768d0a724	feat(webapp): run the dashboard agent through AWS Bedrock behind an env switch (#4609) ... ## What & why The dashboard agent can now run its model calls through AWS Bedrock instead of the direct Anthropic API, chosen by a single env switch. It's **off by default** (`DASHBOARD_AGENT_MODEL_PROVIDER` unset ⇒ `anthropic`), so merging changes nothing at runtime — the Bedrock path is a dormant branch until an operator sets the switch and AWS config. The default Anthropic path is byte-for-byte unchanged. This also carries a related tenant-isolation hardening for the agent's delegated token (kept together deliberately — both land the agent on Bedrock for HIPAA readiness). Refs: TRI-13251, TRI-11032. ## What's inside **Provider seam** — `internal-packages/dashboard-agent/src/model-provider.ts`: the registry now holds both `anthropic` and `bedrock`; `resolveDashboardAgentModel()` maps the canonical `"anthropic:<id>"` strings the managed prompts carry to the active provider, and the cache-breakpoint helpers emit the active provider's shape — Anthropic `cacheControl` vs Bedrock `cachePoint`. Managed prompt strings stay canonical, so stored prompts don't change meaning. Unmapped model ids throw rather than shipping a guaranteed-404 profile. All agent, watch, compaction and title callsites route through the resolver; the `dashboardAgentModelKey` locals override (test mock injection) is preserved. **Cache telemetry** — `step-cache.ts`: cache token usage is read from the active provider (Anthropic reports it on provider metadata; Bedrock reports the write on metadata and the read via standard usage), so `gen_ai.usage.cache_*` is populated on both. This also fixes a latent ordering bug where step attributes could null-overwrite the prompt-cache read count. **Webapp callsites** — `dashboardAgentHeadStart.server.ts` and the head-start route resolve the model and the cache breakpoint through the same seam, so the warm-up prefix and the following turn share one provider. The head-start firing gate is provider-aware: on Bedrock it gates on `AWS_REGION` and lets the SDK resolve credentials (IAM role / static keys / session token / bearer), so a role-based deploy still warms; on Anthropic it stays `Boolean(ANTHROPIC_API_KEY)`. `app/env.server.ts` gains the optional AWS vars and validates `DASHBOARD_AGENT_MODEL_PROVIDER`. `ANTHROPIC_API_KEY` is untouched and not required on a Bedrock deploy. **Tenant-isolation hardening** — `internal-packages/rbac/src/fallback.ts`: for a **scoped** context, the OSS `authenticateUserActor` now applies the same membership floor as the session path — a delegated user-actor token whose user is not a member of the scoped org/project is denied (403). Unscoped tokens keep their prior behavior (no tenant claim, no lookup). The user lookup falls back replica→primary so replication lag can't spuriously 401 a just-joined member. Members and admins are unaffected. Previously this invariant held only through per-route discipline; this makes it structural. ## Enabling Bedrock (later, ops) - Set `DASHBOARD_AGENT_MODEL_PROVIDER=bedrock` **identically** in both the webapp and the agent task container — the webapp warms the cache prefix and the task reads it, so a split would silently miss the cache. - Set `AWS_REGION` and provide credentials the Bedrock SDK can resolve (IAM role preferred). For v1 this runs **without** an Anthropic API key. Note: with no Anthropic key set, rollback is "turn the agent off", not "unset the switch" (unsetting falls back to the Anthropic provider, which then has no key). - Two things to confirm before rollout: the Sonnet inference-profile id is validated against the SDK's own model-id union but still warrants a live smoke test; and Bedrock prompt caching for Sonnet is a 5-minute window (not Anthropic's 1h), so input-token cost rises when flipped. ## Testing Unit tests cover both provider paths: the provider switch and per-provider cache shapes, a structural regex asserting Bedrock ids are real inference profiles (not an echo of the table), the split-metadata cache telemetry, and real-Postgres RBAC tests — member allowed, scoped non-member denied (org-only and project-only), missing user → 401, admin non-member exempt, unscoped success. `typecheck --filter webapp` and the dashboard-agent + rbac suites pass.	2026-08-18 13:14:01 +02:00
Eric Allam	c0b84595a3	feat(webapp): hosted webhook ingress, delivery pipeline, and dashboard (#4344) ... ## Summary The server half of hosted webhooks: the public ingress endpoint, signature verification, the delivery pipeline (Postgres partitioned storage + ClickHouse for ordering), the in-app partition manager, the HTTP API, and the dashboard (Deliveries, Endpoints, and the in-app test console). The public SDK and docs half is #4537. That PR carries the user-facing API (`webhook()`, `chat.event` / `chat.channels`, the `@trigger.dev/slack` connector) and builds on the shared `@trigger.dev/core` schemas that ship here. ## Shipping behind a flag A `WEBHOOK_ENABLED` env var (default off) gates the public ingress route and the engine worker plus partition cron, so merging and deploying this changes nothing in production until it is flipped on per environment. The dashboard is separately gated per org by the `hasWebhooksAccess` feature flag. ## Note on packages This PR includes the `@trigger.dev/core` schema additions the server compiles against, but carries no changeset. Core is not consumed independently of the SDK, so it is released together with the SDK via #4537. Keeping its changeset off `main` means no release cut from `main` publishes it early.	2026-08-16 14:33:42 +01:00
Eric Allam	b98dd79fe4	feat(webapp,run-store,database): env-configurable transaction resilience (maxWait + tx-start retry) (#4623) ... ## What Makes two transaction-resilience behaviors real and env-var configurable, defaults set to the good values, so we can tune during and after the Aug 15 database patch window without a redeploy: - **maxWait 2s → 10s** (TRI-12982): how long Prisma waits to borrow a connection before it can `BEGIN`. A restart freeze holds the pool full, and the only thing that errored was transaction starts giving up at 2s. - **Retry transaction-start P2028-at-acquisition** (TRI-12984): when Prisma can't borrow a connection within `maxWait` it raises P2028 (`Unable to start a transaction in the given time`) and **no SQL ran**, so retrying is safe. Scoped narrowly: only that error (never P2024 pool-exhaustion), 2 attempts, jittered backoff, and a token-bucket budget so a mass freeze can't amplify into a retry storm. ## Env vars (`DATABASE_*` convention) Generic defaults: | var | default | |---|---| | `DATABASE_TRANSACTION_MAX_WAIT_MS` | `10000` | | `DATABASE_TRANSACTION_START_RETRY_ENABLED` | `true` (kill switch) | | `DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS` | `2` | | `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS` | `50` | | `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS` | `250` | | `DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC` | `50` | | `DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST` | `100` | Per-writer-pool overrides, each falling back to the generic when unset (same pattern as the per-client pool/connect-timeout work): `RUN_OPS_DATABASE_TRANSACTION_*` and `RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all 7 knobs each). Transactions only open on writer pools, so those are the only pools with their own knobs. Each pool gets its **own** token bucket, so a storm on one pool can't drain another's retry budget. ## Design - The retry primitives live in `internal-packages/database` and never read `process.env` (IoC): a P2028-at-acquisition classifier, a `TokenBucketRetryBudget`, and `withTransactionStartRetry`, folded into the `$transaction` helper via a new `startRetry` option. Config is resolved at the app boundary and threaded in. - The `$transaction` helper is the chokepoint (wraps the whole transaction), not the per-statement `$allOperations` extension. - The run engine's writes go through `PostgresRunStore`'s own `.$transaction(...)`, not the webapp helper, so both the helper and the two `PostgresRunStore` sites apply maxWait + retry (sharing the per-pool config). Builds on the `options?: { timeout, maxWait }` seam added in #4514. - Webapp `$transaction` call sites get the default `maxWait` + retry injected at one merge point, so no call site needed editing. ## Evidence - Unit red/green in `internal-packages/database`: reverting the helper wiring turned the acquisition-retry test red (`Unable to start a transaction in the given time`), re-applying it green. Full package suite 25/25. Covers: classifier (P2028-acq yes, P2024 no, in-tx P2028 no), retry (retry-then-succeed, no-retry P2024, stop at maxAttempts, disabled, budget-exhausted, jitter bounds), token bucket, and `$transaction` wiring. - Typecheck clean: webapp, run-store, run-engine. - Full-stack run: bounded queue-ay pass (15 projects, real dev runs through the run-engine `PostgresRunStore` transaction path). 13 pass; the 2 failures are one documented known-failure and one stale-worker-state flake that passes 2/2 with this change active on a fresh app. - Boots cleanly with per-pool overrides set. ## Configuration & rollout Ship **inert** first (zero behavior change), then flip to the good values **live via env** — no redeploy needed for either. ### Inert — behaves exactly as today ``` DATABASE_TRANSACTION_MAX_WAIT_MS=2000 # Prisma's built-in default (change defaults to 10000) DATABASE_TRANSACTION_START_RETRY_ENABLED=false # disable the new retry entirely ``` `maxWait=2000` is what every path used before (Prisma's default; the run-store sites and the helper passed no maxWait). `retry=false` short-circuits `withTransactionStartRetry` to a single run and makes the serialization-retry exclusion a no-op. Verified on the pooler-freeze rig: identical fail-fast P2028 at ~2003ms with zero retries — byte-for-byte current behavior, across all pools. ### Production ("good") — the baked defaults Rely on defaults (nothing to set) or set explicitly: ``` DATABASE_TRANSACTION_MAX_WAIT_MS=10000 DATABASE_TRANSACTION_START_RETRY_ENABLED=true DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS=3 # 3 attempts (2 retries); ~30s acquisition tolerance covers a ~20-25s freeze DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS=50 DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS=250 DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC=50 DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST=100 ``` Per-pool overrides `RUN_OPS_DATABASE_TRANSACTION_*` and `RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all seven knobs each) are optional and fall back to the generic set — not needed for v1; the generic set covers the control-plane, run-ops, and run-ops-legacy writer pools. Readers open no transactions and take nothing. **Guardrail:** the retry only engages when a pool's `pool_timeout` > `maxWait`. Prod is fine (`DATABASE_POOL_TIMEOUT=60` >> 10). Do not set any writer pool's `pool_timeout` at or under `maxWait`, or saturation failures flip from retryable P2028 to non-retryable P2024 and the retry silently stops helping. ### Rollback Env flip (set inert) or revert. Retry only fires where no SQL ran, and the per-pool token bucket caps a storm. No migration. refs TRI-13295, TRI-12982, TRI-12984	2026-08-15 09:03:10 +01:00
Eric Allam	dc8f90e66e	fix(run-engine,webapp): resolve dequeue worker version fresh per task (#4622) ... ## Summary After a deployment promotion or rollback, newly triggered runs could keep dispatching onto the previously deployed version for up to 30 seconds. Runs now resolve the current version fresh on every dequeue, so a promotion or rollback takes effect immediately. ## Fix The dequeue path resolved the worker version through a 30s in-process cache that nothing invalidated on promotion, and it loaded the worker's entire task and queue set only to keep the single row matching the run. Both go away: the resolve now fetches just the matched task and queue by unique index and reads them fresh, so there is no cache left to serve a stale version. ``` - cache.get(env:current) # 30s TTL, never invalidated -> stale - worker + ALL tasks + ALL queues + worker + one task WHERE slug=... + one queue WHERE id/name=... # fresh ``` A kill-switch env var (`RUN_OPS_WORKER_VERSION_FRESH_READ_ENABLED`, default on) falls back to the old cached path without a code deploy. Verified end-to-end on an isolated stack: a run triggered after a mid-stream promotion now dequeues onto the new version, with the previous stale behavior reproduced first.	2026-08-14 17:32:43 +01:00
Chris Arderne	ed1bb72fb8	feat: implement cron window spread backend (#4566) ... - New DB fields on Schedule and ScheduleInstance - Use `queueTimestamp` for the "effectiveAt" delayed start time, propagate it to Clickhouse TaskRun table - Disable fastpath for delayed jobs - Add schedule timing logic, API endpoints with windows, persistence - Calculate phase for every schedule, only persist when window is non-null - Additional o11y for phased rollout	2026-08-12 12:24:32 +01:00
Katia Bulatova	9a3bee0288	feat(webapp): dashboard agent — UI (#4529) ... Stacked on #4418. Merge that first. The UI slice of the dashboard agent: the side panel, the chat transport wiring, message and card rendering, suggested prompts, and chat history. #4418 works without this — the system is simply invisible. The diff is mostly components, so the notes below cover only the three decisions you can't read off the markup. Behavior and a hands-on walkthrough live in GUIDEBOOK.md, which lands with #4525. ## Decisions worth knowing - **Action rows always render at the end of a turn.** The model's emission order isn't trusted for layout, so action blocks are split out of the stream and appended last. Display only — `answered` stays keyed on the emission index. - **The last-chat memory is org-true.** It's keyed by the chat's own organization, and a foreign or deleted chat comes back as a 404 the client treats as gone, rather than an empty chat it keeps around. - **A dead stream self-heals from the settled transcript.** Terminal records are written to the chat row after the client's stream closes, so the panel re-reads it. The poll gate is any unfinished turn — a dangling tool part, not just an open investigation. ## Notes - Gated by `canAccessDashboardAgent`; no behavior change with the flag off. - Page marks: `handle.agentPageContext` on 47 routes, ~20 lines each. - Entry points: Ask Trigger button, ⌘J, Help & Feedback. The old ⌘I and `?aiHelp=` links keep working. ## Screenshots <img width="1440" height="788" alt="Screenshot 2026-08-07 at 15 14 29" src="https://github.com/user-attachments/assets/f4e89e8d-13ed-4be3-a88d-d5cca3ece0fa" />	2026-08-12 08:38:59 +02:00
Katia Bulatova	4569657923	feat(webapp): dashboard agent — chat, reports, investigate (#4418) ... ## What & why This is the system behind the Dashboard Agent — an assistant that answers questions about a project's runs, errors, queues, deploys and health, and can investigate failures end to end. The agent runs as a chat.agent task in its own Trigger project. It has no access to the main database or ClickHouse; all platform data is read through the public API using a delegated, read-only user token. Everything here is behind `canAccessDashboardAgent` and inert with the flag off. The UI that mounts the panel lands in #4529. ## Stack `#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook gallery. The scenario/contract reference for the whole stack is `internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch branch): it states, per feature, what makes each thing happen and where that is decided. ## What's inside **Agent runtime and tools** — `internal-packages/dashboard-agent`: prompt, tool set (API reads, TRQL query, docs, navigation, evidence/investigations, repo source), conversation compaction, a prompt-prefix token budget pinned by snapshot test, and sampled LLM-judged turn evals. The package cannot import webapp server code, which is what makes the "no DB access" claim structural rather than a convention. **Contracts** — `internal-packages/dashboard-agent-contracts`: `trigger://` URIs, intents, and the block envelope every rendered card travels in. **Conversation store** — `internal-packages/dashboard-agent-db`: drizzle over postgres-js in its own `trigger_dashboard_agent` Postgres schema, plus one additive migration. **Auth boundary** — the user-actor token gains an optional environment claim; one guard (`userActorEnvironment.server.ts`) enforces it so routes don't each re-derive the rule. Token minting, cap ceiling, and the RBAC fallback path for self-hosted. **Transport** — webapp resource routes that mint the token and proxy each turn, and SDK-side mid-turn reconnect. **Public API the agent reads through** — orgs, projects, environments, runs, queue metrics, workers, a run's commit metadata, repo snapshot, reports, and `POST /api/v1/query`. **Reports** — the health report's layout is declared once and shared by the card, the markdown surface and the JSON/MCP surface, so the same report reads the same in the dashboard, the terminal and an editor. **Block renderers** — the report and investigation cards the flows above already emit (`app/components/dashboard-agent/`). The panel that hosts them, and the rest of the chat UI, is #4529. **Query safety and CSP** — see below. ## Key decisions - **The agent is a separate Trigger project, not webapp code.** It reads platform data over the public API with a delegated user-actor token whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no webapp imports. - **The PAT-only auth helper now refuses user-actor tokens.** This is an intentional behavioral change: its callers consume only a bare userId and do not enforce delegated-token capabilities. Actor-aware routes continue through the scoped route builders instead. - **RBAC fallback builds a delegated token's ability from its own cap**, never the blanket ability a PAT gets (read-only when the token declares none). Without this, the agent's read-only cap would buy a write JWT on self-hosted. - **Org creation checks RBAC only for user-actor tokens, and only after the env gate**, so an install with `ORG_CREATION_API_ENABLED` off returns 404 rather than 403, and an ordinary PAT never consults an ability the route has no org to scope. Both orderings are pinned by test. - **The query path is read-only in depth.** TRQL rejects write statements at the grammar level (they don't parse, rather than being filtered), ClickHouse runs with `readonly=1`, and the org/project/env filters are injected server-side from the credential — the request body cannot widen scope. An unparseable query denies instead of falling through to the permissive resource. - **Document-wide img-src CSP.** Remote images are an outbound-request/exfiltration surface, so the policy permits only own-origin/data/blob, the required SSO avatar hosts, and the favicon endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST; wildcard hosts and bare schemes are intentionally not allowed. - **The chat transport reconnects on a mid-turn EOF** (`@trigger.dev/sdk`). A body that ends without a turn-complete is terminal only when the server says `X-Session-Settled: true`; otherwise the transport resubscribes from `lastEventId` with bounded backoff, and any record re-earns the budget. Previously a closed long-poll window or a proxy restart left the reply stuck as if still generating. - **Conversations live in their own datastore**, schema-scoped and foreign-key-free (it references `organizationId`/`userId` by id, because in cloud it is a different database). It is a display read-model for the History tab and transport resume; `chat.agent`'s object-store snapshot remains the model's source of truth. - **Deterministic first.** Reports and health checks contain no LLM — they are computed from the same data the dashboard shows, and the model only narrates and links them. That is what makes a number in an answer auditable. ## Testing - 63 new test files, run with `pnpm run test --filter webapp` and per-package vitest. Heaviest coverage on the auth boundary (`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`, `contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the report layout, and the SDK reconnect. - The agent package has a separate eval lane (`pnpm run test:evals`, `vitest.eval.config.ts`) that hits the real model, so it never runs in `pnpm test`. - Live-tested against a local stack scenario by scenario; the GUIDEBOOK lists the condition each behaviour is expected under, which is what those runs were checked against. ## Changelog `.server-changes/dashboard-agent.md`, plus changesets for `@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat reconnect) and the CLI's `mint-token` help text.	2026-08-11 18:56:14 +02:00
Saadi Myftija	02de2e693f	feat(api): separate rate limit budget for deployment endpoints (#4565) ... Most deploy-flow API calls shared the general per-environment rate limit bucket with all of that environment's runtime traffic, so an org with heavy API usage could intermittently 429 its own deploys; the `/api/v*/deployments` endpoints themselves were fully exempt from rate limits as a stopgap ([#2774](https://github.com/triggerdotdev/trigger.dev/pull/2774)), which promised a dedicated limiter as the follow-up. This is that follow-up: the whole deploy-flow group now runs on its own budget, separate from runtime API limits. ### Design A new `deploymentRateLimiter` covers every endpoint the deploy flow depends on: the `/api/v*/deployments` group, the env API key exchange (`/api/v1/projects/:ref/:env`), build-time env var resolution and sync (`/envvars`, `/envvars/:slug/import`), preview branches, `/api/v1/remote-build-provider-status` and `/api/v1/artifacts`. The general API limiter whitelists the same shared path list, so exactly one limiter applies to each path and the two can't drift apart. Buckets are keyed per environment for environment API keys and per token for the PAT-authenticated phase of a CLI deploy (whoami, key exchange, branches). The deploy budget is controlled via the `DEPLOYMENT_RATE_LIMIT_*` env vars.	2026-08-11 17:51:31 +02:00
Eric Allam	820c079145	perf(webapp): read per-run environment config from the replica at dequeue (#4560) ... ## Summary Adds an opt-in path to serve a run's per-run configuration reads from the control-plane read replica instead of the primary, reducing primary database load during task execution. The managed-worker dequeue resolves each run's environment, organization, and environment variables before starting the run; those rows are stable for the life of a run, so they can safely come from the replica. Gated by `CONTROL_PLANE_DEQUEUE_READS_FROM_REPLICA`, defaulting to `"0"` (reads from the primary, unchanged from today). Set it to `"1"` to route the reads to the replica. The env-var read is scoped to the dequeue/resolution path (`resolveVariablesForEnvironment`); dashboard env-var reads and writes always stay on the primary. When no read replica is configured, `$replica` transparently falls back to the writer, so single-database self-host is unchanged either way. Verified end-to-end against a real primary/replica split, in both `trigger dev` and deployed (managed-worker) runs: with the flag on, env vars inject correctly and a value set immediately before triggering a deployed run is present on the run.	2026-08-10 17:45:57 +01:00
Eric Allam	90e8bd5c12	feat(webapp,database): opt-in per-client Prisma driver adapters (#4539) ... ## What Adds an opt-in path to run each Prisma client through **`@prisma/adapter-pg`** (the node-postgres driver) instead of the built-in engine driver, controlled by a **per-client env var, all off by default**: | env var | client | |---|---| | `CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER` | control-plane writer | | `CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER` | control-plane replica | | `RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER` | new run-ops writer | | `RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER` | new run-ops replica | | `RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER` | legacy run-ops writer | | `RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER` | legacy run-ops replica | With every flag unset the construction path is byte-identical to today (`datasources` URL + Rust engine), so this is inert until a flag is turned on. Per-client granularity allows enabling the adapter only where it's wanted. ## How - Enables the `driverAdapters` preview feature on both schemas (`@trigger.dev/database` and `@internal/run-ops-database`). This keeps the **Rust query engine** — it does NOT add `queryCompiler` — so query behavior, result types, and engine tracing spans are unchanged. - A shared `buildDriverAdapterPool` builds each client's `pg.Pool` with an explicit `max`, a bounded `connectionTimeoutMillis` (the node-postgres pool otherwise waits unbounded on acquire), and an `onPoolError` handler (an unhandled idle-connection error would otherwise crash the process). Threaded through all four client builders via a `useDriverAdapter` flag. - Adds `@prisma/adapter-pg` + `@types/pg` to the webapp; `pg` is already pinned at `8.15.6` (adapter-pg 6.x requires `pg < 8.17`). ## Connect-failure handling (the important correctness/security bit) Under the adapter an unreachable DB no longer surfaces as `PrismaClientInitializationError` / `P1001`; it becomes a `P2010` "Database not reachable: <host>" (or a raw `ECONNREFUSED`/`ENOTFOUND`-class error). Two handlers are updated so a client on the adapter behaves like today: - **`isInfrastructureError`** now recognizes those shapes (P2010 with a connectivity message, and raw connectivity errno codes). Without this, the DB **hostname would leak into API-client-facing errors** and the failure would go unlogged. Security-relevant. - **`isPrismaRetriableError`** treats the adapter's pool-acquire timeout ("timeout exceeded when trying to connect") as retriable, preserving the `P2024` retry behavior the adapter otherwise drops. ## Evidence Validated on an isolated stack that mirrors the production DB topology (chained PgBouncers in front of writer + reader): - **Behavioral parity:** raw-query results and Prisma error codes/`meta` are byte-identical between the engine driver and the adapter across the queried shapes (unique-constraint `meta.target`, record-not-found, transaction-timeout, serialization-failure, etc.). - **Feature matrix:** a full 380-project queue-ay pass shows no adapter-caused regressions — pass/fail parity between adapter-off and adapter-on, with the residual failures being pre-existing known-failures/flakes common to both. ## Rollout / rollback All flags default off; enable per client via env var, roll back by unsetting and redeploying (no data migration). Recommended first target is a single writer; enable one client at a time. ## Follow-ups (not in this PR) - `$metrics`-based pool observability is removed under the adapter (the Prometheus route + `db.pool.connections.*` instruments); the metrics replacement (via `pg.Pool` counters) lands in a separate PR. - Note for operators: on the adapter path, interactive-transaction `maxWait` does not bound pool acquisition — `connectionTimeoutMillis` does. ## Note on connection-string parameters The adapter pool is built from the base DSN, so Prisma-specific DSN parameters that node-postgres does not understand are not honored when a client is on the adapter: - **Prisma TLS spellings** (`sslaccept`, `sslcert`, etc.) — node-postgres uses `sslmode`/`ssl` instead. Our production DSNs do not use these Prisma-specific TLS params, but any deployment whose DSN relies on them must be checked before enabling a flag. - `pgbouncer=true` and `statement_cache_size` — effectively moot under the adapter, which uses no persistent named prepared statements. `connection_limit`, `pool_timeout`, and `schema` are handled explicitly (passed as `max`/`connectionTimeoutMillis` and PrismaPg's `{schema}` option). refs TRI-13039 --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-08-08 21:27:20 +01:00
Matt Aitken	04f9c4e1a5	fix(webapp,run-engine,core): drop the hidden debounce ceiling, fail fast on an unusable maxDelay (#4521) ... Debouncing with a `delay` longer than an hour did nothing at all. The engine applied a server-side ceiling on how long a debounced run could be pushed back, measured from the run's `createdAt` and defaulting to one hour. A run is only pushed back while its new execution time stays inside that ceiling, so a `delay` at or above it could never push anything: the waiting run was released, the trigger started its own run, and the next trigger repeated it. A `delay: "12h"` produced one run per trigger, each correctly delayed by 12h, with no error raised and nothing on the run to show the debounce key had been ignored. The ceiling is now unset by default. A debounce key with no `maxDelay` keeps collapsing triggers for as long as they keep arriving, which is what the docs have always described. Self-hosters who want a bound can still set `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS`. That has a consequence worth stating plainly, so the docs now carry a warning for it: with no `maxDelay`, a continuously triggered key never executes. Set `maxDelay` when the work has to happen eventually. **Failing fast on an unusable `maxDelay`.** A caller who sets `maxDelay` no longer than their `delay` hits exactly the dead end described above, so that pair is now rejected at trigger time instead of silently behaving as if no debounce were set: ``` debounce.maxDelay (1h) must be longer than debounce.delay (12h). A debounced run is only pushed back while it stays inside maxDelay, so with these values every trigger would create its own run. ``` An unparseable `maxDelay` is rejected too, rather than quietly falling back to no bound at all, and so is a `delay` given as a date rather than a duration, which could never work because the value is re-applied on every push. The same check runs against a configured server ceiling, so a self-hosted deployment that sets `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS` gets the error rather than the silent failure this PR is about. With no `maxDelay` and no configured ceiling, which is the default, there is nothing to conflict with and nothing is rejected. The docs, the `TriggerOptions` JSDoc and the engine option all now state that the room available to push is the gap between `delay` and `maxDelay`. The run engine suite gains the case that motivated this: four triggers on one key with a 12h delay now collapse to a single run.	2026-08-07 07:55:35 +00:00
Eric Allam	58bf4e2833	feat(webapp): per-client database pool and connect timeout overrides (#4515) ... ## Summary Follow-on to #4513. The database connect timeout is now honored, but a single global value has to serve three separate databases at once (control-plane, legacy run-ops, and run-ops). This adds optional per-client overrides for the Prisma pool and connect timeouts, one pair for the writer and one for the read replica of each of the three databases, each falling back to the shared `DATABASE_POOL_TIMEOUT` / `DATABASE_CONNECTION_TIMEOUT` when unset. That lets one database's clients run a fail-fast connect timeout (with a bounded pool wait) while another keeps more headroom, without a single knob forcing the same tradeoff everywhere. No behavior change until an override is set. It also tags each client's queries with its specific datasource (`control-plane` / `legacy-run-ops` / `run-ops`, writer or replica) via the `db.datasource` span attribute, so telemetry can attribute connection behavior to a specific database instead of just writer-vs-replica.	2026-08-05 17:26:41 +01:00
Eric Allam	db6228dd1e	chore(webapp,core,sdk): upgrade @s2-dev/streamstore to 0.25 and migrate S2 hosts (#4349)	2026-08-01 11:33:34 +01:00
Eric Allam	0445b8ec27	fix(webapp,clickhouse): keep the rest of a ClickHouse batch when one run or span has un-ingestable JSON (#4358) ... ## Summary A single run output, trace span, or payload carrying JSON that ClickHouse can't ingest (for example nesting past its depth limit) used to fail the whole insert batch, so unrelated runs and spans silently disappeared from the runs list, traces, and logs. This keeps the rest of the batch and handles the offending row instead of dropping everything around it. ## Fix Recovery is per-table, matched to what each table needs: - **Runs** (`task_runs_v2`) keep their status. We follow ClickHouse's failing-row hint to strip just the un-ingestable JSON column(s) so the run still lands (its output reads from Postgres on the detail page), up to a configurable limit (`RUN_REPLICATION_MAX_POISON_STRIPS_PER_BATCH`, default `1`). Past the limit we stop and land the batch with `allow_errors` in a single pass, skipping the remainder. Cost stays a fixed handful of inserts no matter how large or poisoned a flush is. - **Trace events and payloads** (high volume, append-only) recover with a single `allow_errors` insert: the good rows land in one pass and only the un-ingestable rows are skipped. Before falling back, a lightweight sanitizer still repairs what it can losslessly (lone UTF-16 surrogates, out-of-range integers) so a repairable row lands in full. To read the failing-row hint we patch `@clickhouse/client-common`: its error parser truncates the server response and discards the `(at row N)` position, so the patch preserves the full text for the recovery path to read.	2026-08-01 09:17:20 +01:00
Eric Allam	c72ebf9084	fix(webapp,run-engine): stop batchTriggerAndWait hanging when item streaming never completes (#4397) ... ## Summary `batchTriggerAndWait()` could leave a parent run waiting forever. The 2-phase batch API blocks the parent on the batch's waitpoint as soon as the batch is created, but the batch is only sealed at the end of item streaming. If streaming never completed, nothing sealed the batch, nothing completed the waitpoint, and the parent stayed suspended with no timeout and no way to recover. Supersedes #4016, which added the reaper alone. ## Fix Admission for item streaming was being decided twice. Batch creation passes its own rate limiter, which fixes `expectedCount` and blocks the parent, and then the item stream had to pass the general API limiter as well, competing with unrelated traffic. A second limiter could therefore veto work the first had already committed the parent to. Creation now mints a bounded grant that the item stream spends, so an admitted batch can finish streaming. The grant is capped per batch rather than exempting the path, and every failure mode (no grant, spent grant, unreachable store) falls back to the normal limiter. That makes stranding much rarer but not impossible, since a request timeout or a crash can still end streaming for good. So a seal-timeout reaper aborts any batch still unsealed after `BATCH_SEAL_TIMEOUT_MS` and completes the parent's waitpoint with an error, letting `batchTriggerAndWait()` reject instead of hang. It is race-safe against a late seal, and it is only scheduled for batches that actually block a parent, so fire-and-forget batches cost nothing. Finally, the batches page used to report "Batch completion checked." for these batches while doing nothing, because the completion path returns early on an unsealed batch. It now says the batch cannot be resumed. Rate limiting is no longer the reason a batch strands, so the reaper's default stays at 30 minutes, comfortably above the SDK's worst-case stream-retry budget. ## Verification Unit and container tests cover the grant cap, the bypass ordering (it runs after the authorization check, so it can never skip authentication), and the reaper's abort, seal race, idempotency, and no-waitpoint cases. Also verified end-to-end against a running stack. With the general limit exhausted, batch creation and other API calls returned 429 while a granted batch still streamed and sealed; an ungranted batch id was rate limited rather than bypassed; and the grant cut off exactly at its configured attempt count. Reproducing the stranded state on a real parent run, the batch was aborted at the timeout, the waitpoint completed with an error, and the parent resumed and finished instead of hanging. A parentless batch left unsealed was untouched well past the reaper window. ## Verified against deployed runs The reaper was proven end to end with a real deployed run (locally-run supervisor, containerised run) and a real network fault, rather than a simulated one: toxiproxy severs the phase 2 item stream mid-flight so every SDK stream retry genuinely fails, while phase 1 still succeeds. Only the batch calls traverse the fault, so control-plane traffic is untouched. The reproduction is the shape that actually strands a parent: the task catches the `BatchTriggerError` the SDK throws and carries on, so the phase 1 block outlives the thrown error and the parent hangs at its next suspension point. With the reaper disabled, the parent sat in `EXECUTING_WITH_WAITPOINTS` for over 24 minutes holding two blockers, and stayed stuck across a full infrastructure restart: ``` type | status | has_timeout BATCH | PENDING | f <- orphan, completedAfter NULL DATETIME | COMPLETED | t <- the wait already elapsed ``` With the reaper enabled the same task under the same fault completed in about 75 seconds with zero blockers left, the batch `ABORTED`, and its waitpoint completed carrying the error. Two conditions are required to observe this at all, which is worth knowing for any future test: the run must be deployed rather than `trigger dev` (dev runs execute in process and finish while still holding blocker rows), and the wait after the caught error must exceed the checkpoint threshold, or it is served in process and never suspends. ### Why completing the batch waitpoint is sufficient `batchTriggerAndWait` runs create, then stream, then wait. A phase 2 failure throws before the wait is ever reached, and the reaper only fires on an unsealed batch, so the parent is never suspended awaiting the batch when it runs. The parent therefore does not need a synthetic result, only to stop being blocked. Note this reasoning depends on that ordering: if the wait were ever reached with an unsealed batch, completing the batch waitpoint alone would not settle the caller. ## Follow-ups - Batches stranded before this ships still need a one-off recovery; the reaper only schedules at creation time. - That same property leaves a gap if the process dies between creating the batch and scheduling the job. A periodic sweep would close it, but wants a supporting index. - When a partially streamed batch aborts, children already enqueued keep running while the parent fails. Left as-is deliberately, since cancelling triggered work is a bigger semantic call.	2026-07-31 11:55:25 +01:00
Eric Allam	4eb9292cbe	feat(webapp,run-engine): queue metrics and health dashboard (#4131) ... ## Summary Three related changes, each independently gated: **Queue metrics and health.** Per-queue depth, throughput (enqueued, started, completed), concurrency, whether a queue is throttled, and scheduling delay (how long a run waits between becoming eligible and actually starting), plus a per concurrency-key breakdown for keyed queues. Collected from inside the run queue itself, stored in ClickHouse, and surfaced on the Queues list, a new per-queue detail page, the task pages, and the run inspector. The question it answers is "does this queue have enough concurrency to keep up, and if not, which key or which limit is the constraint". **Percent-based queue concurrency limits.** A queue's concurrency override can now be expressed as a percentage of the environment limit, stored as the source of truth and re-materialized whenever the environment limit changes. Absolute overrides above the environment limit are now **rejected with a 400** instead of being silently capped, which is a behavior change on `POST /api/v1/queues/:queue/concurrency/override`. **The `health` report.** A server-computed verdict on whether work is flowing, whether the runs that do start are healthy, and whether telemetry is fresh, rendered as text with sparklines. Available as `GET /api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool (plus a `report` MCP prompt, which shows up as a slash command in hosts that support prompts). With the flags off, the Queues page renders the pre-metrics component verbatim, nothing is emitted, and nothing is written to ClickHouse. ## Configuration Two independent gates, on purpose. Emission is global so data accrues for everyone before anyone can look at it; the view is per organization so it can be turned on for one org at a time without a deploy. **Runtime flags (no restart)** | Flag | Store | Gates | | --- | --- | --- | | `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by default) | All emission, gauges and counters. Cached in-process for 10s with stale-while-revalidate, warmed eagerly at boot so the first op after a deploy is not dropped. | | `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` | Fraction of queue ops that emit a gauge. Counters are never sampled, so throughput stays exact at any rate. | | `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag` row, per-org `Organization.featureFlags` override wins | Whether an org sees the metrics view at all: the Queues list variant, the queue detail route, the built-in Queues dashboard, the concurrency-keys endpoint, and the metrics blocks on task pages and the run inspector. Off by default; a gated org gets a 404 on the detail route rather than an empty page. | Both Redis keys are readable and writable from `/admin/queue-metrics` (super-admin UI, with a live per-shard stream-health table) and `GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface uses its own Redis client, so it works on any instance regardless of whether that instance runs the emitter or the consumer. **Environment variables (boot time)** | Variable | Default | Notes | | --- | --- | --- | | `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and injects it into the run engine. Without it the run queue has no emitter at all. | | `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on this instance. Independent of emission, so consumers can be sized separately from the API. | | `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per queue. | | `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals insert batch, so an ack can never outrun a write. | | `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` | falls back to the run-queue Redis | Set `HOST` to move the metrics stream onto a dedicated instance so a metrics backlog cannot compete with the run queue for memory. Self-hosters can leave it unset and get a single-Redis deployment. | | `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000` dedicated | Bound on how much a stalled consumer can hold. The default is deliberately lower when the stream shares the queue-critical Redis. | | `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the per-queue cumulative counter key, refreshed on every write, so only queues idle for the whole window are purged. | | `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue names tracked per environment; overflow collapses into `__overflow__`. | | `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea one level down, per queue. | | `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live sample-rate key above. | | `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics tables in the Query page, its schema docs, the schema API and the AI query context. Off keeps them unlisted while the feature is dark; a query naming them still runs either way. | | `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring | Runs queue metrics on their own ClickHouse service: the consumer's inserts and every queue-metrics read go through it, so a metrics-heavy chart refresh never competes with runs-list or trace reads. Unset reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`, reads on the query pool). | | `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split, so the consumer's inserts can never land on a read endpoint. | | `QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}` | `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other per-workload ClickHouse clients. | Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a Postgres migration adding the nullable `TaskQueue.concurrencyLimitOverridePercent`. Both are additive. ## How collection works Queue operations produce two kinds of signal, and they have opposite failure modes, so they are handled differently. **Gauges** (queued, running, queue limit, env queued, env running, env limit, throttled, plus keys-with-backlog and worst-key wait on keyed queues) are read *inside* the same Redis script that performs the enqueue or dequeue, so the reading is atomic with the operation it describes rather than a racy follow-up read. The script returns them on its reply and the app forwards them to the stream. Gauges are sampled and drop-tolerant: they are aggregated with `max`, so a lost reading costs resolution, never correctness. **Counters** (enqueued, started, completed, plus nack and dead-lettered) are cumulative odometers. Each event increments a per-queue key on the metrics Redis and emits the absolute total, and ClickHouse takes the difference across buckets at read time. This is the important property of the design: a summed-delta counter undercounts permanently on any lost event, while a cumulative one self-heals, because the next surviving reading restates the whole total. Only bucket granularity can be lost, never the total. A queue returning after its odometer TTL expired restarts at 1 and reset detection handles it, which is safe precisely because expiry only spans a window with no activity. Both land on one sharded Redis stream. A consumer reads it with a consumer group, reclaims stale pending entries on a 15s interval rather than on every poll, maps one entry to one or two ClickHouse rows (whole-queue and, for keyed queues, per-key), and acks only after the insert lands. Each batch carries a dedup token derived from its stream-entry ids, and the target tables set `non_replicated_deduplication_window`, so a retried batch cannot double-count either the raw rows or the aggregates that hang off them. Consumer and emitter both emit OTel metrics (`queue_metrics.emitter.emitted`, `queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`); stream depth and group lag are the two worth alerting on, and `lag_unknown` exists because Redis can report a null lag after a trim, which must not be read as zero. ## Storage and read path `queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four aggregate tiers are materialized straight from raw, never cascaded off each other, each with a 30 day TTL: - `queue_metrics_v1`, 10 second buckets per queue, the default read path - `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and cross-queue ranking - `env_metrics_v1`, 10 second buckets per environment, queue-independent so it stays cheap at any range - `queue_metrics_ck_v1`, 10 second buckets per concurrency key Every tier is an MV from raw because the counter states do not survive a cascade: their merge is order sensitive, so a `-MergeState` chain off the 10s table inflates the result, and the same property means an aggregate state may only be merged inside one queue. That constraint is now enforced by the query engine rather than by reviewer discipline: a column can declare a `mergeGroupKey`, and any query that references it without grouping by, or pinning to a single value of, every named key fails to compile with an actionable message. On the read side, TRQL gains three tables (`queue_metrics`, `env_metrics`, and a `queue_metrics_by_key` that is hidden from the editor, schema docs and schema API but still queryable, so per-key rows can never silently merge into a plain per-queue query), plus `deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level optimizations ride along: a table can declare coarser rollups, so a query whose bucket interval is 5 minutes or wider is routed to the 5m table with no change to the query itself, and it can opt into the ClickHouse query cache with time bounds floored to a fixed grid, so the auto-refreshing dashboards actually share cache entries instead of missing on every tick. Both are caller-side substitutions, so the printer stays unaware of physical layout. All of this can also live on its own ClickHouse service. A table declares the pool its reads run on, the three queue-metrics tables name the dedicated one, and the ingestion consumer writes through the same client, so both directions move together with one env var and nothing else routes differently. The other engine change is opt-in gap filling: charts can request rows for empty buckets, where counters zero-fill and gauges carry forward. Grouped gauge series are densified per group and carried inside a partition, so a quiet queue's line holds its last value without bleeding another queue's value into it. ## Queue concurrency limits `concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth when an override is set as a percentage; the absolute `concurrencyLimit` is materialized from it (floored, clamped to at least 1 so a percentage can never act as a pause, and never above the environment limit). Every path that changes an environment limit now recalculates the environment's percent-based overrides afterwards, outside the transaction, and pushes changed limits to the engine. The push is attempted even when the stored value did not change, so a previously failed sync self-heals rather than leaving the database and the engine diverged; paused queues are skipped so a recalculation cannot effectively unpause one. The API accepts exactly one of `concurrencyLimit` or `percent`, and the reject-instead-of-clamp change above means a request asking for more than the environment allows now fails loudly. The percent bound (greater than 0, at most 100) is defined once and shared by the zod schema, the dashboard mutation handler and the service, so the three cannot drift. The concurrency-keys table on a queue is now paginated against the ClickHouse per-key tier, ranked by peak backlog with the total on every row from a single scan, and only the keys on the current page are enriched with live counts from Redis. That replaces a hard top-50 cap with something whose cost is a function of page size rather than key cardinality. ## The health report `GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The verdict is computed on the server and is deterministic, not model-generated. Three independent analyzers run over one input snapshot: flow (is work moving, and if not, is the cause a limit, throttling, one bad queue, or dead-lettering), execution (are the runs that start succeeding, and at what latency), and liveness (how fresh is the telemetry). When telemetry is genuinely stale, the first two are forced to unknown and every actionable field is stripped, so no surface ever advises action off stale data. Authorization is per query table rather than a blanket query grant: a JWT must be scoped to every table the report reads (`runs`, `env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull a report that reads more than it was granted. `period` is validated as a shorthand with a 90 day ceiling at the edge. The report catalog is a registry of `{ load, interpret }` entries, so the next report is a new entry and no change to the route, the view model, the renderers, the CLI or the MCP tool. `trigger mcp` no longer launches the install wizard when stdout is a TTY, which fixed a real failure: hosts spawn the server over a PTY, so the wizard would open and the client would time out waiting for a server that never started. The wizard now needs `trigger mcp --install`. ## The part that is live regardless of every flag The enqueue and dequeue scripts now return a 2-tuple so a gauge reading can ride back on the reply. Every return site in the eight affected scripts is wrapped, and a `nil` original is converted to `false` on the way out, because a raw `nil` in the first slot would make Lua truncate the multi-bulk reply and silently drop the gauge on the throttled and empty-queue paths. The reply shape and the destructuring on the app side are exercised on every queue operation whether or not metrics are enabled, so that is the part of `run-engine` worth the closest review. One behavior fix in the same area: the scheduling-delay anchor is set only on a run's first entry into the queue. Anchoring it to trigger time on re-enqueues made waitpoint and checkpoint resumes report the entire wait as scheduling delay. Queue ordering is untouched, so a re-enqueued run keeps its position, and nacks deliberately keep the original anchor because a rolled-back dequeue is the same continuous wait. A pending-version promotion still anchors to trigger time, on purpose: that promotion is the run's first real entry into the queue, since the trigger deliberately held it back waiting for a worker version, and the TTL is armed at the same point for the same reason. The consequence is worth naming, because it is a judgement call: a run that waits on a deployment reports that wait as scheduling delay on its queue, which is time unrelated to queue capacity. ## Verification Unit and integration suites across the new package, the run queue, the mapping layer, the query engine and ClickHouse (including a test that applies migration 036 through the same splitter CI uses, and a regression test that inserts the same batch three times to prove the aggregates do not inflate). Beyond that, the whole path was driven end to end against a live stack with real runs: emitter to Redis stream to consumer to ClickHouse to the dashboards, for both the local dev path and the deployed path where a supervisor drives the dequeue, with assertions on exact counter reconstruction per queue and per concurrency key, throttling, environment saturation, scheduling delay, and a deliberate mid-stream reading drop to confirm the cumulative counters still reconstruct the correct total. The gated-off state was checked on every touched surface. The dedicated ClickHouse service was verified against a second, separately-schema'd instance: with it configured, the driven counters reconstruct exactly on the dedicated instance, the shared instance gains no rows for that window, a read through the query API returns the value that exists only on the dedicated instance, and a `runs` query still succeeds (it would fail outright if it were mis-routed to a service without that table). With the variable unset, the full suite passes unchanged. --------- Co-authored-by: Katia Bulatova <katia@trigger.dev> Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com> Co-authored-by: James Ritchie <james@trigger.dev>	2026-07-29 16:45:24 +01:00
Saadi Myftija	44eca4d166	feat(webapp): org-gated internal API origin in run env vars (#4366) ... Adds an opt-in way for operators to route deployed runs' API traffic through a different origin than the public one, per organization. Set `INTERNAL_API_ORIGIN` on the webapp and enable the `internalApiOriginEnabled` feature flag (globally or per org, with the org override winning in both directions): deployed runs for enabled orgs then get `TRIGGER_API_URL` set to the internal origin instead of `API_ORIGIN`. Useful for gradually moving run traffic onto a private network path. ## Design The origin is resolved when an attempt starts, so flag changes take effect on the next attempt and roll back the same way, with no task redeploys. The org override is read fresh per attempt; the global default comes from the cached flags registry (a cold read fails safe to the public origin). When `INTERNAL_API_ORIGIN` is unset the flag is a no-op and no extra queries run, so existing deployments are unaffected. Dev runs always use the public origin, and `TRIGGER_STREAM_URL` remains unchanged.	2026-07-28 11:28:09 +02:00
Eric Allam	d3906241a5	feat(webapp): read realtime run rows from the primary, not the replica (#4378) ... ## Summary The realtime runs feed hydrates run rows from read replicas, which means it needs a replica-lag gate to avoid serving a run's previous state right after a write. Setting `REALTIME_BACKEND_NATIVE_RUN_READS_FROM_PRIMARY=1` reads those rows from each run store's primary instead, so there is no lag to gate against: no probe, no wake delay, no stale-read retries. Off by default, so nothing changes unless you set it. ## Design The run stores already decide replica-vs-primary from the *brand* on the read client they are handed: a branded replica keeps the read on the owning store's replica, an unbranded writer escalates it to that store's own primary. So this is a one-line choice at the hydrator, and it stays correct across topologies. With the run-ops split on, each leg lands on its own writer and the caller's client is never forwarded across databases; with the split off, it is the single database's primary. ```ts const runReader = new RunHydrator({ readClient: runReadsFromPrimary ? prisma : $replica, runStore, }); ``` The same flag skips constructing the lag estimator, since probing a replica the feed no longer reads would be measuring the wrong thing. Independently, `AuroraReplicaLagSource` detected Aurora by letting `aurora_replica_status()` fail, on the assumption that the app-level catch made that free. It isn't: an unresolvable function is a query error the driver reports to the error log on every sample, so a non-Aurora replica produced a continuous stream of error events while the estimator quietly fell through to its next candidate. It now resolves the function with `to_regproc` and memoizes the answer, so the unparseable call never reaches the wire.	2026-07-26 19:43:41 +01:00
Chris Arderne	6997aeb05e	fix: security release 2026-07-08 (#4316)	2026-07-21 12:00:58 +01:00
Daniel Sutton	ae96b6c175	fix: read-your-writes + global-scope idempotency correctness under the run-ops split (#4284) ... ## What & why Two related correctness fixes for the run-ops DB split. Under the split, run-store reads can route to a **lagging read replica**; a just-written run/waitpoint/batch can then be missed, causing a wrong decision. **1. Read-your-writes → owning primary.** Surfaced first as an intermittent `wait.until({ idempotencyKey })` re-wait on retry. Auditing the run-store read surface found the same class at sibling sites (some gating mutations or returning spurious 404s, others tolerable/self-healing). Reads that must observe their own writes now route to the owning **primary** (`findRun`/`findWaitpoint`/`findBatchTaskRunByFriendlyId` → `*OnPrimary`, a primary re-read on a miss, or a retryable 404 where the SDK polls). Read-view reads stay on the replica. All additive — the happy path is unchanged. **2. Global-scope idempotency across the split.** A `global`-scope key carries no per-run salt, so the same `(env, task, key)` triggered concurrently from parents resident on **different** run-ops DBs could dedup-miss on each DB and create a duplicate (the per-DB unique index can't enforce cross-DB uniqueness). Such triggers (global scope, or scope-absent, while split is active) are serialized through the existing Redis idempotency claim, the loser resolves the winner by id across both DBs, and the claim is reacquired on the expired/failed clear-and-recreate path. `run`/`attempt` scope embed the run id and never contend. ## Stacked for review This is the **base** of a 2-PR stack, split so review is easier: - **This PR** — production code only (34 files). - **Stacked tests PR → https://github.com/triggerdotdev/trigger.dev/pull/4285** — the caller-driven guards (55 test files) on top of this branch. ## Validation Local run-ops split, **both 2-DB and 3-DB**, fresh boot on this branch: SDK canary 64/71 (only the known concurrency/input-streams/s3 failures), quarantine sweep **0 unexpected** (340 pass / 16 known / 4 local) in each topology, dashboard e2e 0 failed. No product regressions.	2026-07-19 17:57:41 +01:00
nicktrn	890dd66eb5	feat(webapp): route ClickHouse reads to an optional read replica (#4081) ... ## Summary Adds optional configuration to send ClickHouse read traffic to a separate instance (for example a read replica) while writes stay on the primary `CLICKHOUSE_URL`. This lets operators offload read load (runs list, traces, logs, queries) from the cluster that handles inserts. Fully backwards compatible: with nothing new set, every client resolves to `CLICKHOUSE_URL` exactly as before. ## What it adds - `CLICKHOUSE_READER_URL` (optional): a single reader endpoint that the read-only clients fall back to. Read clients resolve `<own URL> ?? CLICKHOUSE_READER_URL ?? CLICKHOUSE_URL`. The task-events client (which both inserts events and reads traces, spans, and logs) is built as a reader/writer pair so queries use the reader while inserts stay on `CLICKHOUSE_URL`. - `RUNS_LIST_CLICKHOUSE_URL` (optional): a dedicated client for the runs list (dashboard list, runs list API, live reload, child-status counts), so the highest-traffic read path can target its own instance. ## Safety Only read-only clients fall back to the reader: logs, query, admin, runs list, the pending-version lookup, and the realtime run-id resolver. The query page is constrained to read-only (the TSQL parser rejects anything that is not a `SELECT`, and a `readonly` setting is applied). The task-events client routes inserts to the writer and queries to the reader per method, so a write can never reach the reader. Pure-write clients (event inserts, replication) always use `CLICKHOUSE_URL`. Note: this PR targets a baseline branch rather than `main` so the diff stays scoped to the read-replica changes. It will be retargeted to `main` before merge. --------- Co-authored-by: Eric Allam <eallam@icloud.com>	2026-07-15 14:59:42 +01:00
nicktrn	976171ea16	feat(webapp): management API for orgs, projects, members, and settings (#4146) ... ## Summary Adds a set of PAT-authenticated management API endpoints so orgs, projects, members/invites, environment variables, and a few project/environment settings can be managed programmatically (scripting, automation) rather than only through the dashboard. Each route is a thin wrapper over the **existing** service the dashboard already uses, with the same authorization applied at the route layer - no new business logic. ## Endpoints **Organizations** - `POST /api/v1/orgs` - create an org (`createOrganization`) - `PATCH /api/v1/orgs/:orgParam` - rename (title) - `DELETE /api/v1/orgs/:orgParam` - soft-delete (`DeleteOrganizationService`; keeps the active-subscription guard) **Members & invites** - `GET /api/v1/orgs/:orgParam/members` - list members + pending invites - `DELETE /api/v1/orgs/:orgParam/members/:memberId` - remove a member (last-member guarded) - `POST /api/v1/orgs/:orgParam/invites` - invite by email (`inviteMembers`, sends the invite email) - `DELETE /api/v1/orgs/:orgParam/invites/:inviteId` - revoke an invite **Projects** - `PATCH /api/v1/projects/:projectRef` - rename (`ProjectSettingsService`) - `DELETE /api/v1/projects/:projectRef` - soft-delete (`DeleteProjectService`) - `PUT /api/v1/projects/:projectRef/default-region` - set the default region by worker-group name (`SetDefaultRegionService`) - project GET/list now return `defaultRegion` (worker-group name, or null when unset) **Environments** - `POST /api/v1/projects/:projectRef/:env/pause` and `/resume` (`PauseEnvironmentService`) - `POST /api/v1/projects/:projectRef/:env/regenerate-api-key` - rotate the env secret key (`regenerateApiKey`, RBAC `write:apiKeys`) - env var create now accepts an optional `isSecret` flag ## Auth & authorization - All routes authenticate with a **Personal Access Token** (`Authorization: Bearer tr_pat_...`). - Org/project routes are built on the PAT route builders in `apiBuilder.server.ts`: `createLoaderPATApiRoute` (already existed) and **`createActionPATApiRoute`** (added here - the loader builder had no mutation counterpart). The builder runs auth, resolves the org/project role-floor via `context`, and enforces a declarative `authorization` block using the same RBAC actions the dashboard applies (`manage:organization` / `read:members` / `manage:members` / `manage:project`). Handlers keep a membership-scoped query as the floor, so a non-member gets a 404. This also gives these routes `tenantContext` user attribution (Sentry) and `ServiceValidationError`-to-status mapping for free. - **Membership floor (important).** The OSS RBAC fallback grants a permissive ability, so `ability.can(...)` can't reject a non-member on self-hosted. Every handler therefore resolves the target scoped to the caller's membership (`members: { some: { userId } }`) → 404 for non-members. `authorization` is the *role* gate; this is the *tenant* gate. `resolveOrganizationForApiUser` (`organizationApiAccess.server.ts`) is the org-tier version of the existing `findProjectByRef` - org-addressed PAT routes are new, so no such helper existed before. - Env-tier routes reuse the existing `authorizePatEnvironmentAccess` (`write:apiKeys`). ### What `createActionPATApiRoute` gives you A route is pure declaration - the builder handles auth, RBAC, validation, tracing, and error mapping: ```ts export const action = createActionPATApiRoute( { method: "PUT", // one verb, or ["PATCH", "DELETE"] for multi-verb routes params: ParamsSchema, body: SetDefaultRegionRequestBody, // zod-validated context: async ({ projectRef }) => { // resolve the org for the RBAC role-floor const project = await prisma.project.findFirst({ where: { externalRef: projectRef, deletedAt: null }, select: { organizationId: true }, }); return project ? { organizationId: project.organizationId } : {}; }, authorization: { action: "manage", resource: () => ({ type: "project" }) }, }, async ({ params, body, authentication, ability }) => { // auth + authz already enforced. Just do the work. // `throw new ServiceValidationError("Region not found", 400)` → mapped to that status. return json({ ok: true }); } ); ``` Handled for you, so handlers stay thin: - **Method allowlist** - `method` accepts a verb or an array; any other verb → `405` with an `Allow` header, *before* auth runs: ```ts const allowedMethods = method ? (Array.isArray(method) ? method : [method]) : undefined; if (allowedMethods && !(allowedMethods as string[]).includes(request.method.toUpperCase())) { return json({ error: "Method not allowed" }, { status: 405, headers: { Allow: allowedMethods.join(", ") } }); } ``` - **PAT / user-actor auth** in a single roundtrip → `401` on missing/invalid/revoked token. - **RBAC** - `context` computes the caller's role-floor for the target org/project; `authorization` gates it → `403` with a structured error body. - **Sentry attribution** - `tenantContext.enrich({ userId })` so events from the handler carry the acting user. - **Typed errors** - a thrown `ServiceValidationError` is mapped to its `.status` (default 400); anything else → `500`, and expected boundary errors are logged as `warn` (kept out of Sentry). - **Validation** - params / query / headers / body are all zod-checked → `400` with details. ## Notes for reviewers - Everything wraps an existing service; the intent is API parity for things that are currently dashboard-only, not new behaviour. - `createActionPATApiRoute` is new shared infra (the PAT + RBAC mutation builder that didn't exist). It's self-contained - the loader builder and existing routes are untouched. - `@trigger.dev/core` gets one additive field (`defaultRegion` on the project response, optional/nullable for client-server version skew) - changeset included, patch. - `removeTeamMember`'s last-member guard is now atomic (Serializable transaction via the `$transaction` helper, with retry), so the dashboard and API both get it server-side. Added a `## Transactions` rule to `apps/webapp/CLAUDE.md` (always use the `$transaction` helper); migrating the remaining direct usages is tracked in TRI-11698. ## Open questions - ~~Is PAT the right auth (vs OAT for automation)?~~ **Resolved: PAT.** Organization Access Tokens are currently internal-only (used by the image builder) and not user-accessible, so they can't back this yet. - Should any of these be gated behind a flag or scope? - Naming/shape of the routes.	2026-07-15 10:20:08 +01:00
Daniel Sutton	a1ca64613b	fix(webapp): reuse the primary db pool for legacy run-ops when DSNs match (#4253) ... ## Summary When the run-ops split is enabled, the legacy run-ops database client was always constructed as its own connection pool, even when it points at the same database as the primary (control-plane) client. On setups where those two DSNs resolve to the same physical database, this opened a second, redundant pool and doubled the number of connections used against that database. This change makes the legacy client reuse the primary client's pool whenever their DSNs point at the same database, and only open a separate pool when they genuinely differ. ## Fix A small `sameDatabaseTarget` comparison (host, port, database name, user) decides whether the legacy DSN points at the same database as the primary. When it does, the legacy handle reuses the primary client by reference, so no second pool is opened. When the DSNs diverge, the legacy client is built independently as before, so the split still works once the databases are actually separate. Two smaller changes ride along: - An optional per-pool limit for the run-ops read replica, which connects unpooled and so draws raw backend connections; unset, it falls back to the existing default and behaviour is unchanged. - A startup warning about a missing legacy replica URL is now suppressed when the legacy client shares the primary pool, where it would be misleading. ## Verification Booted the webapp end-to-end in three modes and confirmed the pools opened as expected via the client's own startup logs and live backend connection counts: split off (single pool), split on with a shared database (legacy reuses the primary pool, no doubling), and split on with separate databases (legacy opens its own pool).	2026-07-14 11:20:21 +01:00
Chris Arderne	64e5d732ad	chore(webapp,core): remove the unused ResourceMonitor server logging helper (#4244) ... The `ResourceMonitor` server-side logging helper is no longer used. It periodically logged the webapp process own memory, disk, and CPU usage behind the `RESOURCE_MONITOR_ENABLED` flag (off by default), and was also exported from `@trigger.dev/core/v3/serverOnly` with no other consumers. This removes the helper, its `@trigger.dev/core` export, the webapp wiring, and the `RESOURCE_MONITOR_ENABLED` env var. The supervisor has its own unrelated `ResourceMonitor` class, which is left untouched.	2026-07-13 20:32:29 +01:00
Eric Allam	29598a77b8	feat(webapp): add option to disable PostgreSQL task-event writes (#4242) ... ## Summary Adds `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` (default off), which makes the task-event store skip all PostgreSQL `TaskEvent` writes. It's for deployments that store task events in ClickHouse (`EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2`) and no longer want the PostgreSQL copy. ## How it works The guard sits at the single postgres write boundary, `TaskEventStore.create` / `createMany`, so it covers every write path (OTLP ingestion and run-lifecycle events) with one check. Reads are untouched (`findMany` / trace queries / streaming), so existing PostgreSQL events remain readable. Leave it off unless the default store is `clickhouse_v2`, otherwise task events for any run still routed to PostgreSQL would be dropped.	2026-07-13 17:17:02 +01:00
Daniel Sutton	bea7e2be90	feat(webapp,run-store): route run-graph reads and writes through the run-store router (#4237) ... ## Summary Run-graph data (runs, batches, waitpoints, and their related tables) can now live in a database separate from the control plane, with every read and write routed to the correct database by each run's residency. This makes reading and writing run data more reliable once the two are split, and is a no-op for single-database installs. ## Design - Run-graph table access goes through the run-store router, which selects the legacy or the new run-ops store per run instead of assuming one shared client. - The legacy run-ops client is now independently pointable, so legacy run data can be served from its own database (and replica) rather than the control-plane connection. - Run-graph writes go straight to the run-graph database instead of being forwarded through the control plane, and replication targets are split so runs in the new database still replicate to analytics without under-counting. - Read-through slots refuse the control-plane client, so a missing residency fails loudly instead of silently reading the wrong database. - Migration `20260710120000_drop_remaining_run_graph_seam_foreign_keys` drops the foreign keys that still crossed the run-graph / control-plane seam, which is what lets the two live in separate databases. The split stays off unless explicitly enabled and the two databases are confirmed physically distinct; startup fails closed otherwise. Verified by running the full dashboard end-to-end suite against both a single-database configuration and a three-database configuration (control plane, the new database, and a physically separate legacy database), with runs on both residencies. No misrouted reads in either configuration.	2026-07-13 13:54:54 +01:00
Eric Allam	5ba8557a51	chore(webapp,core): remove the end-of-life v3 (engine V1) execution stack (#4236) ... ## Summary v3 (the engine that ran the SDK v3 era, internally `RunEngineVersion.V1`) is end-of-life. Following the removal of the v3 execution apps ([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and the legacy dev websocket ([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this removes the remaining v3 execution stack from the server. Clients still on v3 (an old SDK or CLI that has not upgraded) keep getting a clear "upgrade to v4" response. Triggers, batch triggers, reschedules, and deploys that resolve to v3 are rejected with a graceful 4xx pointing at the migration guide, never a 5xx, so a stale client cannot affect server health. Self-hosted instances still running v3 should stay on the 4.5.x release line until they migrate. ## What is removed - The MarQS queue and its shared/dev queue consumers. - The v3 socket.io namespaces (coordinator, provider, shared-queue) and the v3 run lifecycle services (attempt, checkpoint, and batch-resume). - The graphile-worker background job system; all live jobs already run on `@trigger.dev/redis-worker`. - The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally, so the flag is gone. - Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace` subpath and the legacy socket message catalogs) and the now-dead MarQS environment variables. ## What stays The v4 engine is untouched. The graceful v3 rejection boundary stays, `determineEngineVersion` still detects a v3 project so it can reject it, and the batch service plus batch-completion worker stay for current clients. Live queue concurrency limits and metrics now read from the v4 run engine instead of MarQS, and a brand-new dev environment now defaults to v4. ## Dependency cleanup Removes webapp dependencies left unused by this change: `seedrandom` and `semver` (only the removed v3 code used them) plus a set that was already dead, their orphaned `@types` packages, and two dead files. Adds a `knip:deps` script and a `knip.json` config so unused dependencies can be found the same way going forward.	2026-07-13 11:32:06 +01:00
Daniel Sutton	c601739d35	perf(webapp,run-store): grouped run-ops reads + mint-kind flip grace (#4227) ... ## Summary Two threads on the run-ops split path. Read path: per-item run reads are batched into grouped queries, a waitpoint's connected-run reads are bounded, and the dedicated-schema relation hydrators fetch only the requested columns instead of whole rows. Retrieve also falls back to the other database when a routed read misses, so a run whose physical residency diverges from its id shape is still found rather than returning a spurious not-found. Fewer and lighter queries on the run read path, with no change to results. Mint-kind flip safety: flipping which database new runs mint to is now a deterministic wall-clock cutover, for both per-org and global flips. For a grace window every process resolves the same database, so a flip cannot route two concurrent triggers that share an idempotency key to different databases (which would bypass the per-database unique constraint and create a duplicate run). Supersedes the earlier #4205 and #4208. Draft: validation in progress.	2026-07-13 10:17:12 +01:00
Eric Allam	45527e317a	feat(webapp): opt-in worker pool for OTLP ingest transform (#4232) ... ## Summary Under high OTLP ingest volume, the whole decode, transform, and enrich pipeline runs on the request event loop, so a single CPU core becomes the ceiling while the rest sit idle. This adds an opt-in worker pool that moves decode, transform, and LLM-cost enrichment onto worker threads, keeping the main thread free for I/O. It is off by default (`OTEL_TRANSFORM_WORKER_POOL_ENABLED`), so behavior is unchanged unless enabled. ## Design Workers do decode, filter, convert, and enrich (including LLM pricing match). The main thread stays the single database reader: it loads the pricing registry and broadcasts the compiled model rows to the workers (re-broadcasting on every reload), so workers never touch the database. The pure transform is extracted into a dependency-light module (no Prisma/Redis/ClickHouse imports) so it can run inside a worker. Importantly, the main thread keeps the existing single consolidated insert path, so ClickHouse insert batching and part count are unchanged. The parallelism buys CPU headroom, not more insert streams (which would add merge pressure). The worker is bundled as a standalone file at build time and ships in the existing image with no Dockerfile change. In local load testing the pool sustained roughly 2.6x the throughput of the single-thread path and kept the main thread responsive under load.	2026-07-11 13:38:12 +01:00
Eric Allam	5d0e9d9dc5	feat(webapp): make the default realtime backend configurable (#4231) ... ## Summary The default realtime backend was hardcoded to Electric. This adds a `REALTIME_BACKEND_DEFAULT` env var (`electric` | `native` | `shadow`, default `electric`) that chooses the backend for any environment whose org has no `realtimeBackend` override. Behavior is unchanged unless you set it; per-org overrides still win. The default is applied at every point where the per-org flag falls through: the initial value, the flag lookup default, and the error fallback.	2026-07-11 09:36:09 +01:00
Matt Aitken	b64b54c74e	feat(webapp): pass database writer and reader config to auth plugins (#4229) ... ## Summary The RBAC and SSO auth plugins can own their own database client, but they could only read `DATABASE_URL`, so every connection they opened landed on the primary. The host webapp now resolves writer and read-replica URLs from its env (the same fallback chain its own Prisma clients use: control-plane URL first, then the default) and passes them to the plugins at create time via a shared `PluginDatabaseConfig`, along with separate connection limits for writes (default 2) and reads (default 5, tunable via `RBAC_DATABASE_*_CONNECTION_LIMIT` and `SSO_DATABASE_*_CONNECTION_LIMIT`). A plugin can then route hot-path reads (per-request auth checks, login routing) to the read replica and keep only rare mutations on the primary. With no replica configured, or no plugin installed, nothing changes: the OSS fallback ignores the new option and keeps reading through the Prisma clients it is already given. 🤖 Generated with [Claude Code](https://claude.com/claude-code)	2026-07-10 17:25:42 +01:00
Eric Allam	1a0198cc5e	perf(webapp,clickhouse): move runs empty-state check to ClickHouse (#4202) ... ## Summary The runs page's empty-state check (whether an environment has ever had a run, which decides between the "getting started" and "no runs match your filters" states) ran a `findFirst` against the Postgres `TaskRun` table. This moves it to ClickHouse, the same store the runs list itself reads from, so the check no longer queries `TaskRun`. ## Design Only the runs list triggers the check now (via an `includeHasAnyRuns` flag); the other presenters that reuse `NextRunListPresenter` (API, schedule detail, waitpoint detail, error group) no longer issue it. When the list is empty it runs `SELECT 1 FROM task_runs_v2 ... LIMIT 1` filtered on the full `(organization_id, project_id, environment_id)` sort-key prefix with a configurable `created_at` lower bound (`RUN_LIST_HAS_RUNS_LOOKBACK_DAYS`, default 30), so it hits the primary index and reads minimal granules. Results are cached in a tiered memory + Redis SWR cache. Only positive ("has runs") results are cached, so an environment with no runs is always re-checked and its first run shows up immediately.	2026-07-09 15:24:31 +01:00
DKP	00ee0751ec	feat(webapp): proxy PostHog through a same-origin /ph path (#4183) ... ## Summary posthog-js sent product analytics to PostHog Cloud directly from the browser. This points `api_host` at a same-origin `/ph` path that forwards to PostHog Cloud EU server-side, following PostHog's standard first-party reverse-proxy setup. ## How it works A resource route forwards each request server-side, splitting by path: `/ph/static/*` and `/ph/array/*` go to the asset host, everything else (analytics events, feature flags) goes to the ingest host. It rewrites the `Host` header, strips the `/ph` prefix, and streams the response back. Only PostHog's own cookies are forwarded, so the app session cookie stays first-party. Upstream hosts default to PostHog Cloud EU, overridable via `POSTHOG_INGEST_HOST` / `POSTHOG_ASSETS_HOST`. It also sets `cross_subdomain_cookie` so a single PostHog session is shared across the marketing site and app. Verified locally: static assets return 200 from the EU asset host, and analytics events return 200 through the ingest host.	2026-07-08 11:35:04 +01:00
Chris Arderne	aa74e68c71	feat(sdk): add bulk replay to api and sdk (#4105) ... ## Summary Adds SDK and API support for run bulk actions. You can now create bulk cancel or replay actions from `@trigger.dev/sdk` using run IDs or the same filters as `runs.list()`, then retrieve, list, poll, or abort the action by its `bulk_` handle. Tests, docs, changesets added. ## Design The dashboard bulk action service now accepts structured filters instead of reading directly from a dashboard request, so the dashboard and API share the same creation path. Replay actions created through the API are attributed with the existing `api` trigger source, while dashboard-created actions keep `dashboard`. The SDK exposes the new surface under `runs.bulk.*`, including `targetRegion` for replay region overrides and cursor pagination for listing bulk actions. ## Filters and runIds Nuance on filters. If `filter` is provided, it MUST have at least one key. This is to remove the footgun of passing no filter and selecting all runs. ```typescript { action: "cancel", runIds: ["run_1"] } // valid { action: "cancel", runIds: [] } // invalid, min(1) { action: "cancel", filter: { status: "FAILED" } } // valid { action: "cancel", filter: {} } // invalid { action: "cancel", filter: {}, runIds: ["run_1"] } // invalid ```	2026-07-07 15:43:30 +01:00
Daniel Sutton	712c7c3b1a	feat(webapp): add RUN_REPLICATION_RUN_OPS_DATABASE_URL for the runs-replication source (#4160) ... ## Summary The run-ops runs-replication source now takes its connection URL from `RUN_REPLICATION_RUN_OPS_DATABASE_URL`, required whenever the run-ops split is enabled. The runs replicator speaks the Postgres streaming replication protocol, which cannot run through a transaction pooler, so it needs its own direct endpoint separate from the app's `RUN_OPS_DATABASE_URL` (which may point at a pooler). When the split is on and this is unset, boot fails via `SplitReplicationMisconfiguredError` rather than silently falling back to a wrong endpoint.	2026-07-05 13:29:22 +00:00
Daniel Sutton	092b9ef07a	fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154) ... ## Problem The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**. The supervisor writes the run id into the Kubernetes pod name (`runner-<id>`), and pod names must be DNS-1123 labels (lowercase `[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and **those runs never launch** (they loop in `PENDING_EXECUTING` until the heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding collides distinct ids and destroys sort order. ## Fix: change the encoding, not the structure Mint a **26-char lowercase base32hex** run id: ``` run_<24-char base32hex core><region char><version char> [ 6-byte ms timestamp ][ 9 CSPRNG bytes ] ``` - **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase, order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding. Hand-rolled encode/decode (no new dependency). - **48-bit ms timestamp** in the leading bytes → plain string sort == creation order at millisecond resolution. - **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no retry loop). - **region / version** are raw positional chars (read via one `charAt` before decoding/routing), version = `"1"`. DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** — `runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still recovers the exact id incl. region+version. ## Residency discriminator: length → version char `classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a third format. It now discriminates on the **version char at a fixed position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex alphabet) → NEW; everything else → LEGACY. Total, never throws. The `Residency` (NEW/LEGACY) contract the routing store consumes is unchanged; the `"ksuid"` `ResidencyKind` label is retained only because it's the persisted `runOpsMintKsuid` feature-flag value. ## Scope / verification - Generator + discriminator in `@trigger.dev/core` isomorphic; mint path + all id-shape call sites swept (~40 webapp files); changeset added (`@trigger.dev/core` patch). - Core unit tests (encode/decode round-trip + property, generator shape, ms sort-order incl. intra-second, parse partitioned-vs-legacy, firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp typechecks; format/lint clean. ## Open decisions (flagged, not silently chosen) 1. **Backward-compat**: existing 27-char base62 KSUID runs now classify LEGACY. On test cloud these are the broken/looping runs that never completed, so this is acceptable — but worth a conscious call before prod. No transitional length-recognition added (keeps the discriminator clean). 2. **Storage collation**: the sort guarantee is byte-order — if the run-ops id column is `TEXT` with default locale collation it's silently not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the run-ops schema. 3. **Region sourcing** wiring — see `regionCharForRegion` / `REGION_CODES`. --- ## ⚠️ Required migration — deploy in lockstep This PR renames a persisted feature-flag key/value and an env var. These are **not** changed by the code alone and must be migrated when this deploys, or affected orgs silently fall back to `cuid` minting (no crash — `defaultValue: "cuid"`): 1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` → `RUN_OPS_MINT_ENABLED` (carry the value over). 2. **DB** `organization.featureFlags`: migrate both the key and value together: - key `runOpsMintKsuid` → `runOpsMintKind` - value `"ksuid"` → `"runOpsId"` Until an org's flag row is migrated, its `runOpsMintKind` lookup misses and it mints `cuid` (legacy) — so no NEW-store ids for that org until the data lands. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 10:05:54 +01:00
Daniel Sutton	962bc48738	feat(run-ops): automatically migrate the dedicated run-ops database (#4150) ... ## What Adds the ability to **automatically migrate the dedicated run-ops database** (the NEW DB in the run-ops split), matching how every other database in the system is migrated. Follow-up to the run-ops split activation. ## Changes - **Migrate runner** — new `internal-packages/run-ops-database/scripts/migrate.mjs`, exposed as `db:migrate:deploy` / `db:migrate:status`. Connects via `RUN_OPS_DATABASE_URL` (the same var the app uses) and expands `${VAR}` refs like Prisma's dotenv. - **Self-host** — `docker/scripts/entrypoint.sh` runs the run-ops migration on boot when the DB is configured, gated by `SKIP_RUN_OPS_MIGRATIONS`. Single-DB installs never set the URL, so it's a clean no-op. - **Single env-var family** — the run-ops DB is now addressed by one canonical `RUN_OPS_*` family, connect path and migrations resolving the identical URL: - `RUN_OPS_DATABASE_URL` (writer) — replaces `TASK_RUN_DATABASE_URL` - `RUN_OPS_LEGACY_DATABASE_URL` — replaces `TASK_RUN_LEGACY_DATABASE_URL` - `RUN_OPS_DATABASE_READ_REPLICA_URL` — replaces `TASK_RUN_DATABASE_READ_REPLICA_URL` - the old `TASK_RUN_*` aliases, the `??` coalesce, the `runOpsNewDatabaseUrl` indirection, and the migrate-only `directUrl` are all removed (consumers read `env.RUN_OPS_DATABASE_URL` directly). `directUrl` was dropped because it was only ever used by `prisma migrate` (never the app runtime) to bypass a pooler for advisory locks — premature here since the run-ops connection isn't wired to the app yet. If a pooler is later introduced for the app, a direct URL can be reintroduced then. ## Safety - **Pure rename** — nothing deployed sets any `TASK_RUN_*` var (the split isn't activated anywhere yet; `.env.example`, docker-compose, and cloud already use `RUN_OPS_*`), so there is no config migration. - **Single-DB / self-host** — no new required env var; entrypoint and migrate are no-ops when `RUN_OPS_DATABASE_URL` is unset. - **Cloud** — runs migrations as pre-deploy ECS tasks (companion cloud PR), calling these same `db:migrate:deploy` / `db:migrate:status` commands. ## Verification - Live migration against a fresh scratch DB with only `RUN_OPS_DATABASE_URL` set: both migrations applied, no `P1012`/`P1013`; `${VAR}` expansion, idempotent re-run, `status`, and no-op skip all pass. - Schema parity 4/4; `typecheck --filter webapp` 18/18; affected split/replication tests 34/34. ## Scope This delivers automatic migrations only. Enabling the app to *use* the new DB (setting `RUN_OPS_DATABASE_URL` + `RUN_OPS_SPLIT_ENABLED` on the service) is a separate activation step. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-04 15:44:47 +00:00
Daniel Sutton	8465ac5ac3	feat(run-ops): webapp db topology, flags, and split-mode resolver wiring (#4117) ... ## What Wires the run-ops split into the webapp: database topology, environment flags, split-mode gating, and the control-plane resolver/cache layer that the run-store and run-engine seams from the previous PR plug into. - **DB topology & env** (`apps/webapp/app/db.server.ts`, `env.server.ts`, `entry.server.tsx`): adds the run-ops database clients/topology and the environment variables that configure and gate the split. - **runOpsMigration module** (new `apps/webapp/app/v3/runOpsMigration/`): the webapp-side machinery — `splitMode.server.ts`, `controlPlaneResolver.server.ts` + `controlPlaneCache.server.ts`, `readThrough.server.ts`, `crossSeamGuard.server.ts`, `distinctDbSentinel.server.ts`, id-minting helpers (`mintBatchFriendlyId`, `runOpsMintKind`, `resolveInheritedMintKind`), `runOpsCascadeCleanup.server.ts`, the split read gate, and route/unblock catalogs. - **Store/engine wiring** (`app/v3/runStore.server.ts`, `runEngine.server.ts`, `runEngineHandlers.server.ts` + new `runEngineHandlersShared.server.ts`): points the webapp's store/engine construction at the resolver, and factors shared handler logic out so both seams use one path. - **Read-path touch-ups**: `runtimeEnvironment.server.ts`, `eventRepository/index.server.ts`, `taskRunHeartbeatFailed.server.ts`, `engineVersion.server.ts` route their run/environment lookups read-through the resolver. - `413a94511` — interlocks split mode against the native realtime backend so the two aren't enabled in an incompatible combination (see `.server-changes/run-ops-split-realtime-interlock.md`). - `dc74c57fd` — drops the earlier "known-migrated" read layer; residency is determined by id-shape only. ## Why PR5 of the run-ops split stack. This is the webapp foundation layer: it stands up the DB topology, flags, and resolver/cache the rest of the stack depends on, and repoints webapp read paths through the resolver. Additive when the split is not enabled (existing single-DB behavior preserved behind flags); behavior-changing on the read-through paths and the realtime interlock. ## Tests New vitest coverage across `apps/webapp/test/` and colocated `*.server.test.ts` files: db topology, split mode, split read gate, cross-seam guard, mint cutover / flip latency, control-plane cache, control-plane resolver, distinct-db sentinel, read-through loaders (route loaders, run-detail loaders, `findEnvironmentFromRun`), and the run-engine handlers. Testcontainers-backed; no mocks. `pnpm-lock.yaml` synced for the two new webapp deps. ## Notes Draft, **stacked on #4116** (`runops/pr04-store-engine`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-03 18:02:22 +01:00
Eric Allam	ea9d6f7430	feat(webapp): pin the dashboard agent to a deployed version (#4128) ... ## Summary The dashboard agent runs as a `chat.agent` in its own Trigger.dev project, deployed separately from the app that starts its sessions. This adds independent, non-disruptive deploys for it: a new workflow deploys the agent with `--skip-promotion` (so a deploy never becomes the current version on its own), and the app pins its sessions to a chosen version. ## How it works `DASHBOARD_AGENT_VERSION` (unset by default) pins agent sessions to a specific deployed version; when unset, sessions run on the project environment's current version. The pin is passed on session start and head start and is forwarded to every continuation run, so a pinned session stays on its version for its whole life. Cutting over to a new build becomes a config change (set the version) rather than a redeploy, and rollback is flipping it back. The workflow (`dashboard-agent-deploy.yml`) runs a leg per environment (staging and prod), each gated by its own environment, and triggers on pushes to `main` that touch the agent or its store (also available via manual dispatch). Deploy versions are per-environment, so each environment pins to its own leg's version.	2026-07-03 10:19:38 +01:00
Oskar Otwinowski	c7f6ed501c	feat(cli,core): add opt-in dev-only telnet log streaming (#4110) ... Stream dev logs over a local telnet/TCP socket. `trigger dev` mirrors its terminal output on port 6767 by default (override with --telnet-logs-port or TRIGGER_DEV_TELNET_LOGS_PORT, 0 disables). webapp, supervisor, and coordinator each expose an opt-in stream gated on a per-service *_TELNET_LOGS_PORT env var. New @trigger.dev/core/v3/telnetLogServer module (localhost-only, backpressure-safe, plain-text) plus optional static Logger.onLog / SimpleStructuredLogger.onLog sinks. Then you (or your agent) can use `nc` to connect and filter out the stream. <img width="1103" height="239" alt="image" src="https://github.com/user-attachments/assets/b4d47efc-8a57-4185-a159-10f2806627ae" />	2026-07-02 19:16:14 +00:00
Eric Allam	f1bd11a7ef	feat(webapp): gracefully shut down the v3 engine behind a flag (#4017) ... ## Summary Adds a single env flag, `DEPRECATE_V3_ENABLED` (default off), that gracefully winds down the v3 engine (`RunEngineVersion.V1`). While it's off nothing changes, so self-hosted instances still on v3 keep working. When it's on: - Triggers that resolve to v3 are rejected with a clear, actionable error pointing at the [v4 migration guide](https://trigger.dev/docs/migrating-from-v3), instead of silently creating runs that never execute. This covers single triggers, batches, scheduled fires, replays, and `triggerAndWait`, which all funnel through one place. - The legacy `trigger dev` websocket used by v3 CLIs is closed with an upgrade message (v4 CLIs use a different dev transport). - The v3 shared-queue consumer refuses to start, so no deployed v3 runs are dequeued. - The v3 run-lifecycle background jobs (heartbeat timeout, TTL expiry, retry, resume batch/dependency, delayed-run enqueue, and scheduled fires) become no-ops, so abandoned v3 runs stop generating database load. This builds on the existing deploy deprecation flag, which already rejects v3 CLI deploys. ## Design Enforcement is read through one helper, `isV3Disabled()`. Every gate combines it with a per-run or per-project engine check (`isV3Disabled() && engine === "V1"`), so a v4 run that happens to reach a shared service behaves exactly as before. v4 (V2) is never affected. The flag is a hard switch, not a drain: when it's on, in-flight v3 runs are abandoned in place rather than failed or expired, which is the intended behaviour for the final shutdown.	2026-06-27 15:51:18 +01:00