7768 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Eric Allam	26cdedda1c	perf(webapp): scope env var create pre-check to submitted keys (#4579) ... ## Summary Setting or importing environment variables ran a conflict pre-check that loaded every variable in the project and every value across all of its environments, only to decide whether the submitted keys already had a value in the target environments. On projects with many variables and environments that meant reading tens of thousands of rows on each create/import call. This scopes the pre-check to the submitted keys and target environments, so it reads only the rows it actually inspects (submitted keys × target envs), wrapped in `boundedIn` to keep the prepared-statement cache stable. Same conflict detection, a handful of rows instead of the whole project's env-var values.	2026-08-12 08:15:27 +01:00
Katia Bulatova	9a3bee0288	feat(webapp): dashboard agent — UI (#4529) ... Stacked on #4418. Merge that first. The UI slice of the dashboard agent: the side panel, the chat transport wiring, message and card rendering, suggested prompts, and chat history. #4418 works without this — the system is simply invisible. The diff is mostly components, so the notes below cover only the three decisions you can't read off the markup. Behavior and a hands-on walkthrough live in GUIDEBOOK.md, which lands with #4525. ## Decisions worth knowing - **Action rows always render at the end of a turn.** The model's emission order isn't trusted for layout, so action blocks are split out of the stream and appended last. Display only — `answered` stays keyed on the emission index. - **The last-chat memory is org-true.** It's keyed by the chat's own organization, and a foreign or deleted chat comes back as a 404 the client treats as gone, rather than an empty chat it keeps around. - **A dead stream self-heals from the settled transcript.** Terminal records are written to the chat row after the client's stream closes, so the panel re-reads it. The poll gate is any unfinished turn — a dangling tool part, not just an open investigation. ## Notes - Gated by `canAccessDashboardAgent`; no behavior change with the flag off. - Page marks: `handle.agentPageContext` on 47 routes, ~20 lines each. - Entry points: Ask Trigger button, ⌘J, Help & Feedback. The old ⌘I and `?aiHelp=` links keep working. ## Screenshots <img width="1440" height="788" alt="Screenshot 2026-08-07 at 15 14 29" src="https://github.com/user-attachments/assets/f4e89e8d-13ed-4be3-a88d-d5cca3ece0fa" />	2026-08-12 08:38:59 +02:00
Katia Bulatova	4569657923	feat(webapp): dashboard agent — chat, reports, investigate (#4418) ... ## What & why This is the system behind the Dashboard Agent — an assistant that answers questions about a project's runs, errors, queues, deploys and health, and can investigate failures end to end. The agent runs as a chat.agent task in its own Trigger project. It has no access to the main database or ClickHouse; all platform data is read through the public API using a delegated, read-only user token. Everything here is behind `canAccessDashboardAgent` and inert with the flag off. The UI that mounts the panel lands in #4529. ## Stack `#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook gallery. The scenario/contract reference for the whole stack is `internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch branch): it states, per feature, what makes each thing happen and where that is decided. ## What's inside **Agent runtime and tools** — `internal-packages/dashboard-agent`: prompt, tool set (API reads, TRQL query, docs, navigation, evidence/investigations, repo source), conversation compaction, a prompt-prefix token budget pinned by snapshot test, and sampled LLM-judged turn evals. The package cannot import webapp server code, which is what makes the "no DB access" claim structural rather than a convention. **Contracts** — `internal-packages/dashboard-agent-contracts`: `trigger://` URIs, intents, and the block envelope every rendered card travels in. **Conversation store** — `internal-packages/dashboard-agent-db`: drizzle over postgres-js in its own `trigger_dashboard_agent` Postgres schema, plus one additive migration. **Auth boundary** — the user-actor token gains an optional environment claim; one guard (`userActorEnvironment.server.ts`) enforces it so routes don't each re-derive the rule. Token minting, cap ceiling, and the RBAC fallback path for self-hosted. **Transport** — webapp resource routes that mint the token and proxy each turn, and SDK-side mid-turn reconnect. **Public API the agent reads through** — orgs, projects, environments, runs, queue metrics, workers, a run's commit metadata, repo snapshot, reports, and `POST /api/v1/query`. **Reports** — the health report's layout is declared once and shared by the card, the markdown surface and the JSON/MCP surface, so the same report reads the same in the dashboard, the terminal and an editor. **Block renderers** — the report and investigation cards the flows above already emit (`app/components/dashboard-agent/`). The panel that hosts them, and the rest of the chat UI, is #4529. **Query safety and CSP** — see below. ## Key decisions - **The agent is a separate Trigger project, not webapp code.** It reads platform data over the public API with a delegated user-actor token whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no webapp imports. - **The PAT-only auth helper now refuses user-actor tokens.** This is an intentional behavioral change: its callers consume only a bare userId and do not enforce delegated-token capabilities. Actor-aware routes continue through the scoped route builders instead. - **RBAC fallback builds a delegated token's ability from its own cap**, never the blanket ability a PAT gets (read-only when the token declares none). Without this, the agent's read-only cap would buy a write JWT on self-hosted. - **Org creation checks RBAC only for user-actor tokens, and only after the env gate**, so an install with `ORG_CREATION_API_ENABLED` off returns 404 rather than 403, and an ordinary PAT never consults an ability the route has no org to scope. Both orderings are pinned by test. - **The query path is read-only in depth.** TRQL rejects write statements at the grammar level (they don't parse, rather than being filtered), ClickHouse runs with `readonly=1`, and the org/project/env filters are injected server-side from the credential — the request body cannot widen scope. An unparseable query denies instead of falling through to the permissive resource. - **Document-wide img-src CSP.** Remote images are an outbound-request/exfiltration surface, so the policy permits only own-origin/data/blob, the required SSO avatar hosts, and the favicon endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST; wildcard hosts and bare schemes are intentionally not allowed. - **The chat transport reconnects on a mid-turn EOF** (`@trigger.dev/sdk`). A body that ends without a turn-complete is terminal only when the server says `X-Session-Settled: true`; otherwise the transport resubscribes from `lastEventId` with bounded backoff, and any record re-earns the budget. Previously a closed long-poll window or a proxy restart left the reply stuck as if still generating. - **Conversations live in their own datastore**, schema-scoped and foreign-key-free (it references `organizationId`/`userId` by id, because in cloud it is a different database). It is a display read-model for the History tab and transport resume; `chat.agent`'s object-store snapshot remains the model's source of truth. - **Deterministic first.** Reports and health checks contain no LLM — they are computed from the same data the dashboard shows, and the model only narrates and links them. That is what makes a number in an answer auditable. ## Testing - 63 new test files, run with `pnpm run test --filter webapp` and per-package vitest. Heaviest coverage on the auth boundary (`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`, `contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the report layout, and the SDK reconnect. - The agent package has a separate eval lane (`pnpm run test:evals`, `vitest.eval.config.ts`) that hits the real model, so it never runs in `pnpm test`. - Live-tested against a local stack scenario by scenario; the GUIDEBOOK lists the condition each behaviour is expected under, which is what those runs were checked against. ## Changelog `.server-changes/dashboard-agent.md`, plus changesets for `@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat reconnect) and the CLI's `mint-token` help text.	2026-08-11 18:56:14 +02:00
Saadi Myftija	02de2e693f	feat(api): separate rate limit budget for deployment endpoints (#4565) ... Most deploy-flow API calls shared the general per-environment rate limit bucket with all of that environment's runtime traffic, so an org with heavy API usage could intermittently 429 its own deploys; the `/api/v*/deployments` endpoints themselves were fully exempt from rate limits as a stopgap ([#2774](https://github.com/triggerdotdev/trigger.dev/pull/2774)), which promised a dedicated limiter as the follow-up. This is that follow-up: the whole deploy-flow group now runs on its own budget, separate from runtime API limits. ### Design A new `deploymentRateLimiter` covers every endpoint the deploy flow depends on: the `/api/v*/deployments` group, the env API key exchange (`/api/v1/projects/:ref/:env`), build-time env var resolution and sync (`/envvars`, `/envvars/:slug/import`), preview branches, `/api/v1/remote-build-provider-status` and `/api/v1/artifacts`. The general API limiter whitelists the same shared path list, so exactly one limiter applies to each path and the two can't drift apart. Buckets are keyed per environment for environment API keys and per token for the PAT-authenticated phase of a CLI deploy (whoami, key exchange, branches). The deploy budget is controlled via the `DEPLOYMENT_RATE_LIMIT_*` env vars.	2026-08-11 17:51:31 +02:00
claude[bot]	8819e25751	fix(webapp): hard-navigate after creating an organization (#4530)	2026-08-11 16:00:06 +01:00
DKP	336f515001	docs(ai): add LLM observability page (#4568) ... ## Summary Adds a docs page for LLM observability: every opted-in Vercel AI SDK call inside a task becomes its own span in the run trace, carrying the model, provider, token counts, cost, and latency. The page covers turning it on per call with `experimental_telemetry: { isEnabled: true }`, what each span inspector tab shows (Overview, Messages, Tools, and a Prompt tab when linked), linking a call to its prompt version with `toAISDKTelemetry()`, and querying usage across runs with TRQL against the `llm_metrics` table. It sits in the AI dropdown under Features, next to [Prompts](https://trigger.dev/docs/ai/prompts), and cross-links the [Query](https://trigger.dev/docs/observability/query) page. It is explicit that capture is opt-in per call (not automatic) and only covers Vercel AI SDK calls, and notes the `@ai-sdk/otel` requirement on AI SDK 7. Every API name, span tab, and TRQL column was checked against the SDK and the live query schema. ## Also in this PR Corrects one bullet in the [AI Agents overview](https://trigger.dev/docs/ai-chat/overview): it claimed an in-progress chat resumes on the new version after a redeploy, which contradicts the version-upgrades and backend pages. Chat agent runs are pinned to the version they started on; moving onto new code is an explicit version upgrade.	2026-08-11 14:40:31 +00:00
DKP	e367899510	docs(ai-agents): add chat.agent guide and refresh the AI agent guides (#4524) ... ## Summary Adds a "Build a chat agent" guide to the AI agents section, surfaces the ClickHouse chat agent example in the guides index and the AI agents overview, and refreshes the five existing workflow guides so their code is current. ## Details The pattern guides (prompt chaining, routing, parallelization, orchestrator, evaluator-optimizer) still used retired models and dated APIs. Updated them to current Anthropic Claude models (claude-haiku-4-5 for lightweight classifier roles, claude-sonnet-4-5 for the main work) and modernized the code: - route-question uses generateObject for the routing decision instead of generateText plus manual JSON parsing. - verify-news-article uses ModelMessage in place of the renamed CoreMessage. - Fixed translate-and-refine discarding its recursive refinement result, so refined translations never returned to the caller. - Fixed an invalid JSON test payload in generate-translate-copy. The pattern concepts are unchanged; only the example code was stale.	2026-08-11 14:32:32 +00:00
Eric Allam	6449a644b9	feat(webapp,cli,database): track real dev onboarding progress (#4563) ... ## Summary The dev environment "Get set up" panel used to be a static list of CLI commands that only disappeared once your tasks registered, so nothing ever changed after you ran `init` and people assumed it was stuck. It now tracks real progress: `trigger init` records the project as initialized, so step 1 checks off, and the panel updates live as the dev server connects and your tasks register. It also adds a prominent "Copy AI agent prompt" button, presented as a clear alternative ("or") to the manual CLI steps, that copies a ready-to-paste setup prompt pre-filled with your project reference for Claude Code, Cursor, or any coding agent. ## Notes - Adds a `Project.initializedAt` column (migration `20260811065646_add_project_initialized_at`); the CLI `init` command calls a new project-scoped `POST /api/v1/projects/:ref/init` best-effort at the end of setup. - The `init` scaffold now imports from `@trigger.dev/sdk` instead of the deprecated `/v3` subpath. ## Screenshots <img width="2400" height="1794" alt="v7-redesigned-card" src="https://github.com/user-attachments/assets/c2fb4fa1-9484-4700-8bd3-110d66f5a44e" />	2026-08-11 11:43:33 +01:00
Eric Allam	820c079145	perf(webapp): read per-run environment config from the replica at dequeue (#4560) ... ## Summary Adds an opt-in path to serve a run's per-run configuration reads from the control-plane read replica instead of the primary, reducing primary database load during task execution. The managed-worker dequeue resolves each run's environment, organization, and environment variables before starting the run; those rows are stable for the life of a run, so they can safely come from the replica. Gated by `CONTROL_PLANE_DEQUEUE_READS_FROM_REPLICA`, defaulting to `"0"` (reads from the primary, unchanged from today). Set it to `"1"` to route the reads to the replica. The env-var read is scoped to the dequeue/resolution path (`resolveVariablesForEnvironment`); dashboard env-var reads and writes always stay on the primary. When no read replica is configured, `$replica` transparently falls back to the writer, so single-database self-host is unchanged either way. Verified end-to-end against a real primary/replica split, in both `trigger dev` and deployed (managed-worker) runs: with the flag on, env vars inject correctly and a value set immediately before triggering a deployed run is present on the run.	2026-08-10 17:45:57 +01:00
Iss	1038641b15	chore: vouch Jakub-Vacek (#4559) ... Adds [Jakub-Vacek](https://github.com/Jakub-Vacek) to the list of vouched outside contributors so their PRs aren't auto-closed by the vouch check.	2026-08-10 15:57:50 +01:00
Eric Allam	ce368dd8e0	perf(database): index EnvironmentVariableValue.valueReferenceId so secret deletes stop seq-scanning (#4555) ... ## Why this change `EnvironmentVariableValue.valueReference` is an `onDelete: SetNull` foreign key. Deleting a `SecretReference` (the env var edit/delete path for secret values) fires the cascade `UPDATE ONLY "EnvironmentVariableValue" SET "valueReferenceId" = NULL WHERE $1 = "valueReferenceId"`. That cascade is scan-shaped: with no index on `valueReferenceId`, it reads the entire table to find the rows referencing the deleted secret. The parent `SecretReference` delete does almost no work itself; its latency is dominated by this cascade. ## Diagnosis `EnvironmentVariableValue` was indexed on `environmentId` and `(variableId, environmentId)`, but not on `valueReferenceId`. The SET NULL cascade therefore did a full sequential scan of the whole table. Two sibling SET NULL cascades on the same delete (`OrganizationIntegration.tokenReferenceId`, `User.mfaSecretReferenceId`) are index-backed and stay fast, which isolates the missing index as the cause. ## Change Add `@@index([valueReferenceId])` on `EnvironmentVariableValue`, created with `CREATE INDEX CONCURRENTLY IF NOT EXISTS` so `prisma migrate deploy` stays safe on a live table. ## Benchmark (local, seeded) Local Postgres seeded with 1,000,000 `EnvironmentVariableValue` rows, `EXPLAIN (ANALYZE, BUFFERS)` on the SET NULL cascade with zero matching rows (the worst case: reads the whole table, affects nothing): | | before | after | |---|---|---| | plan | Seq Scan (1M rows) | Bitmap Index Scan | | execution | 183 ms | 2.8 ms | In a variant where the secret matched several thousand rows, the parent `SecretReference` delete's `EnvironmentVariableValue_valueReferenceId_fkey` trigger dropped from 216 ms to 88 ms (the residual is the heap work of nulling those rows). ## Expected impact The cascade drops from a full-table sequential scan to a targeted index lookup. The win grows with the table, so the benefit is larger than the seeded numbers above. ## Risks - One extra btree to maintain on `EnvironmentVariableValue` writes; small, single-column, and it should be pre-created before the migration deploys (per the repo index rules). - No behavior change: same rows nulled, no ordering or result-set change, read paths untouched. Companion to the same fix on `ProjectAlert.channelId`.	2026-08-10 13:54:18 +01:00
Eric Allam	4c58091973	perf(database): index ProjectAlert.channelId so alert-channel deletes stop seq-scanning (#4554) ... ## Why this change Deleting a `ProjectAlertChannel` fires the FK cascade `DELETE FROM ONLY "ProjectAlert" WHERE $1 = "channelId"`. That cascade is scan-shaped: with no index on `channelId`, it reads the entire `ProjectAlert` table to find the few child rows belonging to the deleted channel. The parent `DELETE ProjectAlertChannel` does almost no work itself; its latency is dominated by this cascade. `ProjectAlert` is append-heavy and grows over time, so the scan cost only increases. ## Diagnosis `ProjectAlert` had no index on `channelId` (only `pkey` + a `friendlyId` unique). The cascade therefore did a full sequential scan of the whole table. The sibling `ProjectAlertStorage` cascade on the same delete is index-backed and stays fast, which isolates the missing index as the cause. ## Change Add `@@index([channelId])` on `ProjectAlert`, created with `CREATE INDEX CONCURRENTLY IF NOT EXISTS` so `prisma migrate deploy` stays safe on a live table. ## Benchmark (local, seeded) Local Postgres seeded with 1,000,000 `ProjectAlert` rows across 50 channels (~20k rows per channel), `EXPLAIN (ANALYZE, BUFFERS)` on the cascade delete: | | before | after | |---|---|---| | plan | Seq Scan (1M rows) | Bitmap Index Scan | | direct child delete | 740 ms | 22 ms | | parent delete `ProjectAlert_channelId_fkey` trigger | 77.7 ms | 23.8 ms | ## Expected impact The cascade drops from a full-table sequential scan to a targeted index lookup. The win grows with the table: the more rows in `ProjectAlert`, the more a scan costs and the more the index saves, so the benefit is larger than the seeded numbers above. ## Risks - One extra btree to maintain on every `ProjectAlert` insert; acceptable for a single-column index on a high-insert table, and it should be pre-created before the migration deploys (per the repo index rules). - No behavior change: no rows orphaned, no ordering or result-set change, read paths untouched. ## Follow-up `ProjectAlert`'s other cascade FK columns (`projectId`, `environmentId`, `workerDeploymentId`) are also unindexed, but their parents are soft-deleted rather than physically removed, so those cascades do not currently fire. Lower priority unless a hard-delete path is introduced.	2026-08-10 13:54:15 +01:00
Eric Allam	951d8e8d7b	feat(webapp): per-client database pool metrics that survive the driver adapter (#4541) ... ## What Follow-up to #4539. The driver-adapter work is inert until a client flips to the pg driver adapter, but the moment one does, our database observability degrades: the OTel metrics pipeline reads pool stats from Prisma's `$metrics`, which is owned by the Rust engine's `quaint` pool. Under the adapter, `pg.Pool` owns the pool, so those gauges read zero. The pipeline also only ever scraped a single client (the control-plane writer singleton). This PR makes database metrics driver-agnostic and per-client: - Every configured client registers a metrics source: control-plane writer/replica, run-ops writer/replica, legacy writer/replica. Previously only the control-plane writer singleton was scraped. - Each OTel instrument is observed per client with `db_client` and `db_driver` (`quaint` | `pg-adapter`) attributes. `db_client` uses our canonical datasource-role labels (`control-plane-writer`, `control-plane-replica`, `run-ops-writer`, `run-ops-replica`, `legacy-run-ops-writer`, `legacy-run-ops-replica`) — the same strings used for the `db.datasource` span attribute, so a metric and a trace point at the same pool. - Pool figures come from the authoritative source per driver: - **pg-adapter**: `pg.Pool` (`totalCount`/`idleCount`/`waitingCount`, plus cumulative opened/closed from `connect`/`remove` events). - **quaint**: the Rust engine's `$metrics` pool gauges/counters, exactly as before. - Query counters and duration histograms still come from `$metrics` for both drivers (the Rust engine executes queries in both cases). - New `db.pool.connections.waiting` gauge (pg.Pool exposes this; quaint reports 0). - Stops exporting Prisma metrics from the Prometheus `/metrics` route. Pool observability now lives entirely in the OTel pipeline, per driver, per client. ## Why So we can flip any client (including the control-plane writer, the primary desync-fix target) to the driver adapter without losing pool visibility. Existing dashboards keyed on the same metric names keep working; they gain a per-client dimension. ## Testing Unit (`apps/webapp/app/utils/databaseMetrics.server.test.ts`): the pure normalizer — quaint reads pool from `$metrics`; adapter reads pool from `pg.Pool` and keeps engine query metrics; `busy` never goes negative; graceful zeroing when `$metrics` is unavailable (adapter still reports live pool figures). Live smoke test against a prod-shaped local stack: three physically-distinct Postgres DBs (control-plane, run-ops, legacy) behind dual PgBouncers, split mode on, with a mix of adapter and quaint clients. Reading the actual emitted OTel metrics, every pool shows up as its own series: ``` db.pool.connections.total{db_client="control-plane-writer", db_driver="pg-adapter"} = 1 db.pool.connections.total{db_client="control-plane-replica", db_driver="quaint"} = 1 db.pool.connections.total{db_client="run-ops-writer", db_driver="pg-adapter"} = 1 db.pool.connections.total{db_client="run-ops-replica", db_driver="quaint"} = 1 db.pool.connections.total{db_client="legacy-run-ops-writer", db_driver="quaint"} = 1 db.pool.connections.total{db_client="legacy-run-ops-replica",db_driver="quaint"} = 1 db.client.queries.total{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing db.client.queries.duration.count{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing ``` Confirms: metrics are attributed per pool with the correct driver; adapter pools' figures come from `pg.Pool`; and query counters/duration histograms keep incrementing under the pg adapter. Also verified `/metrics` (Prometheus) now returns zero `prisma_*` series while still serving the app's own metrics. `pnpm run typecheck --filter webapp` passes. ## Notes - `/metrics` (Prometheus) no longer includes `prisma_*` series. Anything scraping that endpoint for Prisma metrics should read the equivalent `db.*` metrics from the OTel exporter instead. - **PgBouncer + `?schema=` gotcha (separate from this PR, worth flagging for rollout):** since #4539 parses `?schema=` from the DSN and passes `{ schema }` to the adapter, node-postgres sends `search_path` as a startup parameter. A transaction-mode PgBouncer rejects that with `FATAL: unsupported startup parameter: search_path`. Our prod control-plane DSNs use the default `public` schema with no `?schema=` param, so this is latent, but any client we flip to the adapter must not carry `?schema=` in its DSN (or the pooler needs `ignore_startup_parameters = search_path`). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-08-10 13:54:06 +01:00
Saadi Myftija	bd8ce4a50f	feat(deployments): split project dependencies and code into separate layers (#4551) ... Deploy images previously shipped node_modules and the bundled task code in a single layer, so every deploy re-pushed and re-pulled the full dependency tree even when nothing in it changed. The generated Containerfile now copies `/app/node_modules` as its own layer and the app files separately. With unchanged dependencies the dependency layer is identical across deploys, so registries and workers already have it and only the code layer moves.	2026-08-10 14:44:11 +02:00
Katia Bulatova	c00fb9c36c	fix(webapp): report start latency as unknown when there is no data (#4544) ... When the health report had no start-latency measurement for the window, it printed a confident "p95 0ms" and graded it healthy. It now shows "unknown" for that metric and skips grading it, so an absent measurement can't read as a green signal. A genuinely measured 0ms is still shown as 0ms: the loader keeps "no measurement" distinct from a measured zero instead of coercing both to 0.	2026-08-10 13:53:23 +02:00
nicktrn	6e00aaf92b	chore(deps): bump transitive mermaid to 11.16.1 (#4553) ... ## Summary Bumps the transitive `mermaid` in the lockfile from `11.14.0` to `11.16.1`. `mermaid` has no direct dependents here. It arrives through `streamdown`, which declares it as a hard dependency even though diagram rendering is gated behind the optional `@streamdown/mermaid` plugin, which we don't install. `streamdown@2.5.0` is its latest release, and its declared range (`^11.12.2`) already permits `11.16.1`, so this was a stale lockfile pin rather than a range conflict. Done as a scoped override rather than a bare lockfile refresh, so the floor survives a lockfile regenerated from an older base: ```json "mermaid@>=11 <11.16.1": "^11.16.1" ``` Net effect is 96 fewer lockfile lines, contained to mermaid's own subtree. `11.16.1` swapped out its parser, so the `langium` / `chevrotain@12` / `vscode-languageserver-*` chain drops in favour of a single `@chevrotain/types`, and `lodash-es` and `uuid@11` are no longer pulled at all. The override goes away once `streamdown` makes `mermaid` an optional peer of its diagram plugin instead of a hard dependency.	2026-08-10 12:52:49 +01:00
Eric Allam	90e8bd5c12	feat(webapp,database): opt-in per-client Prisma driver adapters (#4539) ... ## What Adds an opt-in path to run each Prisma client through **`@prisma/adapter-pg`** (the node-postgres driver) instead of the built-in engine driver, controlled by a **per-client env var, all off by default**: | env var | client | |---|---| | `CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER` | control-plane writer | | `CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER` | control-plane replica | | `RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER` | new run-ops writer | | `RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER` | new run-ops replica | | `RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER` | legacy run-ops writer | | `RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER` | legacy run-ops replica | With every flag unset the construction path is byte-identical to today (`datasources` URL + Rust engine), so this is inert until a flag is turned on. Per-client granularity allows enabling the adapter only where it's wanted. ## How - Enables the `driverAdapters` preview feature on both schemas (`@trigger.dev/database` and `@internal/run-ops-database`). This keeps the **Rust query engine** — it does NOT add `queryCompiler` — so query behavior, result types, and engine tracing spans are unchanged. - A shared `buildDriverAdapterPool` builds each client's `pg.Pool` with an explicit `max`, a bounded `connectionTimeoutMillis` (the node-postgres pool otherwise waits unbounded on acquire), and an `onPoolError` handler (an unhandled idle-connection error would otherwise crash the process). Threaded through all four client builders via a `useDriverAdapter` flag. - Adds `@prisma/adapter-pg` + `@types/pg` to the webapp; `pg` is already pinned at `8.15.6` (adapter-pg 6.x requires `pg < 8.17`). ## Connect-failure handling (the important correctness/security bit) Under the adapter an unreachable DB no longer surfaces as `PrismaClientInitializationError` / `P1001`; it becomes a `P2010` "Database not reachable: <host>" (or a raw `ECONNREFUSED`/`ENOTFOUND`-class error). Two handlers are updated so a client on the adapter behaves like today: - **`isInfrastructureError`** now recognizes those shapes (P2010 with a connectivity message, and raw connectivity errno codes). Without this, the DB **hostname would leak into API-client-facing errors** and the failure would go unlogged. Security-relevant. - **`isPrismaRetriableError`** treats the adapter's pool-acquire timeout ("timeout exceeded when trying to connect") as retriable, preserving the `P2024` retry behavior the adapter otherwise drops. ## Evidence Validated on an isolated stack that mirrors the production DB topology (chained PgBouncers in front of writer + reader): - **Behavioral parity:** raw-query results and Prisma error codes/`meta` are byte-identical between the engine driver and the adapter across the queried shapes (unique-constraint `meta.target`, record-not-found, transaction-timeout, serialization-failure, etc.). - **Feature matrix:** a full 380-project queue-ay pass shows no adapter-caused regressions — pass/fail parity between adapter-off and adapter-on, with the residual failures being pre-existing known-failures/flakes common to both. ## Rollout / rollback All flags default off; enable per client via env var, roll back by unsetting and redeploying (no data migration). Recommended first target is a single writer; enable one client at a time. ## Follow-ups (not in this PR) - `$metrics`-based pool observability is removed under the adapter (the Prometheus route + `db.pool.connections.*` instruments); the metrics replacement (via `pg.Pool` counters) lands in a separate PR. - Note for operators: on the adapter path, interactive-transaction `maxWait` does not bound pool acquisition — `connectionTimeoutMillis` does. ## Note on connection-string parameters The adapter pool is built from the base DSN, so Prisma-specific DSN parameters that node-postgres does not understand are not honored when a client is on the adapter: - **Prisma TLS spellings** (`sslaccept`, `sslcert`, etc.) — node-postgres uses `sslmode`/`ssl` instead. Our production DSNs do not use these Prisma-specific TLS params, but any deployment whose DSN relies on them must be checked before enabling a flag. - `pgbouncer=true` and `statement_cache_size` — effectively moot under the adapter, which uses no persistent named prepared statements. `connection_limit`, `pool_timeout`, and `schema` are handled explicitly (passed as `max`/`connectionTimeoutMillis` and PrismaPg's `{schema}` option). refs TRI-13039 --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-08-08 21:27:20 +01:00
Eric Allam	c526528d8f	feat(webapp,database): bound Prisma list filter arity (#4480) ... ## Summary Prisma expands `in` / `notIn` into one bind parameter per element, so every distinct list length is a separate prepared statement. Where the length tracks data volume (a batch size, a run-graph fan-out, a prior query's id set) one call site can mint hundreds of them. Each is used about once, but inserting it evicts an entry that was being reused, so the cost lands on unrelated queries sharing the pooler's statement cache. An unbounded list also risks the 65535 bind-parameter ceiling. `boundedIn()` pads a filter list to the next power of two by repeating its last element. `IN` and `NOT IN` ignore duplicates, so results are unchanged, and a call site drops from one statement per length to at most `log2(cap)`. Applied to all existing sites. ## Enforcement Two oxlint rules require the helper: a list filter must be an inline array literal or a `boundedIn()` call. - The first covers filters reached through `where` / `having` / `cursor`, and deliberately never descends into `data`, `create`, `update`, `set` or `equals`. A key named `in` in those positions is user data, not a predicate, and rewriting it would corrupt what gets stored or compared. - The second covers bare filter objects passed to where-building helpers, which the first cannot see. It found five sites in the run-graph batch loaders that were otherwise invisible. Both rules follow filters through the shapes they are actually written in: conditional expressions, logical-and objects, spread-conditional properties, computed keys, and call arguments. An array literal only counts as fixed-arity when nothing spreads into it, since `[...new Set(ids)]` has a runtime length. Twelve sites were hidden behind those shapes until the rules handled them. Scoped to `in` and `notIn`. The scalar-list filters `hasSome` and `hasEvery` compile to `&& $1` and `@> $1`, passing the whole array as a single bind parameter, so their arity never reaches the statement text and there is nothing to bound. Both rules are `error`, so new call sites fail CI. That ratchet has already caught four sites added by other PRs while this one was in review. ## Notes `boundedIn` pads by repeating rather than with null: `x NOT IN (a, b, NULL)` is never true, so null-padding a `notIn` filter would silently return no rows. Lists above 32768 are returned unchanged so padding can never push a query past the parameter limit. Route modules reach the helper through `~/db.server` rather than importing the database barrel directly, since a value import of that barrel into a module that also exports a React component is only safe while dead-code elimination prunes it. Measured on a local rig: 300 distinct list lengths produce 300 prepared statements unpadded, 10 padded. Verified end-to-end against a local stack with the full task-suite sweep, which surfaced no regressions. re2-test-supervisor-main-c526528 re2-prod-supervisor-heatwave-dualwrite	2026-08-07 16:39:58 +01:00
Eric Allam	63176a6d69	fix(webapp): stop api inheriting inbound sampled traceparents so trace sampling applies (#4532) ... ## What The internal tracing `ParentBasedSampler` in `tracer.server.ts` left `remoteParentSampled` at its default of `AlwaysOn`. Any request arriving with a `traceparent` whose sampled flag was set got recorded in full, bypassing `INTERNAL_OTEL_TRACE_SAMPLING_RATE` entirely. Because the SDK propagates its (always-sampled) trace context on calls back to the platform from inside running tasks, the large majority of API server spans inherited a sampled parent and ignored the divisor. The sampling knob was effectively inert on the busiest service. This registers a custom propagator (`NonInheritingTraceContextPropagator`) that stops adopting the inbound trace as the parent: - `inject` still delegates to the standard W3C trace + baggage propagators, so outbound propagation is unchanged. - `extract` drops the parent span (`trace.deleteSpan`) while preserving baggage, so every incoming request roots its own trace and the ratio sampler applies uniformly. `remoteParentSampled` is also set to the ratio sampler as a belt-and-suspenders fallback, in case an inbound sampled parent ever reaches the sampler another way. Two effects: the divisor becomes effective on the API server, and the API no longer stitches onto (and inflates) the propagated task-run traces, which is where the very large, un-thinnable trace chains came from. Rooting each request removes those chains rather than only diluting them. Only the internal APM trace pipeline (`INTERNAL_OTEL_TRACE_EXPORTER_URL`) is affected. The user-facing run-trace pipeline (`otel.v1.traces` -> ClickHouse) is a separate path and is untouched. The only consumer of the global propagator's `extract` is the OTel HTTP/Express auto-instrumentation, so the blast radius is inbound-request trace shape. ## Evidence (local full-stack red/green, divisor 10) A local OTLP/JSON sink counting spans; a driver fires N requests at a real endpoint, each carrying a distinct sampled `traceparent`, then counts how many spans/traces carry that run's marker. | run | code | sent | kept traces | kept fraction | | --- | --- | --- | --- | --- | | before | unmodified | 500 | 500 | 1.00 | | after | this PR | 500 | 67 | 0.134 | | after | this PR | 2000 | 213 | 0.1065 | Before: 100% of inherited-sampled requests kept, divisor ignored. After: ~10% kept (the divisor), converging on it at larger N. In every after-run each kept request is a single self-rooted trace (kept spans == kept distinct traces), confirming the inherited chains are gone, not just thinned. `typecheck` passes. ## Rollout / rollback No flag. Behavior stays governed by the existing `INTERNAL_OTEL_TRACE_SAMPLING_RATE`. Rollback is a straight revert with no data migration. ## Notes Internal dashboards that count raw span or request volume from this pipeline will read lower once this ships. That is expected: those counts were inflated by the bypass, not a real drop in traffic. Latency/percentile monitors retain plenty of samples at the current divisor. refs TRI-13031	2026-08-07 15:13:42 +01:00
Iss	98cdf89c4f	chore: vouch NERLOE (#4531) ... Adds [NERLOE](https://github.com/NERLOE ) to the list of vouched outside contributors so their PRs aren't auto-closed by the vouch check.	2026-08-07 13:54:16 +00:00
github-actions[bot]	72f50c2dad	chore: release v4.5.10 (#4440) helm-v4.5.10 v.docker.4.5.10 v4.5.10	2026-08-07 14:06:43 +01:00
Eric Allam	7246f677db	fix(webapp): strip null bytes from idempotency and debounce keys at trigger (#4527) ... ## What A trigger request carrying a Unicode NUL (`U+0000`) in the **idempotency key** or **debounce key** reached `prisma.taskRun.create()` and failed the insert, so the caller got an opaque 500 and the run was never created. These two keys are stored in `jsonb` columns (`idempotencyKeyOptions`, `debounce`), and Postgres rejects a NUL inside a `jsonb` value with `SQLSTATE 22P05` ("unsupported Unicode escape sequence ... cannot be converted to text"). This fix strips the NUL from both keys at the single trigger-input chokepoint (`#buildEngineTriggerInput`), which every trigger path flows through (single, batch item, mollified, and drainer replay). Stripping matches the existing precedent for run errors and task events. It does not change dedup behaviour: the idempotency **dedup identity** is the hashed key (a clean 64-char digest), computed independently of the raw key we clean, so dedup keeps working exactly as before. For debounce the key is used directly, so the cleaned key also becomes the grouping key, an acceptable change for input that is already malformed. ## Why not payload / metadata / tags Those are `text` columns fed by `JSON.stringify`, which escapes a NUL to a safe escape sequence, so they do not hit this failure on the normal JSON path. (A raw NUL in a `text` column throws a different code, `22021`, and is not what triggers this issue.) The observed failures are the `jsonb` `22P05` variant, which is only reachable via the two key fields. ## Evidence Red then green (containerTest, real Postgres): with the fix reverted, triggering through the real service with a NUL in `idempotencyKeyOptions.key` / `debounce.key` fails with the exact `22P05` signature; with the fix, the run is created and the stored key has the NUL removed. Full-stack e2e (isolated stack, real HTTP): `POST /api/v1/tasks/:taskId/trigger` with a NUL inside `idempotencyKeyOptions.key` (`"acme<NUL>inc"`) and, separately, `debounce.key` (`"grp<NUL>1"`): - both returned `HTTP 200` with a created run (previously `500`) - stored `idempotencyKeyOptions` = `{ "key": "acmeinc", "scope": "run" }` (7 chars, NUL removed) - stored `debounce.key` = `"grp1"` (4 chars, NUL removed) - both runs render in the dashboard Unit tests cover the helper (strip, no-op fast path, object-reference reuse, null/undefined pass-through). ## Rollout / rollback Server-only webapp change, no flag. Zero behaviour change for clean input; only affects inputs that previously 500'd. Rollback is a straight revert, no data migration. ## Known limitation A raw NUL in a plain-string idempotency key (not created via `idempotencyKeys.create()`) lands in a `text` column and throws `22021` instead. That variant is not addressed here because stripping it would change the dedup identity, so it warrants a separate decision. Not observed in practice. refs TRI-13030	2026-08-07 13:28:52 +01:00
claude[bot]	dc529414df	feat(webapp): add /_/* redirect route (#4523)	2026-08-07 13:21:07 +01:00
Chris Arderne	0a44b88b39	fix: security release 2026-07-21 (#4528)	2026-08-07 12:25:40 +01:00
Eric Allam	db67a856fe	perf(webapp,database): index the newest-task-version lookup (#4518) ... Implementing PlanetScale Insights improvement. ## Summary Validating a schedule (creating or updating one through the API or the dashboard, and deploying a project that declares schedules) looks up the newest version of a task by slug. That lookup reads *every* version of the task and sorts them to return one. A project gains a row per task on every deploy, so the work grows with the project's age: the oldest projects pay the most, and dev-mode redeploys make it worse. This was picked because it was the largest single consumer of database time on the schedules path, and the fix is a sort key with no index behind it. ## Fix `BackgroundWorkerTask` is indexed on `(projectId, slug)`, which serves the equality but not the `ORDER BY createdAt DESC`. Postgres seeks the index, then bitmap-scans and top-N sorts the whole group to produce a single row. Adding `createdAt` to the index lets it scan backward and stop at the first row. The same call site also selected all 21 columns, including five JSON blobs, to read one field (`triggerSource`), so it now selects that field alone. ## Benchmark Local Postgres 17, 997,000 seeded rows / 748 MB, group sizes chosen to match the distribution seen in production. | Group size | Before | After | | --- | --- | --- | | 15,000 versions of one task | 11.118 ms, 1,510 buffers, 15,000 rows scanned | 0.027 ms, 4 buffers, 1 row | | 2,000 versions of one task | 2.081 ms, 1,455 buffers, 2,000 rows scanned | 0.022 ms, 4 buffers, 1 row | ``` before: Limit -> Sort (top-N heapsort) -> Bitmap Heap Scan after: Limit -> Index Scan Backward using BackgroundWorkerTask_projectId_slug_createdAt_idx ``` An ascending index scanned backward is enough here, so no descending index is needed. ## Impact and risk Real-world gain lands between the two rows above and scales with how many deploys a project has accumulated. Projects with few deploys will see little change, since there is barely anything to sort. The new index costs noticeably more than the existing two-column one: 43 MB against 7.3 MB on the benchmark rig. Adding `createdAt` makes every key unique, which defeats btree deduplication, so this is a real disk and write cost rather than a rounding error. Writes to this table happen at deploy time, not on the run path, so the write amplification is acceptable. The existing `(projectId, slug)` index is now a redundant prefix and could be dropped, but this PR keeps it so index usage can be observed before removing it. Behavior is unchanged: same predicate, same ordering, same row returned. The narrowed select is the only code change, and the field it keeps is the only one the caller read. Deploy note: the migration is `20260806100000_add_background_worker_task_project_id_slug_created_at_index` and uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it can be pre-applied by hand before the deploy. docs-release-2026-08-07	2026-08-07 11:17:10 +01:00
Eric Allam	6c6e58e6ff	perf(webapp): batch declarative schedule cleanup queries (#4522) ... ## Summary `syncDeclarativeSchedules` runs on every background-worker creation (every deploy, and every file save during `trigger dev`). It issued one instance-delete per declarative schedule the current worker no longer declares, in a loop, and the overwhelming majority of those deletes matched zero rows. This collapses the loop into at most two set-based statements and skips the instance delete entirely when the current environment owns no instance of the schedule. ## Why so many, and mostly no-op The loop runs once per entry in `missingSchedules`, which starts as every DECLARATIVE schedule for the whole project across all its environments (the query filters only by `projectId`). A schedule leaves that set only when a declared task matches it by `taskIdentifier` **and** the schedule already has an instance in the current environment. That last clause is the amplifier. When a task's schedule has no instance in the current environment, the create branch inserts a brand-new `TaskSchedule` row with an instance for this environment rather than adding an instance to the existing row. So the same scheduled task, once it has run in dev and been deployed to prod, exists as two separate schedule rows: one carrying a dev instance, one carrying a prod instance. On a dev worker sync of that project: - the dev-instance row matches the declared task and is removed from the set - the prod-instance row has the same `taskIdentifier` but no dev instance, so it stays in the set and gets `deleteMany(taskScheduleId = prodRow, environmentId = dev)`, which matches zero rows So every declarative task that has been synced in another environment contributes one guaranteed no-op delete per sync, and the count scales with (declarative tasks x environments), plus any leftover rows from renamed or removed tasks. A project does not need to have dropped a schedule to generate these; it just needs the same declarative tasks present in more than one environment, which is the normal develop-in-dev, deploy-to-prod case. ## Fix The candidate schedules are already loaded with their instances, so the branch is decided in memory: - schedules with no instances (or only current-environment instances) are removed in a single `taskSchedule.deleteMany` - schedules that still have another environment's instance have only the current environment's instance detached, in a single `taskScheduleInstance.deleteMany`, and only when such an instance actually exists Behavior is unchanged (cascade delete still removes the instances of a deleted schedule); the difference is statement count. A zero-row delete writes no WAL and creates no dead tuples, so the removed work was pure query and commit overhead. Verified with a testcontainer test (red before, green after) counting the emitted deletes across the no-op, batched-detach, and schedule-delete cases, and end to end through `trigger dev`: three declarative schedules created, surviving a re-sync, then two removed in a single batched delete with the third preserved.	2026-08-07 10:27:39 +01:00
Matt Aitken	04f9c4e1a5	fix(webapp,run-engine,core): drop the hidden debounce ceiling, fail fast on an unusable maxDelay (#4521) ... Debouncing with a `delay` longer than an hour did nothing at all. The engine applied a server-side ceiling on how long a debounced run could be pushed back, measured from the run's `createdAt` and defaulting to one hour. A run is only pushed back while its new execution time stays inside that ceiling, so a `delay` at or above it could never push anything: the waiting run was released, the trigger started its own run, and the next trigger repeated it. A `delay: "12h"` produced one run per trigger, each correctly delayed by 12h, with no error raised and nothing on the run to show the debounce key had been ignored. The ceiling is now unset by default. A debounce key with no `maxDelay` keeps collapsing triggers for as long as they keep arriving, which is what the docs have always described. Self-hosters who want a bound can still set `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS`. That has a consequence worth stating plainly, so the docs now carry a warning for it: with no `maxDelay`, a continuously triggered key never executes. Set `maxDelay` when the work has to happen eventually. **Failing fast on an unusable `maxDelay`.** A caller who sets `maxDelay` no longer than their `delay` hits exactly the dead end described above, so that pair is now rejected at trigger time instead of silently behaving as if no debounce were set: ``` debounce.maxDelay (1h) must be longer than debounce.delay (12h). A debounced run is only pushed back while it stays inside maxDelay, so with these values every trigger would create its own run. ``` An unparseable `maxDelay` is rejected too, rather than quietly falling back to no bound at all, and so is a `delay` given as a date rather than a duration, which could never work because the value is re-applied on every push. The same check runs against a configured server ceiling, so a self-hosted deployment that sets `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS` gets the error rather than the silent failure this PR is about. With no `maxDelay` and no configured ceiling, which is the default, there is nothing to conflict with and nothing is rejected. The docs, the `TriggerOptions` JSDoc and the engine option all now state that the room available to push is the gap between `delay` and `maxDelay`. The run engine suite gains the case that motivated this: four triggers on one key with a 12h delay now collapse to a single run.	2026-08-07 07:55:35 +00:00
Matt Aitken	c084fa6e29	fix(sdk,react-hooks): forward debounce when batch triggering with an array (#4520) ... Passing `debounce` in the per-item options of a batch trigger did nothing when the items were an array. The option was accepted by the types and by the API, then dropped before the request went out, so every item created its own run instead of collapsing onto the debounce key. Four public entry points were affected: `task.batchTrigger`, `task.batchTriggerAndWait`, `tasks.batchTrigger`, and `tasks.batchTriggerAndWait`. The streaming (async iterable) forms of the same calls were already correct, as were `batch.trigger`, `batch.triggerAndWait`, `batch.triggerByTask`, and `batch.triggerByTaskAndWait`. `useTaskTrigger` in `@trigger.dev/react-hooks` had the same silent drop on the single-trigger path, so that is fixed here too. It also drops `machine`, `priority`, `region`, `idempotencyKeyTTL`, and `idempotencyKeyOptions`; those are left alone, since forwarding them is a behaviour change beyond this bug. Each batch item builder constructs its options field by field, which is why one of them could fall behind without anything catching it. TypeScript did not help: the literal is returned from a `.map` callback inside `Promise.all`, so excess-property checking never fired against the `BatchItemNDJSON[]` annotation, and the server's schema silently strips unknown keys. A misspelled option name therefore reproduced this bug with no compile error and no server error. Every builder now ends in `satisfies BatchItemNDJSON`, which does catch it: ``` error TS2561: Object literal may only specify known properties, but 'debounceTYPO' does not exist in type '{ ... debounce?: {...} | undefined; }'. Did you mean to write 'debounce'? ``` The new test drives all six public batch surfaces in both array and async-iterable form and asserts on the NDJSON that actually reaches the wire. Each item carries a distinct debounce key so the test catches a wrong item-to-option pairing, not just a wholesale drop. Fixes #3304	2026-08-06 16:54:24 +01:00
Matt Aitken	f8e1c910f7	docs(ai-chat): guide for migrating an AI SDK route handler to chat.agent (#4519) ... ## Summary Adds a docs page for developers who already have a working Vercel AI SDK chat app (`useChat` on the client, an `app/api/chat/route.ts` calling `streamText`) and want to move it to `chat.agent`. There was no page covering that path. `ai-chat/upgrade-guide` reads like it should be the one, but it covers moving prerelease `chat.agent` code to the Sessions release, which is a different reader. The page is structured around what stays, what goes, and what is new, because the reassuring part of this migration is how much is untouched: the `streamText` call, model config, tool definitions, `useChat`, and all message rendering carry over as-is. What gets deleted is the route handler, the persistence glue wired into it, and any resumable-stream setup. What is new is the agent task, two server actions, and `useTriggerChatTransport`. Covers moving tools onto the agent config so `toModelOutput` survives past turn one, where existing database persistence goes (`hydrateMessages` plus the turn hooks), a short section on what durability you get once you are across, a note that Hono/SvelteKit/Express follow the same shape, and a gotchas list built from the mistakes this specific migration produces. ## Head Start The one thing this migration makes worse is the opening response of a new chat. The route handler answered out of a warm process; the agent run has to be dequeued and booted first. That is the complaint the page has to answer head on, so Head Start gets a full section rather than a closing aside, plus a callout up top next to the "what changes" table so nobody plans the migration without knowing it exists. The section walks the four steps: splitting tool schemas away from tool executes (the bundle-isolation constraint the whole feature rests on), building the handler, mounting it back at `app/api/chat/route.ts` with the original auth check wrapped around it, and the transport option. Both server actions stay, because Head Start only owns the first turn. Three gotchas go with it: a slow first turn without Head Start, Head Start on but the route bundle still heavy, and the route timing out because the handler holds the SSE response open for the whole turn rather than just step 1. The coding-agent prompt names Head Start as explicitly out of scope, so an agent handed the migration does not attempt the tool split unprompted. Also fixes the `chat.headStart` example on `ai-chat/fast-starts`, which set `stopWhen: stepCountIs(15)` after the spread. `toStreamTextOptions()` pins `stopWhen` to `stepCountIs(1)`, so overriding it makes the warm handler run steps the agent is supposed to own (and `stepCountIs` was never imported in that snippet either). ## Migration prompt The page also ships a copy-pasteable prompt for handing the migration to a coding agent. It tells the agent to run `npx trigger.dev@latest skills` first, so it picks up guidance version-pinned to the SDK actually installed in the project, then read `quick-start.md`, `frontend.md`, and `reference.md` (with `llms.txt` as the index) before editing anything. The instructions are explicit about preserving the existing model, prompt, and tool schemas rather than rewriting them. Registered in `docs.json` under Agents, directly after Quick Start, so it is picked up by the generated `llms.txt` and the per-page `.md` variants.	2026-08-06 17:37:23 +02:00
Chris Arderne	088f68b373	feat(webapp): share rate limit bucket across additional API keys per environment (#4508) ... ## What Rate-limit the API by **environment** rather than per API key. Previously the limiter keyed its bucket on the hash of the full `Authorization` header — one bucket per key. With additional environment API keys (`tr_*_sk_*`), an environment can mint many keys and each got its own full bucket, so more keys = higher effective rate limit. This collapses all of an environment's keys onto a single shared per-environment bucket, so the ceiling is exactly the configured limit regardless of key mix. ## How - `authorizationRateLimitMiddleware` now lets the override return `{ config?, identifier? }`. `identifier`, when present, is the rate limit bucket key; otherwise it falls back to the hashed `Authorization` header (unchanged legacy behavior, still used by `engineRateLimiter` and any unauthenticated fallthrough). - `apiRateLimiter`'s override resolves the environment id and uses it as the identifier: - **Additional keys** (`isAdditionalApiKey`) resolve via a new `resolveAdditionalApiKeyRateLimitScope()` — a **scope-agnostic** keyHash → (environmentId, org limiter config) lookup. It is deliberately permissive (restricted keys resolve too) because it's used **only for bucketing, never as an auth decision** — request auth still goes through the RBAC bearer controller, which enforces scopes. Revoked/expired keys are excluded so they can't hold a bucket warm. - **Root/legacy keys** reuse the environment already resolved by `authenticateAuthorizationHeader` and key on `environment.id` too. - The identifier is always the stable environment id, never the secret key (which can rotate and would split the bucket). - The whole override result is cached per key by the existing SWR cache, so **no extra per-request lookup and no separate Redis mapping** is added. ## Behavior notes - Root + additional keys of the same environment now share one bucket (ceiling = configured limit, not a multiple of it). Restricted additional keys are included — they were the biggest gap, since they authenticate via the RBAC controller and previously fell back to per-key buckets. - **Public JWTs** keep their existing fixed-window, per-token bucketing. - One-time bucket reset on deploy (bucket keys change); harmless. ## Tests - New: two tokens resolving to the same identifier share one bucket. - New: with no identifier, bucketing stays per-key (legacy behavior preserved). - Updated existing override tests to the new `{ config }` return shape. Base: `feat/multi-keys-surface`. Closes TRI-12888.	2026-08-06 16:05:27 +01:00
Chris Arderne	9409ddf9bc	feat(webapp): add multiple environment API key management (#4390) ... ## Summary Projects can create, inspect, expire, and revoke multiple API keys for each environment. Plaintext values are shown only at creation; stored credentials are hashed and the API keys page displays only an obfuscated suffix afterward. Self-hosted installations support full-access additional keys by default. Authorization extensions can provide additional access presets and optional task selection. Additional keys can also mint scoped public access tokens through the Trigger.dev API without receiving the environment signing key. ## Feature notes - Only admin+ can create API keys (Developer can make in Development branch). - JWT self-signing will be a server call when used with new `_ak_` keys. - JWTs with long expiry can keep working even with api key deleted (gets priveleges from api key, signed with root key) - Unfiltered session listings intentionally preserve the existing broad task-read behavior. Filtered listings enforce task-level scopes for every requested task. - Buffered runs without a task identifier are not safely authorizable, so cancel/replay requests fail closed rather than resolving an unscoped run. - Batch and waitpoint endpoints intentionally return server-minted, narrowly scoped public tokens to all callers. These tokens have bounded lifetimes and may remain valid until expiry after API-key revocation. ## Deployment notes Deploy the management UI and public-token endpoint with new key creation disabled. Enable creation for selected organizations after the authentication path and released SDK have been verified, then expand availability gradually. Revoking an API key prevents new bearer requests and new token minting. Public tokens already minted by that key remain valid until their own expiration because they are signed by the environment signing key. ## TODO - [x] Add "Created by" to the key table - [x] Document that streamed batch ingestion is non-atomic and may partially accept items before a validation or authorization error. ## Follow-ups - [x] Add an organization-level feature flag for the API key management UI and creation action. - [x] Document rollout ordering: enable additional-key lookup before enabling issuance. - [x] Add a system-wide gate that can stop new key issuance without disabling authentication for existing keys. - [x] Replace the generic SDK compatibility warning with the first published compatible version. Old SDK will mint an unusable token if given an `_ak_` key. - [x] Add public documentation covering creation, storage, expiration, revocation, SDK compatibility, and public-token lifetime behavior. - [x] Add observability for key creation, revocation, policy preparation failures, and public-token mint failures. - [ ] Exercise create, copy-once display, authenticate, mint, expire, and revoke flows end to end before broad enablement.	2026-08-06 15:27:10 +01:00
Katia Bulatova	337dda1e97	feat(webapp): name of the page in tab titles (#4517) ... Adds a shared `pageMeta()` helper and 74 route declarations, so a title reads `run_abc | Runs | Trigger.dev` — the specific thing first, then the page. Org pages also carry the organization: `Team | Acme | Trigger.dev`. Inside a project no scope is added, because the dashboard switches projects in every tab at once. Page names are unchanged; what's new is that a page says which one it is at all. Three wording changes on purpose: the queue page now names the queue, the model page names the model, and entity pages carry their section.	2026-08-06 10:34:30 +02:00
Wes Mason	66940c0384	fix(observability-map): narrow the required check and the report bot's comment lookup (#4507) ... ## Findings addressed - **Report bot edited the wrong comment.** The comment-lookup step matched on the marker body text with no author predicate, so it would silently PATCH a human's comment that happened to quote the marker (GitHub gates comment editing on write access, not authorship, so it never 403'd). Now constrained to `.user.login == "github-actions[bot]"`, the same identity `helm-prerelease.yml` already pins. - **A required check asserted facts about the whole webapp namespace.** `webappSymbols.test.ts` asserted that nobody anywhere in `apps/webapp` (walking locals, params, object keys) declares names like `createJWT`/`updateEnvVars`, so an unrelated PR naming a local variable failed a required check with a message pointing at nothing. Those negative self-tests move onto a package-owned fixture tree; the positive resolution assertions stay required (their absence rotted the tool before) but now name the list to edit. - **The suite ran twice on shared paths.** `obsmap` and `internal` path filters shared four generic paths (`package.json`, both lockfiles, `pr_checks.yml`), so any lockfile bump ran the observability-map suite in both jobs. Dropped from `obsmap` (where `internal` already covers them). The test that should have caught it only checked the package's own source path; it now asserts the two filters' path intersection is empty. - **PR-comment footer** reworded: it said the report gates nothing, which is true of the report but misled now that the tool's test suite does gate webapp PRs. Names both failure directions and where to read the rules. - **Nightly corpus** comment corrected (stale entry count; the failure-notification gap is documented, not silently implied). ## Review Two adversarial reviewers ran over the diff; both findings were verified and fixed: a hollow fixture assertion (a shared name satisfied either walker branch — now one name per declaration form, revert-confirmed) and a filter-intersection test that could be fooled by apostrophes in comment prose (now strips comment lines first). Full package suite green (877 passed), typecheck and format clean.	2026-08-05 22:36:19 +01:00
Eric Allam	b20806247f	fix(run-store): stop run-create failing on a brief write stall (#4514) ... ## Summary On the run-ops store, creating a run could intermittently fail with a "Transaction already closed" error, and the run would never be created. Single-write run creates no longer run inside an interactive transaction, so a brief database write stall can't blow the transaction budget and drop the run. ## Fix The dedicated run-ops `createRun` / `createFailedRun` wrapped a single nested `taskRun.create` in an interactive `$transaction`. Its default 5s budget is wall-clock from `BEGIN`, so when a write briefly stalls the transaction expires before the create completes and throws, even though the statement itself is fast at the database. A single-write create does not need an interactive transaction: Prisma's implicit nested create is already atomic and holds no app-side budget, so it now runs directly. Only the `triggerAndWait` path (run plus its associated waitpoint, two writes that must commit together) keeps an interactive transaction, now with headroom over the default. Verified with a red/green test against the real split topology (reproduces the exact expiry on the unchanged code, green after) and an end-to-end run created and completed through the dedicated store.	2026-08-05 17:35:46 +01:00
Eric Allam	58bf4e2833	feat(webapp): per-client database pool and connect timeout overrides (#4515) ... ## Summary Follow-on to #4513. The database connect timeout is now honored, but a single global value has to serve three separate databases at once (control-plane, legacy run-ops, and run-ops). This adds optional per-client overrides for the Prisma pool and connect timeouts, one pair for the writer and one for the read replica of each of the three databases, each falling back to the shared `DATABASE_POOL_TIMEOUT` / `DATABASE_CONNECTION_TIMEOUT` when unset. That lets one database's clients run a fail-fast connect timeout (with a bounded pool wait) while another keeps more headroom, without a single knob forcing the same tradeoff everywhere. No behavior change until an override is set. It also tags each client's queries with its specific datasource (`control-plane` / `legacy-run-ops` / `run-ops`, writer or replica) via the `db.datasource` span attribute, so telemetry can attribute connection behavior to a specific database instead of just writer-vs-replica.	2026-08-05 17:26:41 +01:00
Chris Arderne	1a16d61a37	fix(build): support decorator metadata with TypeScript 7 (#4505) ... ## Summary Allow projects using TypeScript 7 to enable `emitDecoratorMetadata()` without adding the TypeScript 6 compiler to every Trigger.dev CLI installation. Addresses #4500. ## Fix The extension now resolves TypeScript from the project and feature-detects the legacy compiler API. TypeScript 5 and 6 continue using the project's compiler, while TypeScript 7 projects can install Microsoft's optional `@typescript/typescript6` compatibility package alongside TypeScript 7. When no compatible compiler API is available, the build reports an actionable installation error. The extension documentation includes setup commands for npm, pnpm, and Bun. Verified with TypeScript 5, TypeScript 6, TypeScript 7 with and without the compatibility package, emitted decorator metadata, packed ESM and CommonJS consumers, package export checks, and typechecking.	2026-08-05 16:33:32 +01:00
Eric Allam	771937adf5	fix(webapp): clamp run priority so a large value can't fail run creation (#4512) ... ## Summary Triggering a run with a very large `priority` could fail run creation outright with an opaque database error. `priority` is multiplied by 1000 and stored in a 32-bit integer column, with nothing bounding it, so a big enough value overflowed the column and the create failed. The trigger now caps the value to the highest supported priority instead of erroring, so the run is still created. ## Fix `priorityMs` (the stored `priority * 1000`) now goes through a `clampPriorityMs` helper before the write. It rounds to a whole number and clamps into the column range at both ends, so only a valid integer ever reaches the column and an out-of-range priority caps rather than failing. Single and batch triggers share the write path, so both are covered.	2026-08-05 16:28:22 +01:00
Eric Allam	3039bc14d6	fix(webapp): honor the configured database connect timeout (#4513) ... ## Summary Every Prisma client built its connection URL with a `connection_timeout` query param, but the Postgres connector's parameter is `connect_timeout`. The misspelled param is silently ignored, so all clients fell back to Prisma's 5s default instead of the configured timeout. When establishing a new connection briefly took longer than 5s (for example during connection spikes), it failed with `Can't reach database server` even though the database was healthy. ## Fix All four client builders now construct their connection URL through one shared helper (`buildPrismaConnectionUrl`) that sets `connect_timeout`, so the configured value actually applies, and the parameter name lives in exactly one place. Covered by a unit test.	2026-08-05 15:52:12 +01:00
Chris Arderne	85f5b37c68	chore: upgrade to TypeScript 7 (#4318) ... ## Summary Upgrade the monorepo to TypeScript 7.0.2 and update package build tooling for compatibility with the native compiler. ## Design Package builds now use `tshy` 4, while the packages still using `tsup` move to `tsdown`. The few scripts that depend on the legacy TypeScript compiler API use an explicit TypeScript 6 alias; declaration portability coverage invokes the TypeScript 7 CLI directly. Turbo is updated so workspace tasks can read the regenerated pnpm lockfile. --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-05 15:49:44 +01:00
nicktrn	c01a4f18f4	feat(supervisor): cancel a resumed run's in-flight checkpoint (#4502) ... A run controller must call the continue route to resume, so the supervisor already knows synchronously that any checkpoint still running for that run is pointless. It only acted on that for the compute path. The continue route now cancels it for the Kubernetes path too, matching what completion already does since #4493. Called after the reply so the runner is never delayed, and skipped when there is no checkpoint client or when the compute path owns the run. The request is bounded by a 5s timeout so a hung call cannot leave the handler pending. `checkpoint_cancel_requests_total{result}` records the outcome, using the same label names as the delete path where they overlap: `sent`, `no_client`, `not_applicable`, `http_error`. No changeset: `CheckpointClient` is a server-only internal API, same as #4493. refs TRI-12915	2026-08-05 12:01:14 +01:00
Wes Mason	ca9a74e84a	feat(observability-map): static observability scorer for webapp route entry points (#4455) ... A static observability scorer for the webapp's route entry points, Lighthouse-style. The idea comes from evlog's `map` command, but that tool has no Remix adapter and checks for its own logging API, so the idea is ported rather than the tool. It scans all 427 loader/action entry points in `apps/webapp/app/routes` with the TypeScript compiler API and scores each against five checks: error-classification, auth-boundary, auth-scope, request-context and audit-trail. Current output on the real tree is **19/100** over 412 measured entry points. ``` cd internal-packages/observability-map pnpm exec tsx src/cli.ts # terminal report pnpm exec tsx src/cli.ts --json # machine output pnpm exec tsx src/cli.ts api/v1/token # one entry, per-check detail ``` The two findings at the top of the fix list are real: `/auth/sso` and `/api/v1/authorization-code` mint or exchange credentials unauthenticated, and `/_app/orgs/:organizationSlug/settings/team` resolves its org from a URL slug and gates each mutating branch on an RBAC check alone, which per `apps/webapp/CLAUDE.md` is not the tenant floor on self-hosted. Decisions worth knowing, all with the reasoning in the README: - The score started at 83 during development and fell to 19. Every drop was a perverse incentive being removed, not a regression: routes were being paid for having no error handling, two checks were reading the same fact, suppressing a failure raised the score, and a no-op `catch (e) { throw e }` was worth 50 points a route. - **A mutation corpus is the tool's main defence.** 44 entries apply semantics-preserving edits to a copy of the real route tree and assert the score cannot rise, per route as well as globally, because a mean can hide one route going up by taking another down. One entry runs as a live expected failure: `try { String(0); }` with a deciding catch is a known open hole worth 19 to 44, and it is disclosed rather than quietly excluded. - `audit-trail` and `request-context` are reported as headline figures rather than one finding repeated hundreds of times. Both still count in full where they should. - A cohort change moves the number without anything in the codebase getting better. Widening the sensitive cohort from 26 to 67 took the global from 15 to 19 with no webapp change at all, so the report prints per-check applicability and what the global would be without each one. CI: a report-only job posts a sticky comment when a PR moves the report, and says nothing when it does not. The package's own tests gate through `pr_checks.yml`. The diff-scoped merge gate is still deferred until the report has been used in anger. 524 tests plus the corpus. No runtime or dependency changes to anything that ships. <!-- GitButler Footer Boundary Top --> --- This is **part 1 of 4 in a stack** made with GitButler: - <kbd>&nbsp;4&nbsp;</kbd> #4485 - <kbd>&nbsp;3&nbsp;</kbd> #4484 - <kbd>&nbsp;2&nbsp;</kbd> #4483 - <kbd>&nbsp;1&nbsp;</kbd> #4455 👈 <!-- GitButler Footer Boundary Bottom -->	2026-08-04 15:33:32 +01:00
nicktrn	4f69c43e6b	feat(supervisor): reclaim a run's checkpoint storage when it finishes (#4493) ... When a run reaches a terminal state, ask the checkpoint service to reclaim the storage its checkpoints occupied. Storage for finished runs is not otherwise reclaimed, so nothing frees it today. **Off by default** behind `DELETE_CHECKPOINTS_ON_COMPLETION`, and the service-side handler ships separately, so merging this changes no behaviour. ## Where the tenancy comes from Addressing a run's checkpoints needs org, project, environment, deployment version and run id. All five are already in hand at `attempt.complete`, and three are **signed** by the deployment token: | Value | Source | Trust | | -- | -- | -- | | org | claim `org_id` | signed | | environment | claim `environment_id` | signed | | deployment version | claim `deployment_version` | signed | | project ref | `x-trigger-workload-project-ref` header | runner-supplied | | run | route param | runner-supplied | `authorizeWorkloadRequest` previously returned only `environment_id`, and only in enforce mode, so it now also returns the verified `claims`. That difference is deliberate and documented on the method: claims are used to address a run's **own** resources locally, never to scope the platform, which is why `environmentId` stays enforce-only. The two runner-supplied values are safe because the signed ones are outermost - a runner lying about either can only name something inside its own org and environment, and a project ref that doesn't pair with its signed environment matches nothing. The run id is read from `params.runFriendlyId`, the same value the platform just validated, rather than from the body or a header. Where both a claim and a header exist (`deployment_version`), the claim wins. ## Placement The call sits after `reply.json(...)`, so the runner sees no added latency - the same shape the suspend route already uses. The service enqueues and returns 202, so it is one fast local hop. Terminal means `RUN_FINISHED` **or `RUN_PENDING_CANCEL`** - a run cancelled mid-execution never restores, and skipping it would leave its storage behind. Retries are excluded deliberately: reclamation is per-run, so a retry is covered by the final completion. Also gated on `!snapshotService`, so it stays inert where checkpoints aren't the kind this reclaims. ## Observability `checkpoint_delete_requests_total{result}` counts `sent` **and every reason we decide not to send**: `disabled`, `not_terminal`, `no_claims`, `no_project_ref`, `http_error`. The negative labels are the point - without them, "no requests are happening" looks identical to the feature being switched off. `no_claims` is reachable even under enforcement, since enforce only rejects a *present-but-invalid* token; an absent or legacy id still passes with no claims attached. ## Notes for review - **No changeset**: `CheckpointClient` is `core/v3/serverOnly`, an internal service-to-service API rather than customer-facing surface. - **No `.server-changes/` note**: there is nothing a dashboard user would notice here. Happy to add one if you disagree. - `pnpm run typecheck` can't complete in my checkout - `@trigger.dev/database` fails to build on a missing `tsc` in the pnpm store, unrelated to this diff. Verified with `tsc --noEmit` against the supervisor project instead: **zero errors in `apps/supervisor/src`**. Worth noting it caught a real bug here - the completion response is wrapped, so the status is `data.result.attemptStatus`. refs TRI-12789	2026-08-04 15:01:29 +01:00
Matt Aitken	e8398d13be	chore: vouch Rohan170603 (#4501) ... Adds `Rohan170603` to the list of vouched outside contributors so their PRs aren't auto-closed by the vouch check. Closes #4498	2026-08-04 13:22:24 +00:00
Katia Bulatova	fbd6df33b4	feat(webapp): Themes + contrast settings update (#4206) ... Adds System Preferences, Dark and Light themes, gated by the `hasThemeSwitcher` feature flag (off by default — dark stays the default theme for everyone). Old theme is now "Classic"and set as default. "System preferences" theme has both Light and Dark modes and uses your laptop settings to use a correct one. It has less color accents (specifically less colored text), and they are the same for both modes, only grayscale values change between them. And Light/Dark themes can be used separately. New Contrast setting is available for System Preferences, Dark and Light themes - it changes the contrast for the whole app. All new visual Settings live in Account.	2026-08-03 19:29:33 +02:00
Eric Allam	57254b57fb	fix(webapp): make prop-types a production dependency (#4492) ... ## Summary The webapp's server bundle imports `prop-types` directly, but the package was declared only as a `devDependency`. A production install therefore leaves it out and the built server fails to boot: ``` Failed to start server: Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'prop-types' imported from /triggerdotdev/apps/webapp/build/server/assets/server-build-*.js ``` Moving it to `dependencies` is the whole change. ## Why the bundle imports it Nothing in the webapp's own code uses `prop-types` — there is no reference to it, or to `PropTypes`, anywhere under `apps/webapp/app`. It arrives through `recharts`, whose `react-smooth` dependency still declares `propTypes` on its components. That was invisible until recently. While `recharts` was resolved at runtime, its `prop-types` import was satisfied inside `recharts`' own dependency tree, which is production all the way down. #4486 added `recharts` and `victory-vendor` to `ssr.noExternal` to fix a hydration mismatch on every server-rendered chart; that inlines `react-smooth` into the server bundle, which moves its `prop-types` import into the webapp's own resolution scope — where the package was not available in production. So the bundling change was correct about *which* d3-shape build both sides resolve, and wrong about what the production runtime would be able to find. ## Verification `docker/Dockerfile` builds the runtime dependencies with `pnpm install --prod` against a `turbo prune --scope=webapp --docker` output, so I reproduced exactly that: pruned the workspace, installed with `--prod`, and imported `prop-types` from `apps/webapp`. | | result | | -- | -- | | `main` as it stands (devDependency only) | `FAILS: ERR_MODULE_NOT_FOUND` | | with this change | `prop-types resolves OK` | It resolves both as a CommonJS `require` and as an ESM `import`, which is the form the bundle uses. I also checked this is not one symptom of a wider problem: of the 169 bare specifier roots the server bundle imports, `prop-types` is the **only** one that is a devDependency and not a production dependency. The rest are node builtins or production dependencies. The hydration fix from #4486 is unaffected — the rebuilt bundle still carries the rounding d3-path build. ## Notes `prop-types` is inert in production (its entry point swaps in `factoryWithThrowingShims`), so this adds a 124 KB package that does no work at runtime. It has to be resolvable regardless, because the import is real. An alternative would be adding `prop-types` to `ssr.noExternal` so it is inlined and needs no runtime resolution. That keeps the dependency list honest about the fact that the webapp itself does not use it, at the cost of bundling a CommonJS package into the ESM server output. This route is the smaller, better-understood change. Worth following up separately: a check that every bare import in the server bundle resolves from a production install would have caught this before it landed. Local development installs every devDependency, so the gap is invisible when the built server is run from a working tree.	2026-08-03 16:09:03 +00:00
nicktrn	3fba04573d	fix(supervisor): hold the last backpressure verdict when a read fails (#4444) ... The dequeue brake released the moment its signal became unreadable. `refresh()` caught any error from `source.read()` and set the verdict to `null`, which `computeEngaged()` treats as not-engaged — so a few failed reads dropped an engaged brake, silently, with no log and no metric. That handling was symmetric while the risk is not. A source that has stopped answering correlates with the pressure the brake exists for, so releasing on read failure gives up protection at exactly the wrong moment; holding too long only costs throughput. Now a failed read keeps the last verdict instead of discarding it. The verdict then ages normally, so the existing `maxVerdictAgeMs` check becomes the grace window and still bounds how long a dead source can hold the brake — a permanently unreachable source releases it rather than pinning dequeuing forever. Because `computeEngaged()` only consults staleness for an *engaged* verdict, a released one is unaffected and stays released. The default grace moves from 15s to 120s, comparable to how long the brake normally stays engaged. One guard worth calling out: holding is only safe when something bounds it, so when `maxVerdictAgeMs` is unset the previous discard behaviour is kept. Otherwise an unbounded hold could pin the brake indefinitely. Read failures were previously invisible — the catch block neither logged nor counted. Adds a `read_failures_total` counter, plus an error log on the transition into failure rather than once per tick, since the refresh loop runs every second. The post-release ramp needs no change: it anchors off the engaged-to-released transition, so a grace-window release still ramps back up instead of snapping to full rate, which is what you want after a blind period. Tests cover holding while reads fail, releasing past the max age, and the existing unbounded-config paths are unchanged.	2026-08-03 18:06:23 +02:00
nicktrn	8f9db53350	feat(supervisor): configurable tolerations for run pods (#4491) ... ## Summary Self-hosted Kubernetes deployments can now add tolerations to run pods, so runs can schedule onto tainted nodes. Previously the only way to do this was to patch the supervisor. `KUBERNETES_RUNNER_TOLERATIONS` takes a comma separated list of `key=value:effect`, or `key:effect` to tolerate any value. It applies to every run pod, and for runs from a schedule tree it merges with the existing `KUBERNETES_SCHEDULED_RUN_TOLERATIONS`. Left unset, nothing changes: no tolerations are added and the pod spec leaves the field off entirely. The Helm chart takes it as a list: ```yaml supervisor: config: kubernetes: runnerTolerations: - dedicated=runs:NoSchedule - spot:NoExecute ``` ## Naming The issue proposed `KUBERNETES_WORKER_TOLERATIONS`. This ships as `KUBERNETES_RUNNER_TOLERATIONS` instead, because `RUNNER_*` is already the prefix for run pod settings (`RUNNER_HEARTBEAT_INTERVAL_SECONDS`, `RUNNER_ADDITIONAL_ENV_VARS`, and `DOCKER_RUNNER_NETWORKS` for the Docker equivalent), whereas "worker" refers to the supervisor itself throughout this app. ## Validation Keys and values are checked against the Kubernetes naming rules when the supervisor starts, so `dedicated=prod runs:NoSchedule` fails immediately with a message naming the offending entry. Without that check a bad value is accepted at startup and then rejected by the API server on every pod create, which stops all runs with the cause buried in an API error. `KUBERNETES_WORKER_NODETYPE_LABEL` is trimmed and validated for the same reason: surrounding whitespace is not valid in a label value, so a padded value fails every pod create today. ## Node selector off switch `KUBERNETES_WORKER_NODETYPE_LABEL` accepts an empty string to skip the node selector entirely, so runs schedule on any node. This already worked and the Helm chart has always shipped it empty, but it was not documented. It is now. The issue also asked for general node affinity configuration. That is not included: the node selector off switch plus tolerations covers the reported problem, and a free form affinity setting is a much larger config surface to commit to. Fixes #4458	2026-08-03 15:40:41 +00:00
Eric Allam	9d57aff542	fix(webapp): make the Queues hero charts environment-wide (#4486) ... ## Summary The four charts above the queues table aggregated over **at most the 25 queues on the current page**. They reused the loader's already-paginated queue array as a ClickHouse `queue IN (...)` filter, so paging or re-sorting changed the values, and a name search matching nothing blanked the whole chart row. The stat tiles above them were already environment-wide, so the two rows disagreed. They now read `env_metrics`, the environment-level rollup that already exists for exactly this (the built-in Queues dashboard and the health report read it). That is both correct and queue-count-independent: no `GROUP BY queue` across an entire environment, and no client-side summing. Note this is not only a paging artifact: page 1 under-reported too. On the seeded environment below, page 1 read 82% saturation against a true 87%, because the environment's running total is not the sum of one page of per-queue gauges. Three related fixes ride along. **Scheduling delay and throttling sawed to zero.** Both are event-driven, so at the 10-second bucket a short range picks, most buckets hold no samples at all and were drawn as `0ms`. Measured over a 1-hour window: **232 of 349 buckets had no scheduling-delay samples**. A bucket where nothing started is not a bucket where nothing waited, so the line was both ugly and wrong. TRQL grows a `minBucketSeconds` floor, plumbed through the metric resource route, and the hero tiles set 60s. Buckets that still have no samples render as a gap instead of a dive to zero. **The floor must not feed a width-dependent headline.** Two of the four headlines are not peaks, so widening the plotted buckets moved them: - **Throttled** is a share of buckets that saw any throttling, so a single brief throttle came to mark a whole minute instead of ten seconds: the same seeded events read 17% at 10s and 85% at 60s. - **Scheduling delay p95** is a percentile, and merging quantile states over a wider bucket yields a p95 between the sub-buckets' own. Two 240s samples among twenty in one 10-second sub-bucket give a worst-of-six p95 of 240,000ms against a merged 60-second p95 of 5,000ms — a 48x understatement of a headline whose tooltip claims it is the worst in the window. Both charts keep the floor, since a readable line was the point of it. Their headlines now come from a second query at the range's natural bucket width, via an optional `readout` on the tile, so each means what its tooltip says regardless of how the plotted buckets are sized. Saturation and backlog are genuinely width-invariant (a max of maxes is the same at any width), so they are unchanged and issue no extra query. Both caught by Devin in review; I had wrongly lumped p95 in with the peaks. **Charts reported a hydration mismatch on every render.** Recharts resolved victory-vendor's CJS entry on the server and its ESM entry in the browser. Those bundle different d3-shape builds, and the CJS one predates d3-path's digit rounding, so every server-rendered curve carried full-precision coordinates while the client rounded to 3 decimals: ``` Server: M0,3C0.9305555555555555,3,1.8611111111111112,3,... Client: M0,3C0.931,3,1.861,3,... ``` Bundling recharts for SSR makes both sides resolve the same ESM build. Verified: 45 of 45 server-rendered chart curves now match the client, and the page loads with an empty console. ## Verification An isolated stack with 40 seeded queues (20 heavily loaded, 20 idle) and 90 minutes of 10-second buckets written into `queue_metrics_raw_v1`, so the real materialized views built `queue_metrics_v1`, `env_metrics_v1` and the 5m rollup. Ground truth for the environment: 260 running against a limit of 300 (**87% saturation**), 800 queued. | | before | after | | -- | -- | -- | | Saturation, page 1 | 82% peak | **87% peak** | | Saturation, page 2 | 5% peak | **87% peak** | | Backlog / delay, page 2 | "No activity" | **800 peak / 59.5s** | | Name search matching nothing | all four charts blank | charts stay environment-wide | | Metric refetches on a page change | 4, each painting a skeleton | **0, no skeleton** | | Buckets drawn as 0ms with no samples | 232 of 349 | **0** | | Throttled readout | 17% | **17%**, unchanged by the wider buckets | | Worst-p95 readout source | plotted buckets | **natural width**, so a sub-minute spike is not averaged away | | Crosshair reach, hovering one detail-page chart | 2 of 4 others | **4 of 4** | | SSR chart curves mismatching the client | 45 | **0** | The bucket floor was measured across ranges: it widens 10s to 60s at 30m and 1h, and is correctly a no-op at 12h (300s) and 7d (3600s). One extra request per page load, for the throttled readout. The built-in Queues dashboard, which reads `env_metrics` independently, agrees at 86.7% and 260 of 300. `internal-packages/tsql` suite green (612 tests), including 5 new ones for the floor that fail without it. Webapp typecheck, oxfmt and oxlint clean. Spot-checked the Run metrics dashboard and the per-queue detail page for SSR regressions from bundling recharts: both render, console clean. The queue detail page carries the same event-driven series, so its scheduling delay, throttling and per-key mean delay take the same treatment. ## Screenshots <img width="2540" height="580" alt="after-page1-charts" src="https://github.com/user-attachments/assets/6cd23f9c-e7fd-4918-bcfa-b1d3340b16d1" /> ## Rollout Already behind the per-organization `queueMetricsUiEnabled` flag, so only gated orgs see any of it. Blast radius is chart values on one page plus the SSR bundling of recharts; rollback is a revert with no data migration. ## Stated limitations - `wait_ms_count` and the quantile state both only count `wait_ms > 0`, so "nothing started in this bucket" and "everything started instantly" are indistinguishable in storage. Both render as a gap. Distinguishing them needs a schema change, which is not in this PR. - The queue name search deliberately no longer narrows the charts. It only did so incidentally and incorrectly before (first 25 matches, and blanked on zero matches). Search-scoped charts would need the full unpaginated matching set and a server-side aggregate; worth its own ticket if we want it. - Bundling recharts for SSR grows the server bundle slightly. That is the cost of both sides resolving one d3-shape build. - The plotted delay line is a smoothed 60-second view, so a sub-minute spike above the one-minute warning threshold can fail to colour the line even though the headline reports it and colours itself. - Every chart inside one synced group shares the floor, because the hover crosshair is a reference line on a category x-axis and only draws where the hovered bucket exists in the other chart's own data. That costs the queue detail page's gauges some resolution (1 minute instead of 10 seconds) in exchange for the crosshair working across the row. Separately, while taking the screenshots I found a pre-existing rendering bug unrelated to this change: a **perfectly flat** saturation series draws no line at all (the readout still shows the right percentage), which looks like the threshold gradient's offset degenerating when the series min equals its max. It reproduces on `main`, so it is not a regression here and I have left it alone; filed as its own issue. Refs TRI-12784	2026-08-03 16:19:50 +01:00
Matt Aitken	75df940e4c	chore: vouch Leafgard (#4489) ... Adds `Leafgard` to the list of vouched outside contributors so their PRs aren't auto-closed by the vouch check. Closes #4487	2026-08-03 14:34:17 +00:00
Katia Bulatova	859f30e224	fix(webapp): report message catalogs survive the production bundle (#4488) ... GET /api/v1/reports/health threw `no catalog registered for report "health"` in production (fine in dev): the catalog registered itself as a side effect of a bare import, which the SSR build tree-shakes under `"sideEffects": false`. Verified on the built server bundle — main's is missing the catalog, this branch's carries it. Fix: catalogs are values on the report registry entries; the resolver reads them from there and the mutable register-at-import step is gone.	2026-08-03 16:12:03 +02:00