477 Commits

Author	SHA1	Message	Date
Chris Arderne	ed1bb72fb8	feat: implement cron window spread backend (#4566) ... - New DB fields on Schedule and ScheduleInstance - Use `queueTimestamp` for the "effectiveAt" delayed start time, propagate it to Clickhouse TaskRun table - Disable fastpath for delayed jobs - Add schedule timing logic, API endpoints with windows, persistence - Calculate phase for every schedule, only persist when window is non-null - Additional o11y for phased rollout	2026-08-12 12:24:32 +01:00
Matt Aitken	c2c6e5c705	fix(webapp): keep session runs off the legacy realtime streams backend (#4564) ... ## Summary Runs created for a Session were triggered without a realtime streams version, so they fell through to the `realtimeStreamsVersion` column default of `v1`. A Session's own `.in` / `.out` channels are always `v2`, so any run-scoped `streams.append()` or `streams.pipe()` call made inside a session run wrote to a different backend than the session it belongs to, and stayed there for the life of the run. The API trigger routes were never affected. They call `determineRealtimeStreamsVersion` with the client's `x-trigger-realtime-streams-version` header and always pass an explicit value, so a current SDK asking for v2 gets it. Only the internal callers that build trigger options by hand were leaning on the column default, which no env var can influence because that path never calls the resolver at all. ## The version resolver Fixing the call site exposed a second problem in `determineRealtimeStreamsVersion`. Its two paths disagreed: an explicit `v2` was checked against the S2 configuration first, but when the caller expressed no preference it returned `REALTIME_STREAMS_DEFAULT_VERSION` verbatim with no check. A deployment that set the default to `v2` without configuring S2 therefore stamped runs `v2`, nothing failed at trigger time, and every later read or write against those runs' streams threw `Realtime streams v2 is required for this run but S2 configuration is missing` for the life of the run. Both paths now resolve through one pure function that takes its configuration rather than reading `env`: ```ts const requested = streamVersion ?? config.defaultVersion; if (requested !== "v2") return "v1"; const hasCredentials = Boolean(config.accessToken) || config.skipAccessTokens; return hasCredentials && Boolean(config.basin) ? "v2" : "v1"; ``` ## The basin requirement `resolveStreamBasin` resolves run, session and organization basins ahead of the global setting, so a deployment that provisions a basin per organization can serve v2 with no global basin at all. Gating purely on the global setting would degrade every run there to `v1`. `determineRealtimeStreamsVersion` therefore takes an optional organization basin, and every caller that holds one passes it, including the session path: ```ts basin: organizationBasinName ?? env.REALTIME_STREAMS_S2_BASIN, ``` This is deliberately the resolved basin and not the `REALTIME_STREAMS_PER_ORG_BASINS_ENABLED` flag. The flag says the feature is on, not that a given organization has been provisioned, and provisioning happens out of band. Keying off the flag would stamp `v2` on runs for unprovisioned organizations, recreating the failure this removes. **This widens behaviour for explicit `v2` requests**, which previously required the global basin: a provisioned organization on a per-org deployment now resolves `v2` where it used to get `v1`. That is intentional, and it makes every path agree. ## Scope Only newly created runs change. A run already stamped `v1` keeps that version for its lifetime by design, since readers resolve the backend from the same column and its existing streams have to stay readable. Scheduled runs reach the same column default through `scheduleEngine.server.ts` and are deliberately left alone: that one is a policy question about `REALTIME_STREAMS_DEFAULT_VERSION` rather than an inconsistency inside a single feature. ## Verification A full-stack e2e boots the real webapp plus Postgres, Redis and s2-lite, creates a Session through the public API so the run comes from the real trigger path, appends records the way `streams.append()` does, and asserts three things at once: the version stamped on the run, that the payload is readable from S2, and that no key exists in Redis. It appends at a realistic record size so the route's body cap and S2's per-record cap are both exercised. Reverting the session-path change flips all three observations, so it fails against the old behaviour rather than passing vacuously. Unit tests cover the resolver matrix, including organization-basin-only and credential-only configurations; two of them fail against the previous resolver. Also verified by hand against a local stack: a real `chat.agent` session run writing 8 records of 250KB through `streams.append()` put 2,049,072 bytes into S2 with no Redis key, while the same agent with the session-path change removed put 2,102,360 bytes into Redis and nothing into S2.	2026-08-12 11:01:59 +01:00
Chris Arderne	7b390e5984	feat(cli,webapp): allow deploys with environment API keys (#4561)	2026-08-12 10:11:31 +01:00
Katia Bulatova	0b750d00dd	feat(webapp): dashboard agent — Watch (#4525) ... Watch is the agent noticing something later: you ask it to tell you when a condition holds, and it answers when it does — or when it can't any more. A watch is a **durable one-shot promise**. The condition is checked on a schedule by deterministic code (no LLM in the checks), the answer lands in the chat once, and then the watch is over. Ten kinds: three on a run, five on a queue, error recurrence, health recovery. ## Stack Stacked on **#4529** (UI), which is stacked on **#4418** (chat, reports, investigate). Merge those first. **#4516** (storybook gallery) sits on top of this branch. ## How to review [**GUIDEBOOK.md**](https://github.com/triggerdotdev/trigger.dev/blob/feat/dashboard-agent-flows-watch/internal-packages/dashboard-agent/GUIDEBOOK.md) on this branch is the behaviour reference — it states the conditions rather than the code, so you can predict what happens without running anything. "The ten watch kinds, and what makes each fire" and "Creating a watch" describe exactly this PR, and the tables there are the spec the code is written against. ## What's inside - **Ten watch kinds**, one deterministic check each (`dashboardAgentWatch*Checks.ts`), with the spec union in `dashboard-agent-contracts/src/watch.ts`. - **Scheduling** — each watch schedules its own next check; due watches of one `(environment, cadence)` group can be checked together in one batch pass, with a sweep as the backstop for expiry, redelivery and retention. - **Delivery** — the in-chat wake and card, an optional email alert (new `DASHBOARD_AGENT_WATCH` alert channel, so it shows on the project's Alerts page with one-click unsubscribe), and an optional investigation when the outcome needs attention. - **Submission ledger** — `watch_submissions`, keyed `(chat_id, client_request_id)`, so a retried card submission replays the recorded outcome instead of creating a second watch. - **Watch token** — a delayed-execution credential accepted only by the watch endpoints, re-checked against the user's live access on every tick. - **Unread work** — the panel polls for wakes that landed while it was closed, so a chat can go unread and light the launcher dot. ## Key decisions **A check result is a 4-way, and only two of them are verdicts.** `satisfied` / `terminal_unsatisfied` are answers; `pending` and `unavailable` are not. Any exception inside any check is caught in one place and becomes `unavailable` with an unverified observation — a check that failed is never evidence. **A completed window is an answer, and whether it is good or bad news is declared per kind, never inferred.** There is a table for that in the guidebook: `run_failed` completing its window is *good* news ("hasn't failed"), `backlog_drain` completing it is not. One rule overrides the table: a window that completed on an unverified observation is neutral and says only that the watch ended without a confirmed answer. **An unreadable source is never a negative answer** — and, because investigations only open on `attention`, it never starts one either. **Identity is `(chat, project, environment)` plus the condition,** enforced by a partial unique index over active rows (`watches_chat_active_identity_key`), not by the read-then-insert check. Cadence, window, note and `ticks` are deliberately not part of it. Two different chats may watch the same thing — a watch is a promise to a chat. **The server resolves the target's name, whatever the model calls it.** The model can't tell a task queue (`task/<id>`) from a custom queue, so both spellings are tried and the stored one wins — and the rewrite happens **before** identity and before the row is written, so the identity, the checks, the link and the wording all see one spelling. **Freshness fences.** Depth falls back from the live counter to the newest 60 s ClickHouse bucket, which only counts as current within 60 s of now. A non-current reading at or below the *quiet line* is refused as `unavailable` rather than believed, so a stale empty bucket is never read as "drained". The stall streak is the one piece of carried state: it lives in the previous check's facts and *freezes* on an unreadable reading rather than breaking. **Chain reliability.** There is no shared cron — each watch (or batch group) schedules its own next tick, so the failure mode to review is the chain dying. A failed batch check is caught, the next tick is scheduled anyway and the run resolves rather than failing, so the chain survives a check that couldn't run; the sweep re-arms groups and finalizes anything still active past its deadline, even when delivery isn't configured. Wake redelivery is id-deduped rather than conditional, because the sweep can't know whether the user was already told. Access is re-authorized on **every** check against the primary — replica lag would extend access the user has already lost. **Wording lives in one place.** `watch-wording.ts` is read by the card, banner, toast, email and the agent's own narration, and the numbers come from the frozen observation rather than a fresh read, so a retry produces the same sentence. Replay reproduces the **recorded** decision instead of deciding again — the transcript is append-once, so a second decision would contradict it forever. **Cancellation is the ending without an answer** — no resolution, no wake. One exception, decided during testing: a watch the *user* cancelled leaves a single neutral transcript line ("Stopped watching …"), keyed off the watch id so a retry can't repeat it. The other four reasons stay silent. **Email is opt-in and only a fired watch emails.** An expiry is narrated in the chat and nowhere else. Both gates (agent access, a configured email transport) are checked at subscribe time *and* again at delivery, and the subscription outcome is frozen on the ledger row so a retry replays it. Neither gate is a plan check. **One watch offer per turn.** The prompt and the renderer guard this independently — if the turn already proposed a watch card, the action button is dropped, because the card is the better affordance. Two eval cases pin the prompt side: exactly one offer with the line last and the button after it, and zero offers when the rendered card already carries one — deterministic assertions, over a real-model run. ## Testing Unit tests (vitest, testcontainers, no mocks) under `apps/webapp/test/dashboardAgentWatch*.test.ts` and `internal-packages/dashboard-agent/src/watch-*.test.ts` cover the invariants above: the 4-way check results and the freshness fences, identity/dedup and the submission ledger, queue-name resolution, the batch chain surviving a failed check, sweep boundaries and alert-once, tenancy and the watch token's scope, and the wording snapshot. The load-bearing ones were verified by control-breaking the guard first and checking the test goes red. Live-tested end to end against a local stack, following the guidebook: all ten watch kinds firing and expiring, cancellation, the email pair (a fired watch mails, an expired one does not), and watch recovery from a health report.	2026-08-12 09:51:40 +02:00
Katia Bulatova	9a3bee0288	feat(webapp): dashboard agent — UI (#4529) ... Stacked on #4418. Merge that first. The UI slice of the dashboard agent: the side panel, the chat transport wiring, message and card rendering, suggested prompts, and chat history. #4418 works without this — the system is simply invisible. The diff is mostly components, so the notes below cover only the three decisions you can't read off the markup. Behavior and a hands-on walkthrough live in GUIDEBOOK.md, which lands with #4525. ## Decisions worth knowing - **Action rows always render at the end of a turn.** The model's emission order isn't trusted for layout, so action blocks are split out of the stream and appended last. Display only — `answered` stays keyed on the emission index. - **The last-chat memory is org-true.** It's keyed by the chat's own organization, and a foreign or deleted chat comes back as a 404 the client treats as gone, rather than an empty chat it keeps around. - **A dead stream self-heals from the settled transcript.** Terminal records are written to the chat row after the client's stream closes, so the panel re-reads it. The poll gate is any unfinished turn — a dangling tool part, not just an open investigation. ## Notes - Gated by `canAccessDashboardAgent`; no behavior change with the flag off. - Page marks: `handle.agentPageContext` on 47 routes, ~20 lines each. - Entry points: Ask Trigger button, ⌘J, Help & Feedback. The old ⌘I and `?aiHelp=` links keep working. ## Screenshots <img width="1440" height="788" alt="Screenshot 2026-08-07 at 15 14 29" src="https://github.com/user-attachments/assets/f4e89e8d-13ed-4be3-a88d-d5cca3ece0fa" />	2026-08-12 08:38:59 +02:00
Katia Bulatova	4569657923	feat(webapp): dashboard agent — chat, reports, investigate (#4418) ... ## What & why This is the system behind the Dashboard Agent — an assistant that answers questions about a project's runs, errors, queues, deploys and health, and can investigate failures end to end. The agent runs as a chat.agent task in its own Trigger project. It has no access to the main database or ClickHouse; all platform data is read through the public API using a delegated, read-only user token. Everything here is behind `canAccessDashboardAgent` and inert with the flag off. The UI that mounts the panel lands in #4529. ## Stack `#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook gallery. The scenario/contract reference for the whole stack is `internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch branch): it states, per feature, what makes each thing happen and where that is decided. ## What's inside **Agent runtime and tools** — `internal-packages/dashboard-agent`: prompt, tool set (API reads, TRQL query, docs, navigation, evidence/investigations, repo source), conversation compaction, a prompt-prefix token budget pinned by snapshot test, and sampled LLM-judged turn evals. The package cannot import webapp server code, which is what makes the "no DB access" claim structural rather than a convention. **Contracts** — `internal-packages/dashboard-agent-contracts`: `trigger://` URIs, intents, and the block envelope every rendered card travels in. **Conversation store** — `internal-packages/dashboard-agent-db`: drizzle over postgres-js in its own `trigger_dashboard_agent` Postgres schema, plus one additive migration. **Auth boundary** — the user-actor token gains an optional environment claim; one guard (`userActorEnvironment.server.ts`) enforces it so routes don't each re-derive the rule. Token minting, cap ceiling, and the RBAC fallback path for self-hosted. **Transport** — webapp resource routes that mint the token and proxy each turn, and SDK-side mid-turn reconnect. **Public API the agent reads through** — orgs, projects, environments, runs, queue metrics, workers, a run's commit metadata, repo snapshot, reports, and `POST /api/v1/query`. **Reports** — the health report's layout is declared once and shared by the card, the markdown surface and the JSON/MCP surface, so the same report reads the same in the dashboard, the terminal and an editor. **Block renderers** — the report and investigation cards the flows above already emit (`app/components/dashboard-agent/`). The panel that hosts them, and the rest of the chat UI, is #4529. **Query safety and CSP** — see below. ## Key decisions - **The agent is a separate Trigger project, not webapp code.** It reads platform data over the public API with a delegated user-actor token whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no webapp imports. - **The PAT-only auth helper now refuses user-actor tokens.** This is an intentional behavioral change: its callers consume only a bare userId and do not enforce delegated-token capabilities. Actor-aware routes continue through the scoped route builders instead. - **RBAC fallback builds a delegated token's ability from its own cap**, never the blanket ability a PAT gets (read-only when the token declares none). Without this, the agent's read-only cap would buy a write JWT on self-hosted. - **Org creation checks RBAC only for user-actor tokens, and only after the env gate**, so an install with `ORG_CREATION_API_ENABLED` off returns 404 rather than 403, and an ordinary PAT never consults an ability the route has no org to scope. Both orderings are pinned by test. - **The query path is read-only in depth.** TRQL rejects write statements at the grammar level (they don't parse, rather than being filtered), ClickHouse runs with `readonly=1`, and the org/project/env filters are injected server-side from the credential — the request body cannot widen scope. An unparseable query denies instead of falling through to the permissive resource. - **Document-wide img-src CSP.** Remote images are an outbound-request/exfiltration surface, so the policy permits only own-origin/data/blob, the required SSO avatar hosts, and the favicon endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST; wildcard hosts and bare schemes are intentionally not allowed. - **The chat transport reconnects on a mid-turn EOF** (`@trigger.dev/sdk`). A body that ends without a turn-complete is terminal only when the server says `X-Session-Settled: true`; otherwise the transport resubscribes from `lastEventId` with bounded backoff, and any record re-earns the budget. Previously a closed long-poll window or a proxy restart left the reply stuck as if still generating. - **Conversations live in their own datastore**, schema-scoped and foreign-key-free (it references `organizationId`/`userId` by id, because in cloud it is a different database). It is a display read-model for the History tab and transport resume; `chat.agent`'s object-store snapshot remains the model's source of truth. - **Deterministic first.** Reports and health checks contain no LLM — they are computed from the same data the dashboard shows, and the model only narrates and links them. That is what makes a number in an answer auditable. ## Testing - 63 new test files, run with `pnpm run test --filter webapp` and per-package vitest. Heaviest coverage on the auth boundary (`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`, `contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the report layout, and the SDK reconnect. - The agent package has a separate eval lane (`pnpm run test:evals`, `vitest.eval.config.ts`) that hits the real model, so it never runs in `pnpm test`. - Live-tested against a local stack scenario by scenario; the GUIDEBOOK lists the condition each behaviour is expected under, which is what those runs were checked against. ## Changelog `.server-changes/dashboard-agent.md`, plus changesets for `@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat reconnect) and the CLI's `mint-token` help text.	2026-08-11 18:56:14 +02:00
Eric Allam	6449a644b9	feat(webapp,cli,database): track real dev onboarding progress (#4563) ... ## Summary The dev environment "Get set up" panel used to be a static list of CLI commands that only disappeared once your tasks registered, so nothing ever changed after you ran `init` and people assumed it was stuck. It now tracks real progress: `trigger init` records the project as initialized, so step 1 checks off, and the panel updates live as the dev server connects and your tasks register. It also adds a prominent "Copy AI agent prompt" button, presented as a clear alternative ("or") to the manual CLI steps, that copies a ready-to-paste setup prompt pre-filled with your project reference for Claude Code, Cursor, or any coding agent. ## Notes - Adds a `Project.initializedAt` column (migration `20260811065646_add_project_initialized_at`); the CLI `init` command calls a new project-scoped `POST /api/v1/projects/:ref/init` best-effort at the end of setup. - The `init` scaffold now imports from `@trigger.dev/sdk` instead of the deprecated `/v3` subpath. ## Screenshots <img width="2400" height="1794" alt="v7-redesigned-card" src="https://github.com/user-attachments/assets/c2fb4fa1-9484-4700-8bd3-110d66f5a44e" />	2026-08-11 11:43:33 +01:00
Eric Allam	ce368dd8e0	perf(database): index EnvironmentVariableValue.valueReferenceId so secret deletes stop seq-scanning (#4555) ... ## Why this change `EnvironmentVariableValue.valueReference` is an `onDelete: SetNull` foreign key. Deleting a `SecretReference` (the env var edit/delete path for secret values) fires the cascade `UPDATE ONLY "EnvironmentVariableValue" SET "valueReferenceId" = NULL WHERE $1 = "valueReferenceId"`. That cascade is scan-shaped: with no index on `valueReferenceId`, it reads the entire table to find the rows referencing the deleted secret. The parent `SecretReference` delete does almost no work itself; its latency is dominated by this cascade. ## Diagnosis `EnvironmentVariableValue` was indexed on `environmentId` and `(variableId, environmentId)`, but not on `valueReferenceId`. The SET NULL cascade therefore did a full sequential scan of the whole table. Two sibling SET NULL cascades on the same delete (`OrganizationIntegration.tokenReferenceId`, `User.mfaSecretReferenceId`) are index-backed and stay fast, which isolates the missing index as the cause. ## Change Add `@@index([valueReferenceId])` on `EnvironmentVariableValue`, created with `CREATE INDEX CONCURRENTLY IF NOT EXISTS` so `prisma migrate deploy` stays safe on a live table. ## Benchmark (local, seeded) Local Postgres seeded with 1,000,000 `EnvironmentVariableValue` rows, `EXPLAIN (ANALYZE, BUFFERS)` on the SET NULL cascade with zero matching rows (the worst case: reads the whole table, affects nothing): | | before | after | |---|---|---| | plan | Seq Scan (1M rows) | Bitmap Index Scan | | execution | 183 ms | 2.8 ms | In a variant where the secret matched several thousand rows, the parent `SecretReference` delete's `EnvironmentVariableValue_valueReferenceId_fkey` trigger dropped from 216 ms to 88 ms (the residual is the heap work of nulling those rows). ## Expected impact The cascade drops from a full-table sequential scan to a targeted index lookup. The win grows with the table, so the benefit is larger than the seeded numbers above. ## Risks - One extra btree to maintain on `EnvironmentVariableValue` writes; small, single-column, and it should be pre-created before the migration deploys (per the repo index rules). - No behavior change: same rows nulled, no ordering or result-set change, read paths untouched. Companion to the same fix on `ProjectAlert.channelId`.	2026-08-10 13:54:18 +01:00
Eric Allam	4c58091973	perf(database): index ProjectAlert.channelId so alert-channel deletes stop seq-scanning (#4554) ... ## Why this change Deleting a `ProjectAlertChannel` fires the FK cascade `DELETE FROM ONLY "ProjectAlert" WHERE $1 = "channelId"`. That cascade is scan-shaped: with no index on `channelId`, it reads the entire `ProjectAlert` table to find the few child rows belonging to the deleted channel. The parent `DELETE ProjectAlertChannel` does almost no work itself; its latency is dominated by this cascade. `ProjectAlert` is append-heavy and grows over time, so the scan cost only increases. ## Diagnosis `ProjectAlert` had no index on `channelId` (only `pkey` + a `friendlyId` unique). The cascade therefore did a full sequential scan of the whole table. The sibling `ProjectAlertStorage` cascade on the same delete is index-backed and stays fast, which isolates the missing index as the cause. ## Change Add `@@index([channelId])` on `ProjectAlert`, created with `CREATE INDEX CONCURRENTLY IF NOT EXISTS` so `prisma migrate deploy` stays safe on a live table. ## Benchmark (local, seeded) Local Postgres seeded with 1,000,000 `ProjectAlert` rows across 50 channels (~20k rows per channel), `EXPLAIN (ANALYZE, BUFFERS)` on the cascade delete: | | before | after | |---|---|---| | plan | Seq Scan (1M rows) | Bitmap Index Scan | | direct child delete | 740 ms | 22 ms | | parent delete `ProjectAlert_channelId_fkey` trigger | 77.7 ms | 23.8 ms | ## Expected impact The cascade drops from a full-table sequential scan to a targeted index lookup. The win grows with the table: the more rows in `ProjectAlert`, the more a scan costs and the more the index saves, so the benefit is larger than the seeded numbers above. ## Risks - One extra btree to maintain on every `ProjectAlert` insert; acceptable for a single-column index on a high-insert table, and it should be pre-created before the migration deploys (per the repo index rules). - No behavior change: no rows orphaned, no ordering or result-set change, read paths untouched. ## Follow-up `ProjectAlert`'s other cascade FK columns (`projectId`, `environmentId`, `workerDeploymentId`) are also unindexed, but their parents are soft-deleted rather than physically removed, so those cascades do not currently fire. Lower priority unless a hard-delete path is introduced.	2026-08-10 13:54:15 +01:00
Eric Allam	90e8bd5c12	feat(webapp,database): opt-in per-client Prisma driver adapters (#4539) ... ## What Adds an opt-in path to run each Prisma client through **`@prisma/adapter-pg`** (the node-postgres driver) instead of the built-in engine driver, controlled by a **per-client env var, all off by default**: | env var | client | |---|---| | `CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER` | control-plane writer | | `CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER` | control-plane replica | | `RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER` | new run-ops writer | | `RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER` | new run-ops replica | | `RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER` | legacy run-ops writer | | `RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER` | legacy run-ops replica | With every flag unset the construction path is byte-identical to today (`datasources` URL + Rust engine), so this is inert until a flag is turned on. Per-client granularity allows enabling the adapter only where it's wanted. ## How - Enables the `driverAdapters` preview feature on both schemas (`@trigger.dev/database` and `@internal/run-ops-database`). This keeps the **Rust query engine** — it does NOT add `queryCompiler` — so query behavior, result types, and engine tracing spans are unchanged. - A shared `buildDriverAdapterPool` builds each client's `pg.Pool` with an explicit `max`, a bounded `connectionTimeoutMillis` (the node-postgres pool otherwise waits unbounded on acquire), and an `onPoolError` handler (an unhandled idle-connection error would otherwise crash the process). Threaded through all four client builders via a `useDriverAdapter` flag. - Adds `@prisma/adapter-pg` + `@types/pg` to the webapp; `pg` is already pinned at `8.15.6` (adapter-pg 6.x requires `pg < 8.17`). ## Connect-failure handling (the important correctness/security bit) Under the adapter an unreachable DB no longer surfaces as `PrismaClientInitializationError` / `P1001`; it becomes a `P2010` "Database not reachable: <host>" (or a raw `ECONNREFUSED`/`ENOTFOUND`-class error). Two handlers are updated so a client on the adapter behaves like today: - **`isInfrastructureError`** now recognizes those shapes (P2010 with a connectivity message, and raw connectivity errno codes). Without this, the DB **hostname would leak into API-client-facing errors** and the failure would go unlogged. Security-relevant. - **`isPrismaRetriableError`** treats the adapter's pool-acquire timeout ("timeout exceeded when trying to connect") as retriable, preserving the `P2024` retry behavior the adapter otherwise drops. ## Evidence Validated on an isolated stack that mirrors the production DB topology (chained PgBouncers in front of writer + reader): - **Behavioral parity:** raw-query results and Prisma error codes/`meta` are byte-identical between the engine driver and the adapter across the queried shapes (unique-constraint `meta.target`, record-not-found, transaction-timeout, serialization-failure, etc.). - **Feature matrix:** a full 380-project queue-ay pass shows no adapter-caused regressions — pass/fail parity between adapter-off and adapter-on, with the residual failures being pre-existing known-failures/flakes common to both. ## Rollout / rollback All flags default off; enable per client via env var, roll back by unsetting and redeploying (no data migration). Recommended first target is a single writer; enable one client at a time. ## Follow-ups (not in this PR) - `$metrics`-based pool observability is removed under the adapter (the Prometheus route + `db.pool.connections.*` instruments); the metrics replacement (via `pg.Pool` counters) lands in a separate PR. - Note for operators: on the adapter path, interactive-transaction `maxWait` does not bound pool acquisition — `connectionTimeoutMillis` does. ## Note on connection-string parameters The adapter pool is built from the base DSN, so Prisma-specific DSN parameters that node-postgres does not understand are not honored when a client is on the adapter: - **Prisma TLS spellings** (`sslaccept`, `sslcert`, etc.) — node-postgres uses `sslmode`/`ssl` instead. Our production DSNs do not use these Prisma-specific TLS params, but any deployment whose DSN relies on them must be checked before enabling a flag. - `pgbouncer=true` and `statement_cache_size` — effectively moot under the adapter, which uses no persistent named prepared statements. `connection_limit`, `pool_timeout`, and `schema` are handled explicitly (passed as `max`/`connectionTimeoutMillis` and PrismaPg's `{schema}` option). refs TRI-13039 --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-08-08 21:27:20 +01:00
Eric Allam	c526528d8f	feat(webapp,database): bound Prisma list filter arity (#4480) ... ## Summary Prisma expands `in` / `notIn` into one bind parameter per element, so every distinct list length is a separate prepared statement. Where the length tracks data volume (a batch size, a run-graph fan-out, a prior query's id set) one call site can mint hundreds of them. Each is used about once, but inserting it evicts an entry that was being reused, so the cost lands on unrelated queries sharing the pooler's statement cache. An unbounded list also risks the 65535 bind-parameter ceiling. `boundedIn()` pads a filter list to the next power of two by repeating its last element. `IN` and `NOT IN` ignore duplicates, so results are unchanged, and a call site drops from one statement per length to at most `log2(cap)`. Applied to all existing sites. ## Enforcement Two oxlint rules require the helper: a list filter must be an inline array literal or a `boundedIn()` call. - The first covers filters reached through `where` / `having` / `cursor`, and deliberately never descends into `data`, `create`, `update`, `set` or `equals`. A key named `in` in those positions is user data, not a predicate, and rewriting it would corrupt what gets stored or compared. - The second covers bare filter objects passed to where-building helpers, which the first cannot see. It found five sites in the run-graph batch loaders that were otherwise invisible. Both rules follow filters through the shapes they are actually written in: conditional expressions, logical-and objects, spread-conditional properties, computed keys, and call arguments. An array literal only counts as fixed-arity when nothing spreads into it, since `[...new Set(ids)]` has a runtime length. Twelve sites were hidden behind those shapes until the rules handled them. Scoped to `in` and `notIn`. The scalar-list filters `hasSome` and `hasEvery` compile to `&& $1` and `@> $1`, passing the whole array as a single bind parameter, so their arity never reaches the statement text and there is nothing to bound. Both rules are `error`, so new call sites fail CI. That ratchet has already caught four sites added by other PRs while this one was in review. ## Notes `boundedIn` pads by repeating rather than with null: `x NOT IN (a, b, NULL)` is never true, so null-padding a `notIn` filter would silently return no rows. Lists above 32768 are returned unchanged so padding can never push a query past the parameter limit. Route modules reach the helper through `~/db.server` rather than importing the database barrel directly, since a value import of that barrel into a module that also exports a React component is only safe while dead-code elimination prunes it. Measured on a local rig: 300 distinct list lengths produce 300 prepared statements unpadded, 10 padded. Verified end-to-end against a local stack with the full task-suite sweep, which surfaced no regressions.	2026-08-07 16:39:58 +01:00
Chris Arderne	0a44b88b39	fix: security release 2026-07-21 (#4528)	2026-08-07 12:25:40 +01:00
Eric Allam	db67a856fe	perf(webapp,database): index the newest-task-version lookup (#4518) ... Implementing PlanetScale Insights improvement. ## Summary Validating a schedule (creating or updating one through the API or the dashboard, and deploying a project that declares schedules) looks up the newest version of a task by slug. That lookup reads *every* version of the task and sorts them to return one. A project gains a row per task on every deploy, so the work grows with the project's age: the oldest projects pay the most, and dev-mode redeploys make it worse. This was picked because it was the largest single consumer of database time on the schedules path, and the fix is a sort key with no index behind it. ## Fix `BackgroundWorkerTask` is indexed on `(projectId, slug)`, which serves the equality but not the `ORDER BY createdAt DESC`. Postgres seeks the index, then bitmap-scans and top-N sorts the whole group to produce a single row. Adding `createdAt` to the index lets it scan backward and stop at the first row. The same call site also selected all 21 columns, including five JSON blobs, to read one field (`triggerSource`), so it now selects that field alone. ## Benchmark Local Postgres 17, 997,000 seeded rows / 748 MB, group sizes chosen to match the distribution seen in production. | Group size | Before | After | | --- | --- | --- | | 15,000 versions of one task | 11.118 ms, 1,510 buffers, 15,000 rows scanned | 0.027 ms, 4 buffers, 1 row | | 2,000 versions of one task | 2.081 ms, 1,455 buffers, 2,000 rows scanned | 0.022 ms, 4 buffers, 1 row | ``` before: Limit -> Sort (top-N heapsort) -> Bitmap Heap Scan after: Limit -> Index Scan Backward using BackgroundWorkerTask_projectId_slug_createdAt_idx ``` An ascending index scanned backward is enough here, so no descending index is needed. ## Impact and risk Real-world gain lands between the two rows above and scales with how many deploys a project has accumulated. Projects with few deploys will see little change, since there is barely anything to sort. The new index costs noticeably more than the existing two-column one: 43 MB against 7.3 MB on the benchmark rig. Adding `createdAt` makes every key unique, which defeats btree deduplication, so this is a real disk and write cost rather than a rounding error. Writes to this table happen at deploy time, not on the run path, so the write amplification is acceptable. The existing `(projectId, slug)` index is now a redundant prefix and could be dropped, but this PR keeps it so index usage can be observed before removing it. Behavior is unchanged: same predicate, same ordering, same row returned. The narrowed select is the only code change, and the field it keeps is the only one the caller read. Deploy note: the migration is `20260806100000_add_background_worker_task_project_id_slug_created_at_index` and uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it can be pre-applied by hand before the deploy.	2026-08-07 11:17:10 +01:00
Matt Aitken	04f9c4e1a5	fix(webapp,run-engine,core): drop the hidden debounce ceiling, fail fast on an unusable maxDelay (#4521) ... Debouncing with a `delay` longer than an hour did nothing at all. The engine applied a server-side ceiling on how long a debounced run could be pushed back, measured from the run's `createdAt` and defaulting to one hour. A run is only pushed back while its new execution time stays inside that ceiling, so a `delay` at or above it could never push anything: the waiting run was released, the trigger started its own run, and the next trigger repeated it. A `delay: "12h"` produced one run per trigger, each correctly delayed by 12h, with no error raised and nothing on the run to show the debounce key had been ignored. The ceiling is now unset by default. A debounce key with no `maxDelay` keeps collapsing triggers for as long as they keep arriving, which is what the docs have always described. Self-hosters who want a bound can still set `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS`. That has a consequence worth stating plainly, so the docs now carry a warning for it: with no `maxDelay`, a continuously triggered key never executes. Set `maxDelay` when the work has to happen eventually. **Failing fast on an unusable `maxDelay`.** A caller who sets `maxDelay` no longer than their `delay` hits exactly the dead end described above, so that pair is now rejected at trigger time instead of silently behaving as if no debounce were set: ``` debounce.maxDelay (1h) must be longer than debounce.delay (12h). A debounced run is only pushed back while it stays inside maxDelay, so with these values every trigger would create its own run. ``` An unparseable `maxDelay` is rejected too, rather than quietly falling back to no bound at all, and so is a `delay` given as a date rather than a duration, which could never work because the value is re-applied on every push. The same check runs against a configured server ceiling, so a self-hosted deployment that sets `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS` gets the error rather than the silent failure this PR is about. With no `maxDelay` and no configured ceiling, which is the default, there is nothing to conflict with and nothing is rejected. The docs, the `TriggerOptions` JSDoc and the engine option all now state that the room available to push is the gap between `delay` and `maxDelay`. The run engine suite gains the case that motivated this: four triggers on one key with a 12h delay now collapse to a single run.	2026-08-07 07:55:35 +00:00
Wes Mason	66940c0384	fix(observability-map): narrow the required check and the report bot's comment lookup (#4507) ... ## Findings addressed - **Report bot edited the wrong comment.** The comment-lookup step matched on the marker body text with no author predicate, so it would silently PATCH a human's comment that happened to quote the marker (GitHub gates comment editing on write access, not authorship, so it never 403'd). Now constrained to `.user.login == "github-actions[bot]"`, the same identity `helm-prerelease.yml` already pins. - **A required check asserted facts about the whole webapp namespace.** `webappSymbols.test.ts` asserted that nobody anywhere in `apps/webapp` (walking locals, params, object keys) declares names like `createJWT`/`updateEnvVars`, so an unrelated PR naming a local variable failed a required check with a message pointing at nothing. Those negative self-tests move onto a package-owned fixture tree; the positive resolution assertions stay required (their absence rotted the tool before) but now name the list to edit. - **The suite ran twice on shared paths.** `obsmap` and `internal` path filters shared four generic paths (`package.json`, both lockfiles, `pr_checks.yml`), so any lockfile bump ran the observability-map suite in both jobs. Dropped from `obsmap` (where `internal` already covers them). The test that should have caught it only checked the package's own source path; it now asserts the two filters' path intersection is empty. - **PR-comment footer** reworded: it said the report gates nothing, which is true of the report but misled now that the tool's test suite does gate webapp PRs. Names both failure directions and where to read the rules. - **Nightly corpus** comment corrected (stale entry count; the failure-notification gap is documented, not silently implied). ## Review Two adversarial reviewers ran over the diff; both findings were verified and fixed: a hollow fixture assertion (a shared name satisfied either walker branch — now one name per declaration form, revert-confirmed) and a filter-intersection test that could be fooled by apostrophes in comment prose (now strips comment lines first). Full package suite green (877 passed), typecheck and format clean.	2026-08-05 22:36:19 +01:00
Eric Allam	b20806247f	fix(run-store): stop run-create failing on a brief write stall (#4514) ... ## Summary On the run-ops store, creating a run could intermittently fail with a "Transaction already closed" error, and the run would never be created. Single-write run creates no longer run inside an interactive transaction, so a brief database write stall can't blow the transaction budget and drop the run. ## Fix The dedicated run-ops `createRun` / `createFailedRun` wrapped a single nested `taskRun.create` in an interactive `$transaction`. Its default 5s budget is wall-clock from `BEGIN`, so when a write briefly stalls the transaction expires before the create completes and throws, even though the statement itself is fast at the database. A single-write create does not need an interactive transaction: Prisma's implicit nested create is already atomic and holds no app-side budget, so it now runs directly. Only the `triggerAndWait` path (run plus its associated waitpoint, two writes that must commit together) keeps an interactive transaction, now with headroom over the default. Verified with a red/green test against the real split topology (reproduces the exact expiry on the unchanged code, green after) and an end-to-end run created and completed through the dedicated store.	2026-08-05 17:35:46 +01:00
Chris Arderne	85f5b37c68	chore: upgrade to TypeScript 7 (#4318) ... ## Summary Upgrade the monorepo to TypeScript 7.0.2 and update package build tooling for compatibility with the native compiler. ## Design Package builds now use `tshy` 4, while the packages still using `tsup` move to `tsdown`. The few scripts that depend on the legacy TypeScript compiler API use an explicit TypeScript 6 alias; declaration portability coverage invokes the TypeScript 7 CLI directly. Turbo is updated so workspace tasks can read the regenerated pnpm lockfile. --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-05 15:49:44 +01:00
Wes Mason	ca9a74e84a	feat(observability-map): static observability scorer for webapp route entry points (#4455) ... A static observability scorer for the webapp's route entry points, Lighthouse-style. The idea comes from evlog's `map` command, but that tool has no Remix adapter and checks for its own logging API, so the idea is ported rather than the tool. It scans all 427 loader/action entry points in `apps/webapp/app/routes` with the TypeScript compiler API and scores each against five checks: error-classification, auth-boundary, auth-scope, request-context and audit-trail. Current output on the real tree is **19/100** over 412 measured entry points. ``` cd internal-packages/observability-map pnpm exec tsx src/cli.ts # terminal report pnpm exec tsx src/cli.ts --json # machine output pnpm exec tsx src/cli.ts api/v1/token # one entry, per-check detail ``` The two findings at the top of the fix list are real: `/auth/sso` and `/api/v1/authorization-code` mint or exchange credentials unauthenticated, and `/_app/orgs/:organizationSlug/settings/team` resolves its org from a URL slug and gates each mutating branch on an RBAC check alone, which per `apps/webapp/CLAUDE.md` is not the tenant floor on self-hosted. Decisions worth knowing, all with the reasoning in the README: - The score started at 83 during development and fell to 19. Every drop was a perverse incentive being removed, not a regression: routes were being paid for having no error handling, two checks were reading the same fact, suppressing a failure raised the score, and a no-op `catch (e) { throw e }` was worth 50 points a route. - **A mutation corpus is the tool's main defence.** 44 entries apply semantics-preserving edits to a copy of the real route tree and assert the score cannot rise, per route as well as globally, because a mean can hide one route going up by taking another down. One entry runs as a live expected failure: `try { String(0); }` with a deciding catch is a known open hole worth 19 to 44, and it is disclosed rather than quietly excluded. - `audit-trail` and `request-context` are reported as headline figures rather than one finding repeated hundreds of times. Both still count in full where they should. - A cohort change moves the number without anything in the codebase getting better. Widening the sensitive cohort from 26 to 67 took the global from 15 to 19 with no webapp change at all, so the report prints per-check applicability and what the global would be without each one. CI: a report-only job posts a sticky comment when a PR moves the report, and says nothing when it does not. The package's own tests gate through `pr_checks.yml`. The diff-scoped merge gate is still deferred until the report has been used in anger. 524 tests plus the corpus. No runtime or dependency changes to anything that ships. <!-- GitButler Footer Boundary Top --> --- This is **part 1 of 4 in a stack** made with GitButler: - <kbd>&nbsp;4&nbsp;</kbd> #4485 - <kbd>&nbsp;3&nbsp;</kbd> #4484 - <kbd>&nbsp;2&nbsp;</kbd> #4483 - <kbd>&nbsp;1&nbsp;</kbd> #4455 👈 <!-- GitButler Footer Boundary Bottom -->	2026-08-04 15:33:32 +01:00
Katia Bulatova	fbd6df33b4	feat(webapp): Themes + contrast settings update (#4206) ... Adds System Preferences, Dark and Light themes, gated by the `hasThemeSwitcher` feature flag (off by default — dark stays the default theme for everyone). Old theme is now "Classic"and set as default. "System preferences" theme has both Light and Dark modes and uses your laptop settings to use a correct one. It has less color accents (specifically less colored text), and they are the same for both modes, only grayscale values change between them. And Light/Dark themes can be used separately. New Contrast setting is available for System Preferences, Dark and Light themes - it changes the contrast for the whole app. All new visual Settings live in Account.	2026-08-03 19:29:33 +02:00
Eric Allam	9d57aff542	fix(webapp): make the Queues hero charts environment-wide (#4486) ... ## Summary The four charts above the queues table aggregated over **at most the 25 queues on the current page**. They reused the loader's already-paginated queue array as a ClickHouse `queue IN (...)` filter, so paging or re-sorting changed the values, and a name search matching nothing blanked the whole chart row. The stat tiles above them were already environment-wide, so the two rows disagreed. They now read `env_metrics`, the environment-level rollup that already exists for exactly this (the built-in Queues dashboard and the health report read it). That is both correct and queue-count-independent: no `GROUP BY queue` across an entire environment, and no client-side summing. Note this is not only a paging artifact: page 1 under-reported too. On the seeded environment below, page 1 read 82% saturation against a true 87%, because the environment's running total is not the sum of one page of per-queue gauges. Three related fixes ride along. **Scheduling delay and throttling sawed to zero.** Both are event-driven, so at the 10-second bucket a short range picks, most buckets hold no samples at all and were drawn as `0ms`. Measured over a 1-hour window: **232 of 349 buckets had no scheduling-delay samples**. A bucket where nothing started is not a bucket where nothing waited, so the line was both ugly and wrong. TRQL grows a `minBucketSeconds` floor, plumbed through the metric resource route, and the hero tiles set 60s. Buckets that still have no samples render as a gap instead of a dive to zero. **The floor must not feed a width-dependent headline.** Two of the four headlines are not peaks, so widening the plotted buckets moved them: - **Throttled** is a share of buckets that saw any throttling, so a single brief throttle came to mark a whole minute instead of ten seconds: the same seeded events read 17% at 10s and 85% at 60s. - **Scheduling delay p95** is a percentile, and merging quantile states over a wider bucket yields a p95 between the sub-buckets' own. Two 240s samples among twenty in one 10-second sub-bucket give a worst-of-six p95 of 240,000ms against a merged 60-second p95 of 5,000ms — a 48x understatement of a headline whose tooltip claims it is the worst in the window. Both charts keep the floor, since a readable line was the point of it. Their headlines now come from a second query at the range's natural bucket width, via an optional `readout` on the tile, so each means what its tooltip says regardless of how the plotted buckets are sized. Saturation and backlog are genuinely width-invariant (a max of maxes is the same at any width), so they are unchanged and issue no extra query. Both caught by Devin in review; I had wrongly lumped p95 in with the peaks. **Charts reported a hydration mismatch on every render.** Recharts resolved victory-vendor's CJS entry on the server and its ESM entry in the browser. Those bundle different d3-shape builds, and the CJS one predates d3-path's digit rounding, so every server-rendered curve carried full-precision coordinates while the client rounded to 3 decimals: ``` Server: M0,3C0.9305555555555555,3,1.8611111111111112,3,... Client: M0,3C0.931,3,1.861,3,... ``` Bundling recharts for SSR makes both sides resolve the same ESM build. Verified: 45 of 45 server-rendered chart curves now match the client, and the page loads with an empty console. ## Verification An isolated stack with 40 seeded queues (20 heavily loaded, 20 idle) and 90 minutes of 10-second buckets written into `queue_metrics_raw_v1`, so the real materialized views built `queue_metrics_v1`, `env_metrics_v1` and the 5m rollup. Ground truth for the environment: 260 running against a limit of 300 (**87% saturation**), 800 queued. | | before | after | | -- | -- | -- | | Saturation, page 1 | 82% peak | **87% peak** | | Saturation, page 2 | 5% peak | **87% peak** | | Backlog / delay, page 2 | "No activity" | **800 peak / 59.5s** | | Name search matching nothing | all four charts blank | charts stay environment-wide | | Metric refetches on a page change | 4, each painting a skeleton | **0, no skeleton** | | Buckets drawn as 0ms with no samples | 232 of 349 | **0** | | Throttled readout | 17% | **17%**, unchanged by the wider buckets | | Worst-p95 readout source | plotted buckets | **natural width**, so a sub-minute spike is not averaged away | | Crosshair reach, hovering one detail-page chart | 2 of 4 others | **4 of 4** | | SSR chart curves mismatching the client | 45 | **0** | The bucket floor was measured across ranges: it widens 10s to 60s at 30m and 1h, and is correctly a no-op at 12h (300s) and 7d (3600s). One extra request per page load, for the throttled readout. The built-in Queues dashboard, which reads `env_metrics` independently, agrees at 86.7% and 260 of 300. `internal-packages/tsql` suite green (612 tests), including 5 new ones for the floor that fail without it. Webapp typecheck, oxfmt and oxlint clean. Spot-checked the Run metrics dashboard and the per-queue detail page for SSR regressions from bundling recharts: both render, console clean. The queue detail page carries the same event-driven series, so its scheduling delay, throttling and per-key mean delay take the same treatment. ## Screenshots <img width="2540" height="580" alt="after-page1-charts" src="https://github.com/user-attachments/assets/6cd23f9c-e7fd-4918-bcfa-b1d3340b16d1" /> ## Rollout Already behind the per-organization `queueMetricsUiEnabled` flag, so only gated orgs see any of it. Blast radius is chart values on one page plus the SSR bundling of recharts; rollback is a revert with no data migration. ## Stated limitations - `wait_ms_count` and the quantile state both only count `wait_ms > 0`, so "nothing started in this bucket" and "everything started instantly" are indistinguishable in storage. Both render as a gap. Distinguishing them needs a schema change, which is not in this PR. - The queue name search deliberately no longer narrows the charts. It only did so incidentally and incorrectly before (first 25 matches, and blanked on zero matches). Search-scoped charts would need the full unpaginated matching set and a server-side aggregate; worth its own ticket if we want it. - Bundling recharts for SSR grows the server bundle slightly. That is the cost of both sides resolving one d3-shape build. - The plotted delay line is a smoothed 60-second view, so a sub-minute spike above the one-minute warning threshold can fail to colour the line even though the headline reports it and colours itself. - Every chart inside one synced group shares the floor, because the hover crosshair is a reference line on a category x-axis and only draws where the hovered bucket exists in the other chart's own data. That costs the queue detail page's gauges some resolution (1 minute instead of 10 seconds) in exchange for the crosshair working across the row. Separately, while taking the screenshots I found a pre-existing rendering bug unrelated to this change: a **perfectly flat** saturation series draws no line at all (the readout still shows the right percentage), which looks like the threshold gradient's offset degenerating when the series min equals its max. It reproduces on `main`, so it is not a regression here and I have left it alone; filed as its own issue. Refs TRI-12784	2026-08-03 16:19:50 +01:00
Chris Arderne	763b5dc582	feat(webapp): enforce scopes for environment API keys (#4389) ... ## Summary Environment API keys backed by the additional-key table can authenticate API requests using their stored effective scopes. Revoked and expired keys are rejected, branch environments retain their existing routing behavior, and last-used timestamps are updated on a throttled best-effort basis. ## Design API route builders receive the resolved ability and reject restricted keys on routes without an authorization declaration. Existing deployment, environment variable, queue, run, task, batch, session, and waitpoint routes declare the resources they access. Trigger and batch responses return server-signed public access tokens, so additional keys never need access to the environment signing secret. Root-key rotation also keeps public tokens valid for the existing grace window. ## Feature notes - Root environment keys remain unrestricted for backward compatibility. Additional keys enforce their persisted scopes and fail closed on routes without an authorization declaration. - Machine-key requests never exchange one credential for another. Additional keys cannot retrieve the root key, and rotated root keys are not upgraded during their grace window. - Public JWT validation remains host-owned, while installed RBAC plugins continue to supply root-key abilities. - Unfiltered session and run listings preserve existing broad task-read behavior. Filtered requests enforce the supplied task identifiers. - Related-run summaries remain embedded in run retrieval for API compatibility. Retrieving or mutating a related run independently still requires permission for that run. - Queue management authorizes at collection scope, matching the queue permissions currently issued. - Batch responses deliberately include server-signed public access tokens for all clients. Selected-task credentials continue using their original credential for per-item authorization. - Two-phase batches authorize declared task identifiers before creation and authorize every streamed item. Streaming paths that cannot declare the complete task set remain fail closed. - Authentication telemetry records successful credential resolution separately from subsequent resource-authorization failures. - API keys are high-entropy random tokens. SHA-256 is intentionally used for deterministic indexed lookup, not password hashing. ## Deployment notes The schema migration must be present before this code is deployed. Because bearer resolution runs on every authenticated request, deploy the resolver with additional-key lookup disabled, verify root-key and public-token parity, then enable lookup before any additional keys can be issued. The multi-task authorization tightening changes the result for narrowly scoped tokens that request tasks outside their grants. Observe would-deny results before enforcing that check. Request-idempotency keys are also newly isolated by environment and task, so a retry crossing the deployment boundary may execute once more before old cache entries expire. ## Follow-ups - [x] Add a system-wide kill switch for additional-key lookup, defaulted off for the initial deployment. - [x] Add authentication observability by credential kind, result, latency, and lookup path without recording credential values. - [ ] ~Add would-deny observability and an independent enforcement switch for multi-task authorization.~ - [ ] ~Add an independent switch for server-issued batch tokens while root-key parity is verified.~ - [ ] Confirm every API route reachable by a restricted key has an explicit authorization declaration or intentionally fails closed. - [x] Verify root-key rotation, revoked-key grace, and public-token validation through each bearer resolver path.	2026-08-03 14:00:29 +01:00
Eric Allam	db6228dd1e	chore(webapp,core,sdk): upgrade @s2-dev/streamstore to 0.25 and migrate S2 hosts (#4349)	2026-08-01 11:33:34 +01:00
Eric Allam	f9c8d518c7	perf(webapp,run-engine,database): resolve the newest worker and deployment by createdAt (#4452)	2026-08-01 11:32:21 +01:00
Eric Allam	0445b8ec27	fix(webapp,clickhouse): keep the rest of a ClickHouse batch when one run or span has un-ingestable JSON (#4358) ... ## Summary A single run output, trace span, or payload carrying JSON that ClickHouse can't ingest (for example nesting past its depth limit) used to fail the whole insert batch, so unrelated runs and spans silently disappeared from the runs list, traces, and logs. This keeps the rest of the batch and handles the offending row instead of dropping everything around it. ## Fix Recovery is per-table, matched to what each table needs: - **Runs** (`task_runs_v2`) keep their status. We follow ClickHouse's failing-row hint to strip just the un-ingestable JSON column(s) so the run still lands (its output reads from Postgres on the detail page), up to a configurable limit (`RUN_REPLICATION_MAX_POISON_STRIPS_PER_BATCH`, default `1`). Past the limit we stop and land the batch with `allow_errors` in a single pass, skipping the remainder. Cost stays a fixed handful of inserts no matter how large or poisoned a flush is. - **Trace events and payloads** (high volume, append-only) recover with a single `allow_errors` insert: the good rows land in one pass and only the un-ingestable rows are skipped. Before falling back, a lightweight sanitizer still repairs what it can losslessly (lone UTF-16 surrogates, out-of-range integers) so a repairable row lands in full. To read the failing-row hint we patch `@clickhouse/client-common`: its error parser truncates the server response and discards the `(at row N)` position, so the patch preserves the full text for the recovery path to read.	2026-08-01 09:17:20 +01:00
Eric Allam	f10bc23785	perf(run-engine,run-store): one execution snapshot per triggered run (#4419) ... A non-delayed run used to get two execution snapshots the moment it was triggered: `RUN_CREATED` nested in the run-create transaction, immediately followed by `QUEUED` from its own `BEGIN`/`INSERT`/`COMMIT`. It now gets a single `QUEUED` snapshot written inside the create, and the trigger path only publishes to the queue. One fewer row per run on `TaskRunExecutionSnapshot`, and one fewer round trip on the trigger hot path. `EnqueueSystem` gains a `publishRun` seam that enqueues without writing a snapshot. Every re-enqueue path (waitpoint resume, checkpoint restore, delayed enqueue, pending version, retry requeue) still calls `enqueueRun` and writes its own `QUEUED`, so only the first enqueue changes. The `QUEUED` snapshot still commits before the queue message, so a dequeue sees a dequeueable status exactly as before. Two things for reviewers. Nesting the write skips `createExecutionSnapshot`, which is what emits `executionSnapshotCreated` and therefore the run timeline's `[engine] QUEUED` entry, so the trigger path now emits it directly, the same way the dequeue and attempt-start paths already do for their nested creates. And `RUN_CREATED` is still written when a dequeued run has no background worker yet, so the status and both `statuses.ts` helpers stay live and existing rows keep reading correctly. Delayed runs are untouched: `DELAYED` then `QUEUED` are two genuinely different moments and stay two snapshots. Rollback is a revert. Create-and-enqueue happen in one request in one process, so no in-flight run needs both code paths to agree during a rollout. One note for whoever debugs this path later. The `QUEUED` snapshot now commits before the queue publish, so a failed publish leaves the run recorded as `QUEUED` with no queue message. That state was already reachable, since the publish was never part of the snapshot transaction, but it used to be recorded as `RUN_CREATED`, which was distinctive because it never otherwise persisted. `QUEUED` with no message is indistinguishable from a run waiting on a concurrency slot, so trigger-time publish failure is now one more cause of an apparently stuck queued run.	2026-07-31 16:12:07 +01:00
Eric Allam	c72ebf9084	fix(webapp,run-engine): stop batchTriggerAndWait hanging when item streaming never completes (#4397) ... ## Summary `batchTriggerAndWait()` could leave a parent run waiting forever. The 2-phase batch API blocks the parent on the batch's waitpoint as soon as the batch is created, but the batch is only sealed at the end of item streaming. If streaming never completed, nothing sealed the batch, nothing completed the waitpoint, and the parent stayed suspended with no timeout and no way to recover. Supersedes #4016, which added the reaper alone. ## Fix Admission for item streaming was being decided twice. Batch creation passes its own rate limiter, which fixes `expectedCount` and blocks the parent, and then the item stream had to pass the general API limiter as well, competing with unrelated traffic. A second limiter could therefore veto work the first had already committed the parent to. Creation now mints a bounded grant that the item stream spends, so an admitted batch can finish streaming. The grant is capped per batch rather than exempting the path, and every failure mode (no grant, spent grant, unreachable store) falls back to the normal limiter. That makes stranding much rarer but not impossible, since a request timeout or a crash can still end streaming for good. So a seal-timeout reaper aborts any batch still unsealed after `BATCH_SEAL_TIMEOUT_MS` and completes the parent's waitpoint with an error, letting `batchTriggerAndWait()` reject instead of hang. It is race-safe against a late seal, and it is only scheduled for batches that actually block a parent, so fire-and-forget batches cost nothing. Finally, the batches page used to report "Batch completion checked." for these batches while doing nothing, because the completion path returns early on an unsealed batch. It now says the batch cannot be resumed. Rate limiting is no longer the reason a batch strands, so the reaper's default stays at 30 minutes, comfortably above the SDK's worst-case stream-retry budget. ## Verification Unit and container tests cover the grant cap, the bypass ordering (it runs after the authorization check, so it can never skip authentication), and the reaper's abort, seal race, idempotency, and no-waitpoint cases. Also verified end-to-end against a running stack. With the general limit exhausted, batch creation and other API calls returned 429 while a granted batch still streamed and sealed; an ungranted batch id was rate limited rather than bypassed; and the grant cut off exactly at its configured attempt count. Reproducing the stranded state on a real parent run, the batch was aborted at the timeout, the waitpoint completed with an error, and the parent resumed and finished instead of hanging. A parentless batch left unsealed was untouched well past the reaper window. ## Verified against deployed runs The reaper was proven end to end with a real deployed run (locally-run supervisor, containerised run) and a real network fault, rather than a simulated one: toxiproxy severs the phase 2 item stream mid-flight so every SDK stream retry genuinely fails, while phase 1 still succeeds. Only the batch calls traverse the fault, so control-plane traffic is untouched. The reproduction is the shape that actually strands a parent: the task catches the `BatchTriggerError` the SDK throws and carries on, so the phase 1 block outlives the thrown error and the parent hangs at its next suspension point. With the reaper disabled, the parent sat in `EXECUTING_WITH_WAITPOINTS` for over 24 minutes holding two blockers, and stayed stuck across a full infrastructure restart: ``` type | status | has_timeout BATCH | PENDING | f <- orphan, completedAfter NULL DATETIME | COMPLETED | t <- the wait already elapsed ``` With the reaper enabled the same task under the same fault completed in about 75 seconds with zero blockers left, the batch `ABORTED`, and its waitpoint completed carrying the error. Two conditions are required to observe this at all, which is worth knowing for any future test: the run must be deployed rather than `trigger dev` (dev runs execute in process and finish while still holding blocker rows), and the wait after the caught error must exceed the checkpoint threshold, or it is served in process and never suspends. ### Why completing the batch waitpoint is sufficient `batchTriggerAndWait` runs create, then stream, then wait. A phase 2 failure throws before the wait is ever reached, and the reaper only fires on an unsealed batch, so the parent is never suspended awaiting the batch when it runs. The parent therefore does not need a synthetic result, only to stop being blocked. Note this reasoning depends on that ordering: if the wait were ever reached with an unsealed batch, completing the batch waitpoint alone would not settle the caller. ## Follow-ups - Batches stranded before this ships still need a one-off recovery; the reaper only schedules at creation time. - That same property leaves a gap if the process dies between creating the batch and scheduling the job. A periodic sweep would close it, but wants a supporting index. - When a partially streamed batch aborts, children already enqueued keep running while the parent fails. Left as-is deliberately, since cancelling triggered work is a bigger semantic call.	2026-07-31 11:55:25 +01:00
Chris Arderne	68ed809416	test(clickhouse): keep queue metrics fixtures within TTL (#4428) ... ## Summary Keeps the queue metrics ClickHouse tests stable as wall-clock time advances. ## Root cause The fixtures used fixed timestamps. Once those timestamps crossed the tables' 30-day retention boundary, ClickHouse immediately expired the inserted aggregate rows and all six tests read empty results. The fixtures now derive a recent minute-aligned timestamp once per test file. The second 10-second bucket and ranking window are derived from the same anchor, preserving deterministic bucket relationships while keeping rows inside both the raw and aggregate table TTLs. Verified with `pnpm --filter @internal/clickhouse exec vitest run src/queueMetrics.test.ts`.	2026-07-30 21:20:44 +01:00
Eric Allam	6e5f0f0fe7	fix(webapp,clickhouse): stop invalid customer queries alerting, and isolate Sentry scope per request (#4372) ... ## Summary A query sent to the query API with a typo in it, like a column name that does not exist, was being reported as a server error. That put customer SQL mistakes into our error alerting, where they made up almost all of the volume on one of our noisiest alerts, and it drowned out the failures that are actually ours to fix. This makes the level match who is at fault, and fixes two related problems found alongside it. ## Invalid queries are the caller's, not ours The query API route already got this right. It checks for `QueryError`, logs at warn, and returns a 400, with a comment saying the system handles it gracefully and no alert is needed. The layer underneath ignored that. `executeTSQL` logged every exception out of its catch block at error, including the compile failures the route was about to turn into a 400, and error-level logs are forwarded to error reporting. The TSQL package already draws the line we need: ```ts export class ExposedTSQLError extends BaseTSQLError { /** An exception that can be exposed to the user. */ } export class InternalTSQLError extends BaseTSQLError { /** An internal exception in the TSQL engine. */ } ``` `SyntaxError` and `QueryError` extend the first. So the catch block now branches on `ExposedTSQLError` and logs those at warn, keeping error for `InternalTSQLError` and anything unanticipated, which is a genuine compiler bug. ## SQL the caller wrote is their mistake, not ours The same asymmetry showed up one level down. A query that compiles fine can still be rejected by ClickHouse at execution, and most of those rejections mean the caller's SQL is wrong rather than that we generated something bad. This is where the volume actually is. Checking production, one error group alone, a missing `GROUP BY` on the public query API (`NOT_AN_AGGREGATE`), accounts for over a million events across hundreds of users. It is by far the largest error group in the project, and classifying only by resource limit would have left every one of those at error level. So rejections are split three ways in `ClickhouseClient`, which is the only place holding the parsed `ClickHouseError` and its symbolic type. By the time the error reaches `executeTSQL` it has been wrapped and the type is gone, and the type never appears in the message text, so it cannot be recovered by string matching. - **Resource limits** (memory ceiling, timeout, row/byte caps) log at warn. The query is valid, it just asked for more than it is allowed to spend. - **Invalid SQL** (`NOT_AN_AGGREGATE`, `UNKNOWN_IDENTIFIER`, `SYNTAX_ERROR`, the type and parse families) logs at warn **only when the caller wrote the SQL**. - **Everything else** keeps alerting. That gate matters. The client is shared, so the identical rejection on TRQL *we* generated is our bug and has to stay at error. Callers opt in with `userAuthoredQuery`: | caller | who wrote the SQL | opts in | | --- | --- | --- | | public query API | the customer | yes | | query editor | the customer | yes | | agent charts | the agent's model | yes | | built-in dashboard tiles | us, in code | no | | queue metric cards | us, in code | no | | health report | us, in code | no | The agent is the one judgement call. Its TRQL is not typed by a person, but it is also not something a code fix makes correct, so a query it gets wrong is not worth waking anyone for. The same endpoint serves built-in tiles whose TRQL we do write, so the opt-in lives with the caller rather than the route. Separately, when one of these queries did fail, the log recorded the generated ClickHouse SQL but not the query the caller actually wrote, which made the reports hard to act on. `queryWithStats` takes an optional `logFields` that `executeTSQL` uses to attach the original TSQL. ## Events were attributed to the wrong request Chasing the above turned up something broader: only a tenth of the events on that alert pointed at the query API. The rest were pinned to unrelated requests that happened to be in flight at the same time, so the alert looked like the trigger endpoint was failing. `Sentry.init` runs with `skipOpenTelemetrySetup: true`, because we register our own OTel pipeline. That skips `initOpenTelemetry`, and one of the things it does is: ```js api.context.setGlobalContextManager(new SentryContextManager()); ``` The async-context strategy is still installed, but `withIsolationScope` only marks the OTel context and delegates the actual fork to that context manager: ```js // "We depend on the otelContextManager to handle the context/hub" return api.context.with(ctx.setValue(SENTRY_FORK_ISOLATION_SCOPE_CONTEXT_KEY, true), ...) ``` `provider.register()` installed a plain `AsyncLocalStorageContextManager`, which does not know that key. The lookup found no scopes on the context and fell back to the process-global default isolation scope, so every request wrote its request data into the same object and the last writer won. The tracer now registers `SentryContextManager`, which subclasses `AsyncLocalStorageContextManager`, so OTel behaviour is unchanged. It is also registered on the path where tracing is disabled, which previously never called `register()` at all and so had no context manager of its own. Tenant tags were always correct, because those come from our own async local storage rather than the isolation scope. That is why the attribution being wrong was not obvious. This affects every error report the webapp sends, not just the query API. ## Verification `internal-packages/clickhouse`: 76 tests pass, including eight covering each level decision against a real ClickHouse container. Three pairs pin the gate open and shut at both layers: an invalid query, a compile failure, and a real limit breach driven with `max_rows_to_read` each log at warn with `userAuthoredQuery` and at error without it. The isolation fix has a test that reproduces the leak before asserting the fix. Two overlapping requests each tag their own isolation scope; with the plain context manager the slower one reads back the other's tag, and with `SentryContextManager` each reads back its own. Measured separately against a faithful reproduction of the server's wiring (own OTel pipeline, CommonJS entry) at 200 concurrent requests: per-request attribution goes from 0.5% to 100%, while span nesting, context propagation across awaits, and distinct trace IDs are identical before and after.	2026-07-30 09:04:15 +01:00
Chris Arderne	a81ad4949c	feat(database,rbac): add multiple environment API key foundations (#4388) ... Adds the storage model and authorization contracts needed for multiple environment API keys. Credentials are represented by hashed values, revocation and expiration state, and persisted effective scopes. The built-in authorization fallback exposes full-access policy preparation, while optional authorization extensions can supply additional presets and task-aware scope generation. This change does not create, display, or authenticate additional keys.	2026-07-29 16:24:00 +00:00
Eric Allam	4eb9292cbe	feat(webapp,run-engine): queue metrics and health dashboard (#4131) ... ## Summary Three related changes, each independently gated: **Queue metrics and health.** Per-queue depth, throughput (enqueued, started, completed), concurrency, whether a queue is throttled, and scheduling delay (how long a run waits between becoming eligible and actually starting), plus a per concurrency-key breakdown for keyed queues. Collected from inside the run queue itself, stored in ClickHouse, and surfaced on the Queues list, a new per-queue detail page, the task pages, and the run inspector. The question it answers is "does this queue have enough concurrency to keep up, and if not, which key or which limit is the constraint". **Percent-based queue concurrency limits.** A queue's concurrency override can now be expressed as a percentage of the environment limit, stored as the source of truth and re-materialized whenever the environment limit changes. Absolute overrides above the environment limit are now **rejected with a 400** instead of being silently capped, which is a behavior change on `POST /api/v1/queues/:queue/concurrency/override`. **The `health` report.** A server-computed verdict on whether work is flowing, whether the runs that do start are healthy, and whether telemetry is fresh, rendered as text with sparklines. Available as `GET /api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool (plus a `report` MCP prompt, which shows up as a slash command in hosts that support prompts). With the flags off, the Queues page renders the pre-metrics component verbatim, nothing is emitted, and nothing is written to ClickHouse. ## Configuration Two independent gates, on purpose. Emission is global so data accrues for everyone before anyone can look at it; the view is per organization so it can be turned on for one org at a time without a deploy. **Runtime flags (no restart)** | Flag | Store | Gates | | --- | --- | --- | | `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by default) | All emission, gauges and counters. Cached in-process for 10s with stale-while-revalidate, warmed eagerly at boot so the first op after a deploy is not dropped. | | `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` | Fraction of queue ops that emit a gauge. Counters are never sampled, so throughput stays exact at any rate. | | `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag` row, per-org `Organization.featureFlags` override wins | Whether an org sees the metrics view at all: the Queues list variant, the queue detail route, the built-in Queues dashboard, the concurrency-keys endpoint, and the metrics blocks on task pages and the run inspector. Off by default; a gated org gets a 404 on the detail route rather than an empty page. | Both Redis keys are readable and writable from `/admin/queue-metrics` (super-admin UI, with a live per-shard stream-health table) and `GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface uses its own Redis client, so it works on any instance regardless of whether that instance runs the emitter or the consumer. **Environment variables (boot time)** | Variable | Default | Notes | | --- | --- | --- | | `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and injects it into the run engine. Without it the run queue has no emitter at all. | | `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on this instance. Independent of emission, so consumers can be sized separately from the API. | | `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per queue. | | `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals insert batch, so an ack can never outrun a write. | | `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` | falls back to the run-queue Redis | Set `HOST` to move the metrics stream onto a dedicated instance so a metrics backlog cannot compete with the run queue for memory. Self-hosters can leave it unset and get a single-Redis deployment. | | `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000` dedicated | Bound on how much a stalled consumer can hold. The default is deliberately lower when the stream shares the queue-critical Redis. | | `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the per-queue cumulative counter key, refreshed on every write, so only queues idle for the whole window are purged. | | `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue names tracked per environment; overflow collapses into `__overflow__`. | | `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea one level down, per queue. | | `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live sample-rate key above. | | `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics tables in the Query page, its schema docs, the schema API and the AI query context. Off keeps them unlisted while the feature is dark; a query naming them still runs either way. | | `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring | Runs queue metrics on their own ClickHouse service: the consumer's inserts and every queue-metrics read go through it, so a metrics-heavy chart refresh never competes with runs-list or trace reads. Unset reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`, reads on the query pool). | | `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split, so the consumer's inserts can never land on a read endpoint. | | `QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}` | `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other per-workload ClickHouse clients. | Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a Postgres migration adding the nullable `TaskQueue.concurrencyLimitOverridePercent`. Both are additive. ## How collection works Queue operations produce two kinds of signal, and they have opposite failure modes, so they are handled differently. **Gauges** (queued, running, queue limit, env queued, env running, env limit, throttled, plus keys-with-backlog and worst-key wait on keyed queues) are read *inside* the same Redis script that performs the enqueue or dequeue, so the reading is atomic with the operation it describes rather than a racy follow-up read. The script returns them on its reply and the app forwards them to the stream. Gauges are sampled and drop-tolerant: they are aggregated with `max`, so a lost reading costs resolution, never correctness. **Counters** (enqueued, started, completed, plus nack and dead-lettered) are cumulative odometers. Each event increments a per-queue key on the metrics Redis and emits the absolute total, and ClickHouse takes the difference across buckets at read time. This is the important property of the design: a summed-delta counter undercounts permanently on any lost event, while a cumulative one self-heals, because the next surviving reading restates the whole total. Only bucket granularity can be lost, never the total. A queue returning after its odometer TTL expired restarts at 1 and reset detection handles it, which is safe precisely because expiry only spans a window with no activity. Both land on one sharded Redis stream. A consumer reads it with a consumer group, reclaims stale pending entries on a 15s interval rather than on every poll, maps one entry to one or two ClickHouse rows (whole-queue and, for keyed queues, per-key), and acks only after the insert lands. Each batch carries a dedup token derived from its stream-entry ids, and the target tables set `non_replicated_deduplication_window`, so a retried batch cannot double-count either the raw rows or the aggregates that hang off them. Consumer and emitter both emit OTel metrics (`queue_metrics.emitter.emitted`, `queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`); stream depth and group lag are the two worth alerting on, and `lag_unknown` exists because Redis can report a null lag after a trim, which must not be read as zero. ## Storage and read path `queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four aggregate tiers are materialized straight from raw, never cascaded off each other, each with a 30 day TTL: - `queue_metrics_v1`, 10 second buckets per queue, the default read path - `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and cross-queue ranking - `env_metrics_v1`, 10 second buckets per environment, queue-independent so it stays cheap at any range - `queue_metrics_ck_v1`, 10 second buckets per concurrency key Every tier is an MV from raw because the counter states do not survive a cascade: their merge is order sensitive, so a `-MergeState` chain off the 10s table inflates the result, and the same property means an aggregate state may only be merged inside one queue. That constraint is now enforced by the query engine rather than by reviewer discipline: a column can declare a `mergeGroupKey`, and any query that references it without grouping by, or pinning to a single value of, every named key fails to compile with an actionable message. On the read side, TRQL gains three tables (`queue_metrics`, `env_metrics`, and a `queue_metrics_by_key` that is hidden from the editor, schema docs and schema API but still queryable, so per-key rows can never silently merge into a plain per-queue query), plus `deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level optimizations ride along: a table can declare coarser rollups, so a query whose bucket interval is 5 minutes or wider is routed to the 5m table with no change to the query itself, and it can opt into the ClickHouse query cache with time bounds floored to a fixed grid, so the auto-refreshing dashboards actually share cache entries instead of missing on every tick. Both are caller-side substitutions, so the printer stays unaware of physical layout. All of this can also live on its own ClickHouse service. A table declares the pool its reads run on, the three queue-metrics tables name the dedicated one, and the ingestion consumer writes through the same client, so both directions move together with one env var and nothing else routes differently. The other engine change is opt-in gap filling: charts can request rows for empty buckets, where counters zero-fill and gauges carry forward. Grouped gauge series are densified per group and carried inside a partition, so a quiet queue's line holds its last value without bleeding another queue's value into it. ## Queue concurrency limits `concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth when an override is set as a percentage; the absolute `concurrencyLimit` is materialized from it (floored, clamped to at least 1 so a percentage can never act as a pause, and never above the environment limit). Every path that changes an environment limit now recalculates the environment's percent-based overrides afterwards, outside the transaction, and pushes changed limits to the engine. The push is attempted even when the stored value did not change, so a previously failed sync self-heals rather than leaving the database and the engine diverged; paused queues are skipped so a recalculation cannot effectively unpause one. The API accepts exactly one of `concurrencyLimit` or `percent`, and the reject-instead-of-clamp change above means a request asking for more than the environment allows now fails loudly. The percent bound (greater than 0, at most 100) is defined once and shared by the zod schema, the dashboard mutation handler and the service, so the three cannot drift. The concurrency-keys table on a queue is now paginated against the ClickHouse per-key tier, ranked by peak backlog with the total on every row from a single scan, and only the keys on the current page are enriched with live counts from Redis. That replaces a hard top-50 cap with something whose cost is a function of page size rather than key cardinality. ## The health report `GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The verdict is computed on the server and is deterministic, not model-generated. Three independent analyzers run over one input snapshot: flow (is work moving, and if not, is the cause a limit, throttling, one bad queue, or dead-lettering), execution (are the runs that start succeeding, and at what latency), and liveness (how fresh is the telemetry). When telemetry is genuinely stale, the first two are forced to unknown and every actionable field is stripped, so no surface ever advises action off stale data. Authorization is per query table rather than a blanket query grant: a JWT must be scoped to every table the report reads (`runs`, `env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull a report that reads more than it was granted. `period` is validated as a shorthand with a 90 day ceiling at the edge. The report catalog is a registry of `{ load, interpret }` entries, so the next report is a new entry and no change to the route, the view model, the renderers, the CLI or the MCP tool. `trigger mcp` no longer launches the install wizard when stdout is a TTY, which fixed a real failure: hosts spawn the server over a PTY, so the wizard would open and the client would time out waiting for a server that never started. The wizard now needs `trigger mcp --install`. ## The part that is live regardless of every flag The enqueue and dequeue scripts now return a 2-tuple so a gauge reading can ride back on the reply. Every return site in the eight affected scripts is wrapped, and a `nil` original is converted to `false` on the way out, because a raw `nil` in the first slot would make Lua truncate the multi-bulk reply and silently drop the gauge on the throttled and empty-queue paths. The reply shape and the destructuring on the app side are exercised on every queue operation whether or not metrics are enabled, so that is the part of `run-engine` worth the closest review. One behavior fix in the same area: the scheduling-delay anchor is set only on a run's first entry into the queue. Anchoring it to trigger time on re-enqueues made waitpoint and checkpoint resumes report the entire wait as scheduling delay. Queue ordering is untouched, so a re-enqueued run keeps its position, and nacks deliberately keep the original anchor because a rolled-back dequeue is the same continuous wait. A pending-version promotion still anchors to trigger time, on purpose: that promotion is the run's first real entry into the queue, since the trigger deliberately held it back waiting for a worker version, and the TTL is armed at the same point for the same reason. The consequence is worth naming, because it is a judgement call: a run that waits on a deployment reports that wait as scheduling delay on its queue, which is time unrelated to queue capacity. ## Verification Unit and integration suites across the new package, the run queue, the mapping layer, the query engine and ClickHouse (including a test that applies migration 036 through the same splitter CI uses, and a regression test that inserts the same batch three times to prove the aggregates do not inflate). Beyond that, the whole path was driven end to end against a live stack with real runs: emitter to Redis stream to consumer to ClickHouse to the dashboards, for both the local dev path and the deployed path where a supervisor drives the dequeue, with assertions on exact counter reconstruction per queue and per concurrency key, throttling, environment saturation, scheduling delay, and a deliberate mid-stream reading drop to confirm the cumulative counters still reconstruct the correct total. The gated-off state was checked on every touched surface. The dedicated ClickHouse service was verified against a second, separately-schema'd instance: with it configured, the driven counters reconstruct exactly on the dedicated instance, the shared instance gains no rows for that window, a read through the query API returns the value that exists only on the dedicated instance, and a `runs` query still succeeds (it would fail outright if it were mis-routed to a service without that table). With the variable unset, the full suite passes unchanged. --------- Co-authored-by: Katia Bulatova <katia@trigger.dev> Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com> Co-authored-by: James Ritchie <james@trigger.dev>	2026-07-29 16:45:24 +01:00
claude[bot]	ec562c0e68	fix(webapp): remove unused Electric sync trace routes (#4400) ... <!-- ccr-slack-attribution --> _Requested by **Eric Allam** · [Slack thread](https://triggerdotdev.slack.com/archives/C0AU83M3136/p1785222101937829?thread_ts=1785207509.304669&cid=C0AU83M3136)_ Removes two dead Remix routes and the helpers only they used. `app/routes/sync.traces.runs.$traceId.ts` (`/sync/traces/runs/:traceId`) and `app/routes/sync.traces.$traceId.ts` (`/sync/traces/:traceId`) were added with the original ElectricSQL run page and lost their only consumers when the dashboard hooks that called them were deleted. Nothing in the repo references either route today. Also removed, because the deleted routes were their only callers: - `OtelTraceIdSchema`, `RESERVED_ELECTRIC_SHAPE_PARAMS`, `TraceScope`, `buildElectricTraceWhereClause` from `app/v3/electricShape.server.ts` (the file stays — `UNSAFE_REALTIME_TAG_CHARS` / `sanitizeRealtimeTagForSql` / `sanitizeRealtimeTagsForSql` are still used by `realtime.v1.runs.ts` and `realtimeClient.server.ts`) - the loader-specific cases in `apps/webapp/test/spanTraceRoutes.replicaLag.test.ts` and `internal-packages/run-store/src/runOpsStore.routesSpanTraceReadView.replicaLag.test.ts` `app/utils/longPollingFetch.ts` is untouched — `realtimeClient.server.ts` still uses it. `runOpsStore.ts` / `PostgresRunStore.ts` are untouched too; the unrouted-lookup mechanism there is generic and stays. As a plain code fact: the run lookup these loaders performed keyed on `TaskRun.traceId` alone, which is not an index-backed query shape. That is noted only as context for why the code is not worth keeping around unused. ### Judgement call worth a maintainer's opinion The request was specifically about `/sync/traces/runs/:traceId`, the route that looks up a run by `traceId`. This PR **also** deletes its sibling `/sync/traces/:traceId`. The reasoning: - both routes came in with the same ElectricSQL run-page work - both lost their only consumers in the same later commit - neither has any caller anywhere in the repo - they share the same helper module, so keeping one means keeping the helpers half-used If you would rather keep the sibling, reverting just that one file deletion is easy and does not affect the rest of this PR — say the word and I will restore it along with the helpers it needs. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Verification run locally from the repo root: | Command | Result | | --- | --- | | `pnpm run format` | clean, no changes produced | | `pnpm run lint:fix` | clean | | `pnpm run lint` | pass (exit 0, no findings) | | `pnpm run typecheck --filter webapp` | pass | | `pnpm run typecheck --filter @internal/run-store` | pass | A ripgrep sweep for `sync.traces`, `sync/traces`, `syncTraceRunsLoader`, `buildElectricTraceWhereClause`, `OtelTraceIdSchema` and `RESERVED_ELECTRIC_SHAPE_PARAMS` (excluding `node_modules`) returns zero hits. **Not fully verified:** both edited test files are testcontainers suites and need a Docker runtime, which was not available in my environment. I confirmed each file *collects* correctly with exactly the three intended remaining tests and no import errors — notably, dropping the `session.server` / `controlPlaneResolver.server` / `longPollingFetch` / `env.server` mocks does not break module loading for the surviving loaders. The assertions themselves then failed only on `Could not find a working container runtime strategy`. CI should be the real signal here. Per `apps/webapp/CLAUDE.md`, `pnpm run build --filter webapp` was deliberately not run. --- ## Changelog Removed two unused sync routes left over from the original ElectricSQL run page, along with the helpers and tests that existed only to serve them. No behaviour change — neither route had any caller. --- ## Screenshots _n/a — no user-visible surface changes._ 💯 --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-28 09:00:48 +01:00
Eric Allam	fc576436e2	perf(run-ops-database): index BatchTaskRun for the batches list on the dedicated schema (#4396) ... ## Summary The batches list page orders by `(createdAt DESC, id DESC)`, which is why [#4361](https://github.com/triggerdotdev/trigger.dev/pull/4361) added a matching index on `BatchTaskRun`. That index only landed in `@trigger.dev/database`. The dedicated run-ops database has its own migration history, so it never received the index. `BatchListPresenter` reads both databases and merges, so for environments whose batches live in the dedicated database the page kept falling back to a scan and in-memory sort, which is the exact behaviour #4361 set out to fix. ## Fix Adds the index to the run-ops schema with its own migration. `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it is a no-op where the index already exists and still records its ledger row. The second half is the interesting part. Because the two packages own separate migration histories, a run-graph schema change has to be authored twice, and nothing made the miss visible: the run-ops status check truthfully reports "up to date" against its own history, so the apply step just skips. `schemaParity.test.ts` compares the physical shape of every model the run-ops schema declares against its counterpart in `@trigger.dev/database`: scalar fields with their attributes, plus `@@index`, `@@unique`, `@@id` and `@@map`. Relation navigation fields are excluded, since the run-ops schema deliberately drops relations that would cross a database boundary while keeping the scalar FK column. A field counts as a relation when its type resolves to a model name, which keeps enum-typed columns in scope. Two models are listed as run-ops-only: `CompletedWaitpoint` and `WaitpointRunConnection`, both explicit FK-free replacements for a control-plane implicit many-to-many, since an implicit m2m carries a foreign key that cannot resolve across databases. The test also asserts that exception list is exhaustive, so a new unpaired model fails rather than being silently skipped. Confirmed the guard actually fails: reverting the index turns `BatchTaskRun` red with the missing `@@index` named in the diff.	2026-07-27 16:59:43 +01:00
Eric Allam	9c85e0ecdc	perf(database): index BatchTaskRun on (runtimeEnvironmentId, createdAt, id) for the batches list (#4361) ... ## Summary The batches list page orders by `createdAt DESC, id DESC` filtered by environment and a created-at window, but the only supporting index on `BatchTaskRun` was `(runtimeEnvironmentId, id)`. That index can't satisfy the `createdAt` ordering, so on environments with a large number of batches the query fell back to a full table scan and in-memory sort, which could run long enough to hit the statement timeout. ## Fix Adds `(runtimeEnvironmentId, createdAt DESC, id DESC)` on `BatchTaskRun`. The query now reads straight from the index in order with no sort step, returning a page with only a handful of heap fetches instead of scanning the whole environment slice. The migration uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it takes no table lock and is a no-op if the index already exists.	2026-07-24 11:52:34 +01:00
Eric Allam	e9ac98b7a1	perf(run-store): route id-set reads to the owning store, not both DBs (#4342) ... ## Summary The split run-store's id-set read path (`#findRunsByIdSet`, used by the runs-list hydrate, the realtime hydrator, and engine sweeps) queried the new store for the entire id set and then probed the legacy store for the misses. A run's residency is a total function of its id (run-ops ids live in the new store, every other id in legacy), so each id belongs to exactly one store. Route each id to its owner and query each store only for its own ids, in parallel. Same result set, and while a split is active with most runs still on legacy it removes a wasted new-store query from every id-set read. ## Change `#findRunsByIdSet` now partitions the ids by `classifyResidency` and runs one bounded query per store (skipping an empty side), in parallel, mirroring `expireRunsBatch` and the single-run `#route`. `finalizeRows` still applies orderBy/take/skip globally over the merged set. This drops the id-set path's cross-store fallback, which existed to prefer the new-store copy when the same id was present in both stores. That collision cannot arise when each id maps to exactly one store (nothing writes a legacy-shaped id into the new store), so the fallback is dead code. The two id-set tests that asserted "new copy wins on collision" now assert the routing invariant: a legacy-shaped id resolves to the legacy store and the path never consults the new store. The open-predicate path (`#findRunsOpen`) is unchanged: an open `where` has no id to route on, so it still unions both stores and dedupes.	2026-07-22 23:03:24 +01:00
Chris Arderne	6642c8b785	fix(webapp): prevent duplicate envs from provision race (#4261) ... Fixes TRI-12078 ## Summary Prevents concurrent environment setup requests from creating duplicate Staging and Preview environments. ## Fix Adds database-enforced uniqueness for root Staging and Preview environments. If two requests race, the losing request loads the environment created by the winner and continues successfully instead of creating a duplicate or returning an error.	2026-07-21 15:40:48 +01:00
Katia Bulatova	d05f1a7398	chore(webapp): migrate from Remix compiler to Vite (#4188) ... Replaces Remix compiler with the Vite plugin. The Express server (cluster, socket.io, ws) and the Docker image contract are unchanged.	2026-07-21 15:57:13 +02:00
Chris Arderne	dc87b884e7	chore: upgrade to typescript 6 (#4310) ... ## Summary Upgrades the workspace to TypeScript 6.0.3 and applies the compiler, type, and build configuration changes required to preserve package layouts and existing runtime behavior, apart from correcting the HTTP status field used for deployment connection errors. ## Compatibility - Centralizes TypeScript 6.0.3 through the pnpm workspace catalog. - Replaces compiler options and module resolution modes that TypeScript 6 no longer accepts. - Restores explicit Node types where TypeScript 6 no longer includes them transitively. - Adds explicit declaration build roots that preserve each package's existing output layout. - Patches tsup to stop injecting the removed `baseUrl` option during declaration builds. - Uses type-only assertions for stricter typed-array and stream definitions without changing runtime behavior. - Reads the EventSource v3 HTTP status from `code`, so deployment connection errors include it correctly. - Keeps standalone CLI compatibility fixtures pinned to their existing TypeScript version and lockfiles. `turbo run typecheck` and the complete PR test suite are green.	2026-07-21 13:57:52 +01:00
Chris Arderne	6997aeb05e	fix: security release 2026-07-08 (#4316)	2026-07-21 12:00:58 +01:00
Daniel Sutton	a7c734c223	test: caller-driven replica-lag + idempotency guards (stacked on #4284) (#4285) ... ## Stacked on #4284 — tests only This PR contains **only the tests** that guard the production fixes in #4284 (its base). Review #4284 first; this branch adds no production code. ## What Caller-driven replica-lag and idempotency guards for every fixed site: - Each guard **drives the real exported caller** (route loader/action, presenter `.call()`, service, or engine method) against a **real Postgres** with the owning replica frozen via the shared `laggingReplica` testcontainer primitive — never a store-seam reimplementation. - For a **fixed** site the guard goes **RED when the production change is reverted**; for a **tolerated read-view** site it's a caller-driven **GREEN** proof the miss self-heals (returns null/empty, no mutation, row live on primary). - The **global-scope idempotency** guard drives the real dedup + claim path through a **real `MollifierBuffer` over a Redis testcontainer** (real SETNX/poll/publish), and covers the cross-DB **andWait** waitpoint wiring and the **expired/failed clear-and-recreate** reacquire cases. Run with `vitest --no-file-parallelism` (testcontainers). Verified GREEN, and revert→RED verified per fixed site.	2026-07-19 17:06:45 +00:00
Daniel Sutton	ae96b6c175	fix: read-your-writes + global-scope idempotency correctness under the run-ops split (#4284) ... ## What & why Two related correctness fixes for the run-ops DB split. Under the split, run-store reads can route to a **lagging read replica**; a just-written run/waitpoint/batch can then be missed, causing a wrong decision. **1. Read-your-writes → owning primary.** Surfaced first as an intermittent `wait.until({ idempotencyKey })` re-wait on retry. Auditing the run-store read surface found the same class at sibling sites (some gating mutations or returning spurious 404s, others tolerable/self-healing). Reads that must observe their own writes now route to the owning **primary** (`findRun`/`findWaitpoint`/`findBatchTaskRunByFriendlyId` → `*OnPrimary`, a primary re-read on a miss, or a retryable 404 where the SDK polls). Read-view reads stay on the replica. All additive — the happy path is unchanged. **2. Global-scope idempotency across the split.** A `global`-scope key carries no per-run salt, so the same `(env, task, key)` triggered concurrently from parents resident on **different** run-ops DBs could dedup-miss on each DB and create a duplicate (the per-DB unique index can't enforce cross-DB uniqueness). Such triggers (global scope, or scope-absent, while split is active) are serialized through the existing Redis idempotency claim, the loser resolves the winner by id across both DBs, and the claim is reacquired on the expired/failed clear-and-recreate path. `run`/`attempt` scope embed the run id and never contend. ## Stacked for review This is the **base** of a 2-PR stack, split so review is easier: - **This PR** — production code only (34 files). - **Stacked tests PR → https://github.com/triggerdotdev/trigger.dev/pull/4285** — the caller-driven guards (55 test files) on top of this branch. ## Validation Local run-ops split, **both 2-DB and 3-DB**, fresh boot on this branch: SDK canary 64/71 (only the known concurrency/input-streams/s3 failures), quarantine sweep **0 unexpected** (340 pass / 16 known / 4 local) in each topology, dashboard e2e 0 failed. No product regressions.	2026-07-19 17:57:41 +01:00
Daniel Sutton	821972176d	fix(run-store,webapp): correct split-database read routing, write residency, and batches list ordering (#4272) ... ## Summary Correctness and performance fixes for deployments that split run data across more than one database. Single-database / self-hosted deployments are unaffected (they collapse to a single read/write path). - **Batches list (dashboard):** for some organizations the Batches list could hide older batches or show them out of order. It now orders and paginates by creation time (with the id as a stable tiebreak), so every batch appears exactly once, newest first. The pagination cursor format changes; older in-flight cursors simply restart from the first page. - **Reads:** waitpoint and snapshot lookups that are keyed by a single run now read only the database that holds that run instead of querying both, removing redundant queries on hot paths (unblock, snapshot reads). - **Writes:** environment-scoped writes with no owning run (standalone wait tokens, waitpoint tags, idempotency-key resets) now land in the same database as that environment's runs, rather than defaulting to the other one. An idempotency-key reset also falls back to the other database when it matches nothing, so a reset still clears the key wherever the run actually lives. ## Notes Verified end-to-end against multi-database setups: run-keyed reads and env-scoped writes land on the correct database with no cross-database writes, and the batches list surfaces every batch in creation order. New tests cover the batches ordering/reachability and the write-residency routing.	2026-07-17 16:26:56 +01:00
Chris Arderne	d7ec75d5ad	feat(runtime): add experimental Node.js 24 and 26 task runtimes (#4085) ... ## Summary Adds experimental Node.js 24 and 26 task runtimes through the `experimental-node-24` and `experimental-node-26` config values. Existing runtime defaults and the `node`, `node-22`, and `bun` behavior remain unchanged. The unprefixed `node-24` and `node-26` config values remain unavailable until the runtimes are ready for general use. ## Design Experimental config values normalize to canonical runtime identifiers before build manifests are created, keeping deployment metadata and execution behavior consistent. Kubernetes task pods also use the runtime-default seccomp profile so modern Node.js versions fall back from io_uring to checkpoint-compatible system calls.	2026-07-16 12:19:03 +01:00
Eric Allam	43250522a5	fix(run-store): fix batch idempotency lookup on the dedicated run-ops store (#4271) ... ## Summary `batchTrigger` requests that set a per-item `idempotencyKey` failed with a 500 when the run-store is split across databases: the per-item idempotency lookup errored before any run was created. Batches without per-item keys, single `trigger` idempotency, and batch-level (`idempotency-key` header) idempotency were unaffected. ## Root cause `findRunsByIdempotencyKeys` built its `UNION ALL` of per-key point-lookups with `@trigger.dev/database`'s `Prisma.sql` / `Prisma.join`, then executed it on whichever store client it was handed. On the dedicated run-ops store that client is a *separate* generated Prisma client, and a `Sql` object from a different generated client is not recognized: the bare `$queryRaw(Prisma.join(...))` form dropped the query text entirely (`Argument \`query\` is missing`). The tagged-template form is no better here: joining nested `Prisma.sql` fragments across the two clients mis-numbers the bound parameters (`syntax error at or near "$1"`). ## Fix Build the lookup as a plain parameterized string and run it via `$queryRawUnsafe` with positional placeholders and bound values, so it no longer depends on which generated client executes it. The query text contains only static SQL and integer placeholders; every value (`runtimeEnvironmentId`, `taskIdentifier`, each key) is bound, so it is not a raw-interpolation site. Same per-key point-lookup shape as before, no change on the single-client path. Verified end-to-end against a bundled build with the run-store split enabled: before the fix, `batchTrigger` with a per-item key 500s; after, it returns the runs and dedups correctly across fresh, repeat, and mixed batches.	2026-07-15 19:36:12 +01:00
Chris Arderne	b902e65dfb	chore: standardise internal node on 24.18.0 (#4254) ... ## Summary Updates the internal development, CI, and runtime-image Node version to 24.18.0. SDK compatibility coverage continues to include Node 20, 22, 24, and 26. The Node type definitions and the package-manager lockfiles now resolve against Node 24 types.	2026-07-15 12:49:12 +01:00
Eric Allam	1ab5066ed0	perf(webapp,run-store): point-lookup batch idempotency keys (#4255) ... ## Summary Batch triggers that use per-item idempotency keys could take seconds instead of milliseconds when the target task had a large run history. This keeps the idempotency lookup fast regardless of how many runs a task has accumulated. ## Root cause The batch path checks which items already have runs by looking up their idempotency keys with a single `WHERE runtimeEnvironmentId = ? AND taskIdentifier = ? AND idempotencyKey IN (...)` query. On a very large `TaskRun` table Postgres underestimates the row count of a specific `(environment, task)` pair, so once the `IN` list grows past a handful of keys it stops doing per-key index probes and instead scans every run for that `(environment, task)` and filters the keys in memory. The cost is then flat and large regardless of how many keys are being checked, and a routine `ANALYZE` does not correct the estimate at that table size. ## Fix Look each idempotency key up on its own, batched into a `UNION ALL` of point lookups (chunked, run with bounded concurrency). Each branch is an equality on all three columns of the unique index, so the planner can only do a per-key index probe and can never fall back to the range scan. Same results, same columns, confined to the batch trigger path.	2026-07-15 08:25:41 +01:00
nicktrn	022e5c1ad0	chore(deps): pin transitive deps and upgrade nodemailer to 9 (#4243) ... Routine dependency maintenance. - Pin a few high-fanout transitive deps to current patched versions via `pnpm.overrides`: `form-data`, `ws`, `undici`, `hono`. Lockfile-only (no published-package dependency changes); net shrinks via dedup. - Upgrade `nodemailer` 8 → 9 in `internal-packages/emails` (private package). The SES transport already uses SESv2 and the `createTransport`/`sendMail` API is unchanged, so no code changes were needed. `@types/nodemailer` stays at 8 (no 9.x published yet; types are compatible). Verified locally: `pnpm i` clean; `pnpm run typecheck --filter emails` and `--filter webapp` both pass.	2026-07-13 15:49:27 +01:00
Daniel Sutton	bea7e2be90	feat(webapp,run-store): route run-graph reads and writes through the run-store router (#4237) ... ## Summary Run-graph data (runs, batches, waitpoints, and their related tables) can now live in a database separate from the control plane, with every read and write routed to the correct database by each run's residency. This makes reading and writing run data more reliable once the two are split, and is a no-op for single-database installs. ## Design - Run-graph table access goes through the run-store router, which selects the legacy or the new run-ops store per run instead of assuming one shared client. - The legacy run-ops client is now independently pointable, so legacy run data can be served from its own database (and replica) rather than the control-plane connection. - Run-graph writes go straight to the run-graph database instead of being forwarded through the control plane, and replication targets are split so runs in the new database still replicate to analytics without under-counting. - Read-through slots refuse the control-plane client, so a missing residency fails loudly instead of silently reading the wrong database. - Migration `20260710120000_drop_remaining_run_graph_seam_foreign_keys` drops the foreign keys that still crossed the run-graph / control-plane seam, which is what lets the two live in separate databases. The split stays off unless explicitly enabled and the two databases are confirmed physically distinct; startup fails closed otherwise. Verified by running the full dashboard end-to-end suite against both a single-database configuration and a three-database configuration (control plane, the new database, and a physically separate legacy database), with runs on both residencies. No misrouted reads in either configuration.	2026-07-13 13:54:54 +01:00
Eric Allam	5ba8557a51	chore(webapp,core): remove the end-of-life v3 (engine V1) execution stack (#4236) ... ## Summary v3 (the engine that ran the SDK v3 era, internally `RunEngineVersion.V1`) is end-of-life. Following the removal of the v3 execution apps ([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and the legacy dev websocket ([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this removes the remaining v3 execution stack from the server. Clients still on v3 (an old SDK or CLI that has not upgraded) keep getting a clear "upgrade to v4" response. Triggers, batch triggers, reschedules, and deploys that resolve to v3 are rejected with a graceful 4xx pointing at the migration guide, never a 5xx, so a stale client cannot affect server health. Self-hosted instances still running v3 should stay on the 4.5.x release line until they migrate. ## What is removed - The MarQS queue and its shared/dev queue consumers. - The v3 socket.io namespaces (coordinator, provider, shared-queue) and the v3 run lifecycle services (attempt, checkpoint, and batch-resume). - The graphile-worker background job system; all live jobs already run on `@trigger.dev/redis-worker`. - The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally, so the flag is gone. - Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace` subpath and the legacy socket message catalogs) and the now-dead MarQS environment variables. ## What stays The v4 engine is untouched. The graceful v3 rejection boundary stays, `determineEngineVersion` still detects a v3 project so it can reject it, and the batch service plus batch-completion worker stay for current clients. Live queue concurrency limits and metrics now read from the v4 run engine instead of MarQS, and a brand-new dev environment now defaults to v4. ## Dependency cleanup Removes webapp dependencies left unused by this change: `seedrandom` and `semver` (only the removed v3 code used them) plus a set that was already dead, their orphaned `@types` packages, and two dead files. Adds a `knip:deps` script and a `knip.json` config so unused dependencies can be found the same way going forward.	2026-07-13 11:32:06 +01:00
Daniel Sutton	c601739d35	perf(webapp,run-store): grouped run-ops reads + mint-kind flip grace (#4227) ... ## Summary Two threads on the run-ops split path. Read path: per-item run reads are batched into grouped queries, a waitpoint's connected-run reads are bounded, and the dedicated-schema relation hydrators fetch only the requested columns instead of whole rows. Retrieve also falls back to the other database when a routed read misses, so a run whose physical residency diverges from its id shape is still found rather than returning a spurious not-found. Fewer and lighter queries on the run read path, with no change to results. Mint-kind flip safety: flipping which database new runs mint to is now a deterministic wall-clock cutover, for both per-org and global flips. For a grace window every process resolves the same database, so a flip cannot route two concurrent triggers that share an idempotency key to different databases (which would bypass the per-database unique constraint and create a duplicate run). Supersedes the earlier #4205 and #4208. Draft: validation in progress.	2026-07-13 10:17:12 +01:00
Eric Allam	45527e317a	feat(webapp): opt-in worker pool for OTLP ingest transform (#4232) ... ## Summary Under high OTLP ingest volume, the whole decode, transform, and enrich pipeline runs on the request event loop, so a single CPU core becomes the ceiling while the rest sit idle. This adds an opt-in worker pool that moves decode, transform, and LLM-cost enrichment onto worker threads, keeping the main thread free for I/O. It is off by default (`OTEL_TRANSFORM_WORKER_POOL_ENABLED`), so behavior is unchanged unless enabled. ## Design Workers do decode, filter, convert, and enrich (including LLM pricing match). The main thread stays the single database reader: it loads the pricing registry and broadcasts the compiled model rows to the workers (re-broadcasting on every reload), so workers never touch the database. The pure transform is extracted into a dependency-light module (no Prisma/Redis/ClickHouse imports) so it can run inside a worker. Importantly, the main thread keeps the existing single consolidated insert path, so ClickHouse insert batching and part count are unchanged. The parallelism buys CPU headroom, not more insert streams (which would add merge pressure). The worker is bundled as a standalone file at build time and ships in the existing image with no Dockerfile change. In local load testing the pool sustained roughly 2.6x the throughput of the single-thread path and kept the main thread responsive under load.	2026-07-11 13:38:12 +01:00