docs-live
538 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Eric Allam
|
c0b84595a3 |
feat(webapp): hosted webhook ingress, delivery pipeline, and dashboard (#4344)
## Summary The server half of hosted webhooks: the public ingress endpoint, signature verification, the delivery pipeline (Postgres partitioned storage + ClickHouse for ordering), the in-app partition manager, the HTTP API, and the dashboard (Deliveries, Endpoints, and the in-app test console). The public SDK and docs half is #4537. That PR carries the user-facing API (`webhook()`, `chat.event` / `chat.channels`, the `@trigger.dev/slack` connector) and builds on the shared `@trigger.dev/core` schemas that ship here. ## Shipping behind a flag A `WEBHOOK_ENABLED` env var (default off) gates the public ingress route and the engine worker plus partition cron, so merging and deploying this changes nothing in production until it is flipped on per environment. The dashboard is separately gated per org by the `hasWebhooksAccess` feature flag. ## Note on packages This PR includes the `@trigger.dev/core` schema additions the server compiles against, but carries no changeset. Core is not consumed independently of the SDK, so it is released together with the SDK via #4537. Keeping its changeset off `main` means no release cut from `main` publishes it early. |
||
|
Chris Arderne
|
3e7964e7fa |
feat: surface cron windows in webapp, cli, sdk (#4572)
## Summary Adds execution-window product surfaces for both declarative and imperative schedules. - Declarative schedules can set `window` through `schedules.task()`, with support for whole-minute, hour, and percentage values. - Imperative schedules can create, update, clear, and inspect windows through the API and dashboard. - Schedule API responses preserve `nextRun` as the nominal CRON time and expose `nextRunEffectiveAt` as the stable assigned time. - The dashboard displays configured windows alongside assigned upcoming-run times. - Deploy output summarizes declarative schedules and suggests adding a wider window when the default 60-second placement range is used. ## Design Window validation remains authoritative on the server and ensures each window is compatible with the schedule cadence. Omitting a window uses the default 60-second range, while explicit zero-duration windows remain supported. Deployment summaries are derived from the deployment's stored task metadata, so they reflect the declarations associated with that deployment. |
||
|
Chris Arderne
|
d98f64bb00 | fix(webapp): hide misleading root API key creation dates (#4612) | ||
|
Eric Allam
|
eefe0a378d |
perf(webapp): bound environment loads in the env layout and batches list (#4606)
## Summary Follow-up to #4595. Dashboard pages under an environment loaded every environment in the project on each page just to resolve the one named in the URL. On projects with many preview branches that meant reading hundreds of (mostly archived) rows on every page load. ## Fix The environment-scoped layout loader now scopes its lookup to the slug in the URL (`where: { slug: envParam }`), resolving the current environment through the `projectId, slug` composite index instead of loading the whole project. Archived branches stay viewable by slug. `BatchListPresenter` is bounded to the current environment, since every batch in that list already belongs to it. Verified on a project seeded with 2,000 archived branch environments: the layout lookup drops from all environments to one, and both a normal environment page and an archived branch page render correctly. |
||
|
claude[bot]
|
f6f3b75547 | chore: remove obsolete v3/v4 version copy from the dashboard (#4589) | ||
|
Eric Allam
|
c6ef5f3959 |
perf(webapp): paginate the environment variables settings page (#4597)
## What
The environment variables settings page loaded **every** variable in the
project in one shot, with a nested `values` read plus a `valueReference`
(SecretReference) sub-load that was selected but never read. For a
project with many variables this pulled `variables × environments` value
rows (~18k for large projects) on every page load, plus a matching
~18k-row `SecretReference IN` query.
This paginates the presenter by variable key and removes the dead
include.
- Remove the never-read `valueReference: { select: { key } }` include →
the `SecretReference` query is gone entirely.
- Paginate the parent variable query: `count` + `orderBy key` +
`skip/take`, page size 50 → the value read is bounded to `pageSize ×
environments` per page.
- Scope the count and the page to variables that have a value in a
displayed environment (`values: { some: { environmentId: { in } } }`),
so `totalCount`/`totalPages` and the `skip/take` window match what
actually renders (no phantom empty pages from variables that live only
in archived branches or another member's dev env).
- Display order comes from the DB `orderBy: { key: "asc" }` — the
presenter no longer re-sorts each page with `localeCompare`, which under
pagination could disagree with the DB collation at page boundaries.
- The secret-value lookup (`SecretStore` keys) and the updater lookup
(`user` by id) are now scoped to the current page instead of the whole
project.
- Search moves server-side (variable key, case-insensitive) and drives
both the count and the page; the UI gains standard pagination controls.
## Why
The two correlated ~18k-row control-plane queries flagged in the ticket
come from this settings-page presenter, not from any hot path. Both are
index-covered (`rows_read == rows_returned`); the issue is the sheer
volume fetched in one burst. Bounding it per page removes the burst.
## Evidence
Measured on an isolated stack with a seeded project of 1000 variables ×
3 environments (3000 value rows), using Prisma's emitted-SQL log:
| | SecretReference query | value rows fetched |
| --- | --- | --- |
| before | 1 | 3000 |
| after | **0** | **150** (page 1) + one `count` |
`EXPLAIN` on Prisma's verbatim statements (index confirmed via
`enable_seqscan=off`; the local table is too small for the planner to
choose them by default):
- `count` (`WHERE projectId AND EXISTS(values in displayed envs)`) →
Hash Join: Index Scan on `EnvironmentVariable_pkey` + Bitmap Index Scan
on `EnvironmentVariableValue_environmentId_idx`
- paginated parent (`WHERE projectId AND EXISTS(...) ORDER BY key
LIMIT/OFFSET`) → Nested Loop Semi Join: Index Scan on
`EnvironmentVariable_projectId_key_key` (**no Sort node**) driving an
Index-Only Scan on
`EnvironmentVariableValue_variableId_environmentId_key`
- nested values (`variableId = ANY … AND environmentId = ANY …`) → index
scan on `EnvironmentVariableValue_environmentId_idx`
- `SecretStore` keys (`key = ANY …`) → index scan on
`SecretStore_key_idx`
No new index required. Verified in the browser on the seeded project: 20
pages, page navigation, server-side search (matches across all pages),
last page renders, no app console errors. `typecheck`, `oxlint`, `oxfmt`
all clean.
## Behavior change
The previous client-side search matched variable **name and value** (and
environment type / branch name). Values are encrypted at rest and
resolved separately, so they cannot be searched server-side under
pagination. Search is now **variable-name only**, server-side,
case-insensitive. Projects with fewer than one page of variables see no
pagination bar and no visible change.
## Rollout / rollback
Pure read-path change on a dashboard loader, no schema or data
migration. Rollback is a straight revert.
## Screenshots
<img width="2400" height="1794" alt="01-page1"
src="https://github.com/user-attachments/assets/d4a7effd-d167-4dd6-92f4-6e9174818acd"
/>
<img width="2400" height="1794" alt="02-search-single"
src="https://github.com/user-attachments/assets/cd113ca9-ff87-431f-b2f6-7f7d36f2b32a"
/>
|
||
|
Eric Allam
|
8d0f693186 |
perf(webapp): drop archived branch environments from project env loads (#4595)
## What
Several project pages loaded **every** `RuntimeEnvironment` row for a
project, including the archived preview-branch environments that are
never shown in the UI. On a project with heavy preview-branch usage that
means thousands of rows per load, producing a large result set and a
rare multi-second tail on the environment lookup (~30s outlier observed
via Insights on `RuntimeEnvironment` projectId lookups, fingerprint
`f2b3ecab…`).
The tail is dominated by the size of the result being
parsed/transferred, not by the query plan (it already used
`RuntimeEnvironment_projectId_idx` with no over-read). So the fix is to
stop returning archived branch environments.
## Diagnosis correction
The ticket framed this as a "large `projectId IN` list" and suggested
bounding the IN list / cursor pagination. It's actually a Prisma
**nested relation load** on a *single-project* `project.findFirst`, so
the `IN (...)` holds one projectId and the trailing `OFFSET $1` is
Prisma's relation-subquery artifact. The 4,644 rows in the observed
execution were **one project with ~4,644 environments** (accumulated
archived branches), not many projects.
## Change
Filter the `environments` relation load to `archivedAt: null` (base envs
never archive, so only archived preview branches are excluded):
- `ProjectPresenter.server.ts`
-
`orgs.$organizationSlug.projects.$projectParam.{concurrency,apikeys,environment-variables,settings}.ts`
(best-env resolvers)
And remove an **unused** `environments` select from
`DeploymentListPresenter.server.ts` (it was selected but never read).
`loadProjectEnvironments` (replay route) already filters `archivedAt:
null` + env type; this change follows that existing precedent.
## Evidence (isolated stack, seeded one project with 2,000 archived
branch envs + 4 active)
`EXPLAIN (ANALYZE)` of the exact presenter sub-select:
| | rows returned | index |
|---|---|---|
| before (unfiltered) | **2004** | `RuntimeEnvironment_projectId_idx` |
| after (`archivedAt IS NULL`) | **4** (`Rows Removed by Filter: 2000`)
| same index, no plan change |
500x fewer rows to the client, which is what removes the parse-on-load
tail. No new index needed. `typecheck --filter webapp` clean. UI
verified: project layout, Deploys page, and the concurrency best-env
redirect all render with the 2,000 archived branches present in the DB
and zero console errors.
## Rollout / rollback
Straight deploy, no migration. Rollback is revert-only (read-path
filter, no data change). Old and in-flight rows read correctly under
both the old and new code.
## Limitation
A project with thousands of *active* branches would still load them all;
in practice active branches are few (branches are archived when their
work merges). Hard-bounding active branches would be a larger change and
is out of scope here.
|
||
|
Eric Allam
|
db0ca9eb40 |
fix(webapp): drop unused OrgMember _count aggregate from org-list presenter (#4587)
## What `OrganizationsPresenter.#getOrganizations` selected a Prisma `_count.members` relation on every org-list load (hit on nearly every dashboard navigation). Prisma lowers that relation `_count` to a whole-`OrgMember`-table `GROUP BY organizationId` aggregate joined onto `Organization`. The computed `membersCount` field is read by **nothing** in the webapp, so the entire aggregate scan is wasted work. This removes the `_count` select and the `membersCount` field. The query keeps only the indexed `EXISTS` membership filter and the org/project selects. ## Why it's safe - `membersCount` has zero consumers (whole-webapp grep finds the name only at the point of assignment). It was added in #1796 (2023) and has been unused since. - The member count shown on the org settings/team page comes from a separate presenter query, not this one. No user-visible change. ## Evidence (generated SQL, before/after, seeded isolated stack) Before (with `_count.members`): ```sql SELECT ..., COALESCE(aggr._aggr_count_members, 0) FROM "Organization" LEFT JOIN (SELECT "organizationId", COUNT(*) AS _aggr_count_members FROM "OrgMember" GROUP BY "organizationId") aggr ON ... WHERE EXISTS (... "userId" = $1 ...) AND "deletedAt" IS NULL ORDER BY "createdAt" DESC ``` After: ```sql SELECT id, slug, title, avatar, "featureFlags" FROM "Organization" WHERE EXISTS (... "userId" = $1 ...) AND "deletedAt" IS NULL ORDER BY "createdAt" DESC ``` The whole-table `GROUP BY` aggregate is gone. The only remaining `OrgMember` access is the `EXISTS` on the caller's own membership (indexed by `userId`, a handful of rows). This is the single largest read-amplification query on the control-plane database (~1.39B rows read/day, ~719s DB CPU/day per Insights); removing it takes that portion to zero. Webapp typecheck passes. ## Rollout Straight deploy, zero blast radius. Rollback is a plain revert, no data migration. refs TRI-13170 |
||
|
Chris Arderne
|
ed1bb72fb8 |
feat: implement cron window spread backend (#4566)
- New DB fields on Schedule and ScheduleInstance - Use `queueTimestamp` for the "effectiveAt" delayed start time, propagate it to Clickhouse TaskRun table - Disable fastpath for delayed jobs - Add schedule timing logic, API endpoints with windows, persistence - Calculate phase for every schedule, only persist when window is non-null - Additional o11y for phased rollout |
||
|
Katia Bulatova
|
0b750d00dd |
feat(webapp): dashboard agent — Watch (#4525)
Watch is the agent noticing something later: you ask it to tell you when a condition holds, and it answers when it does — or when it can't any more. A watch is a **durable one-shot promise**. The condition is checked on a schedule by deterministic code (no LLM in the checks), the answer lands in the chat once, and then the watch is over. Ten kinds: three on a run, five on a queue, error recurrence, health recovery. ## Stack Stacked on **#4529** (UI), which is stacked on **#4418** (chat, reports, investigate). Merge those first. **#4516** (storybook gallery) sits on top of this branch. ## How to review [**GUIDEBOOK.md**](https://github.com/triggerdotdev/trigger.dev/blob/feat/dashboard-agent-flows-watch/internal-packages/dashboard-agent/GUIDEBOOK.md) on this branch is the behaviour reference — it states the conditions rather than the code, so you can predict what happens without running anything. "The ten watch kinds, and what makes each fire" and "Creating a watch" describe exactly this PR, and the tables there are the spec the code is written against. ## What's inside - **Ten watch kinds**, one deterministic check each (`dashboardAgentWatch*Checks.ts`), with the spec union in `dashboard-agent-contracts/src/watch.ts`. - **Scheduling** — each watch schedules its own next check; due watches of one `(environment, cadence)` group can be checked together in one batch pass, with a sweep as the backstop for expiry, redelivery and retention. - **Delivery** — the in-chat wake and card, an optional email alert (new `DASHBOARD_AGENT_WATCH` alert channel, so it shows on the project's Alerts page with one-click unsubscribe), and an optional investigation when the outcome needs attention. - **Submission ledger** — `watch_submissions`, keyed `(chat_id, client_request_id)`, so a retried card submission replays the recorded outcome instead of creating a second watch. - **Watch token** — a delayed-execution credential accepted only by the watch endpoints, re-checked against the user's live access on every tick. - **Unread work** — the panel polls for wakes that landed while it was closed, so a chat can go unread and light the launcher dot. ## Key decisions **A check result is a 4-way, and only two of them are verdicts.** `satisfied` / `terminal_unsatisfied` are answers; `pending` and `unavailable` are not. Any exception inside any check is caught in one place and becomes `unavailable` with an unverified observation — a check that failed is never evidence. **A completed window is an answer, and whether it is good or bad news is declared per kind, never inferred.** There is a table for that in the guidebook: `run_failed` completing its window is *good* news ("hasn't failed"), `backlog_drain` completing it is not. One rule overrides the table: a window that completed on an unverified observation is neutral and says only that the watch ended without a confirmed answer. **An unreadable source is never a negative answer** — and, because investigations only open on `attention`, it never starts one either. **Identity is `(chat, project, environment)` plus the condition,** enforced by a partial unique index over active rows (`watches_chat_active_identity_key`), not by the read-then-insert check. Cadence, window, note and `ticks` are deliberately not part of it. Two different chats may watch the same thing — a watch is a promise to a chat. **The server resolves the target's name, whatever the model calls it.** The model can't tell a task queue (`task/<id>`) from a custom queue, so both spellings are tried and the stored one wins — and the rewrite happens **before** identity and before the row is written, so the identity, the checks, the link and the wording all see one spelling. **Freshness fences.** Depth falls back from the live counter to the newest 60 s ClickHouse bucket, which only counts as current within 60 s of now. A non-current reading at or below the *quiet line* is refused as `unavailable` rather than believed, so a stale empty bucket is never read as "drained". The stall streak is the one piece of carried state: it lives in the previous check's facts and *freezes* on an unreadable reading rather than breaking. **Chain reliability.** There is no shared cron — each watch (or batch group) schedules its own next tick, so the failure mode to review is the chain dying. A failed batch check is caught, the next tick is scheduled anyway and the run resolves rather than failing, so the chain survives a check that couldn't run; the sweep re-arms groups and finalizes anything still active past its deadline, even when delivery isn't configured. Wake redelivery is id-deduped rather than conditional, because the sweep can't know whether the user was already told. Access is re-authorized on **every** check against the primary — replica lag would extend access the user has already lost. **Wording lives in one place.** `watch-wording.ts` is read by the card, banner, toast, email and the agent's own narration, and the numbers come from the frozen observation rather than a fresh read, so a retry produces the same sentence. Replay reproduces the **recorded** decision instead of deciding again — the transcript is append-once, so a second decision would contradict it forever. **Cancellation is the ending without an answer** — no resolution, no wake. One exception, decided during testing: a watch the *user* cancelled leaves a single neutral transcript line ("Stopped watching …"), keyed off the watch id so a retry can't repeat it. The other four reasons stay silent. **Email is opt-in and only a fired watch emails.** An expiry is narrated in the chat and nowhere else. Both gates (agent access, a configured email transport) are checked at subscribe time *and* again at delivery, and the subscription outcome is frozen on the ledger row so a retry replays it. Neither gate is a plan check. **One watch offer per turn.** The prompt and the renderer guard this independently — if the turn already proposed a watch card, the action button is dropped, because the card is the better affordance. Two eval cases pin the prompt side: exactly one offer with the line last and the button after it, and zero offers when the rendered card already carries one — deterministic assertions, over a real-model run. ## Testing Unit tests (vitest, testcontainers, no mocks) under `apps/webapp/test/dashboardAgentWatch*.test.ts` and `internal-packages/dashboard-agent/src/watch-*.test.ts` cover the invariants above: the 4-way check results and the freshness fences, identity/dedup and the submission ledger, queue-name resolution, the batch chain surviving a failed check, sweep boundaries and alert-once, tenancy and the watch token's scope, and the wording snapshot. The load-bearing ones were verified by control-breaking the guard first and checking the test goes red. Live-tested end to end against a local stack, following the guidebook: all ten watch kinds firing and expiring, cancellation, the email pair (a fired watch mails, an expired one does not), and watch recovery from a health report. |
||
|
Katia Bulatova
|
4569657923 |
feat(webapp): dashboard agent — chat, reports, investigate (#4418)
## What & why This is the system behind the Dashboard Agent — an assistant that answers questions about a project's runs, errors, queues, deploys and health, and can investigate failures end to end. The agent runs as a chat.agent task in its own Trigger project. It has no access to the main database or ClickHouse; all platform data is read through the public API using a delegated, read-only user token. Everything here is behind `canAccessDashboardAgent` and inert with the flag off. The UI that mounts the panel lands in #4529. ## Stack `#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook gallery. The scenario/contract reference for the whole stack is `internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch branch): it states, per feature, what makes each thing happen and where that is decided. ## What's inside **Agent runtime and tools** — `internal-packages/dashboard-agent`: prompt, tool set (API reads, TRQL query, docs, navigation, evidence/investigations, repo source), conversation compaction, a prompt-prefix token budget pinned by snapshot test, and sampled LLM-judged turn evals. The package cannot import webapp server code, which is what makes the "no DB access" claim structural rather than a convention. **Contracts** — `internal-packages/dashboard-agent-contracts`: `trigger://` URIs, intents, and the block envelope every rendered card travels in. **Conversation store** — `internal-packages/dashboard-agent-db`: drizzle over postgres-js in its own `trigger_dashboard_agent` Postgres schema, plus one additive migration. **Auth boundary** — the user-actor token gains an optional environment claim; one guard (`userActorEnvironment.server.ts`) enforces it so routes don't each re-derive the rule. Token minting, cap ceiling, and the RBAC fallback path for self-hosted. **Transport** — webapp resource routes that mint the token and proxy each turn, and SDK-side mid-turn reconnect. **Public API the agent reads through** — orgs, projects, environments, runs, queue metrics, workers, a run's commit metadata, repo snapshot, reports, and `POST /api/v1/query`. **Reports** — the health report's layout is declared once and shared by the card, the markdown surface and the JSON/MCP surface, so the same report reads the same in the dashboard, the terminal and an editor. **Block renderers** — the report and investigation cards the flows above already emit (`app/components/dashboard-agent/`). The panel that hosts them, and the rest of the chat UI, is #4529. **Query safety and CSP** — see below. ## Key decisions - **The agent is a separate Trigger project, not webapp code.** It reads platform data over the public API with a delegated user-actor token whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no webapp imports. - **The PAT-only auth helper now refuses user-actor tokens.** This is an intentional behavioral change: its callers consume only a bare userId and do not enforce delegated-token capabilities. Actor-aware routes continue through the scoped route builders instead. - **RBAC fallback builds a delegated token's ability from its own cap**, never the blanket ability a PAT gets (read-only when the token declares none). Without this, the agent's read-only cap would buy a write JWT on self-hosted. - **Org creation checks RBAC only for user-actor tokens, and only after the env gate**, so an install with `ORG_CREATION_API_ENABLED` off returns 404 rather than 403, and an ordinary PAT never consults an ability the route has no org to scope. Both orderings are pinned by test. - **The query path is read-only in depth.** TRQL rejects write statements at the grammar level (they don't parse, rather than being filtered), ClickHouse runs with `readonly=1`, and the org/project/env filters are injected server-side from the credential — the request body cannot widen scope. An unparseable query denies instead of falling through to the permissive resource. - **Document-wide img-src CSP.** Remote images are an outbound-request/exfiltration surface, so the policy permits only own-origin/data/blob, the required SSO avatar hosts, and the favicon endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST; wildcard hosts and bare schemes are intentionally not allowed. - **The chat transport reconnects on a mid-turn EOF** (`@trigger.dev/sdk`). A body that ends without a turn-complete is terminal only when the server says `X-Session-Settled: true`; otherwise the transport resubscribes from `lastEventId` with bounded backoff, and any record re-earns the budget. Previously a closed long-poll window or a proxy restart left the reply stuck as if still generating. - **Conversations live in their own datastore**, schema-scoped and foreign-key-free (it references `organizationId`/`userId` by id, because in cloud it is a different database). It is a display read-model for the History tab and transport resume; `chat.agent`'s object-store snapshot remains the model's source of truth. - **Deterministic first.** Reports and health checks contain no LLM — they are computed from the same data the dashboard shows, and the model only narrates and links them. That is what makes a number in an answer auditable. ## Testing - 63 new test files, run with `pnpm run test --filter webapp` and per-package vitest. Heaviest coverage on the auth boundary (`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`, `contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the report layout, and the SDK reconnect. - The agent package has a separate eval lane (`pnpm run test:evals`, `vitest.eval.config.ts`) that hits the real model, so it never runs in `pnpm test`. - Live-tested against a local stack scenario by scenario; the GUIDEBOOK lists the condition each behaviour is expected under, which is what those runs were checked against. ## Changelog `.server-changes/dashboard-agent.md`, plus changesets for `@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat reconnect) and the CLI's `mint-token` help text. |
||
|
Eric Allam
|
6449a644b9 |
feat(webapp,cli,database): track real dev onboarding progress (#4563)
## Summary
The dev environment "Get set up" panel used to be a static list of CLI
commands that only disappeared once your tasks registered, so nothing
ever changed after you ran `init` and people assumed it was stuck. It
now tracks real progress: `trigger init` records the project as
initialized, so step 1 checks off, and the panel updates live as the dev
server connects and your tasks register.
It also adds a prominent "Copy AI agent prompt" button, presented as a
clear alternative ("or") to the manual CLI steps, that copies a
ready-to-paste setup prompt pre-filled with your project reference for
Claude Code, Cursor, or any coding agent.
## Notes
- Adds a `Project.initializedAt` column (migration
`20260811065646_add_project_initialized_at`); the CLI `init` command
calls a new project-scoped `POST /api/v1/projects/:ref/init` best-effort
at the end of setup.
- The `init` scaffold now imports from `@trigger.dev/sdk` instead of the
deprecated `/v3` subpath.
## Screenshots
<img width="2400" height="1794" alt="v7-redesigned-card"
src="https://github.com/user-attachments/assets/c2fb4fa1-9484-4700-8bd3-110d66f5a44e"
/>
|
||
|
Katia Bulatova
|
c00fb9c36c |
fix(webapp): report start latency as unknown when there is no data (#4544)
When the health report had no start-latency measurement for the window, it printed a confident "p95 0ms" and graded it healthy. It now shows "unknown" for that metric and skips grading it, so an absent measurement can't read as a green signal. A genuinely measured 0ms is still shown as 0ms: the loader keeps "no measurement" distinct from a measured zero instead of coercing both to 0. |
||
|
Eric Allam
|
c526528d8f |
feat(webapp,database): bound Prisma list filter arity (#4480)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
## Summary Prisma expands `in` / `notIn` into one bind parameter per element, so every distinct list length is a separate prepared statement. Where the length tracks data volume (a batch size, a run-graph fan-out, a prior query's id set) one call site can mint hundreds of them. Each is used about once, but inserting it evicts an entry that was being reused, so the cost lands on unrelated queries sharing the pooler's statement cache. An unbounded list also risks the 65535 bind-parameter ceiling. `boundedIn()` pads a filter list to the next power of two by repeating its last element. `IN` and `NOT IN` ignore duplicates, so results are unchanged, and a call site drops from one statement per length to at most `log2(cap)`. Applied to all existing sites. ## Enforcement Two oxlint rules require the helper: a list filter must be an inline array literal or a `boundedIn()` call. - The first covers filters reached through `where` / `having` / `cursor`, and deliberately never descends into `data`, `create`, `update`, `set` or `equals`. A key named `in` in those positions is user data, not a predicate, and rewriting it would corrupt what gets stored or compared. - The second covers bare filter objects passed to where-building helpers, which the first cannot see. It found five sites in the run-graph batch loaders that were otherwise invisible. Both rules follow filters through the shapes they are actually written in: conditional expressions, logical-and objects, spread-conditional properties, computed keys, and call arguments. An array literal only counts as fixed-arity when nothing spreads into it, since `[...new Set(ids)]` has a runtime length. Twelve sites were hidden behind those shapes until the rules handled them. Scoped to `in` and `notIn`. The scalar-list filters `hasSome` and `hasEvery` compile to `&& $1` and `@> $1`, passing the whole array as a single bind parameter, so their arity never reaches the statement text and there is nothing to bound. Both rules are `error`, so new call sites fail CI. That ratchet has already caught four sites added by other PRs while this one was in review. ## Notes `boundedIn` pads by repeating rather than with null: `x NOT IN (a, b, NULL)` is never true, so null-padding a `notIn` filter would silently return no rows. Lists above 32768 are returned unchanged so padding can never push a query past the parameter limit. Route modules reach the helper through `~/db.server` rather than importing the database barrel directly, since a value import of that barrel into a module that also exports a React component is only safe while dead-code elimination prunes it. Measured on a local rig: 300 distinct list lengths produce 300 prepared statements unpadded, 10 padded. Verified end-to-end against a local stack with the full task-suite sweep, which surfaced no regressions. |
||
|
Chris Arderne
|
088f68b373 |
feat(webapp): share rate limit bucket across additional API keys per environment (#4508)
## What
Rate-limit the API by **environment** rather than per API key.
Previously the limiter keyed its bucket on the hash of the full
`Authorization` header — one bucket per key. With additional environment
API keys (`tr_*_sk_*`), an environment can mint many keys and each got
its own full bucket, so more keys = higher effective rate limit. This
collapses all of an environment's keys onto a single shared
per-environment bucket, so the ceiling is exactly the configured limit
regardless of key mix.
## How
- `authorizationRateLimitMiddleware` now lets the override return `{
config?, identifier? }`. `identifier`, when present, is the rate limit
bucket key; otherwise it falls back to the hashed `Authorization` header
(unchanged legacy behavior, still used by `engineRateLimiter` and any
unauthenticated fallthrough).
- `apiRateLimiter`'s override resolves the environment id and uses it as
the identifier:
- **Additional keys** (`isAdditionalApiKey`) resolve via a new
`resolveAdditionalApiKeyRateLimitScope()` — a **scope-agnostic** keyHash
→ (environmentId, org limiter config) lookup. It is deliberately
permissive (restricted keys resolve too) because it's used **only for
bucketing, never as an auth decision** — request auth still goes through
the RBAC bearer controller, which enforces scopes. Revoked/expired keys
are excluded so they can't hold a bucket warm.
- **Root/legacy keys** reuse the environment already resolved by
`authenticateAuthorizationHeader` and key on `environment.id` too.
- The identifier is always the stable environment id, never the secret
key (which can rotate and would split the bucket).
- The whole override result is cached per key by the existing SWR cache,
so **no extra per-request lookup and no separate Redis mapping** is
added.
## Behavior notes
- Root + additional keys of the same environment now share one bucket
(ceiling = configured limit, not a multiple of it). Restricted
additional keys are included — they were the biggest gap, since they
authenticate via the RBAC controller and previously fell back to per-key
buckets.
- **Public JWTs** keep their existing fixed-window, per-token bucketing.
- One-time bucket reset on deploy (bucket keys change); harmless.
## Tests
- New: two tokens resolving to the same identifier share one bucket.
- New: with no identifier, bucketing stays per-key (legacy behavior
preserved).
- Updated existing override tests to the new `{ config }` return shape.
Base: `feat/multi-keys-surface`. Closes TRI-12888.
|
||
|
Chris Arderne
|
9409ddf9bc |
feat(webapp): add multiple environment API key management (#4390)
## Summary Projects can create, inspect, expire, and revoke multiple API keys for each environment. Plaintext values are shown only at creation; stored credentials are hashed and the API keys page displays only an obfuscated suffix afterward. Self-hosted installations support full-access additional keys by default. Authorization extensions can provide additional access presets and optional task selection. Additional keys can also mint scoped public access tokens through the Trigger.dev API without receiving the environment signing key. ## Feature notes - Only admin+ can create API keys (Developer can make in Development branch). - JWT self-signing will be a server call when used with new `_ak_` keys. - JWTs with long expiry can keep working even with api key deleted (gets priveleges from api key, signed with root key) - Unfiltered session listings intentionally preserve the existing broad task-read behavior. Filtered listings enforce task-level scopes for every requested task. - Buffered runs without a task identifier are not safely authorizable, so cancel/replay requests fail closed rather than resolving an unscoped run. - Batch and waitpoint endpoints intentionally return server-minted, narrowly scoped public tokens to all callers. These tokens have bounded lifetimes and may remain valid until expiry after API-key revocation. ## Deployment notes Deploy the management UI and public-token endpoint with new key creation disabled. Enable creation for selected organizations after the authentication path and released SDK have been verified, then expand availability gradually. Revoking an API key prevents new bearer requests and new token minting. Public tokens already minted by that key remain valid until their own expiration because they are signed by the environment signing key. ## TODO - [x] Add "Created by" to the key table - [x] Document that streamed batch ingestion is non-atomic and may partially accept items before a validation or authorization error. ## Follow-ups - [x] Add an organization-level feature flag for the API key management UI and creation action. - [x] Document rollout ordering: enable additional-key lookup before enabling issuance. - [x] Add a system-wide gate that can stop new key issuance without disabling authentication for existing keys. - [x] Replace the generic SDK compatibility warning with the first published compatible version. Old SDK will mint an unusable token if given an `_ak_` key. - [x] Add public documentation covering creation, storage, expiration, revocation, SDK compatibility, and public-token lifetime behavior. - [x] Add observability for key creation, revocation, policy preparation failures, and public-token mint failures. - [ ] Exercise create, copy-once display, authenticate, mint, expire, and revoke flows end to end before broad enablement. |
||
|
Katia Bulatova
|
859f30e224 |
fix(webapp): report message catalogs survive the production bundle (#4488)
GET /api/v1/reports/health threw `no catalog registered for report "health"` in production (fine in dev): the catalog registered itself as a side effect of a bare import, which the SSR build tree-shakes under `"sideEffects": false`. Verified on the built server bundle — main's is missing the catalog, this branch's carries it. Fix: catalogs are values on the report registry entries; the resolver reads them from there and the mutable register-at-import step is gone. |
||
|
Chris Arderne
|
763b5dc582 |
feat(webapp): enforce scopes for environment API keys (#4389)
## Summary Environment API keys backed by the additional-key table can authenticate API requests using their stored effective scopes. Revoked and expired keys are rejected, branch environments retain their existing routing behavior, and last-used timestamps are updated on a throttled best-effort basis. ## Design API route builders receive the resolved ability and reject restricted keys on routes without an authorization declaration. Existing deployment, environment variable, queue, run, task, batch, session, and waitpoint routes declare the resources they access. Trigger and batch responses return server-signed public access tokens, so additional keys never need access to the environment signing secret. Root-key rotation also keeps public tokens valid for the existing grace window. ## Feature notes - Root environment keys remain unrestricted for backward compatibility. Additional keys enforce their persisted scopes and fail closed on routes without an authorization declaration. - Machine-key requests never exchange one credential for another. Additional keys cannot retrieve the root key, and rotated root keys are not upgraded during their grace window. - Public JWT validation remains host-owned, while installed RBAC plugins continue to supply root-key abilities. - Unfiltered session and run listings preserve existing broad task-read behavior. Filtered requests enforce the supplied task identifiers. - Related-run summaries remain embedded in run retrieval for API compatibility. Retrieving or mutating a related run independently still requires permission for that run. - Queue management authorizes at collection scope, matching the queue permissions currently issued. - Batch responses deliberately include server-signed public access tokens for all clients. Selected-task credentials continue using their original credential for per-item authorization. - Two-phase batches authorize declared task identifiers before creation and authorize every streamed item. Streaming paths that cannot declare the complete task set remain fail closed. - Authentication telemetry records successful credential resolution separately from subsequent resource-authorization failures. - API keys are high-entropy random tokens. SHA-256 is intentionally used for deterministic indexed lookup, not password hashing. ## Deployment notes The schema migration must be present before this code is deployed. Because bearer resolution runs on every authenticated request, deploy the resolver with additional-key lookup disabled, verify root-key and public-token parity, then enable lookup before any additional keys can be issued. The multi-task authorization tightening changes the result for narrowly scoped tokens that request tasks outside their grants. Observe would-deny results before enforcing that check. Request-idempotency keys are also newly isolated by environment and task, so a retry crossing the deployment boundary may execute once more before old cache entries expire. ## Follow-ups - [x] Add a system-wide kill switch for additional-key lookup, defaulted off for the initial deployment. - [x] Add authentication observability by credential kind, result, latency, and lookup path without recording credential values. - [ ] ~Add would-deny observability and an independent enforcement switch for multi-task authorization.~ - [ ] ~Add an independent switch for server-issued batch tokens while root-key parity is verified.~ - [ ] Confirm every API route reachable by a restricted key has an explicit authorization declaration or intentionally fails closed. - [x] Verify root-key rotation, revoked-key grace, and public-token validation through each bearer resolver path. |
||
|
Oskar Otwinowski
|
4efe0a07c4 |
fix(webapp): create dev environments for SSO and Directory Sync members (#4426)
Members added by SSO just-in-time provisioning or Directory Sync never got their per-member DEVELOPMENT environments - only invite acceptance and project creation created them. `trigger dev` returned "Environment not found" for those members and the dashboard had no dev view. ensureOrgMember now queues provisioning for every membership it settles, so both paths are covered and members missing environments are repaired on their next sync. Provisioning runs as a common-worker job to keep sign-in and directory webhooks off the per-project write loop. A failed enqueue surfaces for Directory Sync, whose worker retries the idempotent effect, and is swallowed for sign-in, where the next login enqueues again. Environment creation now tolerates a concurrent creator so the project-creation loop and the job cannot collide on the unique index. Also fixes environment resolution ignoring dev-environment ownership: a member without their own dev environment could be handed a colleague's and have it persisted as their dashboard preference. |
||
|
Chris Arderne
|
ed8f5e1297 | fix(webapp): stop logging every environment on a lookup miss (#4402) | ||
|
Eric Allam
|
4eb9292cbe |
feat(webapp,run-engine): queue metrics and health dashboard (#4131)
## Summary
Three related changes, each independently gated:
**Queue metrics and health.** Per-queue depth, throughput (enqueued,
started, completed), concurrency, whether a queue is throttled, and
scheduling delay (how long a run waits between becoming eligible and
actually starting), plus a per concurrency-key breakdown for keyed
queues. Collected from inside the run queue itself, stored in
ClickHouse, and surfaced on the Queues list, a new per-queue detail
page, the task pages, and the run inspector. The question it answers is
"does this queue have enough concurrency to keep up, and if not, which
key or which limit is the constraint".
**Percent-based queue concurrency limits.** A queue's concurrency
override can now be expressed as a percentage of the environment limit,
stored as the source of truth and re-materialized whenever the
environment limit changes. Absolute overrides above the environment
limit are now **rejected with a 400** instead of being silently capped,
which is a behavior change on `POST
/api/v1/queues/:queue/concurrency/override`.
**The `health` report.** A server-computed verdict on whether work is
flowing, whether the runs that do start are healthy, and whether
telemetry is fresh, rendered as text with sparklines. Available as `GET
/api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool
(plus a `report` MCP prompt, which shows up as a slash command in hosts
that support prompts).
With the flags off, the Queues page renders the pre-metrics component
verbatim, nothing is emitted, and nothing is written to ClickHouse.
## Configuration
Two independent gates, on purpose. Emission is global so data accrues
for everyone before anyone can look at it; the view is per organization
so it can be turned on for one org at a time without a deploy.
**Runtime flags (no restart)**
| Flag | Store | Gates |
| --- | --- | --- |
| `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by
default) | All emission, gauges and counters. Cached in-process for 10s
with stale-while-revalidate, warmed eagerly at boot so the first op
after a deploy is not dropped. |
| `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` |
Fraction of queue ops that emit a gauge. Counters are never sampled, so
throughput stays exact at any rate. |
| `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag`
row, per-org `Organization.featureFlags` override wins | Whether an org
sees the metrics view at all: the Queues list variant, the queue detail
route, the built-in Queues dashboard, the concurrency-keys endpoint, and
the metrics blocks on task pages and the run inspector. Off by default;
a gated org gets a 404 on the detail route rather than an empty page. |
Both Redis keys are readable and writable from `/admin/queue-metrics`
(super-admin UI, with a live per-shard stream-health table) and
`GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface
uses its own Redis client, so it works on any instance regardless of
whether that instance runs the emitter or the consumer.
**Environment variables (boot time)**
| Variable | Default | Notes |
| --- | --- | --- |
| `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and
injects it into the run engine. Without it the run queue has no emitter
at all. |
| `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on
this instance. Independent of emission, so consumers can be sized
separately from the API. |
| `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per
queue. |
| `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals
insert batch, so an ack can never outrun a write. |
| `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` |
falls back to the run-queue Redis | Set `HOST` to move the metrics
stream onto a dedicated instance so a metrics backlog cannot compete
with the run queue for memory. Self-hosters can leave it unset and get a
single-Redis deployment. |
| `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000`
dedicated | Bound on how much a stalled consumer can hold. The default
is deliberately lower when the stream shares the queue-critical Redis. |
| `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the
per-queue cumulative counter key, refreshed on every write, so only
queues idle for the whole window are purged. |
| `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue
names tracked per environment; overflow collapses into `__overflow__`. |
| `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea
one level down, per queue. |
| `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live
sample-rate key above. |
| `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics
tables in the Query page, its schema docs, the schema API and the AI
query context. Off keeps them unlisted while the feature is dark; a
query naming them still runs either way. |
| `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring |
Runs queue metrics on their own ClickHouse service: the consumer's
inserts and every queue-metrics read go through it, so a metrics-heavy
chart refresh never competes with runs-list or trace reads. Unset
reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`,
reads on the query pool). |
| `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split,
so the consumer's inserts can never land on a read endpoint. |
|
`QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}`
| `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other
per-workload ClickHouse clients. |
Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a
Postgres migration adding the nullable
`TaskQueue.concurrencyLimitOverridePercent`. Both are additive.
## How collection works
Queue operations produce two kinds of signal, and they have opposite
failure modes, so they are handled differently.
**Gauges** (queued, running, queue limit, env queued, env running, env
limit, throttled, plus keys-with-backlog and worst-key wait on keyed
queues) are read *inside* the same Redis script that performs the
enqueue or dequeue, so the reading is atomic with the operation it
describes rather than a racy follow-up read. The script returns them on
its reply and the app forwards them to the stream. Gauges are sampled
and drop-tolerant: they are aggregated with `max`, so a lost reading
costs resolution, never correctness.
**Counters** (enqueued, started, completed, plus nack and dead-lettered)
are cumulative odometers. Each event increments a per-queue key on the
metrics Redis and emits the absolute total, and ClickHouse takes the
difference across buckets at read time. This is the important property
of the design: a summed-delta counter undercounts permanently on any
lost event, while a cumulative one self-heals, because the next
surviving reading restates the whole total. Only bucket granularity can
be lost, never the total. A queue returning after its odometer TTL
expired restarts at 1 and reset detection handles it, which is safe
precisely because expiry only spans a window with no activity.
Both land on one sharded Redis stream. A consumer reads it with a
consumer group, reclaims stale pending entries on a 15s interval rather
than on every poll, maps one entry to one or two ClickHouse rows
(whole-queue and, for keyed queues, per-key), and acks only after the
insert lands. Each batch carries a dedup token derived from its
stream-entry ids, and the target tables set
`non_replicated_deduplication_window`, so a retried batch cannot
double-count either the raw rows or the aggregates that hang off them.
Consumer and emitter both emit OTel metrics
(`queue_metrics.emitter.emitted`,
`queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`);
stream depth and group lag are the two worth alerting on, and
`lag_unknown` exists because Redis can report a null lag after a trim,
which must not be read as zero.
## Storage and read path
`queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four
aggregate tiers are materialized straight from raw, never cascaded off
each other, each with a 30 day TTL:
- `queue_metrics_v1`, 10 second buckets per queue, the default read path
- `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and
cross-queue ranking
- `env_metrics_v1`, 10 second buckets per environment, queue-independent
so it stays cheap at any range
- `queue_metrics_ck_v1`, 10 second buckets per concurrency key
Every tier is an MV from raw because the counter states do not survive a
cascade: their merge is order sensitive, so a `-MergeState` chain off
the 10s table inflates the result, and the same property means an
aggregate state may only be merged inside one queue. That constraint is
now enforced by the query engine rather than by reviewer discipline: a
column can declare a `mergeGroupKey`, and any query that references it
without grouping by, or pinning to a single value of, every named key
fails to compile with an actionable message.
On the read side, TRQL gains three tables (`queue_metrics`,
`env_metrics`, and a `queue_metrics_by_key` that is hidden from the
editor, schema docs and schema API but still queryable, so per-key rows
can never silently merge into a plain per-queue query), plus
`deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level
optimizations ride along: a table can declare coarser rollups, so a
query whose bucket interval is 5 minutes or wider is routed to the 5m
table with no change to the query itself, and it can opt into the
ClickHouse query cache with time bounds floored to a fixed grid, so the
auto-refreshing dashboards actually share cache entries instead of
missing on every tick. Both are caller-side substitutions, so the
printer stays unaware of physical layout.
All of this can also live on its own ClickHouse service. A table
declares the pool its reads run on, the three queue-metrics tables name
the dedicated one, and the ingestion consumer writes through the same
client, so both directions move together with one env var and nothing
else routes differently.
The other engine change is opt-in gap filling: charts can request rows
for empty buckets, where counters zero-fill and gauges carry forward.
Grouped gauge series are densified per group and carried inside a
partition, so a quiet queue's line holds its last value without bleeding
another queue's value into it.
## Queue concurrency limits
`concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth
when an override is set as a percentage; the absolute `concurrencyLimit`
is materialized from it (floored, clamped to at least 1 so a percentage
can never act as a pause, and never above the environment limit). Every
path that changes an environment limit now recalculates the
environment's percent-based overrides afterwards, outside the
transaction, and pushes changed limits to the engine. The push is
attempted even when the stored value did not change, so a previously
failed sync self-heals rather than leaving the database and the engine
diverged; paused queues are skipped so a recalculation cannot
effectively unpause one.
The API accepts exactly one of `concurrencyLimit` or `percent`, and the
reject-instead-of-clamp change above means a request asking for more
than the environment allows now fails loudly. The percent bound (greater
than 0, at most 100) is defined once and shared by the zod schema, the
dashboard mutation handler and the service, so the three cannot drift.
The concurrency-keys table on a queue is now paginated against the
ClickHouse per-key tier, ranked by peak backlog with the total on every
row from a single scan, and only the keys on the current page are
enriched with live counts from Redis. That replaces a hard top-50 cap
with something whose cost is a function of page size rather than key
cardinality.
## The health report
`GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The
verdict is computed on the server and is deterministic, not
model-generated. Three independent analyzers run over one input
snapshot: flow (is work moving, and if not, is the cause a limit,
throttling, one bad queue, or dead-lettering), execution (are the runs
that start succeeding, and at what latency), and liveness (how fresh is
the telemetry). When telemetry is genuinely stale, the first two are
forced to unknown and every actionable field is stripped, so no surface
ever advises action off stale data.
Authorization is per query table rather than a blanket query grant: a
JWT must be scoped to every table the report reads (`runs`,
`env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull
a report that reads more than it was granted. `period` is validated as a
shorthand with a 90 day ceiling at the edge. The report catalog is a
registry of `{ load, interpret }` entries, so the next report is a new
entry and no change to the route, the view model, the renderers, the CLI
or the MCP tool.
`trigger mcp` no longer launches the install wizard when stdout is a
TTY, which fixed a real failure: hosts spawn the server over a PTY, so
the wizard would open and the client would time out waiting for a server
that never started. The wizard now needs `trigger mcp --install`.
## The part that is live regardless of every flag
The enqueue and dequeue scripts now return a 2-tuple so a gauge reading
can ride back on the reply. Every return site in the eight affected
scripts is wrapped, and a `nil` original is converted to `false` on the
way out, because a raw `nil` in the first slot would make Lua truncate
the multi-bulk reply and silently drop the gauge on the throttled and
empty-queue paths. The reply shape and the destructuring on the app side
are exercised on every queue operation whether or not metrics are
enabled, so that is the part of `run-engine` worth the closest review.
One behavior fix in the same area: the scheduling-delay anchor is set
only on a run's first entry into the queue. Anchoring it to trigger time
on re-enqueues made waitpoint and checkpoint resumes report the entire
wait as scheduling delay. Queue ordering is untouched, so a re-enqueued
run keeps its position, and nacks deliberately keep the original anchor
because a rolled-back dequeue is the same continuous wait.
A pending-version promotion still anchors to trigger time, on purpose:
that promotion is the run's first real entry into the queue, since the
trigger deliberately held it back waiting for a worker version, and the
TTL is armed at the same point for the same reason. The consequence is
worth naming, because it is a judgement call: a run that waits on a
deployment reports that wait as scheduling delay on its queue, which is
time unrelated to queue capacity.
## Verification
Unit and integration suites across the new package, the run queue, the
mapping layer, the query engine and ClickHouse (including a test that
applies migration 036 through the same splitter CI uses, and a
regression test that inserts the same batch three times to prove the
aggregates do not inflate). Beyond that, the whole path was driven end
to end against a live stack with real runs: emitter to Redis stream to
consumer to ClickHouse to the dashboards, for both the local dev path
and the deployed path where a supervisor drives the dequeue, with
assertions on exact counter reconstruction per queue and per concurrency
key, throttling, environment saturation, scheduling delay, and a
deliberate mid-stream reading drop to confirm the cumulative counters
still reconstruct the correct total. The gated-off state was checked on
every touched surface.
The dedicated ClickHouse service was verified against a second,
separately-schema'd instance: with it configured, the driven counters
reconstruct exactly on the dedicated instance, the shared instance gains
no rows for that window, a read through the query API returns the value
that exists only on the dedicated instance, and a `runs` query still
succeeds (it would fail outright if it were mis-routed to a service
without that table). With the variable unset, the full suite passes
unchanged.
---------
Co-authored-by: Katia Bulatova <katia@trigger.dev>
Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com>
Co-authored-by: James Ritchie <james@trigger.dev>
|
||
|
claude[bot]
|
d91818f198 |
fix(webapp): remove unawaited task list metrics promises (#4380)
<!-- ccr-slack-attribution --> _Requested via [Slack thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785082528841609)_ `TaskListPresenter` created promises that nothing ever consumed. Two of the three deferred metrics promises it returned had no reader, no `await` and no `.catch()`, so when the query behind one of them failed the rejection had nowhere to go. ## Before / After **Before** - `TaskListPresenter.call()` returned four things: `tasks`, `activity`, `runningStats` and `durations`. Its only caller reads `tasks` and `runningStats`. - Every load of the tasks page therefore fired two ClickHouse queries whose results were thrown away. - If either of those two queries failed, the resulting promise rejection was unhandled — nothing was awaiting it and nothing had attached an error handler, so it surfaced as an unhandled rejection at the process level rather than as an error anyone could attribute to a request. **After** - `TaskListPresenter.call()` returns `tasks` and `runningStats` only. - Two fewer queries run per tasks-page load. - There is no longer an unconsumed promise that can reject without a handler. `runningStats` is awaited by its caller, so its failures continue to be handled the way they always were. Nothing changes on screen: the tasks page renders `hourlyActivity` and `runningStates`, and neither of the removed values fed either of those. ## How The removed values were verified unreferenced before deleting anything: - `TaskListPresenter` has exactly one caller, `UnifiedTaskListPresenter`, which reads `taskResult.tasks` and `taskResult.runningStats` and nothing else. - No file anywhere in the repo — app code, tests, or type re-exports — reads an `activity` or `durations` field off the presenter's result. - `UnifiedTaskListPresenter` builds its own `unifiedTaskListHourlyActivity` query for the 24h chart the page actually renders, which is what made the presenter's separate 7-day daily activity data redundant. - `getDailyTaskActivity` and `getAverageDurations` on `ClickHouseEnvironmentMetricsRepository` had no callers other than the two lines being deleted, so they and their now-orphaned helpers and types were removed too. Changes: - `apps/webapp/app/presenters/v3/TaskListPresenter.server.ts` — drop the `activity` and `durations` fields (both from the main return and from the no-current-worker early return) and the two repository calls behind them. Drop the unreferenced `TaskActivity` type alias. The "don't await this" comment on the remaining `runningStats` promise now spells out that the caller has to consume it. - `apps/webapp/app/services/environmentMetricsRepository.server.ts` — remove `getDailyTaskActivity` and `getAverageDurations` from the `EnvironmentMetricsRepository` interface and its ClickHouse implementation, along with `fillInDailyTaskActivity` and the `DailyTaskActivity` / `AverageDurations` types. `getCurrentRunningStats` is the control that shows the diagnosis is right. It throws on query failure in exactly the same way as the two removed methods — `if (queryError) throw queryError` — but it never produced an unhandled rejection, because `UnifiedTaskListPresenter` passes its promise into a `Promise.all(...).then(...)` chain that the route then awaits. Same failure mode, opposite outcome, and the only difference is whether anything consumes the promise. Follow-ups, not in this PR: - `AgentListPresenter` returns three sparkline promises in the same shape and they look similarly unconsumed. Left alone here to keep this change reviewable. - With these two callers gone, the `getTaskActivity` and `getAverageDurations` query builders in `@internal/clickhouse` have no remaining callers in this repo. Whether to remove them is a separate call for someone who owns that package. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing - `pnpm run typecheck --filter webapp` — passes. This is the meaningful check here: it proves nothing still references the removed fields, methods or types. - `pnpm run format` and `pnpm run lint:fix` — clean, no changes produced. - No test file referenced the removed symbols, so no test needed updating. --- ## Changelog Server-only change, so this carries a `.server-changes/` note rather than a changeset: `.server-changes/task-list-remove-unused-metrics-queries.md`. > The tasks page no longer runs two queries whose results were never displayed, cutting wasted work on every page load and removing a source of hidden server errors --- ## Screenshots _No visual change — the removed data was never rendered._ Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Katia Bulatova
|
d05f1a7398 |
chore(webapp): migrate from Remix compiler to Vite (#4188)
Replaces Remix compiler with the Vite plugin. The Express server (cluster, socket.io, ws) and the Docker image contract are unchanged. |
||
|
Daniel Sutton
|
ae96b6c175 |
fix: read-your-writes + global-scope idempotency correctness under the run-ops split (#4284)
## What & why
Two related correctness fixes for the run-ops DB split. Under the split,
run-store reads can route to a **lagging read replica**; a just-written
run/waitpoint/batch can then be missed, causing a wrong decision.
**1. Read-your-writes → owning primary.** Surfaced first as an
intermittent `wait.until({ idempotencyKey })` re-wait on retry. Auditing
the run-store read surface found the same class at sibling sites (some
gating mutations or returning spurious 404s, others
tolerable/self-healing). Reads that must observe their own writes now
route to the owning **primary**
(`findRun`/`findWaitpoint`/`findBatchTaskRunByFriendlyId` →
`*OnPrimary`, a primary re-read on a miss, or a retryable 404 where the
SDK polls). Read-view reads stay on the replica. All additive — the
happy path is unchanged.
**2. Global-scope idempotency across the split.** A `global`-scope key
carries no per-run salt, so the same `(env, task, key)` triggered
concurrently from parents resident on **different** run-ops DBs could
dedup-miss on each DB and create a duplicate (the per-DB unique index
can't enforce cross-DB uniqueness). Such triggers (global scope, or
scope-absent, while split is active) are serialized through the existing
Redis idempotency claim, the loser resolves the winner by id across both
DBs, and the claim is reacquired on the expired/failed
clear-and-recreate path. `run`/`attempt` scope embed the run id and
never contend.
## Stacked for review
This is the **base** of a 2-PR stack, split so review is easier:
- **This PR** — production code only (34 files).
- **Stacked tests PR →
https://github.com/triggerdotdev/trigger.dev/pull/4285** — the
caller-driven guards (55 test files) on top of this branch.
## Validation
Local run-ops split, **both 2-DB and 3-DB**, fresh boot on this branch:
SDK canary 64/71 (only the known concurrency/input-streams/s3 failures),
quarantine sweep **0 unexpected** (340 pass / 16 known / 4 local) in
each topology, dashboard e2e 0 failed. No product regressions.
|
||
|
Chris Arderne
|
cecdfd94be | fix: only count preview branches toward the preview branch limit (#4283) | ||
|
Daniel Sutton
|
821972176d |
fix(run-store,webapp): correct split-database read routing, write residency, and batches list ordering (#4272)
## Summary Correctness and performance fixes for deployments that split run data across more than one database. Single-database / self-hosted deployments are unaffected (they collapse to a single read/write path). - **Batches list (dashboard):** for some organizations the Batches list could hide older batches or show them out of order. It now orders and paginates by creation time (with the id as a stable tiebreak), so every batch appears exactly once, newest first. The pagination cursor format changes; older in-flight cursors simply restart from the first page. - **Reads:** waitpoint and snapshot lookups that are keyed by a single run now read only the database that holds that run instead of querying both, removing redundant queries on hot paths (unblock, snapshot reads). - **Writes:** environment-scoped writes with no owning run (standalone wait tokens, waitpoint tags, idempotency-key resets) now land in the same database as that environment's runs, rather than defaulting to the other one. An idempotency-key reset also falls back to the other database when it matches nothing, so a reset still clears the key wherever the run actually lives. ## Notes Verified end-to-end against multi-database setups: run-keyed reads and env-scoped writes land on the correct database with no cross-database writes, and the batches list surfaces every batch in creation order. New tests cover the batches ordering/reachability and the write-residency routing. |
||
|
nicktrn
|
890dd66eb5 |
feat(webapp): route ClickHouse reads to an optional read replica (#4081)
## Summary Adds optional configuration to send ClickHouse read traffic to a separate instance (for example a read replica) while writes stay on the primary `CLICKHOUSE_URL`. This lets operators offload read load (runs list, traces, logs, queries) from the cluster that handles inserts. Fully backwards compatible: with nothing new set, every client resolves to `CLICKHOUSE_URL` exactly as before. ## What it adds - `CLICKHOUSE_READER_URL` (optional): a single reader endpoint that the read-only clients fall back to. Read clients resolve `<own URL> ?? CLICKHOUSE_READER_URL ?? CLICKHOUSE_URL`. The task-events client (which both inserts events and reads traces, spans, and logs) is built as a reader/writer pair so queries use the reader while inserts stay on `CLICKHOUSE_URL`. - `RUNS_LIST_CLICKHOUSE_URL` (optional): a dedicated client for the runs list (dashboard list, runs list API, live reload, child-status counts), so the highest-traffic read path can target its own instance. ## Safety Only read-only clients fall back to the reader: logs, query, admin, runs list, the pending-version lookup, and the realtime run-id resolver. The query page is constrained to read-only (the TSQL parser rejects anything that is not a `SELECT`, and a `readonly` setting is applied). The task-events client routes inserts to the writer and queries to the reader per method, so a write can never reach the reader. Pure-write clients (event inserts, replication) always use `CLICKHOUSE_URL`. Note: this PR targets a baseline branch rather than `main` so the diff stays scoped to the read-replica changes. It will be retargeted to `main` before merge. --------- Co-authored-by: Eric Allam <eallam@icloud.com> |
||
|
Daniel Sutton
|
bea7e2be90 |
feat(webapp,run-store): route run-graph reads and writes through the run-store router (#4237)
## Summary Run-graph data (runs, batches, waitpoints, and their related tables) can now live in a database separate from the control plane, with every read and write routed to the correct database by each run's residency. This makes reading and writing run data more reliable once the two are split, and is a no-op for single-database installs. ## Design - Run-graph table access goes through the run-store router, which selects the legacy or the new run-ops store per run instead of assuming one shared client. - The legacy run-ops client is now independently pointable, so legacy run data can be served from its own database (and replica) rather than the control-plane connection. - Run-graph writes go straight to the run-graph database instead of being forwarded through the control plane, and replication targets are split so runs in the new database still replicate to analytics without under-counting. - Read-through slots refuse the control-plane client, so a missing residency fails loudly instead of silently reading the wrong database. - Migration `20260710120000_drop_remaining_run_graph_seam_foreign_keys` drops the foreign keys that still crossed the run-graph / control-plane seam, which is what lets the two live in separate databases. The split stays off unless explicitly enabled and the two databases are confirmed physically distinct; startup fails closed otherwise. Verified by running the full dashboard end-to-end suite against both a single-database configuration and a three-database configuration (control plane, the new database, and a physically separate legacy database), with runs on both residencies. No misrouted reads in either configuration. |
||
|
Eric Allam
|
5ba8557a51 |
chore(webapp,core): remove the end-of-life v3 (engine V1) execution stack (#4236)
## Summary v3 (the engine that ran the SDK v3 era, internally `RunEngineVersion.V1`) is end-of-life. Following the removal of the v3 execution apps ([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and the legacy dev websocket ([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this removes the remaining v3 execution stack from the server. Clients still on v3 (an old SDK or CLI that has not upgraded) keep getting a clear "upgrade to v4" response. Triggers, batch triggers, reschedules, and deploys that resolve to v3 are rejected with a graceful 4xx pointing at the migration guide, never a 5xx, so a stale client cannot affect server health. Self-hosted instances still running v3 should stay on the 4.5.x release line until they migrate. ## What is removed - The MarQS queue and its shared/dev queue consumers. - The v3 socket.io namespaces (coordinator, provider, shared-queue) and the v3 run lifecycle services (attempt, checkpoint, and batch-resume). - The graphile-worker background job system; all live jobs already run on `@trigger.dev/redis-worker`. - The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally, so the flag is gone. - Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace` subpath and the legacy socket message catalogs) and the now-dead MarQS environment variables. ## What stays The v4 engine is untouched. The graceful v3 rejection boundary stays, `determineEngineVersion` still detects a v3 project so it can reject it, and the batch service plus batch-completion worker stay for current clients. Live queue concurrency limits and metrics now read from the v4 run engine instead of MarQS, and a brand-new dev environment now defaults to v4. ## Dependency cleanup Removes webapp dependencies left unused by this change: `seedrandom` and `semver` (only the removed v3 code used them) plus a set that was already dead, their orphaned `@types` packages, and two dead files. Adds a `knip:deps` script and a `knip.json` config so unused dependencies can be found the same way going forward. |
||
|
Daniel Sutton
|
c601739d35 |
perf(webapp,run-store): grouped run-ops reads + mint-kind flip grace (#4227)
## Summary Two threads on the run-ops split path. Read path: per-item run reads are batched into grouped queries, a waitpoint's connected-run reads are bounded, and the dedicated-schema relation hydrators fetch only the requested columns instead of whole rows. Retrieve also falls back to the other database when a routed read misses, so a run whose physical residency diverges from its id shape is still found rather than returning a spurious not-found. Fewer and lighter queries on the run read path, with no change to results. Mint-kind flip safety: flipping which database new runs mint to is now a deterministic wall-clock cutover, for both per-org and global flips. For a grace window every process resolves the same database, so a flip cannot route two concurrent triggers that share an idempotency key to different databases (which would bypass the per-database unique constraint and create a duplicate run). Supersedes the earlier #4205 and #4208. Draft: validation in progress. |
||
|
Eric Allam
|
1a0198cc5e |
perf(webapp,clickhouse): move runs empty-state check to ClickHouse (#4202)
## Summary
The runs page's empty-state check (whether an environment has ever had a
run, which decides between the "getting started" and "no runs match your
filters" states) ran a `findFirst` against the Postgres `TaskRun` table.
This moves it to ClickHouse, the same store the runs list itself reads
from, so the check no longer queries `TaskRun`.
## Design
Only the runs list triggers the check now (via an `includeHasAnyRuns`
flag); the other presenters that reuse `NextRunListPresenter` (API,
schedule detail, waitpoint detail, error group) no longer issue it. When
the list is empty it runs `SELECT 1 FROM task_runs_v2 ... LIMIT 1`
filtered on the full `(organization_id, project_id, environment_id)`
sort-key prefix with a configurable `created_at` lower bound
(`RUN_LIST_HAS_RUNS_LOOKBACK_DAYS`, default 30), so it hits the primary
index and reads minimal granules.
Results are cached in a tiered memory + Redis SWR cache. Only positive
("has runs") results are cached, so an environment with no runs is
always re-checked and its first run shows up immediately.
|
||
|
Chris Arderne
|
aa74e68c71 |
feat(sdk): add bulk replay to api and sdk (#4105)
## Summary
Adds SDK and API support for run bulk actions. You can now create bulk
cancel or replay actions from `@trigger.dev/sdk` using run IDs or the
same filters as `runs.list()`, then retrieve, list, poll, or abort the
action by its `bulk_` handle.
Tests, docs, changesets added.
## Design
The dashboard bulk action service now accepts structured filters instead
of reading directly from a dashboard request, so the dashboard and API
share the same creation path. Replay actions created through the API are
attributed with the existing `api` trigger source, while
dashboard-created actions keep `dashboard`.
The SDK exposes the new surface under `runs.bulk.*`, including
`targetRegion` for replay region overrides and cursor pagination for
listing bulk actions.
## Filters and runIds
Nuance on filters. If `filter` is provided, it MUST have at least one
key. This is to remove the footgun of passing no filter and selecting
all runs.
```typescript
{ action: "cancel", runIds: ["run_1"] } // valid
{ action: "cancel", runIds: [] } // invalid, min(1)
{ action: "cancel", filter: { status: "FAILED" } } // valid
{ action: "cancel", filter: {} } // invalid
{ action: "cancel", filter: {}, runIds: ["run_1"] } // invalid
```
|
||
|
Daniel Sutton
|
f101983a70 |
fix(run-store,run-engine): fix run-ops split hangs from wrong-store reads on the resume path (#4163)
## Summary On the run-ops split, NEW-residency runs could hang. Time-based waits (`wait.for`, `wait.until`, `delay`, waitpoint tokens), `batchTriggerAndWait`, and attempt starts stalled and never resumed. Each was a run-ops read or update that hit the wrong database: either the owning store's read replica when it needed read-your-writes, or the wrong store entirely because it routed by an id that does not encode residency. ## Fixes **Waitpoint resume (the main hang).** The managed resume path reads a run's completed waitpoints by snapshot id (`findSnapshotCompletedWaitpointIds`). Snapshot ids are cuids, which always classify to the legacy store, so a NEW run's join rows (which live on the new store) were never found. The resumed run saw zero completed waitpoints and hung. It now fans out across both stores and merges, like its sibling readers. **Batch completion.** Batch item completion (`updateManyBatchTaskRunItems`) routed by the item id, which is also a cuid, so a NEW batch's items were updated on the wrong store, matched zero rows, and the batch was treated as already complete (its parent's `batchTriggerAndWait` then hung). It now routes by the batch id, which does encode residency, matching the sibling `countBatchTaskRunItems`. **Read-your-writes on the resume path.** The block-time pending-waitpoint check (`countPendingWaitpoints`) and the attempt-start lock check (`findRun` in `startRunAttempt`) both read the owning store's replica with no read-your-writes guarantee, so a just-committed waitpoint completion or dequeue lock could be missed under replica lag and strand the run. Both now read the owning primary. Each fix ships with a two-database store or engine test that reproduces the hang and passes with the fix. |
||
|
Daniel Sutton
|
092b9ef07a |
fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154)
## Problem
The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.
## Fix: change the encoding, not the structure
Mint a **26-char lowercase base32hex** run id:
```
run_<24-char base32hex core><region char><version char>
[ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```
- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.
DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.
## Residency discriminator: length → version char
`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.
## Scope / verification
- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.
## Open decisions (flagged, not silently chosen)
1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.
---
## ⚠️ Required migration — deploy in lockstep
This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):
1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
- key `runOpsMintKsuid` → `runOpsMintKind`
- value `"ksuid"` → `"runOpsId"`
Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
0a51341347 | feat(run-ops): read presenters — de-join control-plane relations + read-through hydration (#4122) | ||
|
Daniel Sutton
|
c266e96c87 |
feat(run-ops): run-store routing seam + run-engine read seams (#4116)
## What Introduces the run-store routing seam and the run-engine read seams that let run lifecycle operations be dispatched to either the control-plane database or a separately-generated run-ops database, depending on where a run/batch resides. - **run-store** (`internal-packages/run-store`): adds `runOpsStore.ts` and substantially expands `PostgresRunStore.ts` so the store can resolve residency and route reads/writes to the correct backing client. `types.ts` grows the routing/residency types; `NoopRunStore.ts` is removed. - **run-engine** (`internal-packages/run-engine`): adds `engine/controlPlaneResolver.ts` and routes the per-system read paths (dequeue, enqueue, waitpoint, checkpoint, run-attempt, ttl, delayed-run, execution-snapshot, pending-version, debounce, batch) through the resolver/store instead of talking to a single Prisma client directly. `engine/errors.ts`, `engine/types.ts`, and `engine/index.ts` are extended to support injecting the store/resolver. Three fixes are included on top of the seam work: - `c6cadd85f` — routes read-your-writes to the owning store's **writer**, not its lagging replica, so an operation immediately reading back what it just wrote sees a consistent result. - `05c912e05` — normalizes run-ops-generation Prisma errors to the control-plane error class at the store **write boundary**, so `instanceof` checks and the `P2002` → 422 handling continue to work across the separately-generated run-ops Prisma client. - `88d12907f` — resolves NEW-resident batches in `ApiBatchResultsPresenter` by routing the batch read through the store, so a dedicated-DB batch resolves instead of returning 404. The change is heavily test-first: the bulk of the diff is new unit/integration coverage for the store routing, residency, and each run-engine system's control-plane resolver path. ## Why PR4 of the run-ops split stack (PR1–PR3 land the ClickHouse test-container and earlier plumbing). This PR is the read-path foundation: it adds the seam and read-routing but leaves the write path to route through the same seam in a later PR. Behavior-changing where the three fixes above touch existing read-your-writes / error-normalization / batch-resolution paths; otherwise additive (new store module, new resolver, injectable dependencies with existing single-client behavior preserved when no dedicated store is configured). ## Tests Extensive new vitest coverage under `run-store/src/*.test.ts` (routing, residency, dual-schema select, cross-generation error normalization, read-after-write, idempotency dedup, mixed residency, waitpoint co-location) and `run-engine/src/engine/**/*.test.ts` (per-system `controlPlaneResolver` tests, injectability, block-edge residency, waitpoint read residency, trigger-create routing, lifecycle router). Testcontainers-backed; no mocks. ## Notes Draft, **stacked on #4114** (`runops/pr03-clickhouse-tc`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris Arderne
|
c7861be520 |
chore: activate no-unused-vars and import linters (#4096)
Once this is merged, oxlint is at a pretty sensible baseline. **Enable `no-unused-vars`, `typescript/consistent-type-imports`, and `import/no-duplicates` lint rules** Turns on three previously-disabled oxlint rules across the monorepo and fixes all violations: - **`no-unused-vars`** – enabled as an error with standard ignore patterns: unused function arguments are ignored by default (`args: "none"`), variables/caught errors/destructured array elements prefixed with `_` are allowed, and rest siblings are permitted. - **`typescript/consistent-type-imports`** – enforced as an error; all type-only imports now use the `import type` syntax. - **`import/no-duplicates`** – enforced as an error; duplicate import statements from the same module have been merged. The remaining commits clean up the violations found across the codebase: removing unused variables/imports/type aliases, adding `_` prefixes to intentionally unused bindings, fixing duplicate imports, and converting value imports to `import type` where appropriate. |
||
|
Katia Bulatova
|
cc752cc375 |
fix(webapp): fetch run-scoped trace subtrees for large traces (#4024)
Fixed trace rendering for child and nested runs in large traces. Dashboard and trace API responses now load the requested run's trace subtree instead of depending on the run span appearing in the initial trace slice. |
||
|
James Ritchie
|
9f29eb2090 |
fix(webapp): update task filter icons and the scheduled run icon (#4065)
## Summary Fixes the Tasks icon on the Runs, Logs, Errors, and metrics pages (and the task-derived queue indicators) to use the correct "T" tasks icon, and shows a clock icon for scheduled runs in the run inspector header. ## Before <img width="1440" height="704" alt="CleanShot 2026-06-28 at 21 39 57@2x" src="https://github.com/user-attachments/assets/d00d5cbf-4726-4695-825b-59f5bdd84529" /> ## After <img width="1370" height="604" alt="CleanShot 2026-06-28 at 21 38 59@2x" src="https://github.com/user-attachments/assets/fb709073-0b9f-409d-a852-72979bf14da6" /> |
||
|
James Ritchie
|
b2b4c510e2 |
feat(webapp): improve task and dashboard activity charts (#4064)
## Summary Improves and unifies the run-activity charts by extracting a shared set of chart primitives and adopting them on the three task landing pages (agent, standard, scheduled), with the density and label fixes also carried over to the dashboard and custom query charts. Main changes a reviewer should know about: - **Shared primitives (DRY).** New `ChartCard` (title + maximize/fullscreen), `ChartSyncContext` (cross-chart hover + zoom state), `useXAxisTicks` (width-aware tick selection), `activityTimeAxis`, and `statusColors`, plus a server-side `activitySeries.server.ts` holding `chooseBucketSeconds`, status grouping, and the zero-fill helpers. The three task routes and both presenters were refactored onto these, removing roughly 3x duplicated tick logic, status-color tables, and bucket-ladder code. - **Denser bars on short ranges.** Server-side bucketing now uses `chooseBucketSeconds` (nice-interval ladder, ~72 target, capped at 120 buckets) instead of the hardcoded 1h/6h/1d ladder, so a 5m or 1h range no longer collapses into a single bar. - **Width-aware x-axis labels.** Labels are selected to fit the measured plot width (always first + last, evenly spaced, de-duplicated by rendered text), stay horizontal, and reflow on panel/window resize. Y-axis values default to compact form (8K, 1.2M) in `ChartBar`/`ChartLine`. - **Synced hover line.** Hovering one of the agent page's three charts draws a dashed vertical line at the same bucket on the *other* two, and suppresses it on the hovered chart. It is opt-in via `ChartSyncProvider`, so single-chart pages are unaffected. - **Maximize button.** Each chart gets a fullscreen dialog toggle (reuses the existing dashboard-widget pattern, `v` shortcut while hovered). - **Drag-to-zoom on task pages.** Dragging across a task chart sets the Time/Date filter (`from`/`to` URL params, clearing `period`/`cursor`), with a From/To tooltip shown during the drag. - **Custom query charts.** Long categorical x labels (run IDs, task names) middle-truncate and auto-rotate only when needed, and label thinning is now width-aware for both bar and line variants. Dashboard line-chart label density is also width-aware, tuned by a `TIME_AXIS_LABEL_SPACING_PX` constant. - **Tests.** 46 new unit tests for the pure logic (bucket selection, tick spacing, time-axis formatting, zoom range, truncation). ## Intentionally unchanged - **No click/drag zoom on the dashboard or custom query charts.** Drag-to-zoom is wired up on the task landing pages only; zooming the dashboard and custom charts is deliberately deferred to a separate follow-up PR. A plain click (without a drag) on a task chart is a no-op. - **The 25 mini activity charts on the Task list (`_index`) page are untouched.** They are hand-rolled raw-Recharts sparklines kept deliberately lightweight and do not use these primitives. - **Other raw-Recharts sparklines are untouched** (the usage sparkline, errors and prompts pages). - **No ClickHouse query semantics changed** beyond the bucket-interval parameter (same filters, same FINAL / `_is_deleted` handling). - **Webapp-only.** No public package (`packages/*`) changes, so there is no changeset; the `.server-changes/` entries cover it. --- ## Testing Added 46 unit tests covering server-side bucket selection, width-aware tick spacing, time-axis formatting, zoom-range math, and categorical label truncation (`pnpm --filter webapp run test`), and `pnpm run typecheck --filter webapp` passes. Manually exercised each task landing page (agent, standard, scheduled) plus the dashboard and custom query charts, stepping the Date/Time filter through 5m, 1h, 24h, 7d, and 30d to confirm dense bars on short ranges, non-overlapping labels that reflow on resize, the synced hover line across the agent charts, the maximize button, and drag-to-zoom updating the filter. --- ## Changelog The activity charts on the task landing pages and the dashboard and custom query charts now share one set of reusable primitives. X-axis labels are width-aware so they never overlap and reflow when a panel resizes, y-axis values are abbreviated (8K, 1.2M), and short time ranges render dense bars instead of collapsing into a single bar. Hovering any agent chart mirrors a vertical line on the others, every chart gains a maximize button, and dragging across a task chart zooms the Time/Date filter. Long categorical labels such as run IDs and task names middle-truncate and auto-rotate only when needed. --- https://github.com/user-attachments/assets/6be09e38-3a0e-4947-b6e3-4839daa2fbe0 |
||
|
Katia Bulatova
|
b1987dc090 |
feat(webapp): billing limits — pause, reject, recovery, and settings UI (#3996)
## Summary Adds Billing Limits to the webapp. Customers can set a monthly spend cap. When usage crosses the limit, billable environments enter a grace period. If the limit is not resolved before grace expires, new triggers are rejected until the organization increases or removes the limit. |
||
|
Chris Arderne
|
4fde283e76 |
chore: format and lint webapp also (#4056)
#3977 added formatting and linting everywhere else. This extends it to the webapp. |
||
|
Chris Arderne
|
df78ef96d9 |
feat: multi dev branches (#4023)
Closes this feature request: [https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances) ### Feature notes: - CLI `trigger dev` works as before - `trigger dev --branch my-branch` to create a new branch and run against it. - `trigger dev archive --branch my-branch` to archive (or in webapp). - New webapp page to manage and archive dev branches, currently feature flagged. ### Implementation details: - No changes to data model, no backfill. `isBranchableEnvironment` column is ignored for dev branches, we use `parentEnvironmentId IS NULL` instead. - `x-trigger-branch` overloaded for preview and dev branches - New `TRIGGER_DEV_BRANCH` env var available locally. `TRIGGER_PREVIEW_BRANCH` overloaded for child runs. - Lots of new glue code to sanitise the branch checks. ### Rollout - Deploy webapp/API changes (all backwards compatible) - Manual tests on some orgs - Deploy docs, release CLI, flip feature flag for webapp feature ### NB - `api.v1.projects.$projectRef.environments.ts` will return `isBranchableEnvironment: true` for all dev environments. ### Prerequisites - [x] Typecheck will not pass until we make a new release of `@trigger.dev/platform` and bump it here |
||
|
Eric Allam
|
c06005b353 |
feat(webapp,sdk): in-dashboard AI agent (#4018)
## Summary Adds an in-dashboard AI agent: a chat panel, reachable from any environment page, that answers questions about your runs, errors, tasks, and analytics, diagnoses why a run failed, charts your data, reads your connected repo's source, and answers product and how-to questions. It is gated behind the `hasDashboardAgentAccess` feature flag (global or per-org, default off), so this PR ships disabled: the launcher is hidden unless the flag is enabled. ## Design The agent runs as a standalone `chat.agent` Trigger task in its own internal package, with no access to the webapp database, Prisma, or ClickHouse. It reads the user's data over the public API, acting as the user via a short-lived delegated user-actor token minted server-side each turn (never in the browser), building on [#3997](https://github.com/triggerdotdev/trigger.dev/pull/3997). The error and analytics tools use [#4005](https://github.com/triggerdotdev/trigger.dev/pull/4005) and the TRQL query API. The first turn of a new chat streams from a warm webapp route (Head Start) while the durable agent boots in parallel. Structured answers (a run-failure diagnosis card, a live chart) render through a small typed view catalog rather than arbitrary markup. A knowledge lane forwards product and how-to questions to the support assistant. Conversation history lives in a separate Drizzle-backed store on its own Postgres schema, kept as a display read-model so it can never corrupt the agent's model context. The SDK changes add an `apiClient` option to `chat.createStartSessionAction` and `chat.headStart`, and keep the Head Start tool-approval tail intact across a custom `prepareMessages` hook so prompt caching and Head Start compose. |
||
|
James Ritchie
|
a90a495542 |
feat(webapp,database): show a Test column for agent sessions (#4011)
## Summary Sessions started from the agent Test playground were tagged with a `"playground"` tag that rendered in the Sessions table's Tags column. They are now flagged with a real `Session.isTest` boolean (mirroring `TaskRun.isTest`) and surfaced as a dedicated **Test** column with a check icon, to the left of Tags, on both the Sessions page and the Agent landing page, plus a matching **Test** property on the session detail page. This mirrors how Standard and Scheduled task runs already indicate test runs. ## Design `isTest` is a new `Session` column (Postgres) replicated into ClickHouse `sessions_v1` alongside the existing fields. The Sessions list reads `isTest` from Postgres for display (ClickHouse only supplies the ordered session IDs), so the column renders correctly without a ClickHouse backfill. The playground action now sets `isTest: true` on session create instead of writing the `"playground"` tag. The triggered run still carries `playground:true` in its own tags (unchanged). A migration backfills existing sessions, setting `isTest = true` and stripping the now-redundant `"playground"` tag where it is present, so the list and detail views render consistently without read-time tag filtering. |
||
|
Eric Allam
|
c6f0769299 |
fix(webapp): bound logs search memory and fix pagination at scale (#4012)
## Summary The logs search page (behind a feature flag) ran ClickHouse out of memory when browsing back over long time ranges. This keeps it within bounded memory and fixes a pagination bug that could skip or duplicate rows at a page boundary. ## Fix Memory: the list query reads in sort-key order, which opens one read stream per part in the window, and on object storage those per-part read buffers dominate peak memory, so it scaled with the number of parts scanned. Two changes bound it: - The logs ClickHouse client caps the per-part read buffers via new env-tunable settings. The object-storage-only setting is opt-in, so it is never sent to a ClickHouse version that lacks it. - Recent-first window narrowing: rows come back newest first, so the presenter probes the most recent window and only widens toward the full requested range when a page is short. A busy environment fills a page from a few recent parts instead of scanning the whole range; a quiet one still returns every row in a couple of cheap reads. Correctness: the keyset cursor ordered on (triggered_timestamp, trace_id), which is not unique because the spans of a trace share both, so rows at a tie could be skipped or duplicated across pages. The cursor and ORDER BY now include span_id, and the cursor is versioned so stale cursors reset to the first page. Guards: the effective page size is capped, and the existing per-query memory limit lets a pathological wide browse fail with an error instead of taking the node down. ## ClickHouse 26.2 The memory fix relies on lazy materialization deferring the wide attributes column to the output rows, which only holds on 26.x. Cloud already runs 26.2, so this moves the dev stack, testcontainers, and CI to match. The ClickHouse test suite passes on 26.2. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Daniel Sutton
|
65c545da4e |
refactor(run-store,webapp,run-engine): route Postgres TaskRun reads through the run store (#3990)
## Summary Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`, `findRuns`) and routes every Postgres read of `TaskRun` through them, mirroring how writes already go through the store. Behavior-preserving: each relocated read keeps its exact query, field selection, and database client (writer, replica, or transaction). This lets `TaskRun` reads be retargeted to a different backing store later without touching call sites. Stacked on #3981 (the write adapter); that PR is the base of this one. ## Scope In scope: the run engine, webapp services, presenters, and route loaders. Three reads that pulled `TaskRun` in through a parent model's relation `include` (alert delivery, batch results, attempt-dependency cancellation) are decomposed to fetch the run(s) through the store and stitch them back, since a relation include would not follow `TaskRun` to a new table. Left reading the existing table (out of scope): the legacy MarQS paths, the legacy trigger idempotency read, and one raw-SQL recovery script (commented for revisiting at cutover). ## Notes Reads default to the read replica; callers pass the writer or a transaction client wherever the original read did, so writer-vs-replica behavior is unchanged. |
||
|
Eric Allam
|
5052d895b3 |
feat(webapp,core): add a public HTTP API for errors (#4005)
## Summary
Adds an environment-scoped HTTP API over the Errors feature, mirroring
the runs API. Task-run failures are grouped by a fingerprint into "error
groups," and this exposes everything you can do with them in the
dashboard:
- `GET /api/v1/errors` lists error groups, with
`filter[taskIdentifier]`, `filter[version]`, `filter[status]`
(`unresolved`/`resolved`/`ignored`), `filter[search]`, a time range, and
cursor pagination.
- `GET /api/v1/errors/{errorId}` retrieves a single group (summary,
lifecycle state, affected versions).
- `POST /api/v1/errors/{errorId}/{resolve,ignore,unresolve}` changes its
state.
- `GET /api/v1/runs?filter[error]={errorId}` lists the runs behind a
group.
Request and response schemas are exported from `@trigger.dev/core/v3` so
the SDK can reuse them, and all endpoints are documented in the API
reference (OpenAPI). `errorId` is the `error_<fingerprint>` friendly id.
## Attribution
State changes record who made them. A plain environment API key has no
user, so `resolvedBy`/`ignoredByUserId` stay null. When the caller uses
an environment JWT obtained by exchanging a personal access token or a
delegated user token at `POST /api/v1/projects/:ref/:env/jwt`, that
exchange now stamps an `act` delegation claim, and the write endpoints
read `act.sub` to attribute the change to the acting user. This is the
first endpoint to consume the `act` claim, so two small pieces of
plumbing ride along: the exchange stamps `act` for personal-access-token
subjects too (it was delegated-token-only), and the public-JWT
bearer-auth path surfaces `act.sub` to the handler.
Built on the delegated-token work in #3997.
|
||
|
nicktrn
|
7aa871f37b |
feat(webapp): plan-aware compute migration (#3957)
Adds an opt-in mechanism to route a configurable percentage of organizations onto the compute (MicroVM) backing of their region at trigger time, without changing their stored region settings. Routing is gated by three global feature flags - `computeMigrationEnabled`, `computeMigrationFreePercentage`, `computeMigrationPaidPercentage` - plus a per-org `computeMigrationEnabled` override that wins in both directions. A region's compute backing is resolved from a new `WorkerInstanceGroup.region` column: a container group and its MicroVM group share one geo `region`, so the migration swaps the resolved worker queue to the backing group's queue. Orgs are bucketed deterministically by id, so ramping a percentage down keeps a strict subset rather than reshuffling, and a region with no compute backing is never touched. Everything is off by default - behaviour is unchanged unless the flags are set. The flags and the worker-region groups are read on the trigger hot path from in-memory snapshots rather than the database: a small `createReloadingRegistry` helper loads each at startup and refreshes them on an interval, so no per-trigger query is added and a percentage or kill-switch change propagates within the reload interval. A cold replica whose snapshot hasn't loaded yet reads as not-migrated (the container path) and self-corrects on the next load - the same cold-start contract as the datastore / LLM-pricing registries, with a `reloading_registry_loaded` metric so a never-loaded registry is alertable. The same migration decision is consulted at deploy-time template creation so a migrated org gets a compute template built ahead of its first run. This runs in shadow mode (best-effort, never fails the deploy) by default, or - when the `computeMigrationRequireTemplate` flag is on - in required mode, built synchronously at deploy so the first run never builds on-demand and template errors surface at deploy time. So operators keep "which runs ran where" while customers only see geography: the run's actual worker queue is stored raw, and the geo region is stamped separately on `TaskRun.region` (and a new ClickHouse `region` column) at trigger time. Read surfaces - the dashboard, the API, and the Query/Logs page - show the geo region, falling back to the worker queue for runs written before the column existed. Minor follow-ups left out of scope: the percentage flags render as text inputs on the admin flags page (the catalog UI has no numeric control type yet), and `createReloadingRegistry` could later gain pub/sub for sub-second cross-replica propagation if the reload interval proves too slow. |
||
|
Eric Allam
|
07a0e4ade9 |
feat(webapp): split Models into Your models and Model library tabs (#3958)
## Summary The Models page is now split into two tabs. **Your models** shows the models your project has actually used in the selected time range, with usage charts (cost over time, tokens over time, calls by model), a per-model table of calls / cost / avg TTFC / avg tokens-per-sec, and calls/tokens trend sparklines. **Model library** is the full catalog, reordered from alphabetical to a relevance-based provider order (Anthropic, OpenAI, Google, then the rest), newest models first within each provider, with a "New" badge on models released in the last 7 days. One time-range selector drives the whole Your models tab, so the charts, the table, and the sparklines all share the same window. Opening a model shows its own metrics with an independent range picker and a "View in AI metrics" link that opens the AI metrics dashboard filtered to that model. The active tab is kept in the URL so it survives a refresh and is shareable. ## Prompt caching & cost accuracy Both the Your models tab and the AI metrics dashboard now surface prompt-cache usage: a cache-savings column plus per-model cached-tokens and cache-hit-rate views, and a caching section on the dashboard (hit rate, cached tokens, estimated savings, and hit rate by model). Building this surfaced a cost bug. `input_tokens` is the total prompt count and already includes cache-read and cache-creation tokens, but the cost pipeline charged the full input at the input price and then added a separate cache line, so cached tokens were billed twice (and on Anthropic, cache reads were never discounted because their price is keyed differently). The input price now applies only to the non-cached remainder, with cache prices resolved across the provider-specific keys, so LLM cost and the cache hit-rate metric are accurate. Hit rate is computed as cached reads over total input. ## Notes Also fixes React "invalid DOM property" console warnings from the provider icons (the Llama and DeepSeek SVGs used raw `fill-rule` / `clip-rule` / `clip-path` attributes), which this page surfaces by rendering more provider icons. ## Screenshots **Your models tab:** usage charts and a per-model table with calls/tokens trend sparklines. <img width="2560" height="1267" alt="1-your-models-tab" src="https://github.com/user-attachments/assets/859bd24f-9047-4828-8bbb-83e5882846d6" /> **Model library:** provider-relevance ordering with a "New" badge on models released in the last 7 days. <img width="2560" height="1267" alt="2-model-library-tab" src="https://github.com/user-attachments/assets/46dd54b9-80f9-4922-ade9-5935b08dfebc" /> **Model detail, Metrics tab:** per-model range picker and a "View in AI metrics" link. <img width="2560" height="1267" alt="3-model-detail-metrics" src="https://github.com/user-attachments/assets/0f65d9d0-6142-4918-93f0-110bb277101a" /> **View in AI metrics:** the dashboard deep-linked and filtered to the selected model. <img width="2560" height="1267" alt="4-ai-metrics-filtered" src="https://github.com/user-attachments/assets/821f256c-e305-493c-98c7-eafaf2f57f83" /> |