4675 Commits

Author	SHA1	Message	Date
claude[bot]	802d23836d	fix(webapp): show the toast when saving project general settings (#4601)	2026-08-13 12:17:48 +01:00
Katia Bulatova	ee854480fe	fix(webapp): dashboard agent maintenance moves into the agent project (#4599) ... ## What & why The dashboard agent's upkeep — retention deletes and the investigation sweep — ran as cron jobs on the webapp's common worker, even though it only touches the agent's own datastore. This moves that upkeep into the agent's Trigger project as scheduled tasks (TRI-13182). ## What's inside **Retention** — `internal-packages/dashboard-agent/src/maintenance.ts`, a daily task (03:00 UTC). Deletes turn evals older than 30 days, hard-deletes chats soft-deleted more than 30 days ago, and purges terminal watches and submission rows older than 7 days. It used to run every 5 minutes; nothing needs a hard delete that fast, so it is daily now, draining in bounded batches and warning if it hits the cap. It retries (3 attempts) because the next run is a day away. It connects with `DASHBOARD_AGENT_DATABASE_URL`, falling back to `DATABASE_URL` like every other task in the package (the deletes are confined to the agent's own Postgres schema), and skips when neither is set. **Investigation sweep** — `src/investigation-sweep.ts`, every 5 minutes, same as before: settles investigation cards stuck `in_progress` (30-minute window, attempt cap, force-abandon note). It keeps the fast cadence because it fixes live state the UI is showing. **What stays in the webapp.** The watch finalize/deliver sweep and batch rearm: they cover a dead agent-side tick chain — a backstop can't live inside the thing it backstops — and they need the main database and the alerts worker. The org-deletion chat purge also stays: deletion must not depend on the agent project being deployed. The removed cron job keeps a cron-less tombstone entry so already-queued items drain cleanly; remove it in a follow-up. **Test plumbing** — the drizzle migration replayer that webapp tests hand-rolled is now exported once from `@internal/dashboard-agent-db/testing`; the moved tests live in the agent package as `src/*.test.ts` against real Postgres. ## Testing Agent package: retention passes (backlog drain, batch cap, no-op guard, chat-delete cascade) and the sweep, on testcontainers Postgres. Webapp: the watch/chat suites, plus a test that a settlement card stops the dashboard spinner. Full typecheck on both.	2026-08-13 13:13:02 +02:00
Eric Allam	aca234d1c3	perf(webapp): bound checkSchedule environment load to the requested ids (#4598) ... ## What `CheckScheduleService.call` loaded **every** environment of a project (`{ id, type, archivedAt }`, no filter) and then immediately narrowed to just the requested `environmentIds` via `resolveProjectScopedEnvironments`. It only ever uses the requested envs (to reject foreign env ids and reject archived branches). On a preview-heavy project that meant loading hundreds of archived branch rows to validate one, on a path called in a per-scheduled-task loop on the deploy path (`createBackgroundWorker` -> `syncDeclarativeSchedules`) and from `upsertTaskSchedule`. The query is index-backed and individually fast (rows_read/returned = 1 per predicate), so this is about result-set width / egress and wasted work at scale (~580k calls/24h observed via Insights), not a slow plan. ## Change Bound the `environments` relation load to `boundedIn(environmentIds)`: ```ts environments: { where: { id: { in: boundedIn(environmentIds) } }, select: { id: true, type: true, archivedAt: true }, } ``` Returns `<=` the number of requested envs (usually 1) instead of the whole project. Both existing behaviors are preserved: - **Foreign-id rejection**: the relation is still scoped to the project, so a requested id belonging to another project never comes back and `resolveProjectScopedEnvironments` reports it as `foreign` (a missing requested id is already treated as foreign). - **Archived-branch rejection**: a requested id that is an archived branch still comes back with `archivedAt` set, so the downstream `Can't add or edit a schedule for an archived branch` check still fires. `archivedAt` is kept in the select deliberately, so this bounds by id rather than filtering archived rows out. ## Evidence (isolated stack, seeded 1 prod env + 40 archived branch envs) Local `EXPLAIN (ANALYZE)` of the exact environments sub-select: | | rows returned | buffers | |---|---|---| | before (unbounded) | **41** | shared hit=12 | | after (`id IN (requested)`) | **1** (`Rows Removed by Filter: 40`) | shared hit=4 | Same `RuntimeEnvironment_projectId_idx`, no plan change. Rows to the client drop to `len(environmentIds)`, which is the point. **Unit (vitest, testcontainers, real Postgres):** `apps/webapp/test/checkSchedule.test.ts` extended to prove, on real rows, that the bounded load returns only the requested env (1 of 10), still reports a foreign id as foreign, and still surfaces an archived branch when it is the requested one. 5/5 pass. **Full e2e (both execution modes, real stack):** a purpose-built project with two declarative `schedules.task`s. - `trigger dev`: dev worker created, both schedules synced through the edited `checkSchedule` loop, no errors. - `trigger deploy` (managed deployment): PRODUCTION worker registered, both schedules synced against the **prod** environment through the same loop, prod + dev schedule instances active, no errors. `typecheck --filter webapp` clean. ## Rollout / rollback Straight deploy, no flag, no migration. Rollback is revert-only (read-path narrowing, no data change). Old and in-flight rows read correctly under both the old and new code. ## Out of scope The two lower-priority sibling reads in the ticket (the Query/metrics env id->slug map and the env-var repository fan-out) are left for follow-ups; they need caching / per-method scoping rather than this single bound.	2026-08-13 07:36:07 +01:00
Eric Allam	c6ef5f3959	perf(webapp): paginate the environment variables settings page (#4597) ... ## What The environment variables settings page loaded **every** variable in the project in one shot, with a nested `values` read plus a `valueReference` (SecretReference) sub-load that was selected but never read. For a project with many variables this pulled `variables × environments` value rows (~18k for large projects) on every page load, plus a matching ~18k-row `SecretReference IN` query. This paginates the presenter by variable key and removes the dead include. - Remove the never-read `valueReference: { select: { key } }` include → the `SecretReference` query is gone entirely. - Paginate the parent variable query: `count` + `orderBy key` + `skip/take`, page size 50 → the value read is bounded to `pageSize × environments` per page. - Scope the count and the page to variables that have a value in a displayed environment (`values: { some: { environmentId: { in } } }`), so `totalCount`/`totalPages` and the `skip/take` window match what actually renders (no phantom empty pages from variables that live only in archived branches or another member's dev env). - Display order comes from the DB `orderBy: { key: "asc" }` — the presenter no longer re-sorts each page with `localeCompare`, which under pagination could disagree with the DB collation at page boundaries. - The secret-value lookup (`SecretStore` keys) and the updater lookup (`user` by id) are now scoped to the current page instead of the whole project. - Search moves server-side (variable key, case-insensitive) and drives both the count and the page; the UI gains standard pagination controls. ## Why The two correlated ~18k-row control-plane queries flagged in the ticket come from this settings-page presenter, not from any hot path. Both are index-covered (`rows_read == rows_returned`); the issue is the sheer volume fetched in one burst. Bounding it per page removes the burst. ## Evidence Measured on an isolated stack with a seeded project of 1000 variables × 3 environments (3000 value rows), using Prisma's emitted-SQL log: | | SecretReference query | value rows fetched | | --- | --- | --- | | before | 1 | 3000 | | after | **0** | **150** (page 1) + one `count` | `EXPLAIN` on Prisma's verbatim statements (index confirmed via `enable_seqscan=off`; the local table is too small for the planner to choose them by default): - `count` (`WHERE projectId AND EXISTS(values in displayed envs)`) → Hash Join: Index Scan on `EnvironmentVariable_pkey` + Bitmap Index Scan on `EnvironmentVariableValue_environmentId_idx` - paginated parent (`WHERE projectId AND EXISTS(...) ORDER BY key LIMIT/OFFSET`) → Nested Loop Semi Join: Index Scan on `EnvironmentVariable_projectId_key_key` (**no Sort node**) driving an Index-Only Scan on `EnvironmentVariableValue_variableId_environmentId_key` - nested values (`variableId = ANY … AND environmentId = ANY …`) → index scan on `EnvironmentVariableValue_environmentId_idx` - `SecretStore` keys (`key = ANY …`) → index scan on `SecretStore_key_idx` No new index required. Verified in the browser on the seeded project: 20 pages, page navigation, server-side search (matches across all pages), last page renders, no app console errors. `typecheck`, `oxlint`, `oxfmt` all clean. ## Behavior change The previous client-side search matched variable **name and value** (and environment type / branch name). Values are encrypted at rest and resolved separately, so they cannot be searched server-side under pagination. Search is now **variable-name only**, server-side, case-insensitive. Projects with fewer than one page of variables see no pagination bar and no visible change. ## Rollout / rollback Pure read-path change on a dashboard loader, no schema or data migration. Rollback is a straight revert. ## Screenshots <img width="2400" height="1794" alt="01-page1" src="https://github.com/user-attachments/assets/d4a7effd-d167-4dd6-92f4-6e9174818acd" /> <img width="2400" height="1794" alt="02-search-single" src="https://github.com/user-attachments/assets/cd113ca9-ff87-431f-b2f6-7f7d36f2b32a" />	2026-08-12 23:51:14 +01:00
Eric Allam	8d0f693186	perf(webapp): drop archived branch environments from project env loads (#4595) ... ## What Several project pages loaded **every** `RuntimeEnvironment` row for a project, including the archived preview-branch environments that are never shown in the UI. On a project with heavy preview-branch usage that means thousands of rows per load, producing a large result set and a rare multi-second tail on the environment lookup (~30s outlier observed via Insights on `RuntimeEnvironment` projectId lookups, fingerprint `f2b3ecab…`). The tail is dominated by the size of the result being parsed/transferred, not by the query plan (it already used `RuntimeEnvironment_projectId_idx` with no over-read). So the fix is to stop returning archived branch environments. ## Diagnosis correction The ticket framed this as a "large `projectId IN` list" and suggested bounding the IN list / cursor pagination. It's actually a Prisma **nested relation load** on a *single-project* `project.findFirst`, so the `IN (...)` holds one projectId and the trailing `OFFSET $1` is Prisma's relation-subquery artifact. The 4,644 rows in the observed execution were **one project with ~4,644 environments** (accumulated archived branches), not many projects. ## Change Filter the `environments` relation load to `archivedAt: null` (base envs never archive, so only archived preview branches are excluded): - `ProjectPresenter.server.ts` - `orgs.$organizationSlug.projects.$projectParam.{concurrency,apikeys,environment-variables,settings}.ts` (best-env resolvers) And remove an **unused** `environments` select from `DeploymentListPresenter.server.ts` (it was selected but never read). `loadProjectEnvironments` (replay route) already filters `archivedAt: null` + env type; this change follows that existing precedent. ## Evidence (isolated stack, seeded one project with 2,000 archived branch envs + 4 active) `EXPLAIN (ANALYZE)` of the exact presenter sub-select: | | rows returned | index | |---|---|---| | before (unfiltered) | **2004** | `RuntimeEnvironment_projectId_idx` | | after (`archivedAt IS NULL`) | **4** (`Rows Removed by Filter: 2000`) | same index, no plan change | 500x fewer rows to the client, which is what removes the parse-on-load tail. No new index needed. `typecheck --filter webapp` clean. UI verified: project layout, Deploys page, and the concurrency best-env redirect all render with the 2,000 archived branches present in the DB and zero console errors. ## Rollout / rollback Straight deploy, no migration. Rollback is revert-only (read-path filter, no data change). Old and in-flight rows read correctly under both the old and new code. ## Limitation A project with thousands of *active* branches would still load them all; in practice active branches are few (branches are archived when their work merges). Hard-bounding active branches would be a larger change and is out of scope here.	2026-08-12 23:50:52 +01:00
Matt Aitken	bc3a33be24	fix(webapp): stop the billing limits page timing out under enforcement (#4594) ... ## Summary Opening the billing limits page while a spend limit was being enforced could time out with no response for organizations with many preview branches. That is exactly the moment the page matters: it is the only self-serve way to raise or resolve the limit. The page now loads fast regardless of how many environments the organization has. ## Root cause and fix The loader's queued-run count ran one ClickHouse count per billable environment, sequentially, with no timeout, and the environment list included every archived preview branch ever created. Thousands of environments times one round trip each held the response open past the edge timeout. The count is now a single org-level ClickHouse query filtered on environment type, capped server-side with max_execution_time. If the count fails, the loader falls back to 0 (the page hides the count label at 0) instead of throwing, so the recovery panel stays reachable even when the count errors. The billing-limit bulk-cancel path also stops enumerating archived environments.	2026-08-12 20:40:38 +01:00
Katia Bulatova	622fa79643	fix(webapp): restore header docs buttons when the dashboard agent is unavailable (#4592) ... Restores the page-header docs buttons removed in #4529 / #4418, shown only when the dashboard agent is unavailable (feature flag off, or pages outside the environment layout). The buttons are restored verbatim at their original spots — 22 sites across 21 files — wrapped in a small `WhenAgentUnavailable` gate that reads the agent context (SSR-safe, no hydration flicker). Also: in the light theme, the query editor's Format/Clear/Copy toolbar gets a translucent white background (`light:bg-white/80`) instead of transparent, so it no longer blends into the code behind it. <img width="1215" height="133" alt="Screenshot 2026-08-12 at 17 26 02" src="https://github.com/user-attachments/assets/4bdba825-9cbd-4df4-b6ca-0ea6691a534b" />	2026-08-12 18:08:44 +02:00
claude[bot]	442702e879	feat(webapp): stay on the same page when switching project or organization (#4585) ... <!-- ccr-slack-attribution --> _Requested by **Eric Allam** · [Slack thread](https://triggerdotdev.slack.com/archives/C0BEM9Z73TM/p1786528784863449)_ **Before:** you're on the API keys page in project X, you switch to project Y in the sidebar, and you land on project Y's Tasks page. Same for switching organization. Every switch threw away the page you were looking at. **After:** you land on project Y's API keys page. Switching organization does the same thing, one project down. Pages that name a single thing — a run, a batch, a queue, a schedule, a deploy, a session, an error group — can't exist in another project, so those take you to the matching list page instead (a run page takes you to Runs). The environment is still chosen exactly as it is today: nothing tries to guess it in the browser. --- ## Testing - New `apps/webapp/app/utils/pageSwitching.test.ts` (35 tests). It reads the compiled Remix route manifest, so the portable-page list can't silently drift from the routes: - every environment page that names no resource survives an environment switch — the same pages the old slug swap kept - the two branch lists are the only pages an environment switch keeps and a project switch drops - the pages gated per organization — Logs, Query and the queue metrics dashboard — travel with an environment or project switch but not an organization switch, and that list is derived from the route sources so a new gated page cannot be missed - every portable page points at a route that exists - every one of the 19 environment routes that takes a resource id truncates to a list page, with the id gone - portable pages resolve to themselves, so switching twice lands in the same place - every rejection case: leading slash, `//`, absolute URL, `..`, percent-encoded traversal, `javascript:`, unknown page — each falls back to Tasks rather than being sanitised into something - Manual: switch project and organization from API keys, project settings, a run page, and a queue page. - `pnpm run typecheck --filter webapp` passes. - The rest of the webapp suite needs Docker for testcontainers, which wasn't available here; all colocated pure unit tests under `app/utils/` pass (15 files, 150 tests). --- ## Changelog Switching project or organization in the sidebar keeps you on the same page instead of sending you back to Tasks. Pages for a specific run, deploy or other single item open the matching list instead. --- ## How The switcher links already pointed at `/orgs/:org/projects/:project` and `/orgs/:org`, whose `_index` loaders resolve the best environment (and, for the organization, the best project) and redirect. So the page travels as a search param on those links, and each loader appends it to the path it already builds: - `app/utils/pageSwitching.ts` — one pure module. `environmentPortablePage(suffix)` and `projectPortablePage(suffix)` walk up the suffix until they find an entry in an allowlist of portable pages, and answer with the environment root if they find none. The result is therefore always a literal from that closed set, which is what makes it safe to concatenate into a redirect target; there is no regex sanitising. The allowlist is built from the landing pages already listed in `deeplinkPages.ts` plus the handful of nested pages that file doesn't know about, so this isn't a new URL-shape table. - `app/hooks/useEnvironmentSwitcher.ts` — `usePageSwitcher()` derives the current page by slicing the environment layout route match's pathname off the current pathname, so there's no route table on the client either. The query string and hash are dropped on a project or organization switch, since filters encode task slugs and ids scoped to the project you're leaving. - Both `_index` loaders re-validate the page through the same function before using it. Two things worth a look: - **The environment switcher's truncation gap is fixed as a side effect.** It had a hand-written switch covering `runs/:runParam`, `deployments/:deploymentParam` and `schedules/:scheduleParam`; the other 16 id-bearing routes carried their id straight into the new environment (e.g. `queues/:queueParam`, `batches/:batchParam`, `errors/:fingerprint`, `sessions/:sessionParam`). All three switchers now share one truncation, and the test asserts it covers every such route in the manifest. - **Portability turned out to be two properties, not one.** Preview branches and dev branches render under any environment slug of their project — both loaders pass a hardcoded environment type and the project slug and never read `envParam` — so an environment switch keeps them, exactly as swapping the slug did before. A project or organization switch still falls back to Tasks, since the project you land in may have no preview branches. A test locks the environment half: every id-free page below an environment has to survive an environment switch. --- ## Screenshots _n/a — no visual change; only where the switcher links point._ 💯 --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-08-12 15:35:31 +01:00
Iss	4b4f6f2071	fix(webapp): accept Plain customers without an external id on customer cards (#4575) ... Plain sends `customer.externalId` as an explicit `null` rather than omitting the key. The schema validated it with `z.string().optional()`, which accepts `undefined` but rejects `null`, so every customer we don't set an `externalId` for got a 400 instead of a card — while the rest worked, which made it look intermittent. `email`, `externalId` and `thread` are now `nullish`. One of email/externalId is still required, and the route's existing email fallback resolves these customers. Three related fixes in the same path: - The route returned `{ cards: [] }` when no user matched. Plain records an integration error for any requested key it doesn't get back, so that surfaced as a broken card rather than a hidden one. Every requested key is now answered, with `components: null` where there's no data. - The impersonation link is offered only when the customer matched on `externalId` — a value we set ourselves. An email match is a weaker claim, since the address on a Plain customer isn't verified and for customers created outside our own writes it comes from whoever sent the message. Email-matched customers get the account rows without a one-click impersonation link. - The not-found log recorded raw customer identifiers; it now keeps presence flags only. The schema and the response helper moved to `app/utils/plainCustomerCards.ts` so they can be unit-tested without pulling in the db and env modules. ## Testing `app/utils/plainCustomerCards.test.ts` — 11 tests covering the null shapes, the every-key-answered response, and the missing-vs-zero distinction. Verified locally. Split out of #4571, which bundled this with an unrelated impersonation fix.	2026-08-12 09:18:04 -04:00
Eric Allam	db0ca9eb40	fix(webapp): drop unused OrgMember _count aggregate from org-list presenter (#4587) ... ## What `OrganizationsPresenter.#getOrganizations` selected a Prisma `_count.members` relation on every org-list load (hit on nearly every dashboard navigation). Prisma lowers that relation `_count` to a whole-`OrgMember`-table `GROUP BY organizationId` aggregate joined onto `Organization`. The computed `membersCount` field is read by **nothing** in the webapp, so the entire aggregate scan is wasted work. This removes the `_count` select and the `membersCount` field. The query keeps only the indexed `EXISTS` membership filter and the org/project selects. ## Why it's safe - `membersCount` has zero consumers (whole-webapp grep finds the name only at the point of assignment). It was added in #1796 (2023) and has been unused since. - The member count shown on the org settings/team page comes from a separate presenter query, not this one. No user-visible change. ## Evidence (generated SQL, before/after, seeded isolated stack) Before (with `_count.members`): ```sql SELECT ..., COALESCE(aggr._aggr_count_members, 0) FROM "Organization" LEFT JOIN (SELECT "organizationId", COUNT(*) AS _aggr_count_members FROM "OrgMember" GROUP BY "organizationId") aggr ON ... WHERE EXISTS (... "userId" = $1 ...) AND "deletedAt" IS NULL ORDER BY "createdAt" DESC ``` After: ```sql SELECT id, slug, title, avatar, "featureFlags" FROM "Organization" WHERE EXISTS (... "userId" = $1 ...) AND "deletedAt" IS NULL ORDER BY "createdAt" DESC ``` The whole-table `GROUP BY` aggregate is gone. The only remaining `OrgMember` access is the `EXISTS` on the caller's own membership (indexed by `userId`, a handful of rows). This is the single largest read-amplification query on the control-plane database (~1.39B rows read/day, ~719s DB CPU/day per Insights); removing it takes that portion to zero. Webapp typecheck passes. ## Rollout Straight deploy, zero blast radius. Rollback is a plain revert, no data migration. refs TRI-13170	2026-08-12 14:03:52 +01:00
Eric Allam	4fd7cc0f55	perf(webapp,database): index RuntimeEnvironment.pauseSource for the billing-limit reconcile tick (#4590) ... ## What The `billingLimit.reconcileTick` worker calls `getOrgIdsWithBillingPauseSource()` on `BILLING_LIMIT_RECONCILE_INTERVAL_MS` (~every 90s) to find which orgs currently have billing-limit-paused environments. Two problems: 1. `RuntimeEnvironment.pauseSource` had no index, so `WHERE pauseSource = 'BILLING_LIMIT'` was a **sequential scan of the whole table** on the control-plane primary, every tick. 2. Prisma `distinct` dedups **after** fetching, so it read every paused row (thousands) to produce a handful of distinct org ids. This PR: - Adds a **partial index** on `RuntimeEnvironment (pauseSource, organizationId) WHERE pauseSource IS NOT NULL`. Nearly all rows have `pauseSource = null`, so the index stays tiny. Second column lets the DB satisfy the distinct-org lookup from the index. Defined in SQL (Prisma can't express partial indexes), matching the existing partial-unique indexes on this model. - Switches the query from `findMany({ distinct })` to `groupBy(["organizationId"])`, pushing DISTINCT into the DB so it returns only the distinct orgs. ## Evidence **Correctness** — colocated `postgresTest` (testcontainers, no mocks): multiple `BILLING_LIMIT` envs in one org collapse to one org id, `pauseSource = null` envs are excluded, each org id returned once. 5/5 tests in `billingLimitReconciliation.test.ts` pass. **Plan change** — `EXPLAIN ANALYZE` on a synthetic table (200k rows, 5,250 `BILLING_LIMIT` across ~40 orgs, mirroring the test-side numbers from the investigation): | | Before (no index) | After (partial index) | |---|---|---| | Plan | Seq Scan (194,750 rows removed by filter) | Bitmap Index Scan on partial index | | Buffers | 1355 | 51 (index 6 + heap 45) | | Exec time | 6.06 ms | 0.59 ms | Index size 56 kB vs table 11 MB. The key win: cost now scales with the paused-env count, not total table size, which matters most on prod where the table is far larger. ## Rollout & rollback - **Index**: `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, in its own migration file. Pre-apply the index manually on the control-plane primary before deploying the migration (the migration is a no-op if the index already exists). - **Query change** is behavior-equivalent (same distinct org set), so no flag needed. - **Rollback**: revert the deploy and drop the index. No data migration either direction. ## Notes / limitations - The planner uses a Bitmap Heap Scan, so `organizationId` is still read from the heap (45 blocks for the matched rows only, not the whole table). A pure index-only scan isn't chosen for the bitmap path; the second index column keeps that open for the index-scan path at negligible cost. refs TRI-13169	2026-08-12 14:03:44 +01:00
Chris Arderne	7b7d48916d	fix(webapp): selfhost apikey role cta (#4586)	2026-08-12 14:14:57 +02:00
Katia Bulatova	480bede0ad	feat(webapp,sdk): dashboard agent plan enforcement, component gallery — and fixes (#4516) ... Plan enforcement for the dashboard agent — message quota and watch limits — plus the component gallery, fixes and test hardening from the same stack (#4548, #4549, #4550, #4552, #4556 merged here). ## Plan enforcement ([TRI-12863](https://linear.app/triggerdotdev/issue/TRI-12863)) **Agent message quota.** The Free-plan allowance becomes a real server-side limit with a durable counter. New `agent_message_usage` table keyed `(organization_id, period)` — deliberately not joined to chats, so deleting a chat can't free quota within the period. Both send paths count one user message (wakes never count) and refuse at the cap with `403 message_quota_reached`, which the client renders as an upgrade panel, never a silent drop. The refusal code is a single shared constant on both sides. **Watch limits.** A watch whose window exceeds the plan's `agentWatchMaxHours`, or that would push the org past its `agentWatchers` count, is refused with `watch_limit_reached` (409 on the API, an upgrade hint on the card). Plan limits only tighten the existing code ceilings (`min(plan, 24h)`, per-chat cap of 3 still applies). A plan limit of zero means zero, not unlimited. Questions answerable instantly are answered before any plan refusal — a one-shot consumes no slot and never sees an upgrade nag. **Fails open by design.** Cloud ships the actual per-plan numbers separately (TRI-12863 P0). Until then absent limits resolve to the unlimited sentinel and the upgrade UI is gated on billing presence — self-hosted sees no cap, no upsell, with tests proving the fallback. Both quotas are nudges, not security boundaries: a failing limit read never blocks a send. ## Component gallery An admin-only gallery of every agent card state: five `storybook.agent-*` pages (chat UI, view blocks, report, investigation, watch) with their shared shell and manifest, demo fixtures, two demo-only cards, toast examples, and the screenshot script. No LLM and no data — every state renders from fixtures under `dashboard-agent/demo/`, never reachable from a production path. Designers and reviewers can look at every state, including the report states, without seeding anything. ## And fixes **SDK: watch-mode chat subscriptions survive quiet windows** (TRI-13065, TRI-13070) — watch mode keeps reconnecting across empty long-poll windows and only stops on abort or a settled session; a passive subscriber can no longer stop a turn it doesn't own (`stopOnAbort` is explicit, default off). Review findings fixed alongside: a superseded stream's async teardown no longer removes the live successor's abort controller or multi-tab claim, and stopping a generation hands the chat back to the user's other tabs. **Query boundary pinned end-to-end** ([TRI-11165](https://linear.app/triggerdotdev/issue/TRI-11165)) — a route-level test drives `api.v1.query` with a real signed environment JWT (writes refused before ClickHouse, a read passes); `readonly=1` made non-overridable; a per-turn cap stops the model burning a turn rewriting a query it can't fix (deterministic SQL errors only — busy/transport rejections don't count). **chat.agent durability regression suite** ([TRI-11166](https://linear.app/triggerdotdev/issue/TRI-11166)) — testcontainers-backed coverage of the two audit criticals (cross-tenant isolation, no duplicate mid-stream turn, both control-broken) plus crash-resume, cursor-based refresh, clean rollback of a mid-write turn failure (torn by a real constraint violation), and OOM-restart replay. **Investigation sweep backoff** — stale investigations get an attempt counter and backoff so a poison row can't pin the sweep queue head (migration `0005`: `sweep_attempts`, `last_sweep_attempt_at`). ## Screenshots <img width="1440" height="791" alt="Screenshot 2026-08-06 at 00 36 19" src="https://github.com/user-attachments/assets/6a68cd42-8580-469d-afe7-e28d1eef18e1" /> 🤖 Generated with [Claude Code](https://claude.com/claude-code)	2026-08-12 13:59:35 +02:00
Chris Arderne	ed1bb72fb8	feat: implement cron window spread backend (#4566) ... - New DB fields on Schedule and ScheduleInstance - Use `queueTimestamp` for the "effectiveAt" delayed start time, propagate it to Clickhouse TaskRun table - Disable fastpath for delayed jobs - Add schedule timing logic, API endpoints with windows, persistence - Calculate phase for every schedule, only persist when window is non-null - Additional o11y for phased rollout	2026-08-12 12:24:32 +01:00
Chris Arderne	3c5bbc1607	fix(webapp): hard-navigate after creating a project (#4584)	2026-08-12 10:44:38 +00:00
Matt Aitken	c2c6e5c705	fix(webapp): keep session runs off the legacy realtime streams backend (#4564) ... ## Summary Runs created for a Session were triggered without a realtime streams version, so they fell through to the `realtimeStreamsVersion` column default of `v1`. A Session's own `.in` / `.out` channels are always `v2`, so any run-scoped `streams.append()` or `streams.pipe()` call made inside a session run wrote to a different backend than the session it belongs to, and stayed there for the life of the run. The API trigger routes were never affected. They call `determineRealtimeStreamsVersion` with the client's `x-trigger-realtime-streams-version` header and always pass an explicit value, so a current SDK asking for v2 gets it. Only the internal callers that build trigger options by hand were leaning on the column default, which no env var can influence because that path never calls the resolver at all. ## The version resolver Fixing the call site exposed a second problem in `determineRealtimeStreamsVersion`. Its two paths disagreed: an explicit `v2` was checked against the S2 configuration first, but when the caller expressed no preference it returned `REALTIME_STREAMS_DEFAULT_VERSION` verbatim with no check. A deployment that set the default to `v2` without configuring S2 therefore stamped runs `v2`, nothing failed at trigger time, and every later read or write against those runs' streams threw `Realtime streams v2 is required for this run but S2 configuration is missing` for the life of the run. Both paths now resolve through one pure function that takes its configuration rather than reading `env`: ```ts const requested = streamVersion ?? config.defaultVersion; if (requested !== "v2") return "v1"; const hasCredentials = Boolean(config.accessToken) || config.skipAccessTokens; return hasCredentials && Boolean(config.basin) ? "v2" : "v1"; ``` ## The basin requirement `resolveStreamBasin` resolves run, session and organization basins ahead of the global setting, so a deployment that provisions a basin per organization can serve v2 with no global basin at all. Gating purely on the global setting would degrade every run there to `v1`. `determineRealtimeStreamsVersion` therefore takes an optional organization basin, and every caller that holds one passes it, including the session path: ```ts basin: organizationBasinName ?? env.REALTIME_STREAMS_S2_BASIN, ``` This is deliberately the resolved basin and not the `REALTIME_STREAMS_PER_ORG_BASINS_ENABLED` flag. The flag says the feature is on, not that a given organization has been provisioned, and provisioning happens out of band. Keying off the flag would stamp `v2` on runs for unprovisioned organizations, recreating the failure this removes. **This widens behaviour for explicit `v2` requests**, which previously required the global basin: a provisioned organization on a per-org deployment now resolves `v2` where it used to get `v1`. That is intentional, and it makes every path agree. ## Scope Only newly created runs change. A run already stamped `v1` keeps that version for its lifetime by design, since readers resolve the backend from the same column and its existing streams have to stay readable. Scheduled runs reach the same column default through `scheduleEngine.server.ts` and are deliberately left alone: that one is a policy question about `REALTIME_STREAMS_DEFAULT_VERSION` rather than an inconsistency inside a single feature. ## Verification A full-stack e2e boots the real webapp plus Postgres, Redis and s2-lite, creates a Session through the public API so the run comes from the real trigger path, appends records the way `streams.append()` does, and asserts three things at once: the version stamped on the run, that the payload is readable from S2, and that no key exists in Redis. It appends at a realistic record size so the route's body cap and S2's per-record cap are both exercised. Reverting the session-path change flips all three observations, so it fails against the old behaviour rather than passing vacuously. Unit tests cover the resolver matrix, including organization-basin-only and credential-only configurations; two of them fail against the previous resolver. Also verified by hand against a local stack: a real `chat.agent` session run writing 8 records of 250KB through `streams.append()` put 2,049,072 bytes into S2 with no Redis key, while the same agent with the session-path change removed put 2,102,360 bytes into Redis and nothing into S2.	2026-08-12 11:01:59 +01:00
Chris Arderne	429c004118	fix(webapp): include Tailwind in production image (#4582) ... fix(webapp): include Tailwind in production image ## Summary Include `tailwindcss` in the webapp production dependencies so self-hosted Docker images can render emails that use React Email's Tailwind component.	2026-08-12 10:52:26 +01:00
Chris Arderne	7b390e5984	feat(cli,webapp): allow deploys with environment API keys (#4561)	2026-08-12 10:11:31 +01:00
Chris Arderne	26a730f908	fix(webapp): externalize kapaai (#4580)	2026-08-12 09:47:51 +01:00
Katia Bulatova	0b750d00dd	feat(webapp): dashboard agent — Watch (#4525) ... Watch is the agent noticing something later: you ask it to tell you when a condition holds, and it answers when it does — or when it can't any more. A watch is a **durable one-shot promise**. The condition is checked on a schedule by deterministic code (no LLM in the checks), the answer lands in the chat once, and then the watch is over. Ten kinds: three on a run, five on a queue, error recurrence, health recovery. ## Stack Stacked on **#4529** (UI), which is stacked on **#4418** (chat, reports, investigate). Merge those first. **#4516** (storybook gallery) sits on top of this branch. ## How to review [**GUIDEBOOK.md**](https://github.com/triggerdotdev/trigger.dev/blob/feat/dashboard-agent-flows-watch/internal-packages/dashboard-agent/GUIDEBOOK.md) on this branch is the behaviour reference — it states the conditions rather than the code, so you can predict what happens without running anything. "The ten watch kinds, and what makes each fire" and "Creating a watch" describe exactly this PR, and the tables there are the spec the code is written against. ## What's inside - **Ten watch kinds**, one deterministic check each (`dashboardAgentWatch*Checks.ts`), with the spec union in `dashboard-agent-contracts/src/watch.ts`. - **Scheduling** — each watch schedules its own next check; due watches of one `(environment, cadence)` group can be checked together in one batch pass, with a sweep as the backstop for expiry, redelivery and retention. - **Delivery** — the in-chat wake and card, an optional email alert (new `DASHBOARD_AGENT_WATCH` alert channel, so it shows on the project's Alerts page with one-click unsubscribe), and an optional investigation when the outcome needs attention. - **Submission ledger** — `watch_submissions`, keyed `(chat_id, client_request_id)`, so a retried card submission replays the recorded outcome instead of creating a second watch. - **Watch token** — a delayed-execution credential accepted only by the watch endpoints, re-checked against the user's live access on every tick. - **Unread work** — the panel polls for wakes that landed while it was closed, so a chat can go unread and light the launcher dot. ## Key decisions **A check result is a 4-way, and only two of them are verdicts.** `satisfied` / `terminal_unsatisfied` are answers; `pending` and `unavailable` are not. Any exception inside any check is caught in one place and becomes `unavailable` with an unverified observation — a check that failed is never evidence. **A completed window is an answer, and whether it is good or bad news is declared per kind, never inferred.** There is a table for that in the guidebook: `run_failed` completing its window is *good* news ("hasn't failed"), `backlog_drain` completing it is not. One rule overrides the table: a window that completed on an unverified observation is neutral and says only that the watch ended without a confirmed answer. **An unreadable source is never a negative answer** — and, because investigations only open on `attention`, it never starts one either. **Identity is `(chat, project, environment)` plus the condition,** enforced by a partial unique index over active rows (`watches_chat_active_identity_key`), not by the read-then-insert check. Cadence, window, note and `ticks` are deliberately not part of it. Two different chats may watch the same thing — a watch is a promise to a chat. **The server resolves the target's name, whatever the model calls it.** The model can't tell a task queue (`task/<id>`) from a custom queue, so both spellings are tried and the stored one wins — and the rewrite happens **before** identity and before the row is written, so the identity, the checks, the link and the wording all see one spelling. **Freshness fences.** Depth falls back from the live counter to the newest 60 s ClickHouse bucket, which only counts as current within 60 s of now. A non-current reading at or below the *quiet line* is refused as `unavailable` rather than believed, so a stale empty bucket is never read as "drained". The stall streak is the one piece of carried state: it lives in the previous check's facts and *freezes* on an unreadable reading rather than breaking. **Chain reliability.** There is no shared cron — each watch (or batch group) schedules its own next tick, so the failure mode to review is the chain dying. A failed batch check is caught, the next tick is scheduled anyway and the run resolves rather than failing, so the chain survives a check that couldn't run; the sweep re-arms groups and finalizes anything still active past its deadline, even when delivery isn't configured. Wake redelivery is id-deduped rather than conditional, because the sweep can't know whether the user was already told. Access is re-authorized on **every** check against the primary — replica lag would extend access the user has already lost. **Wording lives in one place.** `watch-wording.ts` is read by the card, banner, toast, email and the agent's own narration, and the numbers come from the frozen observation rather than a fresh read, so a retry produces the same sentence. Replay reproduces the **recorded** decision instead of deciding again — the transcript is append-once, so a second decision would contradict it forever. **Cancellation is the ending without an answer** — no resolution, no wake. One exception, decided during testing: a watch the *user* cancelled leaves a single neutral transcript line ("Stopped watching …"), keyed off the watch id so a retry can't repeat it. The other four reasons stay silent. **Email is opt-in and only a fired watch emails.** An expiry is narrated in the chat and nowhere else. Both gates (agent access, a configured email transport) are checked at subscribe time *and* again at delivery, and the subscription outcome is frozen on the ledger row so a retry replays it. Neither gate is a plan check. **One watch offer per turn.** The prompt and the renderer guard this independently — if the turn already proposed a watch card, the action button is dropped, because the card is the better affordance. Two eval cases pin the prompt side: exactly one offer with the line last and the button after it, and zero offers when the rendered card already carries one — deterministic assertions, over a real-model run. ## Testing Unit tests (vitest, testcontainers, no mocks) under `apps/webapp/test/dashboardAgentWatch*.test.ts` and `internal-packages/dashboard-agent/src/watch-*.test.ts` cover the invariants above: the 4-way check results and the freshness fences, identity/dedup and the submission ledger, queue-name resolution, the batch chain surviving a failed check, sweep boundaries and alert-once, tenancy and the watch token's scope, and the wording snapshot. The load-bearing ones were verified by control-breaking the guard first and checking the test goes red. Live-tested end to end against a local stack, following the guidebook: all ten watch kinds firing and expiring, cancellation, the email pair (a fired watch mails, an expired one does not), and watch recovery from a health report.	2026-08-12 09:51:40 +02:00
Eric Allam	326e9950f4	perf(webapp): scope declarative schedule sync to the current environment (#4577) ... ## Summary Background worker registration runs on every deploy and every `trigger dev` file save. Its declarative-schedule reconcile loaded every declarative schedule for the whole project across all environments, then re-fetched the deletion candidates it already had in memory. For projects with many scheduled tasks or many environments, that meant reading tens of thousands of rows on each registration. This scopes the load to the environment being registered, drops the redundant re-fetch, and selects only the columns the reconcile needs. It also fixes the schedule-limit count (`getUsedSchedulesCount`), which joined `TaskSchedule` and `RuntimeEnvironment` without a project constraint and could scan those tables in full. Pushing `projectId` onto both joins gives it a project-scoped index path with the same result. Follow-up to [#4522](https://github.com/triggerdotdev/trigger.dev/pull/4522), which batched the delete side of the same reconcile.	2026-08-12 08:16:55 +01:00
Eric Allam	26cdedda1c	perf(webapp): scope env var create pre-check to submitted keys (#4579) ... ## Summary Setting or importing environment variables ran a conflict pre-check that loaded every variable in the project and every value across all of its environments, only to decide whether the submitted keys already had a value in the target environments. On projects with many variables and environments that meant reading tens of thousands of rows on each create/import call. This scopes the pre-check to the submitted keys and target environments, so it reads only the rows it actually inspects (submitted keys × target envs), wrapped in `boundedIn` to keep the prepared-statement cache stable. Same conflict detection, a handful of rows instead of the whole project's env-var values.	2026-08-12 08:15:27 +01:00
Katia Bulatova	9a3bee0288	feat(webapp): dashboard agent — UI (#4529) ... Stacked on #4418. Merge that first. The UI slice of the dashboard agent: the side panel, the chat transport wiring, message and card rendering, suggested prompts, and chat history. #4418 works without this — the system is simply invisible. The diff is mostly components, so the notes below cover only the three decisions you can't read off the markup. Behavior and a hands-on walkthrough live in GUIDEBOOK.md, which lands with #4525. ## Decisions worth knowing - **Action rows always render at the end of a turn.** The model's emission order isn't trusted for layout, so action blocks are split out of the stream and appended last. Display only — `answered` stays keyed on the emission index. - **The last-chat memory is org-true.** It's keyed by the chat's own organization, and a foreign or deleted chat comes back as a 404 the client treats as gone, rather than an empty chat it keeps around. - **A dead stream self-heals from the settled transcript.** Terminal records are written to the chat row after the client's stream closes, so the panel re-reads it. The poll gate is any unfinished turn — a dangling tool part, not just an open investigation. ## Notes - Gated by `canAccessDashboardAgent`; no behavior change with the flag off. - Page marks: `handle.agentPageContext` on 47 routes, ~20 lines each. - Entry points: Ask Trigger button, ⌘J, Help & Feedback. The old ⌘I and `?aiHelp=` links keep working. ## Screenshots <img width="1440" height="788" alt="Screenshot 2026-08-07 at 15 14 29" src="https://github.com/user-attachments/assets/f4e89e8d-13ed-4be3-a88d-d5cca3ece0fa" />	2026-08-12 08:38:59 +02:00
Katia Bulatova	4569657923	feat(webapp): dashboard agent — chat, reports, investigate (#4418) ... ## What & why This is the system behind the Dashboard Agent — an assistant that answers questions about a project's runs, errors, queues, deploys and health, and can investigate failures end to end. The agent runs as a chat.agent task in its own Trigger project. It has no access to the main database or ClickHouse; all platform data is read through the public API using a delegated, read-only user token. Everything here is behind `canAccessDashboardAgent` and inert with the flag off. The UI that mounts the panel lands in #4529. ## Stack `#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook gallery. The scenario/contract reference for the whole stack is `internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch branch): it states, per feature, what makes each thing happen and where that is decided. ## What's inside **Agent runtime and tools** — `internal-packages/dashboard-agent`: prompt, tool set (API reads, TRQL query, docs, navigation, evidence/investigations, repo source), conversation compaction, a prompt-prefix token budget pinned by snapshot test, and sampled LLM-judged turn evals. The package cannot import webapp server code, which is what makes the "no DB access" claim structural rather than a convention. **Contracts** — `internal-packages/dashboard-agent-contracts`: `trigger://` URIs, intents, and the block envelope every rendered card travels in. **Conversation store** — `internal-packages/dashboard-agent-db`: drizzle over postgres-js in its own `trigger_dashboard_agent` Postgres schema, plus one additive migration. **Auth boundary** — the user-actor token gains an optional environment claim; one guard (`userActorEnvironment.server.ts`) enforces it so routes don't each re-derive the rule. Token minting, cap ceiling, and the RBAC fallback path for self-hosted. **Transport** — webapp resource routes that mint the token and proxy each turn, and SDK-side mid-turn reconnect. **Public API the agent reads through** — orgs, projects, environments, runs, queue metrics, workers, a run's commit metadata, repo snapshot, reports, and `POST /api/v1/query`. **Reports** — the health report's layout is declared once and shared by the card, the markdown surface and the JSON/MCP surface, so the same report reads the same in the dashboard, the terminal and an editor. **Block renderers** — the report and investigation cards the flows above already emit (`app/components/dashboard-agent/`). The panel that hosts them, and the rest of the chat UI, is #4529. **Query safety and CSP** — see below. ## Key decisions - **The agent is a separate Trigger project, not webapp code.** It reads platform data over the public API with a delegated user-actor token whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no webapp imports. - **The PAT-only auth helper now refuses user-actor tokens.** This is an intentional behavioral change: its callers consume only a bare userId and do not enforce delegated-token capabilities. Actor-aware routes continue through the scoped route builders instead. - **RBAC fallback builds a delegated token's ability from its own cap**, never the blanket ability a PAT gets (read-only when the token declares none). Without this, the agent's read-only cap would buy a write JWT on self-hosted. - **Org creation checks RBAC only for user-actor tokens, and only after the env gate**, so an install with `ORG_CREATION_API_ENABLED` off returns 404 rather than 403, and an ordinary PAT never consults an ability the route has no org to scope. Both orderings are pinned by test. - **The query path is read-only in depth.** TRQL rejects write statements at the grammar level (they don't parse, rather than being filtered), ClickHouse runs with `readonly=1`, and the org/project/env filters are injected server-side from the credential — the request body cannot widen scope. An unparseable query denies instead of falling through to the permissive resource. - **Document-wide img-src CSP.** Remote images are an outbound-request/exfiltration surface, so the policy permits only own-origin/data/blob, the required SSO avatar hosts, and the favicon endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST; wildcard hosts and bare schemes are intentionally not allowed. - **The chat transport reconnects on a mid-turn EOF** (`@trigger.dev/sdk`). A body that ends without a turn-complete is terminal only when the server says `X-Session-Settled: true`; otherwise the transport resubscribes from `lastEventId` with bounded backoff, and any record re-earns the budget. Previously a closed long-poll window or a proxy restart left the reply stuck as if still generating. - **Conversations live in their own datastore**, schema-scoped and foreign-key-free (it references `organizationId`/`userId` by id, because in cloud it is a different database). It is a display read-model for the History tab and transport resume; `chat.agent`'s object-store snapshot remains the model's source of truth. - **Deterministic first.** Reports and health checks contain no LLM — they are computed from the same data the dashboard shows, and the model only narrates and links them. That is what makes a number in an answer auditable. ## Testing - 63 new test files, run with `pnpm run test --filter webapp` and per-package vitest. Heaviest coverage on the auth boundary (`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`, `contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the report layout, and the SDK reconnect. - The agent package has a separate eval lane (`pnpm run test:evals`, `vitest.eval.config.ts`) that hits the real model, so it never runs in `pnpm test`. - Live-tested against a local stack scenario by scenario; the GUIDEBOOK lists the condition each behaviour is expected under, which is what those runs were checked against. ## Changelog `.server-changes/dashboard-agent.md`, plus changesets for `@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat reconnect) and the CLI's `mint-token` help text.	2026-08-11 18:56:14 +02:00
Saadi Myftija	02de2e693f	feat(api): separate rate limit budget for deployment endpoints (#4565) ... Most deploy-flow API calls shared the general per-environment rate limit bucket with all of that environment's runtime traffic, so an org with heavy API usage could intermittently 429 its own deploys; the `/api/v*/deployments` endpoints themselves were fully exempt from rate limits as a stopgap ([#2774](https://github.com/triggerdotdev/trigger.dev/pull/2774)), which promised a dedicated limiter as the follow-up. This is that follow-up: the whole deploy-flow group now runs on its own budget, separate from runtime API limits. ### Design A new `deploymentRateLimiter` covers every endpoint the deploy flow depends on: the `/api/v*/deployments` group, the env API key exchange (`/api/v1/projects/:ref/:env`), build-time env var resolution and sync (`/envvars`, `/envvars/:slug/import`), preview branches, `/api/v1/remote-build-provider-status` and `/api/v1/artifacts`. The general API limiter whitelists the same shared path list, so exactly one limiter applies to each path and the two can't drift apart. Buckets are keyed per environment for environment API keys and per token for the PAT-authenticated phase of a CLI deploy (whoami, key exchange, branches). The deploy budget is controlled via the `DEPLOYMENT_RATE_LIMIT_*` env vars.	2026-08-11 17:51:31 +02:00
claude[bot]	8819e25751	fix(webapp): hard-navigate after creating an organization (#4530)	2026-08-11 16:00:06 +01:00
Eric Allam	6449a644b9	feat(webapp,cli,database): track real dev onboarding progress (#4563) ... ## Summary The dev environment "Get set up" panel used to be a static list of CLI commands that only disappeared once your tasks registered, so nothing ever changed after you ran `init` and people assumed it was stuck. It now tracks real progress: `trigger init` records the project as initialized, so step 1 checks off, and the panel updates live as the dev server connects and your tasks register. It also adds a prominent "Copy AI agent prompt" button, presented as a clear alternative ("or") to the manual CLI steps, that copies a ready-to-paste setup prompt pre-filled with your project reference for Claude Code, Cursor, or any coding agent. ## Notes - Adds a `Project.initializedAt` column (migration `20260811065646_add_project_initialized_at`); the CLI `init` command calls a new project-scoped `POST /api/v1/projects/:ref/init` best-effort at the end of setup. - The `init` scaffold now imports from `@trigger.dev/sdk` instead of the deprecated `/v3` subpath. ## Screenshots <img width="2400" height="1794" alt="v7-redesigned-card" src="https://github.com/user-attachments/assets/c2fb4fa1-9484-4700-8bd3-110d66f5a44e" />	2026-08-11 11:43:33 +01:00
Eric Allam	820c079145	perf(webapp): read per-run environment config from the replica at dequeue (#4560) ... ## Summary Adds an opt-in path to serve a run's per-run configuration reads from the control-plane read replica instead of the primary, reducing primary database load during task execution. The managed-worker dequeue resolves each run's environment, organization, and environment variables before starting the run; those rows are stable for the life of a run, so they can safely come from the replica. Gated by `CONTROL_PLANE_DEQUEUE_READS_FROM_REPLICA`, defaulting to `"0"` (reads from the primary, unchanged from today). Set it to `"1"` to route the reads to the replica. The env-var read is scoped to the dequeue/resolution path (`resolveVariablesForEnvironment`); dashboard env-var reads and writes always stay on the primary. When no read replica is configured, `$replica` transparently falls back to the writer, so single-database self-host is unchanged either way. Verified end-to-end against a real primary/replica split, in both `trigger dev` and deployed (managed-worker) runs: with the flag on, env vars inject correctly and a value set immediately before triggering a deployed run is present on the run.	2026-08-10 17:45:57 +01:00
Eric Allam	951d8e8d7b	feat(webapp): per-client database pool metrics that survive the driver adapter (#4541) ... ## What Follow-up to #4539. The driver-adapter work is inert until a client flips to the pg driver adapter, but the moment one does, our database observability degrades: the OTel metrics pipeline reads pool stats from Prisma's `$metrics`, which is owned by the Rust engine's `quaint` pool. Under the adapter, `pg.Pool` owns the pool, so those gauges read zero. The pipeline also only ever scraped a single client (the control-plane writer singleton). This PR makes database metrics driver-agnostic and per-client: - Every configured client registers a metrics source: control-plane writer/replica, run-ops writer/replica, legacy writer/replica. Previously only the control-plane writer singleton was scraped. - Each OTel instrument is observed per client with `db_client` and `db_driver` (`quaint` | `pg-adapter`) attributes. `db_client` uses our canonical datasource-role labels (`control-plane-writer`, `control-plane-replica`, `run-ops-writer`, `run-ops-replica`, `legacy-run-ops-writer`, `legacy-run-ops-replica`) — the same strings used for the `db.datasource` span attribute, so a metric and a trace point at the same pool. - Pool figures come from the authoritative source per driver: - **pg-adapter**: `pg.Pool` (`totalCount`/`idleCount`/`waitingCount`, plus cumulative opened/closed from `connect`/`remove` events). - **quaint**: the Rust engine's `$metrics` pool gauges/counters, exactly as before. - Query counters and duration histograms still come from `$metrics` for both drivers (the Rust engine executes queries in both cases). - New `db.pool.connections.waiting` gauge (pg.Pool exposes this; quaint reports 0). - Stops exporting Prisma metrics from the Prometheus `/metrics` route. Pool observability now lives entirely in the OTel pipeline, per driver, per client. ## Why So we can flip any client (including the control-plane writer, the primary desync-fix target) to the driver adapter without losing pool visibility. Existing dashboards keyed on the same metric names keep working; they gain a per-client dimension. ## Testing Unit (`apps/webapp/app/utils/databaseMetrics.server.test.ts`): the pure normalizer — quaint reads pool from `$metrics`; adapter reads pool from `pg.Pool` and keeps engine query metrics; `busy` never goes negative; graceful zeroing when `$metrics` is unavailable (adapter still reports live pool figures). Live smoke test against a prod-shaped local stack: three physically-distinct Postgres DBs (control-plane, run-ops, legacy) behind dual PgBouncers, split mode on, with a mix of adapter and quaint clients. Reading the actual emitted OTel metrics, every pool shows up as its own series: ``` db.pool.connections.total{db_client="control-plane-writer", db_driver="pg-adapter"} = 1 db.pool.connections.total{db_client="control-plane-replica", db_driver="quaint"} = 1 db.pool.connections.total{db_client="run-ops-writer", db_driver="pg-adapter"} = 1 db.pool.connections.total{db_client="run-ops-replica", db_driver="quaint"} = 1 db.pool.connections.total{db_client="legacy-run-ops-writer", db_driver="quaint"} = 1 db.pool.connections.total{db_client="legacy-run-ops-replica",db_driver="quaint"} = 1 db.client.queries.total{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing db.client.queries.duration.count{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing ``` Confirms: metrics are attributed per pool with the correct driver; adapter pools' figures come from `pg.Pool`; and query counters/duration histograms keep incrementing under the pg adapter. Also verified `/metrics` (Prometheus) now returns zero `prisma_*` series while still serving the app's own metrics. `pnpm run typecheck --filter webapp` passes. ## Notes - `/metrics` (Prometheus) no longer includes `prisma_*` series. Anything scraping that endpoint for Prisma metrics should read the equivalent `db.*` metrics from the OTel exporter instead. - **PgBouncer + `?schema=` gotcha (separate from this PR, worth flagging for rollout):** since #4539 parses `?schema=` from the DSN and passes `{ schema }` to the adapter, node-postgres sends `search_path` as a startup parameter. A transaction-mode PgBouncer rejects that with `FATAL: unsupported startup parameter: search_path`. Our prod control-plane DSNs use the default `public` schema with no `?schema=` param, so this is latent, but any client we flip to the adapter must not carry `?schema=` in its DSN (or the pooler needs `ignore_startup_parameters = search_path`). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-08-10 13:54:06 +01:00
Katia Bulatova	c00fb9c36c	fix(webapp): report start latency as unknown when there is no data (#4544) ... When the health report had no start-latency measurement for the window, it printed a confident "p95 0ms" and graded it healthy. It now shows "unknown" for that metric and skips grading it, so an absent measurement can't read as a green signal. A genuinely measured 0ms is still shown as 0ms: the loader keeps "no measurement" distinct from a measured zero instead of coercing both to 0.	2026-08-10 13:53:23 +02:00
Eric Allam	90e8bd5c12	feat(webapp,database): opt-in per-client Prisma driver adapters (#4539) ... ## What Adds an opt-in path to run each Prisma client through **`@prisma/adapter-pg`** (the node-postgres driver) instead of the built-in engine driver, controlled by a **per-client env var, all off by default**: | env var | client | |---|---| | `CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER` | control-plane writer | | `CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER` | control-plane replica | | `RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER` | new run-ops writer | | `RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER` | new run-ops replica | | `RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER` | legacy run-ops writer | | `RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER` | legacy run-ops replica | With every flag unset the construction path is byte-identical to today (`datasources` URL + Rust engine), so this is inert until a flag is turned on. Per-client granularity allows enabling the adapter only where it's wanted. ## How - Enables the `driverAdapters` preview feature on both schemas (`@trigger.dev/database` and `@internal/run-ops-database`). This keeps the **Rust query engine** — it does NOT add `queryCompiler` — so query behavior, result types, and engine tracing spans are unchanged. - A shared `buildDriverAdapterPool` builds each client's `pg.Pool` with an explicit `max`, a bounded `connectionTimeoutMillis` (the node-postgres pool otherwise waits unbounded on acquire), and an `onPoolError` handler (an unhandled idle-connection error would otherwise crash the process). Threaded through all four client builders via a `useDriverAdapter` flag. - Adds `@prisma/adapter-pg` + `@types/pg` to the webapp; `pg` is already pinned at `8.15.6` (adapter-pg 6.x requires `pg < 8.17`). ## Connect-failure handling (the important correctness/security bit) Under the adapter an unreachable DB no longer surfaces as `PrismaClientInitializationError` / `P1001`; it becomes a `P2010` "Database not reachable: <host>" (or a raw `ECONNREFUSED`/`ENOTFOUND`-class error). Two handlers are updated so a client on the adapter behaves like today: - **`isInfrastructureError`** now recognizes those shapes (P2010 with a connectivity message, and raw connectivity errno codes). Without this, the DB **hostname would leak into API-client-facing errors** and the failure would go unlogged. Security-relevant. - **`isPrismaRetriableError`** treats the adapter's pool-acquire timeout ("timeout exceeded when trying to connect") as retriable, preserving the `P2024` retry behavior the adapter otherwise drops. ## Evidence Validated on an isolated stack that mirrors the production DB topology (chained PgBouncers in front of writer + reader): - **Behavioral parity:** raw-query results and Prisma error codes/`meta` are byte-identical between the engine driver and the adapter across the queried shapes (unique-constraint `meta.target`, record-not-found, transaction-timeout, serialization-failure, etc.). - **Feature matrix:** a full 380-project queue-ay pass shows no adapter-caused regressions — pass/fail parity between adapter-off and adapter-on, with the residual failures being pre-existing known-failures/flakes common to both. ## Rollout / rollback All flags default off; enable per client via env var, roll back by unsetting and redeploying (no data migration). Recommended first target is a single writer; enable one client at a time. ## Follow-ups (not in this PR) - `$metrics`-based pool observability is removed under the adapter (the Prometheus route + `db.pool.connections.*` instruments); the metrics replacement (via `pg.Pool` counters) lands in a separate PR. - Note for operators: on the adapter path, interactive-transaction `maxWait` does not bound pool acquisition — `connectionTimeoutMillis` does. ## Note on connection-string parameters The adapter pool is built from the base DSN, so Prisma-specific DSN parameters that node-postgres does not understand are not honored when a client is on the adapter: - **Prisma TLS spellings** (`sslaccept`, `sslcert`, etc.) — node-postgres uses `sslmode`/`ssl` instead. Our production DSNs do not use these Prisma-specific TLS params, but any deployment whose DSN relies on them must be checked before enabling a flag. - `pgbouncer=true` and `statement_cache_size` — effectively moot under the adapter, which uses no persistent named prepared statements. `connection_limit`, `pool_timeout`, and `schema` are handled explicitly (passed as `max`/`connectionTimeoutMillis` and PrismaPg's `{schema}` option). refs TRI-13039 --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-08-08 21:27:20 +01:00
Eric Allam	c526528d8f	feat(webapp,database): bound Prisma list filter arity (#4480) ... ## Summary Prisma expands `in` / `notIn` into one bind parameter per element, so every distinct list length is a separate prepared statement. Where the length tracks data volume (a batch size, a run-graph fan-out, a prior query's id set) one call site can mint hundreds of them. Each is used about once, but inserting it evicts an entry that was being reused, so the cost lands on unrelated queries sharing the pooler's statement cache. An unbounded list also risks the 65535 bind-parameter ceiling. `boundedIn()` pads a filter list to the next power of two by repeating its last element. `IN` and `NOT IN` ignore duplicates, so results are unchanged, and a call site drops from one statement per length to at most `log2(cap)`. Applied to all existing sites. ## Enforcement Two oxlint rules require the helper: a list filter must be an inline array literal or a `boundedIn()` call. - The first covers filters reached through `where` / `having` / `cursor`, and deliberately never descends into `data`, `create`, `update`, `set` or `equals`. A key named `in` in those positions is user data, not a predicate, and rewriting it would corrupt what gets stored or compared. - The second covers bare filter objects passed to where-building helpers, which the first cannot see. It found five sites in the run-graph batch loaders that were otherwise invisible. Both rules follow filters through the shapes they are actually written in: conditional expressions, logical-and objects, spread-conditional properties, computed keys, and call arguments. An array literal only counts as fixed-arity when nothing spreads into it, since `[...new Set(ids)]` has a runtime length. Twelve sites were hidden behind those shapes until the rules handled them. Scoped to `in` and `notIn`. The scalar-list filters `hasSome` and `hasEvery` compile to `&& $1` and `@> $1`, passing the whole array as a single bind parameter, so their arity never reaches the statement text and there is nothing to bound. Both rules are `error`, so new call sites fail CI. That ratchet has already caught four sites added by other PRs while this one was in review. ## Notes `boundedIn` pads by repeating rather than with null: `x NOT IN (a, b, NULL)` is never true, so null-padding a `notIn` filter would silently return no rows. Lists above 32768 are returned unchanged so padding can never push a query past the parameter limit. Route modules reach the helper through `~/db.server` rather than importing the database barrel directly, since a value import of that barrel into a module that also exports a React component is only safe while dead-code elimination prunes it. Measured on a local rig: 300 distinct list lengths produce 300 prepared statements unpadded, 10 padded. Verified end-to-end against a local stack with the full task-suite sweep, which surfaced no regressions.	2026-08-07 16:39:58 +01:00
Eric Allam	63176a6d69	fix(webapp): stop api inheriting inbound sampled traceparents so trace sampling applies (#4532) ... ## What The internal tracing `ParentBasedSampler` in `tracer.server.ts` left `remoteParentSampled` at its default of `AlwaysOn`. Any request arriving with a `traceparent` whose sampled flag was set got recorded in full, bypassing `INTERNAL_OTEL_TRACE_SAMPLING_RATE` entirely. Because the SDK propagates its (always-sampled) trace context on calls back to the platform from inside running tasks, the large majority of API server spans inherited a sampled parent and ignored the divisor. The sampling knob was effectively inert on the busiest service. This registers a custom propagator (`NonInheritingTraceContextPropagator`) that stops adopting the inbound trace as the parent: - `inject` still delegates to the standard W3C trace + baggage propagators, so outbound propagation is unchanged. - `extract` drops the parent span (`trace.deleteSpan`) while preserving baggage, so every incoming request roots its own trace and the ratio sampler applies uniformly. `remoteParentSampled` is also set to the ratio sampler as a belt-and-suspenders fallback, in case an inbound sampled parent ever reaches the sampler another way. Two effects: the divisor becomes effective on the API server, and the API no longer stitches onto (and inflates) the propagated task-run traces, which is where the very large, un-thinnable trace chains came from. Rooting each request removes those chains rather than only diluting them. Only the internal APM trace pipeline (`INTERNAL_OTEL_TRACE_EXPORTER_URL`) is affected. The user-facing run-trace pipeline (`otel.v1.traces` -> ClickHouse) is a separate path and is untouched. The only consumer of the global propagator's `extract` is the OTel HTTP/Express auto-instrumentation, so the blast radius is inbound-request trace shape. ## Evidence (local full-stack red/green, divisor 10) A local OTLP/JSON sink counting spans; a driver fires N requests at a real endpoint, each carrying a distinct sampled `traceparent`, then counts how many spans/traces carry that run's marker. | run | code | sent | kept traces | kept fraction | | --- | --- | --- | --- | --- | | before | unmodified | 500 | 500 | 1.00 | | after | this PR | 500 | 67 | 0.134 | | after | this PR | 2000 | 213 | 0.1065 | Before: 100% of inherited-sampled requests kept, divisor ignored. After: ~10% kept (the divisor), converging on it at larger N. In every after-run each kept request is a single self-rooted trace (kept spans == kept distinct traces), confirming the inherited chains are gone, not just thinned. `typecheck` passes. ## Rollout / rollback No flag. Behavior stays governed by the existing `INTERNAL_OTEL_TRACE_SAMPLING_RATE`. Rollback is a straight revert with no data migration. ## Notes Internal dashboards that count raw span or request volume from this pipeline will read lower once this ships. That is expected: those counts were inflated by the bypass, not a real drop in traffic. Latency/percentile monitors retain plenty of samples at the current divisor. refs TRI-13031	2026-08-07 15:13:42 +01:00
Eric Allam	7246f677db	fix(webapp): strip null bytes from idempotency and debounce keys at trigger (#4527) ... ## What A trigger request carrying a Unicode NUL (`U+0000`) in the **idempotency key** or **debounce key** reached `prisma.taskRun.create()` and failed the insert, so the caller got an opaque 500 and the run was never created. These two keys are stored in `jsonb` columns (`idempotencyKeyOptions`, `debounce`), and Postgres rejects a NUL inside a `jsonb` value with `SQLSTATE 22P05` ("unsupported Unicode escape sequence ... cannot be converted to text"). This fix strips the NUL from both keys at the single trigger-input chokepoint (`#buildEngineTriggerInput`), which every trigger path flows through (single, batch item, mollified, and drainer replay). Stripping matches the existing precedent for run errors and task events. It does not change dedup behaviour: the idempotency **dedup identity** is the hashed key (a clean 64-char digest), computed independently of the raw key we clean, so dedup keeps working exactly as before. For debounce the key is used directly, so the cleaned key also becomes the grouping key, an acceptable change for input that is already malformed. ## Why not payload / metadata / tags Those are `text` columns fed by `JSON.stringify`, which escapes a NUL to a safe escape sequence, so they do not hit this failure on the normal JSON path. (A raw NUL in a `text` column throws a different code, `22021`, and is not what triggers this issue.) The observed failures are the `jsonb` `22P05` variant, which is only reachable via the two key fields. ## Evidence Red then green (containerTest, real Postgres): with the fix reverted, triggering through the real service with a NUL in `idempotencyKeyOptions.key` / `debounce.key` fails with the exact `22P05` signature; with the fix, the run is created and the stored key has the NUL removed. Full-stack e2e (isolated stack, real HTTP): `POST /api/v1/tasks/:taskId/trigger` with a NUL inside `idempotencyKeyOptions.key` (`"acme<NUL>inc"`) and, separately, `debounce.key` (`"grp<NUL>1"`): - both returned `HTTP 200` with a created run (previously `500`) - stored `idempotencyKeyOptions` = `{ "key": "acmeinc", "scope": "run" }` (7 chars, NUL removed) - stored `debounce.key` = `"grp1"` (4 chars, NUL removed) - both runs render in the dashboard Unit tests cover the helper (strip, no-op fast path, object-reference reuse, null/undefined pass-through). ## Rollout / rollback Server-only webapp change, no flag. Zero behaviour change for clean input; only affects inputs that previously 500'd. Rollback is a straight revert, no data migration. ## Known limitation A raw NUL in a plain-string idempotency key (not created via `idempotencyKeys.create()`) lands in a `text` column and throws `22021` instead. That variant is not addressed here because stripping it would change the dedup identity, so it warrants a separate decision. Not observed in practice. refs TRI-13030	2026-08-07 13:28:52 +01:00
claude[bot]	dc529414df	feat(webapp): add /_/* redirect route (#4523)	2026-08-07 13:21:07 +01:00
Chris Arderne	0a44b88b39	fix: security release 2026-07-21 (#4528)	2026-08-07 12:25:40 +01:00
Eric Allam	db67a856fe	perf(webapp,database): index the newest-task-version lookup (#4518) ... Implementing PlanetScale Insights improvement. ## Summary Validating a schedule (creating or updating one through the API or the dashboard, and deploying a project that declares schedules) looks up the newest version of a task by slug. That lookup reads *every* version of the task and sorts them to return one. A project gains a row per task on every deploy, so the work grows with the project's age: the oldest projects pay the most, and dev-mode redeploys make it worse. This was picked because it was the largest single consumer of database time on the schedules path, and the fix is a sort key with no index behind it. ## Fix `BackgroundWorkerTask` is indexed on `(projectId, slug)`, which serves the equality but not the `ORDER BY createdAt DESC`. Postgres seeks the index, then bitmap-scans and top-N sorts the whole group to produce a single row. Adding `createdAt` to the index lets it scan backward and stop at the first row. The same call site also selected all 21 columns, including five JSON blobs, to read one field (`triggerSource`), so it now selects that field alone. ## Benchmark Local Postgres 17, 997,000 seeded rows / 748 MB, group sizes chosen to match the distribution seen in production. | Group size | Before | After | | --- | --- | --- | | 15,000 versions of one task | 11.118 ms, 1,510 buffers, 15,000 rows scanned | 0.027 ms, 4 buffers, 1 row | | 2,000 versions of one task | 2.081 ms, 1,455 buffers, 2,000 rows scanned | 0.022 ms, 4 buffers, 1 row | ``` before: Limit -> Sort (top-N heapsort) -> Bitmap Heap Scan after: Limit -> Index Scan Backward using BackgroundWorkerTask_projectId_slug_createdAt_idx ``` An ascending index scanned backward is enough here, so no descending index is needed. ## Impact and risk Real-world gain lands between the two rows above and scales with how many deploys a project has accumulated. Projects with few deploys will see little change, since there is barely anything to sort. The new index costs noticeably more than the existing two-column one: 43 MB against 7.3 MB on the benchmark rig. Adding `createdAt` makes every key unique, which defeats btree deduplication, so this is a real disk and write cost rather than a rounding error. Writes to this table happen at deploy time, not on the run path, so the write amplification is acceptable. The existing `(projectId, slug)` index is now a redundant prefix and could be dropped, but this PR keeps it so index usage can be observed before removing it. Behavior is unchanged: same predicate, same ordering, same row returned. The narrowed select is the only code change, and the field it keeps is the only one the caller read. Deploy note: the migration is `20260806100000_add_background_worker_task_project_id_slug_created_at_index` and uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it can be pre-applied by hand before the deploy.	2026-08-07 11:17:10 +01:00
Eric Allam	6c6e58e6ff	perf(webapp): batch declarative schedule cleanup queries (#4522) ... ## Summary `syncDeclarativeSchedules` runs on every background-worker creation (every deploy, and every file save during `trigger dev`). It issued one instance-delete per declarative schedule the current worker no longer declares, in a loop, and the overwhelming majority of those deletes matched zero rows. This collapses the loop into at most two set-based statements and skips the instance delete entirely when the current environment owns no instance of the schedule. ## Why so many, and mostly no-op The loop runs once per entry in `missingSchedules`, which starts as every DECLARATIVE schedule for the whole project across all its environments (the query filters only by `projectId`). A schedule leaves that set only when a declared task matches it by `taskIdentifier` **and** the schedule already has an instance in the current environment. That last clause is the amplifier. When a task's schedule has no instance in the current environment, the create branch inserts a brand-new `TaskSchedule` row with an instance for this environment rather than adding an instance to the existing row. So the same scheduled task, once it has run in dev and been deployed to prod, exists as two separate schedule rows: one carrying a dev instance, one carrying a prod instance. On a dev worker sync of that project: - the dev-instance row matches the declared task and is removed from the set - the prod-instance row has the same `taskIdentifier` but no dev instance, so it stays in the set and gets `deleteMany(taskScheduleId = prodRow, environmentId = dev)`, which matches zero rows So every declarative task that has been synced in another environment contributes one guaranteed no-op delete per sync, and the count scales with (declarative tasks x environments), plus any leftover rows from renamed or removed tasks. A project does not need to have dropped a schedule to generate these; it just needs the same declarative tasks present in more than one environment, which is the normal develop-in-dev, deploy-to-prod case. ## Fix The candidate schedules are already loaded with their instances, so the branch is decided in memory: - schedules with no instances (or only current-environment instances) are removed in a single `taskSchedule.deleteMany` - schedules that still have another environment's instance have only the current environment's instance detached, in a single `taskScheduleInstance.deleteMany`, and only when such an instance actually exists Behavior is unchanged (cascade delete still removes the instances of a deleted schedule); the difference is statement count. A zero-row delete writes no WAL and creates no dead tuples, so the removed work was pure query and commit overhead. Verified with a testcontainer test (red before, green after) counting the emitted deletes across the no-op, batched-detach, and schedule-delete cases, and end to end through `trigger dev`: three declarative schedules created, surviving a re-sync, then two removed in a single batched delete with the third preserved.	2026-08-07 10:27:39 +01:00
Matt Aitken	04f9c4e1a5	fix(webapp,run-engine,core): drop the hidden debounce ceiling, fail fast on an unusable maxDelay (#4521) ... Debouncing with a `delay` longer than an hour did nothing at all. The engine applied a server-side ceiling on how long a debounced run could be pushed back, measured from the run's `createdAt` and defaulting to one hour. A run is only pushed back while its new execution time stays inside that ceiling, so a `delay` at or above it could never push anything: the waiting run was released, the trigger started its own run, and the next trigger repeated it. A `delay: "12h"` produced one run per trigger, each correctly delayed by 12h, with no error raised and nothing on the run to show the debounce key had been ignored. The ceiling is now unset by default. A debounce key with no `maxDelay` keeps collapsing triggers for as long as they keep arriving, which is what the docs have always described. Self-hosters who want a bound can still set `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS`. That has a consequence worth stating plainly, so the docs now carry a warning for it: with no `maxDelay`, a continuously triggered key never executes. Set `maxDelay` when the work has to happen eventually. **Failing fast on an unusable `maxDelay`.** A caller who sets `maxDelay` no longer than their `delay` hits exactly the dead end described above, so that pair is now rejected at trigger time instead of silently behaving as if no debounce were set: ``` debounce.maxDelay (1h) must be longer than debounce.delay (12h). A debounced run is only pushed back while it stays inside maxDelay, so with these values every trigger would create its own run. ``` An unparseable `maxDelay` is rejected too, rather than quietly falling back to no bound at all, and so is a `delay` given as a date rather than a duration, which could never work because the value is re-applied on every push. The same check runs against a configured server ceiling, so a self-hosted deployment that sets `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS` gets the error rather than the silent failure this PR is about. With no `maxDelay` and no configured ceiling, which is the default, there is nothing to conflict with and nothing is rejected. The docs, the `TriggerOptions` JSDoc and the engine option all now state that the room available to push is the gap between `delay` and `maxDelay`. The run engine suite gains the case that motivated this: four triggers on one key with a 12h delay now collapse to a single run.	2026-08-07 07:55:35 +00:00
Chris Arderne	088f68b373	feat(webapp): share rate limit bucket across additional API keys per environment (#4508) ... ## What Rate-limit the API by **environment** rather than per API key. Previously the limiter keyed its bucket on the hash of the full `Authorization` header — one bucket per key. With additional environment API keys (`tr_*_sk_*`), an environment can mint many keys and each got its own full bucket, so more keys = higher effective rate limit. This collapses all of an environment's keys onto a single shared per-environment bucket, so the ceiling is exactly the configured limit regardless of key mix. ## How - `authorizationRateLimitMiddleware` now lets the override return `{ config?, identifier? }`. `identifier`, when present, is the rate limit bucket key; otherwise it falls back to the hashed `Authorization` header (unchanged legacy behavior, still used by `engineRateLimiter` and any unauthenticated fallthrough). - `apiRateLimiter`'s override resolves the environment id and uses it as the identifier: - **Additional keys** (`isAdditionalApiKey`) resolve via a new `resolveAdditionalApiKeyRateLimitScope()` — a **scope-agnostic** keyHash → (environmentId, org limiter config) lookup. It is deliberately permissive (restricted keys resolve too) because it's used **only for bucketing, never as an auth decision** — request auth still goes through the RBAC bearer controller, which enforces scopes. Revoked/expired keys are excluded so they can't hold a bucket warm. - **Root/legacy keys** reuse the environment already resolved by `authenticateAuthorizationHeader` and key on `environment.id` too. - The identifier is always the stable environment id, never the secret key (which can rotate and would split the bucket). - The whole override result is cached per key by the existing SWR cache, so **no extra per-request lookup and no separate Redis mapping** is added. ## Behavior notes - Root + additional keys of the same environment now share one bucket (ceiling = configured limit, not a multiple of it). Restricted additional keys are included — they were the biggest gap, since they authenticate via the RBAC controller and previously fell back to per-key buckets. - **Public JWTs** keep their existing fixed-window, per-token bucketing. - One-time bucket reset on deploy (bucket keys change); harmless. ## Tests - New: two tokens resolving to the same identifier share one bucket. - New: with no identifier, bucketing stays per-key (legacy behavior preserved). - Updated existing override tests to the new `{ config }` return shape. Base: `feat/multi-keys-surface`. Closes TRI-12888.	2026-08-06 16:05:27 +01:00
Chris Arderne	9409ddf9bc	feat(webapp): add multiple environment API key management (#4390) ... ## Summary Projects can create, inspect, expire, and revoke multiple API keys for each environment. Plaintext values are shown only at creation; stored credentials are hashed and the API keys page displays only an obfuscated suffix afterward. Self-hosted installations support full-access additional keys by default. Authorization extensions can provide additional access presets and optional task selection. Additional keys can also mint scoped public access tokens through the Trigger.dev API without receiving the environment signing key. ## Feature notes - Only admin+ can create API keys (Developer can make in Development branch). - JWT self-signing will be a server call when used with new `_ak_` keys. - JWTs with long expiry can keep working even with api key deleted (gets priveleges from api key, signed with root key) - Unfiltered session listings intentionally preserve the existing broad task-read behavior. Filtered listings enforce task-level scopes for every requested task. - Buffered runs without a task identifier are not safely authorizable, so cancel/replay requests fail closed rather than resolving an unscoped run. - Batch and waitpoint endpoints intentionally return server-minted, narrowly scoped public tokens to all callers. These tokens have bounded lifetimes and may remain valid until expiry after API-key revocation. ## Deployment notes Deploy the management UI and public-token endpoint with new key creation disabled. Enable creation for selected organizations after the authentication path and released SDK have been verified, then expand availability gradually. Revoking an API key prevents new bearer requests and new token minting. Public tokens already minted by that key remain valid until their own expiration because they are signed by the environment signing key. ## TODO - [x] Add "Created by" to the key table - [x] Document that streamed batch ingestion is non-atomic and may partially accept items before a validation or authorization error. ## Follow-ups - [x] Add an organization-level feature flag for the API key management UI and creation action. - [x] Document rollout ordering: enable additional-key lookup before enabling issuance. - [x] Add a system-wide gate that can stop new key issuance without disabling authentication for existing keys. - [x] Replace the generic SDK compatibility warning with the first published compatible version. Old SDK will mint an unusable token if given an `_ak_` key. - [x] Add public documentation covering creation, storage, expiration, revocation, SDK compatibility, and public-token lifetime behavior. - [x] Add observability for key creation, revocation, policy preparation failures, and public-token mint failures. - [ ] Exercise create, copy-once display, authenticate, mint, expire, and revoke flows end to end before broad enablement.	2026-08-06 15:27:10 +01:00
Katia Bulatova	337dda1e97	feat(webapp): name of the page in tab titles (#4517) ... Adds a shared `pageMeta()` helper and 74 route declarations, so a title reads `run_abc | Runs | Trigger.dev` — the specific thing first, then the page. Org pages also carry the organization: `Team | Acme | Trigger.dev`. Inside a project no scope is added, because the dashboard switches projects in every tab at once. Page names are unchanged; what's new is that a page says which one it is at all. Three wording changes on purpose: the queue page now names the queue, the model page names the model, and entity pages carry their section.	2026-08-06 10:34:30 +02:00
Eric Allam	58bf4e2833	feat(webapp): per-client database pool and connect timeout overrides (#4515) ... ## Summary Follow-on to #4513. The database connect timeout is now honored, but a single global value has to serve three separate databases at once (control-plane, legacy run-ops, and run-ops). This adds optional per-client overrides for the Prisma pool and connect timeouts, one pair for the writer and one for the read replica of each of the three databases, each falling back to the shared `DATABASE_POOL_TIMEOUT` / `DATABASE_CONNECTION_TIMEOUT` when unset. That lets one database's clients run a fail-fast connect timeout (with a bounded pool wait) while another keeps more headroom, without a single knob forcing the same tradeoff everywhere. No behavior change until an override is set. It also tags each client's queries with its specific datasource (`control-plane` / `legacy-run-ops` / `run-ops`, writer or replica) via the `db.datasource` span attribute, so telemetry can attribute connection behavior to a specific database instead of just writer-vs-replica.	2026-08-05 17:26:41 +01:00
Eric Allam	771937adf5	fix(webapp): clamp run priority so a large value can't fail run creation (#4512) ... ## Summary Triggering a run with a very large `priority` could fail run creation outright with an opaque database error. `priority` is multiplied by 1000 and stored in a 32-bit integer column, with nothing bounding it, so a big enough value overflowed the column and the create failed. The trigger now caps the value to the highest supported priority instead of erroring, so the run is still created. ## Fix `priorityMs` (the stored `priority * 1000`) now goes through a `clampPriorityMs` helper before the write. It rounds to a whole number and clamps into the column range at both ends, so only a valid integer ever reaches the column and an out-of-range priority caps rather than failing. Single and batch triggers share the write path, so both are covered.	2026-08-05 16:28:22 +01:00
Eric Allam	3039bc14d6	fix(webapp): honor the configured database connect timeout (#4513) ... ## Summary Every Prisma client built its connection URL with a `connection_timeout` query param, but the Postgres connector's parameter is `connect_timeout`. The misspelled param is silently ignored, so all clients fell back to Prisma's 5s default instead of the configured timeout. When establishing a new connection briefly took longer than 5s (for example during connection spikes), it failed with `Can't reach database server` even though the database was healthy. ## Fix All four client builders now construct their connection URL through one shared helper (`buildPrismaConnectionUrl`) that sets `connect_timeout`, so the configured value actually applies, and the parameter name lives in exactly one place. Covered by a unit test.	2026-08-05 15:52:12 +01:00
Chris Arderne	85f5b37c68	chore: upgrade to TypeScript 7 (#4318) ... ## Summary Upgrade the monorepo to TypeScript 7.0.2 and update package build tooling for compatibility with the native compiler. ## Design Package builds now use `tshy` 4, while the packages still using `tsup` move to `tsdown`. The few scripts that depend on the legacy TypeScript compiler API use an explicit TypeScript 6 alias; declaration portability coverage invokes the TypeScript 7 CLI directly. Turbo is updated so workspace tasks can read the regenerated pnpm lockfile. --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-05 15:49:44 +01:00
nicktrn	c01a4f18f4	feat(supervisor): cancel a resumed run's in-flight checkpoint (#4502) ... A run controller must call the continue route to resume, so the supervisor already knows synchronously that any checkpoint still running for that run is pointless. It only acted on that for the compute path. The continue route now cancels it for the Kubernetes path too, matching what completion already does since #4493. Called after the reply so the runner is never delayed, and skipped when there is no checkpoint client or when the compute path owns the run. The request is bounded by a 5s timeout so a hung call cannot leave the handler pending. `checkpoint_cancel_requests_total{result}` records the outcome, using the same label names as the delete path where they overlap: `sent`, `no_client`, `not_applicable`, `http_error`. No changeset: `CheckpointClient` is a server-only internal API, same as #4493. refs TRI-12915	2026-08-05 12:01:14 +01:00
nicktrn	4f69c43e6b	feat(supervisor): reclaim a run's checkpoint storage when it finishes (#4493) ... When a run reaches a terminal state, ask the checkpoint service to reclaim the storage its checkpoints occupied. Storage for finished runs is not otherwise reclaimed, so nothing frees it today. **Off by default** behind `DELETE_CHECKPOINTS_ON_COMPLETION`, and the service-side handler ships separately, so merging this changes no behaviour. ## Where the tenancy comes from Addressing a run's checkpoints needs org, project, environment, deployment version and run id. All five are already in hand at `attempt.complete`, and three are **signed** by the deployment token: | Value | Source | Trust | | -- | -- | -- | | org | claim `org_id` | signed | | environment | claim `environment_id` | signed | | deployment version | claim `deployment_version` | signed | | project ref | `x-trigger-workload-project-ref` header | runner-supplied | | run | route param | runner-supplied | `authorizeWorkloadRequest` previously returned only `environment_id`, and only in enforce mode, so it now also returns the verified `claims`. That difference is deliberate and documented on the method: claims are used to address a run's **own** resources locally, never to scope the platform, which is why `environmentId` stays enforce-only. The two runner-supplied values are safe because the signed ones are outermost - a runner lying about either can only name something inside its own org and environment, and a project ref that doesn't pair with its signed environment matches nothing. The run id is read from `params.runFriendlyId`, the same value the platform just validated, rather than from the body or a header. Where both a claim and a header exist (`deployment_version`), the claim wins. ## Placement The call sits after `reply.json(...)`, so the runner sees no added latency - the same shape the suspend route already uses. The service enqueues and returns 202, so it is one fast local hop. Terminal means `RUN_FINISHED` **or `RUN_PENDING_CANCEL`** - a run cancelled mid-execution never restores, and skipping it would leave its storage behind. Retries are excluded deliberately: reclamation is per-run, so a retry is covered by the final completion. Also gated on `!snapshotService`, so it stays inert where checkpoints aren't the kind this reclaims. ## Observability `checkpoint_delete_requests_total{result}` counts `sent` **and every reason we decide not to send**: `disabled`, `not_terminal`, `no_claims`, `no_project_ref`, `http_error`. The negative labels are the point - without them, "no requests are happening" looks identical to the feature being switched off. `no_claims` is reachable even under enforcement, since enforce only rejects a *present-but-invalid* token; an absent or legacy id still passes with no claims attached. ## Notes for review - **No changeset**: `CheckpointClient` is `core/v3/serverOnly`, an internal service-to-service API rather than customer-facing surface. - **No `.server-changes/` note**: there is nothing a dashboard user would notice here. Happy to add one if you disagree. - `pnpm run typecheck` can't complete in my checkout - `@trigger.dev/database` fails to build on a missing `tsc` in the pnpm store, unrelated to this diff. Verified with `tsc --noEmit` against the supervisor project instead: **zero errors in `apps/supervisor/src`**. Worth noting it caught a real bug here - the completion response is wrapped, so the status is `data.result.attemptStatus`. refs TRI-12789	2026-08-04 15:01:29 +01:00
Katia Bulatova	fbd6df33b4	feat(webapp): Themes + contrast settings update (#4206) ... Adds System Preferences, Dark and Light themes, gated by the `hasThemeSwitcher` feature flag (off by default — dark stays the default theme for everyone). Old theme is now "Classic"and set as default. "System preferences" theme has both Light and Dark modes and uses your laptop settings to use a correct one. It has less color accents (specifically less colored text), and they are the same for both modes, only grayscale values change between them. And Light/Dark themes can be used separately. New Contrast setting is available for System Preferences, Dark and Light themes - it changes the contrast for the whole app. All new visual Settings live in Account.	2026-08-03 19:29:33 +02:00
Eric Allam	57254b57fb	fix(webapp): make prop-types a production dependency (#4492) ... ## Summary The webapp's server bundle imports `prop-types` directly, but the package was declared only as a `devDependency`. A production install therefore leaves it out and the built server fails to boot: ``` Failed to start server: Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'prop-types' imported from /triggerdotdev/apps/webapp/build/server/assets/server-build-*.js ``` Moving it to `dependencies` is the whole change. ## Why the bundle imports it Nothing in the webapp's own code uses `prop-types` — there is no reference to it, or to `PropTypes`, anywhere under `apps/webapp/app`. It arrives through `recharts`, whose `react-smooth` dependency still declares `propTypes` on its components. That was invisible until recently. While `recharts` was resolved at runtime, its `prop-types` import was satisfied inside `recharts`' own dependency tree, which is production all the way down. #4486 added `recharts` and `victory-vendor` to `ssr.noExternal` to fix a hydration mismatch on every server-rendered chart; that inlines `react-smooth` into the server bundle, which moves its `prop-types` import into the webapp's own resolution scope — where the package was not available in production. So the bundling change was correct about *which* d3-shape build both sides resolve, and wrong about what the production runtime would be able to find. ## Verification `docker/Dockerfile` builds the runtime dependencies with `pnpm install --prod` against a `turbo prune --scope=webapp --docker` output, so I reproduced exactly that: pruned the workspace, installed with `--prod`, and imported `prop-types` from `apps/webapp`. | | result | | -- | -- | | `main` as it stands (devDependency only) | `FAILS: ERR_MODULE_NOT_FOUND` | | with this change | `prop-types resolves OK` | It resolves both as a CommonJS `require` and as an ESM `import`, which is the form the bundle uses. I also checked this is not one symptom of a wider problem: of the 169 bare specifier roots the server bundle imports, `prop-types` is the **only** one that is a devDependency and not a production dependency. The rest are node builtins or production dependencies. The hydration fix from #4486 is unaffected — the rebuilt bundle still carries the rounding d3-path build. ## Notes `prop-types` is inert in production (its entry point swaps in `factoryWithThrowingShims`), so this adds a 124 KB package that does no work at runtime. It has to be resolvable regardless, because the import is real. An alternative would be adding `prop-types` to `ssr.noExternal` so it is inlined and needs no runtime resolution. That keeps the dependency list honest about the fact that the webapp itself does not use it, at the cost of bundling a CommonJS package into the ESM server output. This route is the smaller, better-understood change. Worth following up separately: a check that every bare import in the server bundle resolves from a production install would have caught this before it landed. Local development installs every devDependency, so the gap is invisible when the built server is run from a working tree.	2026-08-03 16:09:03 +00:00