332 Commits

Author	SHA1	Message	Date
github-actions[bot]	72f50c2dad	chore: release v4.5.10 (#4440)	2026-08-07 14:06:43 +01:00
Eric Allam	7246f677db	fix(webapp): strip null bytes from idempotency and debounce keys at trigger (#4527) ... ## What A trigger request carrying a Unicode NUL (`U+0000`) in the **idempotency key** or **debounce key** reached `prisma.taskRun.create()` and failed the insert, so the caller got an opaque 500 and the run was never created. These two keys are stored in `jsonb` columns (`idempotencyKeyOptions`, `debounce`), and Postgres rejects a NUL inside a `jsonb` value with `SQLSTATE 22P05` ("unsupported Unicode escape sequence ... cannot be converted to text"). This fix strips the NUL from both keys at the single trigger-input chokepoint (`#buildEngineTriggerInput`), which every trigger path flows through (single, batch item, mollified, and drainer replay). Stripping matches the existing precedent for run errors and task events. It does not change dedup behaviour: the idempotency **dedup identity** is the hashed key (a clean 64-char digest), computed independently of the raw key we clean, so dedup keeps working exactly as before. For debounce the key is used directly, so the cleaned key also becomes the grouping key, an acceptable change for input that is already malformed. ## Why not payload / metadata / tags Those are `text` columns fed by `JSON.stringify`, which escapes a NUL to a safe escape sequence, so they do not hit this failure on the normal JSON path. (A raw NUL in a `text` column throws a different code, `22021`, and is not what triggers this issue.) The observed failures are the `jsonb` `22P05` variant, which is only reachable via the two key fields. ## Evidence Red then green (containerTest, real Postgres): with the fix reverted, triggering through the real service with a NUL in `idempotencyKeyOptions.key` / `debounce.key` fails with the exact `22P05` signature; with the fix, the run is created and the stored key has the NUL removed. Full-stack e2e (isolated stack, real HTTP): `POST /api/v1/tasks/:taskId/trigger` with a NUL inside `idempotencyKeyOptions.key` (`"acme<NUL>inc"`) and, separately, `debounce.key` (`"grp<NUL>1"`): - both returned `HTTP 200` with a created run (previously `500`) - stored `idempotencyKeyOptions` = `{ "key": "acmeinc", "scope": "run" }` (7 chars, NUL removed) - stored `debounce.key` = `"grp1"` (4 chars, NUL removed) - both runs render in the dashboard Unit tests cover the helper (strip, no-op fast path, object-reference reuse, null/undefined pass-through). ## Rollout / rollback Server-only webapp change, no flag. Zero behaviour change for clean input; only affects inputs that previously 500'd. Rollback is a straight revert, no data migration. ## Known limitation A raw NUL in a plain-string idempotency key (not created via `idempotencyKeys.create()`) lands in a `text` column and throws `22021` instead. That variant is not addressed here because stripping it would change the dedup identity, so it warrants a separate decision. Not observed in practice. refs TRI-13030	2026-08-07 13:28:52 +01:00
claude[bot]	dc529414df	feat(webapp): add /_/* redirect route (#4523)	2026-08-07 13:21:07 +01:00
Chris Arderne	0a44b88b39	fix: security release 2026-07-21 (#4528)	2026-08-07 12:25:40 +01:00
Eric Allam	db67a856fe	perf(webapp,database): index the newest-task-version lookup (#4518) ... Implementing PlanetScale Insights improvement. ## Summary Validating a schedule (creating or updating one through the API or the dashboard, and deploying a project that declares schedules) looks up the newest version of a task by slug. That lookup reads *every* version of the task and sorts them to return one. A project gains a row per task on every deploy, so the work grows with the project's age: the oldest projects pay the most, and dev-mode redeploys make it worse. This was picked because it was the largest single consumer of database time on the schedules path, and the fix is a sort key with no index behind it. ## Fix `BackgroundWorkerTask` is indexed on `(projectId, slug)`, which serves the equality but not the `ORDER BY createdAt DESC`. Postgres seeks the index, then bitmap-scans and top-N sorts the whole group to produce a single row. Adding `createdAt` to the index lets it scan backward and stop at the first row. The same call site also selected all 21 columns, including five JSON blobs, to read one field (`triggerSource`), so it now selects that field alone. ## Benchmark Local Postgres 17, 997,000 seeded rows / 748 MB, group sizes chosen to match the distribution seen in production. | Group size | Before | After | | --- | --- | --- | | 15,000 versions of one task | 11.118 ms, 1,510 buffers, 15,000 rows scanned | 0.027 ms, 4 buffers, 1 row | | 2,000 versions of one task | 2.081 ms, 1,455 buffers, 2,000 rows scanned | 0.022 ms, 4 buffers, 1 row | ``` before: Limit -> Sort (top-N heapsort) -> Bitmap Heap Scan after: Limit -> Index Scan Backward using BackgroundWorkerTask_projectId_slug_createdAt_idx ``` An ascending index scanned backward is enough here, so no descending index is needed. ## Impact and risk Real-world gain lands between the two rows above and scales with how many deploys a project has accumulated. Projects with few deploys will see little change, since there is barely anything to sort. The new index costs noticeably more than the existing two-column one: 43 MB against 7.3 MB on the benchmark rig. Adding `createdAt` makes every key unique, which defeats btree deduplication, so this is a real disk and write cost rather than a rounding error. Writes to this table happen at deploy time, not on the run path, so the write amplification is acceptable. The existing `(projectId, slug)` index is now a redundant prefix and could be dropped, but this PR keeps it so index usage can be observed before removing it. Behavior is unchanged: same predicate, same ordering, same row returned. The narrowed select is the only code change, and the field it keeps is the only one the caller read. Deploy note: the migration is `20260806100000_add_background_worker_task_project_id_slug_created_at_index` and uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it can be pre-applied by hand before the deploy.	2026-08-07 11:17:10 +01:00
Eric Allam	6c6e58e6ff	perf(webapp): batch declarative schedule cleanup queries (#4522) ... ## Summary `syncDeclarativeSchedules` runs on every background-worker creation (every deploy, and every file save during `trigger dev`). It issued one instance-delete per declarative schedule the current worker no longer declares, in a loop, and the overwhelming majority of those deletes matched zero rows. This collapses the loop into at most two set-based statements and skips the instance delete entirely when the current environment owns no instance of the schedule. ## Why so many, and mostly no-op The loop runs once per entry in `missingSchedules`, which starts as every DECLARATIVE schedule for the whole project across all its environments (the query filters only by `projectId`). A schedule leaves that set only when a declared task matches it by `taskIdentifier` **and** the schedule already has an instance in the current environment. That last clause is the amplifier. When a task's schedule has no instance in the current environment, the create branch inserts a brand-new `TaskSchedule` row with an instance for this environment rather than adding an instance to the existing row. So the same scheduled task, once it has run in dev and been deployed to prod, exists as two separate schedule rows: one carrying a dev instance, one carrying a prod instance. On a dev worker sync of that project: - the dev-instance row matches the declared task and is removed from the set - the prod-instance row has the same `taskIdentifier` but no dev instance, so it stays in the set and gets `deleteMany(taskScheduleId = prodRow, environmentId = dev)`, which matches zero rows So every declarative task that has been synced in another environment contributes one guaranteed no-op delete per sync, and the count scales with (declarative tasks x environments), plus any leftover rows from renamed or removed tasks. A project does not need to have dropped a schedule to generate these; it just needs the same declarative tasks present in more than one environment, which is the normal develop-in-dev, deploy-to-prod case. ## Fix The candidate schedules are already loaded with their instances, so the branch is decided in memory: - schedules with no instances (or only current-environment instances) are removed in a single `taskSchedule.deleteMany` - schedules that still have another environment's instance have only the current environment's instance detached, in a single `taskScheduleInstance.deleteMany`, and only when such an instance actually exists Behavior is unchanged (cascade delete still removes the instances of a deleted schedule); the difference is statement count. A zero-row delete writes no WAL and creates no dead tuples, so the removed work was pure query and commit overhead. Verified with a testcontainer test (red before, green after) counting the emitted deletes across the no-op, batched-detach, and schedule-delete cases, and end to end through `trigger dev`: three declarative schedules created, surviving a re-sync, then two removed in a single batched delete with the third preserved.	2026-08-07 10:27:39 +01:00
Matt Aitken	04f9c4e1a5	fix(webapp,run-engine,core): drop the hidden debounce ceiling, fail fast on an unusable maxDelay (#4521) ... Debouncing with a `delay` longer than an hour did nothing at all. The engine applied a server-side ceiling on how long a debounced run could be pushed back, measured from the run's `createdAt` and defaulting to one hour. A run is only pushed back while its new execution time stays inside that ceiling, so a `delay` at or above it could never push anything: the waiting run was released, the trigger started its own run, and the next trigger repeated it. A `delay: "12h"` produced one run per trigger, each correctly delayed by 12h, with no error raised and nothing on the run to show the debounce key had been ignored. The ceiling is now unset by default. A debounce key with no `maxDelay` keeps collapsing triggers for as long as they keep arriving, which is what the docs have always described. Self-hosters who want a bound can still set `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS`. That has a consequence worth stating plainly, so the docs now carry a warning for it: with no `maxDelay`, a continuously triggered key never executes. Set `maxDelay` when the work has to happen eventually. **Failing fast on an unusable `maxDelay`.** A caller who sets `maxDelay` no longer than their `delay` hits exactly the dead end described above, so that pair is now rejected at trigger time instead of silently behaving as if no debounce were set: ``` debounce.maxDelay (1h) must be longer than debounce.delay (12h). A debounced run is only pushed back while it stays inside maxDelay, so with these values every trigger would create its own run. ``` An unparseable `maxDelay` is rejected too, rather than quietly falling back to no bound at all, and so is a `delay` given as a date rather than a duration, which could never work because the value is re-applied on every push. The same check runs against a configured server ceiling, so a self-hosted deployment that sets `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS` gets the error rather than the silent failure this PR is about. With no `maxDelay` and no configured ceiling, which is the default, there is nothing to conflict with and nothing is rejected. The docs, the `TriggerOptions` JSDoc and the engine option all now state that the room available to push is the gap between `delay` and `maxDelay`. The run engine suite gains the case that motivated this: four triggers on one key with a 12h delay now collapse to a single run.	2026-08-07 07:55:35 +00:00
Chris Arderne	088f68b373	feat(webapp): share rate limit bucket across additional API keys per environment (#4508) ... ## What Rate-limit the API by **environment** rather than per API key. Previously the limiter keyed its bucket on the hash of the full `Authorization` header — one bucket per key. With additional environment API keys (`tr_*_sk_*`), an environment can mint many keys and each got its own full bucket, so more keys = higher effective rate limit. This collapses all of an environment's keys onto a single shared per-environment bucket, so the ceiling is exactly the configured limit regardless of key mix. ## How - `authorizationRateLimitMiddleware` now lets the override return `{ config?, identifier? }`. `identifier`, when present, is the rate limit bucket key; otherwise it falls back to the hashed `Authorization` header (unchanged legacy behavior, still used by `engineRateLimiter` and any unauthenticated fallthrough). - `apiRateLimiter`'s override resolves the environment id and uses it as the identifier: - **Additional keys** (`isAdditionalApiKey`) resolve via a new `resolveAdditionalApiKeyRateLimitScope()` — a **scope-agnostic** keyHash → (environmentId, org limiter config) lookup. It is deliberately permissive (restricted keys resolve too) because it's used **only for bucketing, never as an auth decision** — request auth still goes through the RBAC bearer controller, which enforces scopes. Revoked/expired keys are excluded so they can't hold a bucket warm. - **Root/legacy keys** reuse the environment already resolved by `authenticateAuthorizationHeader` and key on `environment.id` too. - The identifier is always the stable environment id, never the secret key (which can rotate and would split the bucket). - The whole override result is cached per key by the existing SWR cache, so **no extra per-request lookup and no separate Redis mapping** is added. ## Behavior notes - Root + additional keys of the same environment now share one bucket (ceiling = configured limit, not a multiple of it). Restricted additional keys are included — they were the biggest gap, since they authenticate via the RBAC controller and previously fell back to per-key buckets. - **Public JWTs** keep their existing fixed-window, per-token bucketing. - One-time bucket reset on deploy (bucket keys change); harmless. ## Tests - New: two tokens resolving to the same identifier share one bucket. - New: with no identifier, bucketing stays per-key (legacy behavior preserved). - Updated existing override tests to the new `{ config }` return shape. Base: `feat/multi-keys-surface`. Closes TRI-12888.	2026-08-06 16:05:27 +01:00
Chris Arderne	9409ddf9bc	feat(webapp): add multiple environment API key management (#4390) ... ## Summary Projects can create, inspect, expire, and revoke multiple API keys for each environment. Plaintext values are shown only at creation; stored credentials are hashed and the API keys page displays only an obfuscated suffix afterward. Self-hosted installations support full-access additional keys by default. Authorization extensions can provide additional access presets and optional task selection. Additional keys can also mint scoped public access tokens through the Trigger.dev API without receiving the environment signing key. ## Feature notes - Only admin+ can create API keys (Developer can make in Development branch). - JWT self-signing will be a server call when used with new `_ak_` keys. - JWTs with long expiry can keep working even with api key deleted (gets priveleges from api key, signed with root key) - Unfiltered session listings intentionally preserve the existing broad task-read behavior. Filtered listings enforce task-level scopes for every requested task. - Buffered runs without a task identifier are not safely authorizable, so cancel/replay requests fail closed rather than resolving an unscoped run. - Batch and waitpoint endpoints intentionally return server-minted, narrowly scoped public tokens to all callers. These tokens have bounded lifetimes and may remain valid until expiry after API-key revocation. ## Deployment notes Deploy the management UI and public-token endpoint with new key creation disabled. Enable creation for selected organizations after the authentication path and released SDK have been verified, then expand availability gradually. Revoking an API key prevents new bearer requests and new token minting. Public tokens already minted by that key remain valid until their own expiration because they are signed by the environment signing key. ## TODO - [x] Add "Created by" to the key table - [x] Document that streamed batch ingestion is non-atomic and may partially accept items before a validation or authorization error. ## Follow-ups - [x] Add an organization-level feature flag for the API key management UI and creation action. - [x] Document rollout ordering: enable additional-key lookup before enabling issuance. - [x] Add a system-wide gate that can stop new key issuance without disabling authentication for existing keys. - [x] Replace the generic SDK compatibility warning with the first published compatible version. Old SDK will mint an unusable token if given an `_ak_` key. - [x] Add public documentation covering creation, storage, expiration, revocation, SDK compatibility, and public-token lifetime behavior. - [x] Add observability for key creation, revocation, policy preparation failures, and public-token mint failures. - [ ] Exercise create, copy-once display, authenticate, mint, expire, and revoke flows end to end before broad enablement.	2026-08-06 15:27:10 +01:00
Katia Bulatova	337dda1e97	feat(webapp): name of the page in tab titles (#4517) ... Adds a shared `pageMeta()` helper and 74 route declarations, so a title reads `run_abc | Runs | Trigger.dev` — the specific thing first, then the page. Org pages also carry the organization: `Team | Acme | Trigger.dev`. Inside a project no scope is added, because the dashboard switches projects in every tab at once. Page names are unchanged; what's new is that a page says which one it is at all. Three wording changes on purpose: the queue page now names the queue, the model page names the model, and entity pages carry their section.	2026-08-06 10:34:30 +02:00
Eric Allam	b20806247f	fix(run-store): stop run-create failing on a brief write stall (#4514) ... ## Summary On the run-ops store, creating a run could intermittently fail with a "Transaction already closed" error, and the run would never be created. Single-write run creates no longer run inside an interactive transaction, so a brief database write stall can't blow the transaction budget and drop the run. ## Fix The dedicated run-ops `createRun` / `createFailedRun` wrapped a single nested `taskRun.create` in an interactive `$transaction`. Its default 5s budget is wall-clock from `BEGIN`, so when a write briefly stalls the transaction expires before the create completes and throws, even though the statement itself is fast at the database. A single-write create does not need an interactive transaction: Prisma's implicit nested create is already atomic and holds no app-side budget, so it now runs directly. Only the `triggerAndWait` path (run plus its associated waitpoint, two writes that must commit together) keeps an interactive transaction, now with headroom over the default. Verified with a red/green test against the real split topology (reproduces the exact expiry on the unchanged code, green after) and an end-to-end run created and completed through the dedicated store.	2026-08-05 17:35:46 +01:00
Eric Allam	58bf4e2833	feat(webapp): per-client database pool and connect timeout overrides (#4515) ... ## Summary Follow-on to #4513. The database connect timeout is now honored, but a single global value has to serve three separate databases at once (control-plane, legacy run-ops, and run-ops). This adds optional per-client overrides for the Prisma pool and connect timeouts, one pair for the writer and one for the read replica of each of the three databases, each falling back to the shared `DATABASE_POOL_TIMEOUT` / `DATABASE_CONNECTION_TIMEOUT` when unset. That lets one database's clients run a fail-fast connect timeout (with a bounded pool wait) while another keeps more headroom, without a single knob forcing the same tradeoff everywhere. No behavior change until an override is set. It also tags each client's queries with its specific datasource (`control-plane` / `legacy-run-ops` / `run-ops`, writer or replica) via the `db.datasource` span attribute, so telemetry can attribute connection behavior to a specific database instead of just writer-vs-replica.	2026-08-05 17:26:41 +01:00
Eric Allam	771937adf5	fix(webapp): clamp run priority so a large value can't fail run creation (#4512) ... ## Summary Triggering a run with a very large `priority` could fail run creation outright with an opaque database error. `priority` is multiplied by 1000 and stored in a 32-bit integer column, with nothing bounding it, so a big enough value overflowed the column and the create failed. The trigger now caps the value to the highest supported priority instead of erroring, so the run is still created. ## Fix `priorityMs` (the stored `priority * 1000`) now goes through a `clampPriorityMs` helper before the write. It rounds to a whole number and clamps into the column range at both ends, so only a valid integer ever reaches the column and an out-of-range priority caps rather than failing. Single and batch triggers share the write path, so both are covered.	2026-08-05 16:28:22 +01:00
Eric Allam	3039bc14d6	fix(webapp): honor the configured database connect timeout (#4513) ... ## Summary Every Prisma client built its connection URL with a `connection_timeout` query param, but the Postgres connector's parameter is `connect_timeout`. The misspelled param is silently ignored, so all clients fell back to Prisma's 5s default instead of the configured timeout. When establishing a new connection briefly took longer than 5s (for example during connection spikes), it failed with `Can't reach database server` even though the database was healthy. ## Fix All four client builders now construct their connection URL through one shared helper (`buildPrismaConnectionUrl`) that sets `connect_timeout`, so the configured value actually applies, and the parameter name lives in exactly one place. Covered by a unit test.	2026-08-05 15:52:12 +01:00
Katia Bulatova	fbd6df33b4	feat(webapp): Themes + contrast settings update (#4206) ... Adds System Preferences, Dark and Light themes, gated by the `hasThemeSwitcher` feature flag (off by default — dark stays the default theme for everyone). Old theme is now "Classic"and set as default. "System preferences" theme has both Light and Dark modes and uses your laptop settings to use a correct one. It has less color accents (specifically less colored text), and they are the same for both modes, only grayscale values change between them. And Light/Dark themes can be used separately. New Contrast setting is available for System Preferences, Dark and Light themes - it changes the contrast for the whole app. All new visual Settings live in Account.	2026-08-03 19:29:33 +02:00
nicktrn	3fba04573d	fix(supervisor): hold the last backpressure verdict when a read fails (#4444) ... The dequeue brake released the moment its signal became unreadable. `refresh()` caught any error from `source.read()` and set the verdict to `null`, which `computeEngaged()` treats as not-engaged — so a few failed reads dropped an engaged brake, silently, with no log and no metric. That handling was symmetric while the risk is not. A source that has stopped answering correlates with the pressure the brake exists for, so releasing on read failure gives up protection at exactly the wrong moment; holding too long only costs throughput. Now a failed read keeps the last verdict instead of discarding it. The verdict then ages normally, so the existing `maxVerdictAgeMs` check becomes the grace window and still bounds how long a dead source can hold the brake — a permanently unreachable source releases it rather than pinning dequeuing forever. Because `computeEngaged()` only consults staleness for an *engaged* verdict, a released one is unaffected and stays released. The default grace moves from 15s to 120s, comparable to how long the brake normally stays engaged. One guard worth calling out: holding is only safe when something bounds it, so when `maxVerdictAgeMs` is unset the previous discard behaviour is kept. Otherwise an unbounded hold could pin the brake indefinitely. Read failures were previously invisible — the catch block neither logged nor counted. Adds a `read_failures_total` counter, plus an error log on the transition into failure rather than once per tick, since the refresh loop runs every second. The post-release ramp needs no change: it anchors off the engaged-to-released transition, so a grace-window release still ramps back up instead of snapping to full rate, which is what you want after a blind period. Tests cover holding while reads fail, releasing past the max age, and the existing unbounded-config paths are unchanged.	2026-08-03 18:06:23 +02:00
nicktrn	8f9db53350	feat(supervisor): configurable tolerations for run pods (#4491) ... ## Summary Self-hosted Kubernetes deployments can now add tolerations to run pods, so runs can schedule onto tainted nodes. Previously the only way to do this was to patch the supervisor. `KUBERNETES_RUNNER_TOLERATIONS` takes a comma separated list of `key=value:effect`, or `key:effect` to tolerate any value. It applies to every run pod, and for runs from a schedule tree it merges with the existing `KUBERNETES_SCHEDULED_RUN_TOLERATIONS`. Left unset, nothing changes: no tolerations are added and the pod spec leaves the field off entirely. The Helm chart takes it as a list: ```yaml supervisor: config: kubernetes: runnerTolerations: - dedicated=runs:NoSchedule - spot:NoExecute ``` ## Naming The issue proposed `KUBERNETES_WORKER_TOLERATIONS`. This ships as `KUBERNETES_RUNNER_TOLERATIONS` instead, because `RUNNER_*` is already the prefix for run pod settings (`RUNNER_HEARTBEAT_INTERVAL_SECONDS`, `RUNNER_ADDITIONAL_ENV_VARS`, and `DOCKER_RUNNER_NETWORKS` for the Docker equivalent), whereas "worker" refers to the supervisor itself throughout this app. ## Validation Keys and values are checked against the Kubernetes naming rules when the supervisor starts, so `dedicated=prod runs:NoSchedule` fails immediately with a message naming the offending entry. Without that check a bad value is accepted at startup and then rejected by the API server on every pod create, which stops all runs with the cause buried in an API error. `KUBERNETES_WORKER_NODETYPE_LABEL` is trimmed and validated for the same reason: surrounding whitespace is not valid in a label value, so a padded value fails every pod create today. ## Node selector off switch `KUBERNETES_WORKER_NODETYPE_LABEL` accepts an empty string to skip the node selector entirely, so runs schedule on any node. This already worked and the Helm chart has always shipped it empty, but it was not documented. It is now. The issue also asked for general node affinity configuration. That is not included: the node selector off switch plus tolerations covers the reported problem, and a free form affinity setting is a much larger config surface to commit to. Fixes #4458	2026-08-03 15:40:41 +00:00
Eric Allam	9d57aff542	fix(webapp): make the Queues hero charts environment-wide (#4486) ... ## Summary The four charts above the queues table aggregated over **at most the 25 queues on the current page**. They reused the loader's already-paginated queue array as a ClickHouse `queue IN (...)` filter, so paging or re-sorting changed the values, and a name search matching nothing blanked the whole chart row. The stat tiles above them were already environment-wide, so the two rows disagreed. They now read `env_metrics`, the environment-level rollup that already exists for exactly this (the built-in Queues dashboard and the health report read it). That is both correct and queue-count-independent: no `GROUP BY queue` across an entire environment, and no client-side summing. Note this is not only a paging artifact: page 1 under-reported too. On the seeded environment below, page 1 read 82% saturation against a true 87%, because the environment's running total is not the sum of one page of per-queue gauges. Three related fixes ride along. **Scheduling delay and throttling sawed to zero.** Both are event-driven, so at the 10-second bucket a short range picks, most buckets hold no samples at all and were drawn as `0ms`. Measured over a 1-hour window: **232 of 349 buckets had no scheduling-delay samples**. A bucket where nothing started is not a bucket where nothing waited, so the line was both ugly and wrong. TRQL grows a `minBucketSeconds` floor, plumbed through the metric resource route, and the hero tiles set 60s. Buckets that still have no samples render as a gap instead of a dive to zero. **The floor must not feed a width-dependent headline.** Two of the four headlines are not peaks, so widening the plotted buckets moved them: - **Throttled** is a share of buckets that saw any throttling, so a single brief throttle came to mark a whole minute instead of ten seconds: the same seeded events read 17% at 10s and 85% at 60s. - **Scheduling delay p95** is a percentile, and merging quantile states over a wider bucket yields a p95 between the sub-buckets' own. Two 240s samples among twenty in one 10-second sub-bucket give a worst-of-six p95 of 240,000ms against a merged 60-second p95 of 5,000ms — a 48x understatement of a headline whose tooltip claims it is the worst in the window. Both charts keep the floor, since a readable line was the point of it. Their headlines now come from a second query at the range's natural bucket width, via an optional `readout` on the tile, so each means what its tooltip says regardless of how the plotted buckets are sized. Saturation and backlog are genuinely width-invariant (a max of maxes is the same at any width), so they are unchanged and issue no extra query. Both caught by Devin in review; I had wrongly lumped p95 in with the peaks. **Charts reported a hydration mismatch on every render.** Recharts resolved victory-vendor's CJS entry on the server and its ESM entry in the browser. Those bundle different d3-shape builds, and the CJS one predates d3-path's digit rounding, so every server-rendered curve carried full-precision coordinates while the client rounded to 3 decimals: ``` Server: M0,3C0.9305555555555555,3,1.8611111111111112,3,... Client: M0,3C0.931,3,1.861,3,... ``` Bundling recharts for SSR makes both sides resolve the same ESM build. Verified: 45 of 45 server-rendered chart curves now match the client, and the page loads with an empty console. ## Verification An isolated stack with 40 seeded queues (20 heavily loaded, 20 idle) and 90 minutes of 10-second buckets written into `queue_metrics_raw_v1`, so the real materialized views built `queue_metrics_v1`, `env_metrics_v1` and the 5m rollup. Ground truth for the environment: 260 running against a limit of 300 (**87% saturation**), 800 queued. | | before | after | | -- | -- | -- | | Saturation, page 1 | 82% peak | **87% peak** | | Saturation, page 2 | 5% peak | **87% peak** | | Backlog / delay, page 2 | "No activity" | **800 peak / 59.5s** | | Name search matching nothing | all four charts blank | charts stay environment-wide | | Metric refetches on a page change | 4, each painting a skeleton | **0, no skeleton** | | Buckets drawn as 0ms with no samples | 232 of 349 | **0** | | Throttled readout | 17% | **17%**, unchanged by the wider buckets | | Worst-p95 readout source | plotted buckets | **natural width**, so a sub-minute spike is not averaged away | | Crosshair reach, hovering one detail-page chart | 2 of 4 others | **4 of 4** | | SSR chart curves mismatching the client | 45 | **0** | The bucket floor was measured across ranges: it widens 10s to 60s at 30m and 1h, and is correctly a no-op at 12h (300s) and 7d (3600s). One extra request per page load, for the throttled readout. The built-in Queues dashboard, which reads `env_metrics` independently, agrees at 86.7% and 260 of 300. `internal-packages/tsql` suite green (612 tests), including 5 new ones for the floor that fail without it. Webapp typecheck, oxfmt and oxlint clean. Spot-checked the Run metrics dashboard and the per-queue detail page for SSR regressions from bundling recharts: both render, console clean. The queue detail page carries the same event-driven series, so its scheduling delay, throttling and per-key mean delay take the same treatment. ## Screenshots <img width="2540" height="580" alt="after-page1-charts" src="https://github.com/user-attachments/assets/6cd23f9c-e7fd-4918-bcfa-b1d3340b16d1" /> ## Rollout Already behind the per-organization `queueMetricsUiEnabled` flag, so only gated orgs see any of it. Blast radius is chart values on one page plus the SSR bundling of recharts; rollback is a revert with no data migration. ## Stated limitations - `wait_ms_count` and the quantile state both only count `wait_ms > 0`, so "nothing started in this bucket" and "everything started instantly" are indistinguishable in storage. Both render as a gap. Distinguishing them needs a schema change, which is not in this PR. - The queue name search deliberately no longer narrows the charts. It only did so incidentally and incorrectly before (first 25 matches, and blanked on zero matches). Search-scoped charts would need the full unpaginated matching set and a server-side aggregate; worth its own ticket if we want it. - Bundling recharts for SSR grows the server bundle slightly. That is the cost of both sides resolving one d3-shape build. - The plotted delay line is a smoothed 60-second view, so a sub-minute spike above the one-minute warning threshold can fail to colour the line even though the headline reports it and colours itself. - Every chart inside one synced group shares the floor, because the hover crosshair is a reference line on a category x-axis and only draws where the hovered bucket exists in the other chart's own data. That costs the queue detail page's gauges some resolution (1 minute instead of 10 seconds) in exchange for the crosshair working across the row. Separately, while taking the screenshots I found a pre-existing rendering bug unrelated to this change: a **perfectly flat** saturation series draws no line at all (the readout still shows the right percentage), which looks like the threshold gradient's offset degenerating when the series min equals its max. It reproduces on `main`, so it is not a regression here and I have left it alone; filed as its own issue. Refs TRI-12784	2026-08-03 16:19:50 +01:00
Katia Bulatova	859f30e224	fix(webapp): report message catalogs survive the production bundle (#4488) ... GET /api/v1/reports/health threw `no catalog registered for report "health"` in production (fine in dev): the catalog registered itself as a side effect of a bare import, which the SSR build tree-shakes under `"sideEffects": false`. Verified on the built server bundle — main's is missing the catalog, this branch's carries it. Fix: catalogs are values on the report registry entries; the resolver reads them from there and the mutable register-at-import step is gone.	2026-08-03 16:12:03 +02:00
Chris Arderne	763b5dc582	feat(webapp): enforce scopes for environment API keys (#4389) ... ## Summary Environment API keys backed by the additional-key table can authenticate API requests using their stored effective scopes. Revoked and expired keys are rejected, branch environments retain their existing routing behavior, and last-used timestamps are updated on a throttled best-effort basis. ## Design API route builders receive the resolved ability and reject restricted keys on routes without an authorization declaration. Existing deployment, environment variable, queue, run, task, batch, session, and waitpoint routes declare the resources they access. Trigger and batch responses return server-signed public access tokens, so additional keys never need access to the environment signing secret. Root-key rotation also keeps public tokens valid for the existing grace window. ## Feature notes - Root environment keys remain unrestricted for backward compatibility. Additional keys enforce their persisted scopes and fail closed on routes without an authorization declaration. - Machine-key requests never exchange one credential for another. Additional keys cannot retrieve the root key, and rotated root keys are not upgraded during their grace window. - Public JWT validation remains host-owned, while installed RBAC plugins continue to supply root-key abilities. - Unfiltered session and run listings preserve existing broad task-read behavior. Filtered requests enforce the supplied task identifiers. - Related-run summaries remain embedded in run retrieval for API compatibility. Retrieving or mutating a related run independently still requires permission for that run. - Queue management authorizes at collection scope, matching the queue permissions currently issued. - Batch responses deliberately include server-signed public access tokens for all clients. Selected-task credentials continue using their original credential for per-item authorization. - Two-phase batches authorize declared task identifiers before creation and authorize every streamed item. Streaming paths that cannot declare the complete task set remain fail closed. - Authentication telemetry records successful credential resolution separately from subsequent resource-authorization failures. - API keys are high-entropy random tokens. SHA-256 is intentionally used for deterministic indexed lookup, not password hashing. ## Deployment notes The schema migration must be present before this code is deployed. Because bearer resolution runs on every authenticated request, deploy the resolver with additional-key lookup disabled, verify root-key and public-token parity, then enable lookup before any additional keys can be issued. The multi-task authorization tightening changes the result for narrowly scoped tokens that request tasks outside their grants. Observe would-deny results before enforcing that check. Request-idempotency keys are also newly isolated by environment and task, so a retry crossing the deployment boundary may execute once more before old cache entries expire. ## Follow-ups - [x] Add a system-wide kill switch for additional-key lookup, defaulted off for the initial deployment. - [x] Add authentication observability by credential kind, result, latency, and lookup path without recording credential values. - [ ] ~Add would-deny observability and an independent enforcement switch for multi-task authorization.~ - [ ] ~Add an independent switch for server-issued batch tokens while root-key parity is verified.~ - [ ] Confirm every API route reachable by a restricted key has an explicit authorization declaration or intentionally fails closed. - [x] Verify root-key rotation, revoked-key grace, and public-token validation through each bearer resolver path.	2026-08-03 14:00:29 +01:00
Eric Allam	5f29ae49ab	feat(webapp): default the queue metrics period to 1 hour and remember it (#4438) ... ## Summary The Queues list and queue detail pages opened on a 1 day window, and went back to it every time you navigated between queues or reloaded. They now default to the last hour, and the period you pick is remembered across navigations and refreshes. ## Design The last period is stored in a `queueMetricsPeriod` cookie, written client-side whenever a `period` lands in the URL and read by both loaders. A cookie rather than localStorage because the queues list renders its per-queue metrics columns server-side: with localStorage the page would paint the 1 hour default and then re-fetch, and the picker would flash the wrong window. Both pages resolve the window once, in one place, and pass it down: ```ts period: resolveQueueMetricsPeriod({ period: value("period"), // a usable period in the URL wins from: value("from"), // an absolute range means "no period" to: value("to"), defaultPeriod, // otherwise the remembered default from the loader }), ``` That keeps the picker pill and every chart query on the same value, so no call site falls back to its own default. Periods the picker could never produce (a hand-edited `?period=garbage`, or a window past the 30 day retention) fall back to the default, and the picker renders the resolved window rather than the raw search param so the label can't disagree with the data. Absolute from/to ranges, including drag-to-zoom, are not remembered, since they would pin later visits to a window that has gone stale. While wiring that up: the two queue-metric queries that go straight to ClickHouse (the list table and the concurrency-keys endpoint) never applied the org's `queryPeriodDays` limit, so a hand-typed `?period=` read further back than the plan allows. Everything behind `/resources/metric` is already clipped that way by `executeQuery`; both of these now clip with the same limit, capped at the retention window, and the plan cap is resolved once per load and handed to the page instead of each route deriving its own copy from the client-side subscription. Verified on both pages: default with no cookie is 1 hr, picking 6 hrs survives navigating away and back to a param-free URL and a hard reload, clearing the cookie returns to 1 hr, an oversized period falls back without being remembered, and an absolute range still renders as a range.	2026-08-03 10:09:34 +01:00
Iss	8f66af6e18	fix(webapp): stop the sidebar feedback popover from canceling the submit (#4445) ... The Help & Feedback → "Contact us" form in the sidebar intermittently failed to send. The `<Feedback>` dialog was nested inside the Help popover, so clicking **Send** closed the popover and unmounted the form mid-submit — canceling the `POST /resources/feedback` before it went out. The message was silently lost (the success toast still shows). A race, so it "worked sometimes"; the standalone "I'm stuck!" path was unaffected. **Fix:** host the Feedback dialog *outside* the popover (same pattern as `AskAIRoot`) and open it from the menu item, so closing the popover no longer tears down the form. `Feedback` gains an optional controlled `open`/`setOpen` mode; existing `button`-triggered usages are unchanged. ## Changes - `Feedback.tsx` — optional controlled `open`/`setOpen`; `button` now optional. - `HelpAndFeedbackPopover.tsx` — "Contact us…" opens a `<Feedback>` hosted outside `PopoverContent`. - `.server-changes/fix-sidebar-feedback.md` — user-facing note. ## Testing Webapp typecheck passes. Sidebar "Contact us…" now sends on every attempt (Network: `POST /resources/feedback` → `204`, never `(canceled)`); "I'm stuck!" and the `?feedbackPanel=` open path unchanged.	2026-08-02 14:49:09 +01:00
Eric Allam	f9c8d518c7	perf(webapp,run-engine,database): resolve the newest worker and deployment by createdAt (#4452)	2026-08-01 11:32:21 +01:00
Eric Allam	0445b8ec27	fix(webapp,clickhouse): keep the rest of a ClickHouse batch when one run or span has un-ingestable JSON (#4358) ... ## Summary A single run output, trace span, or payload carrying JSON that ClickHouse can't ingest (for example nesting past its depth limit) used to fail the whole insert batch, so unrelated runs and spans silently disappeared from the runs list, traces, and logs. This keeps the rest of the batch and handles the offending row instead of dropping everything around it. ## Fix Recovery is per-table, matched to what each table needs: - **Runs** (`task_runs_v2`) keep their status. We follow ClickHouse's failing-row hint to strip just the un-ingestable JSON column(s) so the run still lands (its output reads from Postgres on the detail page), up to a configurable limit (`RUN_REPLICATION_MAX_POISON_STRIPS_PER_BATCH`, default `1`). Past the limit we stop and land the batch with `allow_errors` in a single pass, skipping the remainder. Cost stays a fixed handful of inserts no matter how large or poisoned a flush is. - **Trace events and payloads** (high volume, append-only) recover with a single `allow_errors` insert: the good rows land in one pass and only the un-ingestable rows are skipped. Before falling back, a lightweight sanitizer still repairs what it can losslessly (lone UTF-16 surrogates, out-of-range integers) so a repairable row lands in full. To read the failing-row hint we patch `@clickhouse/client-common`: its error parser truncates the server response and discards the `(at row N)` position, so the patch preserves the full text for the recovery path to read.	2026-08-01 09:17:20 +01:00
nicktrn	b42e5c3771	fix(supervisor): count pods from a limit=1 list instead of an aggregate metric (#4442) ... The pod-count backpressure source read `apiserver_storage_objects{resource="pods"}` from an apiserver `/metrics` scrape. That gauge is a periodically-refreshed cached count, and it is served by whichever apiserver replica the scrape lands on — replicas disagree with each other at the same instant, by enough to swamp the engage/release hysteresis band. Engage and release timing was therefore partly a function of scrape routing. This replaces it with a single `limit=1` list of the workload namespace and computes `remainingItemCount + items.length`. One pod object transferred, no informer, no watch cache. Two request-shape constraints are load-bearing and called out in the code: passing a label or field selector makes the apiserver omit `remainingItemCount` entirely, and setting `resourceVersion` serves a cached count rather than a quorum read. Neither is passed. `remainingItemCount` is only set when the list is truncated, so `_continue` is the truncation signal — if it is absent the returned page is the whole collection and `items.length` is already exact. If the list *is* truncated and the count is missing or implausible, the fetcher throws rather than guessing. Failure semantics are unchanged: a throw lands in the monitor's existing catch, exactly as the previous parse did. The hysteresis, verdict shape, and gauge are untouched. RBAC is unchanged — the existing role already grants `pods: list`. The `/metrics` non-resource grant in the deployment role becomes unused, and the scrape-timeout env var is now a slight misnomer. Both left alone deliberately: the grant may be wanted again for other apiserver signals, and renaming the var would need a coordinated config change for no behavioural gain. Tests cover the not-truncated, truncated, missing-count, negative-count and timeout paths.	2026-07-31 19:45:37 +01:00
Eric Allam	f10bc23785	perf(run-engine,run-store): one execution snapshot per triggered run (#4419) ... A non-delayed run used to get two execution snapshots the moment it was triggered: `RUN_CREATED` nested in the run-create transaction, immediately followed by `QUEUED` from its own `BEGIN`/`INSERT`/`COMMIT`. It now gets a single `QUEUED` snapshot written inside the create, and the trigger path only publishes to the queue. One fewer row per run on `TaskRunExecutionSnapshot`, and one fewer round trip on the trigger hot path. `EnqueueSystem` gains a `publishRun` seam that enqueues without writing a snapshot. Every re-enqueue path (waitpoint resume, checkpoint restore, delayed enqueue, pending version, retry requeue) still calls `enqueueRun` and writes its own `QUEUED`, so only the first enqueue changes. The `QUEUED` snapshot still commits before the queue message, so a dequeue sees a dequeueable status exactly as before. Two things for reviewers. Nesting the write skips `createExecutionSnapshot`, which is what emits `executionSnapshotCreated` and therefore the run timeline's `[engine] QUEUED` entry, so the trigger path now emits it directly, the same way the dequeue and attempt-start paths already do for their nested creates. And `RUN_CREATED` is still written when a dequeued run has no background worker yet, so the status and both `statuses.ts` helpers stay live and existing rows keep reading correctly. Delayed runs are untouched: `DELAYED` then `QUEUED` are two genuinely different moments and stay two snapshots. Rollback is a revert. Create-and-enqueue happen in one request in one process, so no in-flight run needs both code paths to agree during a rollout. One note for whoever debugs this path later. The `QUEUED` snapshot now commits before the queue publish, so a failed publish leaves the run recorded as `QUEUED` with no queue message. That state was already reachable, since the publish was never part of the snapshot transaction, but it used to be recorded as `RUN_CREATED`, which was distinctive because it never otherwise persisted. `QUEUED` with no message is indistinguishable from a run waiting on a concurrency slot, so trigger-time publish failure is now one more cause of an apparently stuck queued run.	2026-07-31 16:12:07 +01:00
Eric Allam	c72ebf9084	fix(webapp,run-engine): stop batchTriggerAndWait hanging when item streaming never completes (#4397) ... ## Summary `batchTriggerAndWait()` could leave a parent run waiting forever. The 2-phase batch API blocks the parent on the batch's waitpoint as soon as the batch is created, but the batch is only sealed at the end of item streaming. If streaming never completed, nothing sealed the batch, nothing completed the waitpoint, and the parent stayed suspended with no timeout and no way to recover. Supersedes #4016, which added the reaper alone. ## Fix Admission for item streaming was being decided twice. Batch creation passes its own rate limiter, which fixes `expectedCount` and blocks the parent, and then the item stream had to pass the general API limiter as well, competing with unrelated traffic. A second limiter could therefore veto work the first had already committed the parent to. Creation now mints a bounded grant that the item stream spends, so an admitted batch can finish streaming. The grant is capped per batch rather than exempting the path, and every failure mode (no grant, spent grant, unreachable store) falls back to the normal limiter. That makes stranding much rarer but not impossible, since a request timeout or a crash can still end streaming for good. So a seal-timeout reaper aborts any batch still unsealed after `BATCH_SEAL_TIMEOUT_MS` and completes the parent's waitpoint with an error, letting `batchTriggerAndWait()` reject instead of hang. It is race-safe against a late seal, and it is only scheduled for batches that actually block a parent, so fire-and-forget batches cost nothing. Finally, the batches page used to report "Batch completion checked." for these batches while doing nothing, because the completion path returns early on an unsealed batch. It now says the batch cannot be resumed. Rate limiting is no longer the reason a batch strands, so the reaper's default stays at 30 minutes, comfortably above the SDK's worst-case stream-retry budget. ## Verification Unit and container tests cover the grant cap, the bypass ordering (it runs after the authorization check, so it can never skip authentication), and the reaper's abort, seal race, idempotency, and no-waitpoint cases. Also verified end-to-end against a running stack. With the general limit exhausted, batch creation and other API calls returned 429 while a granted batch still streamed and sealed; an ungranted batch id was rate limited rather than bypassed; and the grant cut off exactly at its configured attempt count. Reproducing the stranded state on a real parent run, the batch was aborted at the timeout, the waitpoint completed with an error, and the parent resumed and finished instead of hanging. A parentless batch left unsealed was untouched well past the reaper window. ## Verified against deployed runs The reaper was proven end to end with a real deployed run (locally-run supervisor, containerised run) and a real network fault, rather than a simulated one: toxiproxy severs the phase 2 item stream mid-flight so every SDK stream retry genuinely fails, while phase 1 still succeeds. Only the batch calls traverse the fault, so control-plane traffic is untouched. The reproduction is the shape that actually strands a parent: the task catches the `BatchTriggerError` the SDK throws and carries on, so the phase 1 block outlives the thrown error and the parent hangs at its next suspension point. With the reaper disabled, the parent sat in `EXECUTING_WITH_WAITPOINTS` for over 24 minutes holding two blockers, and stayed stuck across a full infrastructure restart: ``` type | status | has_timeout BATCH | PENDING | f <- orphan, completedAfter NULL DATETIME | COMPLETED | t <- the wait already elapsed ``` With the reaper enabled the same task under the same fault completed in about 75 seconds with zero blockers left, the batch `ABORTED`, and its waitpoint completed carrying the error. Two conditions are required to observe this at all, which is worth knowing for any future test: the run must be deployed rather than `trigger dev` (dev runs execute in process and finish while still holding blocker rows), and the wait after the caught error must exceed the checkpoint threshold, or it is served in process and never suspends. ### Why completing the batch waitpoint is sufficient `batchTriggerAndWait` runs create, then stream, then wait. A phase 2 failure throws before the wait is ever reached, and the reaper only fires on an unsealed batch, so the parent is never suspended awaiting the batch when it runs. The parent therefore does not need a synthetic result, only to stop being blocked. Note this reasoning depends on that ordering: if the wait were ever reached with an unsealed batch, completing the batch waitpoint alone would not settle the caller. ## Follow-ups - Batches stranded before this ships still need a one-off recovery; the reaper only schedules at creation time. - That same property leaves a gap if the process dies between creating the batch and scheduling the job. A periodic sweep would close it, but wants a supporting index. - When a partially streamed batch aborts, children already enqueued keep running while the parent fails. Left as-is deliberately, since cancelling triggered work is a bigger semantic call.	2026-07-31 11:55:25 +01:00
claude[bot]	debfa2b733	feat(webapp): impersonation consent page and a view-as-user toggle (#4421)	2026-07-30 21:28:44 +01:00
Oskar Otwinowski	4efe0a07c4	fix(webapp): create dev environments for SSO and Directory Sync members (#4426) ... Members added by SSO just-in-time provisioning or Directory Sync never got their per-member DEVELOPMENT environments - only invite acceptance and project creation created them. `trigger dev` returned "Environment not found" for those members and the dashboard had no dev view. ensureOrgMember now queues provisioning for every membership it settles, so both paths are covered and members missing environments are repaired on their next sync. Provisioning runs as a common-worker job to keep sign-in and directory webhooks off the per-project write loop. A failed enqueue surfaces for Directory Sync, whose worker retries the idempotent effect, and is swallowed for sign-in, where the next login enqueues again. Environment creation now tolerates a concurrent creator so the project-creation loop and the job cannot collide on the unique index. Also fixes environment resolution ignoring dev-environment ownership: a member without their own dev environment could be handed a colleague's and have it persisted as their dashboard preference.	2026-07-30 21:50:15 +02:00
Iss	d90f06ba5e	feat(webapp): migrate Plain to @team-plain/graphql + attribute support threads to org tenant (#4368) ... ## What Two changes, shipped together: 1. **SDK migration (TRI-12460).** `@team-plain/typescript-sdk` is deprecated. Move the webapp to its successors — `@team-plain/graphql` (client) and `@team-plain/ui-components` (`uiComponent` builder). Behaviour-preserving: the `PlainClient` customer upsert + thread creation move to the new `client.mutation.*({ input })` shape; the client now throws on failure, so `sendToPlain` wraps its calls and logs, staying best-effort. 2. **Org tenant attribution (TRI-12461).** When org context is available, `sendToPlain` now upserts a Plain tenant keyed by `externalId = org_id`, links the customer to it, and stamps the created thread with that tenant — so support threads become attributable to a Trigger.dev org. Wired into the four add-on quota requests and the plan-cancellation feedback (which already have org context). The tenant steps are isolated in their own try/catch and the thread's `tenantIdentifier` is gated on their success, so a tenant failure never blocks thread creation. ## Not affected - `customer.externalId` stays `User.id` — the customer cards + impersonation link are unchanged. - No ticket content leaves Plain. - Callers without a single org (e.g. the feedback widget) are unchanged — the org params are optional. ## Deploy prerequisite The webapp's Plain API key needs three **new** scopes for attribution to work (it already has `customer:create`, `customer:edit`, `thread:create`): - [x] `tenant:create` - [x] `tenant:edit` - [x] `customerTenantMembership:create` Until granted, nothing breaks — `sendToPlain` logs the forbidden error and creates the thread without attribution. ## Testing - `pnpm typecheck --filter webapp` passes; oxfmt + oxlint clean. - Ran the real `sendToPlain` end-to-end via a throwaway vitest harness against live Plain — confirmed the code path executes; the live write is gated only by the key scopes above.	2026-07-30 14:55:20 -04:00
github-actions[bot]	86b948b47a	chore: release v4.5.9 (#4408)	2026-07-30 10:12:14 +01:00
Eric Allam	6e5f0f0fe7	fix(webapp,clickhouse): stop invalid customer queries alerting, and isolate Sentry scope per request (#4372) ... ## Summary A query sent to the query API with a typo in it, like a column name that does not exist, was being reported as a server error. That put customer SQL mistakes into our error alerting, where they made up almost all of the volume on one of our noisiest alerts, and it drowned out the failures that are actually ours to fix. This makes the level match who is at fault, and fixes two related problems found alongside it. ## Invalid queries are the caller's, not ours The query API route already got this right. It checks for `QueryError`, logs at warn, and returns a 400, with a comment saying the system handles it gracefully and no alert is needed. The layer underneath ignored that. `executeTSQL` logged every exception out of its catch block at error, including the compile failures the route was about to turn into a 400, and error-level logs are forwarded to error reporting. The TSQL package already draws the line we need: ```ts export class ExposedTSQLError extends BaseTSQLError { /** An exception that can be exposed to the user. */ } export class InternalTSQLError extends BaseTSQLError { /** An internal exception in the TSQL engine. */ } ``` `SyntaxError` and `QueryError` extend the first. So the catch block now branches on `ExposedTSQLError` and logs those at warn, keeping error for `InternalTSQLError` and anything unanticipated, which is a genuine compiler bug. ## SQL the caller wrote is their mistake, not ours The same asymmetry showed up one level down. A query that compiles fine can still be rejected by ClickHouse at execution, and most of those rejections mean the caller's SQL is wrong rather than that we generated something bad. This is where the volume actually is. Checking production, one error group alone, a missing `GROUP BY` on the public query API (`NOT_AN_AGGREGATE`), accounts for over a million events across hundreds of users. It is by far the largest error group in the project, and classifying only by resource limit would have left every one of those at error level. So rejections are split three ways in `ClickhouseClient`, which is the only place holding the parsed `ClickHouseError` and its symbolic type. By the time the error reaches `executeTSQL` it has been wrapped and the type is gone, and the type never appears in the message text, so it cannot be recovered by string matching. - **Resource limits** (memory ceiling, timeout, row/byte caps) log at warn. The query is valid, it just asked for more than it is allowed to spend. - **Invalid SQL** (`NOT_AN_AGGREGATE`, `UNKNOWN_IDENTIFIER`, `SYNTAX_ERROR`, the type and parse families) logs at warn **only when the caller wrote the SQL**. - **Everything else** keeps alerting. That gate matters. The client is shared, so the identical rejection on TRQL *we* generated is our bug and has to stay at error. Callers opt in with `userAuthoredQuery`: | caller | who wrote the SQL | opts in | | --- | --- | --- | | public query API | the customer | yes | | query editor | the customer | yes | | agent charts | the agent's model | yes | | built-in dashboard tiles | us, in code | no | | queue metric cards | us, in code | no | | health report | us, in code | no | The agent is the one judgement call. Its TRQL is not typed by a person, but it is also not something a code fix makes correct, so a query it gets wrong is not worth waking anyone for. The same endpoint serves built-in tiles whose TRQL we do write, so the opt-in lives with the caller rather than the route. Separately, when one of these queries did fail, the log recorded the generated ClickHouse SQL but not the query the caller actually wrote, which made the reports hard to act on. `queryWithStats` takes an optional `logFields` that `executeTSQL` uses to attach the original TSQL. ## Events were attributed to the wrong request Chasing the above turned up something broader: only a tenth of the events on that alert pointed at the query API. The rest were pinned to unrelated requests that happened to be in flight at the same time, so the alert looked like the trigger endpoint was failing. `Sentry.init` runs with `skipOpenTelemetrySetup: true`, because we register our own OTel pipeline. That skips `initOpenTelemetry`, and one of the things it does is: ```js api.context.setGlobalContextManager(new SentryContextManager()); ``` The async-context strategy is still installed, but `withIsolationScope` only marks the OTel context and delegates the actual fork to that context manager: ```js // "We depend on the otelContextManager to handle the context/hub" return api.context.with(ctx.setValue(SENTRY_FORK_ISOLATION_SCOPE_CONTEXT_KEY, true), ...) ``` `provider.register()` installed a plain `AsyncLocalStorageContextManager`, which does not know that key. The lookup found no scopes on the context and fell back to the process-global default isolation scope, so every request wrote its request data into the same object and the last writer won. The tracer now registers `SentryContextManager`, which subclasses `AsyncLocalStorageContextManager`, so OTel behaviour is unchanged. It is also registered on the path where tracing is disabled, which previously never called `register()` at all and so had no context manager of its own. Tenant tags were always correct, because those come from our own async local storage rather than the isolation scope. That is why the attribution being wrong was not obvious. This affects every error report the webapp sends, not just the query API. ## Verification `internal-packages/clickhouse`: 76 tests pass, including eight covering each level decision against a real ClickHouse container. Three pairs pin the gate open and shut at both layers: an invalid query, a compile failure, and a real limit breach driven with `max_rows_to_read` each log at warn with `userAuthoredQuery` and at error without it. The isolation fix has a test that reproduces the leak before asserting the fix. Two overlapping requests each tag their own isolation scope; with the plain context manager the slower one reads back the other's tag, and with `SentryContextManager` each reads back its own. Measured separately against a faithful reproduction of the server's wiring (own OTel pipeline, CommonJS entry) at 200 concurrent requests: per-request attribution goes from 0.5% to 100%, while span nesting, context propagation across awaits, and distinct trace IDs are identical before and after.	2026-07-30 09:04:15 +01:00
Eric Allam	4eb9292cbe	feat(webapp,run-engine): queue metrics and health dashboard (#4131) ... ## Summary Three related changes, each independently gated: **Queue metrics and health.** Per-queue depth, throughput (enqueued, started, completed), concurrency, whether a queue is throttled, and scheduling delay (how long a run waits between becoming eligible and actually starting), plus a per concurrency-key breakdown for keyed queues. Collected from inside the run queue itself, stored in ClickHouse, and surfaced on the Queues list, a new per-queue detail page, the task pages, and the run inspector. The question it answers is "does this queue have enough concurrency to keep up, and if not, which key or which limit is the constraint". **Percent-based queue concurrency limits.** A queue's concurrency override can now be expressed as a percentage of the environment limit, stored as the source of truth and re-materialized whenever the environment limit changes. Absolute overrides above the environment limit are now **rejected with a 400** instead of being silently capped, which is a behavior change on `POST /api/v1/queues/:queue/concurrency/override`. **The `health` report.** A server-computed verdict on whether work is flowing, whether the runs that do start are healthy, and whether telemetry is fresh, rendered as text with sparklines. Available as `GET /api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool (plus a `report` MCP prompt, which shows up as a slash command in hosts that support prompts). With the flags off, the Queues page renders the pre-metrics component verbatim, nothing is emitted, and nothing is written to ClickHouse. ## Configuration Two independent gates, on purpose. Emission is global so data accrues for everyone before anyone can look at it; the view is per organization so it can be turned on for one org at a time without a deploy. **Runtime flags (no restart)** | Flag | Store | Gates | | --- | --- | --- | | `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by default) | All emission, gauges and counters. Cached in-process for 10s with stale-while-revalidate, warmed eagerly at boot so the first op after a deploy is not dropped. | | `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` | Fraction of queue ops that emit a gauge. Counters are never sampled, so throughput stays exact at any rate. | | `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag` row, per-org `Organization.featureFlags` override wins | Whether an org sees the metrics view at all: the Queues list variant, the queue detail route, the built-in Queues dashboard, the concurrency-keys endpoint, and the metrics blocks on task pages and the run inspector. Off by default; a gated org gets a 404 on the detail route rather than an empty page. | Both Redis keys are readable and writable from `/admin/queue-metrics` (super-admin UI, with a live per-shard stream-health table) and `GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface uses its own Redis client, so it works on any instance regardless of whether that instance runs the emitter or the consumer. **Environment variables (boot time)** | Variable | Default | Notes | | --- | --- | --- | | `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and injects it into the run engine. Without it the run queue has no emitter at all. | | `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on this instance. Independent of emission, so consumers can be sized separately from the API. | | `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per queue. | | `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals insert batch, so an ack can never outrun a write. | | `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` | falls back to the run-queue Redis | Set `HOST` to move the metrics stream onto a dedicated instance so a metrics backlog cannot compete with the run queue for memory. Self-hosters can leave it unset and get a single-Redis deployment. | | `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000` dedicated | Bound on how much a stalled consumer can hold. The default is deliberately lower when the stream shares the queue-critical Redis. | | `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the per-queue cumulative counter key, refreshed on every write, so only queues idle for the whole window are purged. | | `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue names tracked per environment; overflow collapses into `__overflow__`. | | `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea one level down, per queue. | | `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live sample-rate key above. | | `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics tables in the Query page, its schema docs, the schema API and the AI query context. Off keeps them unlisted while the feature is dark; a query naming them still runs either way. | | `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring | Runs queue metrics on their own ClickHouse service: the consumer's inserts and every queue-metrics read go through it, so a metrics-heavy chart refresh never competes with runs-list or trace reads. Unset reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`, reads on the query pool). | | `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split, so the consumer's inserts can never land on a read endpoint. | | `QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}` | `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other per-workload ClickHouse clients. | Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a Postgres migration adding the nullable `TaskQueue.concurrencyLimitOverridePercent`. Both are additive. ## How collection works Queue operations produce two kinds of signal, and they have opposite failure modes, so they are handled differently. **Gauges** (queued, running, queue limit, env queued, env running, env limit, throttled, plus keys-with-backlog and worst-key wait on keyed queues) are read *inside* the same Redis script that performs the enqueue or dequeue, so the reading is atomic with the operation it describes rather than a racy follow-up read. The script returns them on its reply and the app forwards them to the stream. Gauges are sampled and drop-tolerant: they are aggregated with `max`, so a lost reading costs resolution, never correctness. **Counters** (enqueued, started, completed, plus nack and dead-lettered) are cumulative odometers. Each event increments a per-queue key on the metrics Redis and emits the absolute total, and ClickHouse takes the difference across buckets at read time. This is the important property of the design: a summed-delta counter undercounts permanently on any lost event, while a cumulative one self-heals, because the next surviving reading restates the whole total. Only bucket granularity can be lost, never the total. A queue returning after its odometer TTL expired restarts at 1 and reset detection handles it, which is safe precisely because expiry only spans a window with no activity. Both land on one sharded Redis stream. A consumer reads it with a consumer group, reclaims stale pending entries on a 15s interval rather than on every poll, maps one entry to one or two ClickHouse rows (whole-queue and, for keyed queues, per-key), and acks only after the insert lands. Each batch carries a dedup token derived from its stream-entry ids, and the target tables set `non_replicated_deduplication_window`, so a retried batch cannot double-count either the raw rows or the aggregates that hang off them. Consumer and emitter both emit OTel metrics (`queue_metrics.emitter.emitted`, `queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`); stream depth and group lag are the two worth alerting on, and `lag_unknown` exists because Redis can report a null lag after a trim, which must not be read as zero. ## Storage and read path `queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four aggregate tiers are materialized straight from raw, never cascaded off each other, each with a 30 day TTL: - `queue_metrics_v1`, 10 second buckets per queue, the default read path - `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and cross-queue ranking - `env_metrics_v1`, 10 second buckets per environment, queue-independent so it stays cheap at any range - `queue_metrics_ck_v1`, 10 second buckets per concurrency key Every tier is an MV from raw because the counter states do not survive a cascade: their merge is order sensitive, so a `-MergeState` chain off the 10s table inflates the result, and the same property means an aggregate state may only be merged inside one queue. That constraint is now enforced by the query engine rather than by reviewer discipline: a column can declare a `mergeGroupKey`, and any query that references it without grouping by, or pinning to a single value of, every named key fails to compile with an actionable message. On the read side, TRQL gains three tables (`queue_metrics`, `env_metrics`, and a `queue_metrics_by_key` that is hidden from the editor, schema docs and schema API but still queryable, so per-key rows can never silently merge into a plain per-queue query), plus `deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level optimizations ride along: a table can declare coarser rollups, so a query whose bucket interval is 5 minutes or wider is routed to the 5m table with no change to the query itself, and it can opt into the ClickHouse query cache with time bounds floored to a fixed grid, so the auto-refreshing dashboards actually share cache entries instead of missing on every tick. Both are caller-side substitutions, so the printer stays unaware of physical layout. All of this can also live on its own ClickHouse service. A table declares the pool its reads run on, the three queue-metrics tables name the dedicated one, and the ingestion consumer writes through the same client, so both directions move together with one env var and nothing else routes differently. The other engine change is opt-in gap filling: charts can request rows for empty buckets, where counters zero-fill and gauges carry forward. Grouped gauge series are densified per group and carried inside a partition, so a quiet queue's line holds its last value without bleeding another queue's value into it. ## Queue concurrency limits `concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth when an override is set as a percentage; the absolute `concurrencyLimit` is materialized from it (floored, clamped to at least 1 so a percentage can never act as a pause, and never above the environment limit). Every path that changes an environment limit now recalculates the environment's percent-based overrides afterwards, outside the transaction, and pushes changed limits to the engine. The push is attempted even when the stored value did not change, so a previously failed sync self-heals rather than leaving the database and the engine diverged; paused queues are skipped so a recalculation cannot effectively unpause one. The API accepts exactly one of `concurrencyLimit` or `percent`, and the reject-instead-of-clamp change above means a request asking for more than the environment allows now fails loudly. The percent bound (greater than 0, at most 100) is defined once and shared by the zod schema, the dashboard mutation handler and the service, so the three cannot drift. The concurrency-keys table on a queue is now paginated against the ClickHouse per-key tier, ranked by peak backlog with the total on every row from a single scan, and only the keys on the current page are enriched with live counts from Redis. That replaces a hard top-50 cap with something whose cost is a function of page size rather than key cardinality. ## The health report `GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The verdict is computed on the server and is deterministic, not model-generated. Three independent analyzers run over one input snapshot: flow (is work moving, and if not, is the cause a limit, throttling, one bad queue, or dead-lettering), execution (are the runs that start succeeding, and at what latency), and liveness (how fresh is the telemetry). When telemetry is genuinely stale, the first two are forced to unknown and every actionable field is stripped, so no surface ever advises action off stale data. Authorization is per query table rather than a blanket query grant: a JWT must be scoped to every table the report reads (`runs`, `env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull a report that reads more than it was granted. `period` is validated as a shorthand with a 90 day ceiling at the edge. The report catalog is a registry of `{ load, interpret }` entries, so the next report is a new entry and no change to the route, the view model, the renderers, the CLI or the MCP tool. `trigger mcp` no longer launches the install wizard when stdout is a TTY, which fixed a real failure: hosts spawn the server over a PTY, so the wizard would open and the client would time out waiting for a server that never started. The wizard now needs `trigger mcp --install`. ## The part that is live regardless of every flag The enqueue and dequeue scripts now return a 2-tuple so a gauge reading can ride back on the reply. Every return site in the eight affected scripts is wrapped, and a `nil` original is converted to `false` on the way out, because a raw `nil` in the first slot would make Lua truncate the multi-bulk reply and silently drop the gauge on the throttled and empty-queue paths. The reply shape and the destructuring on the app side are exercised on every queue operation whether or not metrics are enabled, so that is the part of `run-engine` worth the closest review. One behavior fix in the same area: the scheduling-delay anchor is set only on a run's first entry into the queue. Anchoring it to trigger time on re-enqueues made waitpoint and checkpoint resumes report the entire wait as scheduling delay. Queue ordering is untouched, so a re-enqueued run keeps its position, and nacks deliberately keep the original anchor because a rolled-back dequeue is the same continuous wait. A pending-version promotion still anchors to trigger time, on purpose: that promotion is the run's first real entry into the queue, since the trigger deliberately held it back waiting for a worker version, and the TTL is armed at the same point for the same reason. The consequence is worth naming, because it is a judgement call: a run that waits on a deployment reports that wait as scheduling delay on its queue, which is time unrelated to queue capacity. ## Verification Unit and integration suites across the new package, the run queue, the mapping layer, the query engine and ClickHouse (including a test that applies migration 036 through the same splitter CI uses, and a regression test that inserts the same batch three times to prove the aggregates do not inflate). Beyond that, the whole path was driven end to end against a live stack with real runs: emitter to Redis stream to consumer to ClickHouse to the dashboards, for both the local dev path and the deployed path where a supervisor drives the dequeue, with assertions on exact counter reconstruction per queue and per concurrency key, throttling, environment saturation, scheduling delay, and a deliberate mid-stream reading drop to confirm the cumulative counters still reconstruct the correct total. The gated-off state was checked on every touched surface. The dedicated ClickHouse service was verified against a second, separately-schema'd instance: with it configured, the driven counters reconstruct exactly on the dedicated instance, the shared instance gains no rows for that window, a read through the query API returns the value that exists only on the dedicated instance, and a `runs` query still succeeds (it would fail outright if it were mis-routed to a service without that table). With the variable unset, the full suite passes unchanged. --------- Co-authored-by: Katia Bulatova <katia@trigger.dev> Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com> Co-authored-by: James Ritchie <james@trigger.dev>	2026-07-29 16:45:24 +01:00
claude[bot]	639eaf6e82	fix(webapp): don't apply an invite's role to an existing org member (#4409) ... <!-- ccr-slack-attribution --> _Requested via [Slack thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785249693523749)_ ## Summary Accepting an old invitation could change the role of someone who was already in the organization. A long-pending invite can carry a lower role than the member has since been promoted to, so accepting it was a silent demotion. When the accepting user was the organization's only Owner, the role layer refused that demotion, and the refusal (an expected, protective outcome) was logged as an error. An invitation now only sets a role on a membership the accept actually created, and people who are already in an organization are skipped when invitations are sent. ## How `acceptInvite` already skipped the `OrgMember` create when it found an existing membership, but the `rbac.setUserRole` call below it was gated only on `invite.rbacRoleId`. It now also tracks whether this accept created the membership. A create that loses the unique-constraint race counts as pre-existing, since whichever flow won it owns that membership's role. Skipping existing members outright would regress one case: a member with no RBAC role at all would never receive the invitation's role. `ensureOrgMember` handles that with `healMissingRoleAssignment`, which fills in a null role but never overwrites a real one, so `assignInviteRbacRole` takes the same gate. An established role is never touched; an absent one is filled in. `assignInviteRbacRole` branches on the result's machine-readable `code` instead of logging every refusal at `error`. `last_owner` goes to `logger.info`, matching the two directory-sync role paths; everything else, including a refusal that carries no code, goes to `logger.warn`. The helper is best-effort and never throws, so no outcome it produces warrants `error`. No string matching on the error text is involved. `inviteMembers` resolves the organization's members by email and skips those addresses before creating invites. The invite table's `@@unique([organizationId, email])` only dedupes *pending invites*, so it could never catch this. ## Invite surfaces Skipping addresses means a batch can now come back empty, and neither caller handled that: - The dashboard action built its redirect from `invites[0].organization`, so a batch where every address was skipped threw a `TypeError` that reached the admin as a raw error string. It also reported the submitted count rather than the created one. It now names what it skipped ("No invitations sent: 1 already a member of this organization") and counts what it actually created. - The invites API derived `alreadyInvited` as "everything not created", so an existing member was reported as though they had already been invited. `inviteMembers` now returns the two groups separately and the endpoint reports `alreadyMembers` alongside `alreadyInvited`. ## Testing `apps/webapp/test/member.server.test.ts` passes 16/16 locally, up from 12. Getting there needed a harness fix. The `~/db.server` mock did not export `Prisma`, so any code reaching `PrismaNamespace.PrismaClientKnownRequestError` threw before it could branch, leaving every duplicate-key path in `member.server.ts` unreachable from tests. The mock now re-exports the real `Prisma`, and there is a case covering the pending-invite skip. New cases: the invite role is applied when the accept creates the membership; it is not applied when the member already has a role; it is applied when an existing member has no role assigned; the organization is still joined when the assignment is refused with `last_owner`; and `inviteMembers` reports members separately from pending invites. Forcing the gate off fails exactly the "already has a role" case, so the coverage is load-bearing. `pnpm run typecheck --filter webapp` and `oxfmt --check` both pass. ## Changelog Accepting an old invitation could change the role of someone who was already in the organization. An invitation now leaves an existing member's role untouched, people who are already in an organization are no longer sent invitations to it, and the invite form says which addresses it skipped instead of failing with an unhelpful error. --- ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works. ## Screenshots No visual changes. The invite form's toast copy changes, as described above. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Matt Aitken <matt@mattaitken.com>	2026-07-29 15:31:59 +01:00
Saadi Myftija	44eca4d166	feat(webapp): org-gated internal API origin in run env vars (#4366) ... Adds an opt-in way for operators to route deployed runs' API traffic through a different origin than the public one, per organization. Set `INTERNAL_API_ORIGIN` on the webapp and enable the `internalApiOriginEnabled` feature flag (globally or per org, with the org override winning in both directions): deployed runs for enabled orgs then get `TRIGGER_API_URL` set to the internal origin instead of `API_ORIGIN`. Useful for gradually moving run traffic onto a private network path. ## Design The origin is resolved when an attempt starts, so flag changes take effect on the next attempt and roll back the same way, with no task redeploys. The org override is read fresh per attempt; the global default comes from the cached flags registry (a cold read fails safe to the public origin). When `INTERNAL_API_ORIGIN` is unset the flag is a no-op and no extra queries run, so existing deployments are unaffected. Dev runs always use the public origin, and `TRIGGER_STREAM_URL` remains unchanged.	2026-07-28 11:28:09 +02:00
claude[bot]	ec562c0e68	fix(webapp): remove unused Electric sync trace routes (#4400) ... <!-- ccr-slack-attribution --> _Requested by **Eric Allam** · [Slack thread](https://triggerdotdev.slack.com/archives/C0AU83M3136/p1785222101937829?thread_ts=1785207509.304669&cid=C0AU83M3136)_ Removes two dead Remix routes and the helpers only they used. `app/routes/sync.traces.runs.$traceId.ts` (`/sync/traces/runs/:traceId`) and `app/routes/sync.traces.$traceId.ts` (`/sync/traces/:traceId`) were added with the original ElectricSQL run page and lost their only consumers when the dashboard hooks that called them were deleted. Nothing in the repo references either route today. Also removed, because the deleted routes were their only callers: - `OtelTraceIdSchema`, `RESERVED_ELECTRIC_SHAPE_PARAMS`, `TraceScope`, `buildElectricTraceWhereClause` from `app/v3/electricShape.server.ts` (the file stays — `UNSAFE_REALTIME_TAG_CHARS` / `sanitizeRealtimeTagForSql` / `sanitizeRealtimeTagsForSql` are still used by `realtime.v1.runs.ts` and `realtimeClient.server.ts`) - the loader-specific cases in `apps/webapp/test/spanTraceRoutes.replicaLag.test.ts` and `internal-packages/run-store/src/runOpsStore.routesSpanTraceReadView.replicaLag.test.ts` `app/utils/longPollingFetch.ts` is untouched — `realtimeClient.server.ts` still uses it. `runOpsStore.ts` / `PostgresRunStore.ts` are untouched too; the unrouted-lookup mechanism there is generic and stays. As a plain code fact: the run lookup these loaders performed keyed on `TaskRun.traceId` alone, which is not an index-backed query shape. That is noted only as context for why the code is not worth keeping around unused. ### Judgement call worth a maintainer's opinion The request was specifically about `/sync/traces/runs/:traceId`, the route that looks up a run by `traceId`. This PR **also** deletes its sibling `/sync/traces/:traceId`. The reasoning: - both routes came in with the same ElectricSQL run-page work - both lost their only consumers in the same later commit - neither has any caller anywhere in the repo - they share the same helper module, so keeping one means keeping the helpers half-used If you would rather keep the sibling, reverting just that one file deletion is easy and does not affect the rest of this PR — say the word and I will restore it along with the helpers it needs. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Verification run locally from the repo root: | Command | Result | | --- | --- | | `pnpm run format` | clean, no changes produced | | `pnpm run lint:fix` | clean | | `pnpm run lint` | pass (exit 0, no findings) | | `pnpm run typecheck --filter webapp` | pass | | `pnpm run typecheck --filter @internal/run-store` | pass | A ripgrep sweep for `sync.traces`, `sync/traces`, `syncTraceRunsLoader`, `buildElectricTraceWhereClause`, `OtelTraceIdSchema` and `RESERVED_ELECTRIC_SHAPE_PARAMS` (excluding `node_modules`) returns zero hits. **Not fully verified:** both edited test files are testcontainers suites and need a Docker runtime, which was not available in my environment. I confirmed each file *collects* correctly with exactly the three intended remaining tests and no import errors — notably, dropping the `session.server` / `controlPlaneResolver.server` / `longPollingFetch` / `env.server` mocks does not break module loading for the surviving loaders. The assertions themselves then failed only on `Could not find a working container runtime strategy`. CI should be the real signal here. Per `apps/webapp/CLAUDE.md`, `pnpm run build --filter webapp` was deliberately not run. --- ## Changelog Removed two unused sync routes left over from the original ElectricSQL run page, along with the helpers and tests that existed only to serve them. No behaviour change — neither route had any caller. --- ## Screenshots _n/a — no user-visible surface changes._ 💯 --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-28 09:00:48 +01:00
github-actions[bot]	d189ce17d3	chore: release v4.5.8 (#4364) ... ## Summary 2 new features, 9 improvements, 3 bug fixes. ## Highlights - Allow additional environment API keys to create scoped public access tokens through the Trigger.dev API. Use server-issued public access tokens for batch operations so environment-scoped API keys can read batch results. ([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387)) ## Improvements - Preserve the partial assistant message when a chat turn's model stream fails mid-response. `chat.agent` now passes the recovered partial to `onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it before rethrowing, instead of dropping the streamed-so-far output. ([#4348](https://github.com/triggerdotdev/trigger.dev/pull/4348)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Favorite any dashboard page to a new Favorites section in the side menu, and customize the sidebar by renaming favorites, hiding items, and reordering items and sections. ([#4375](https://github.com/triggerdotdev/trigger.dev/pull/4375)) - List API endpoints now clamp the page size to a maximum of 100. Requests asking for a larger page size return up to 100 items and keep paginating, rather than pulling an unbounded page. ([#4360](https://github.com/triggerdotdev/trigger.dev/pull/4360)) - Organizations without billing alerts now get default spend alert thresholds, so you're notified before usage grows unexpectedly. The billing limit page no longer pre-selects an option before you've set a limit and prompts you to configure one. Alert previews now update immediately after you change your billing limit. ([#4328](https://github.com/triggerdotdev/trigger.dev/pull/4328)) - When you create a Personal Access Token, the generated token now shows its first and last few characters instead of being fully hidden, so you can confirm you copied the right value. ([#4363](https://github.com/triggerdotdev/trigger.dev/pull/4363)) - Add metrics to the realtime backend that measure how often a single changed run is served to multiple subscriptions in one batch. ([#4341](https://github.com/triggerdotdev/trigger.dev/pull/4341)) - Realtime run subscriptions can now be configured to read run data straight from the primary database, so a run's latest state is never served from a lagging replica. Off by default; replica reads are unchanged unless you turn it on. ([#4378](https://github.com/triggerdotdev/trigger.dev/pull/4378)) - SSO and Directory Sync are no longer restricted to Enterprise plans — get in touch and we can turn them on for your organization whatever plan you're on. ([#4393](https://github.com/triggerdotdev/trigger.dev/pull/4393)) - Improved supervisor observability: it now reports metrics for its outbound requests, making failed calls to upstream services easier to monitor. ([#4350](https://github.com/triggerdotdev/trigger.dev/pull/4350)) - The runs list on a task's page now updates live — run statuses change and newly triggered runs appear without a manual refresh, matching the main Runs page. ([#4377](https://github.com/triggerdotdev/trigger.dev/pull/4377)) - Speed up the Batches list page for environments with a large number of batches, which could previously time out while loading. ([#4361](https://github.com/triggerdotdev/trigger.dev/pull/4361)) - Container startup no longer prints database and ClickHouse connection strings (with credentials) to the logs. ([#4346](https://github.com/triggerdotdev/trigger.dev/pull/4346)) - The tasks page no longer runs two queries whose results were never displayed, cutting wasted work on every page load and removing a source of hidden server errors ([#4380](https://github.com/triggerdotdev/trigger.dev/pull/4380)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## trigger.dev@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` - `@trigger.dev/build@4.5.8` - `@trigger.dev/schema-to-json@4.5.8` ## @trigger.dev/core@4.5.8 ### Patch Changes - Allow additional environment API keys to create scoped public access tokens through the Trigger.dev API. Use server-issued public access tokens for batch operations so environment-scoped API keys can read batch results. ([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387)) ## @trigger.dev/python@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.8` - `@trigger.dev/core@4.5.8` - `@trigger.dev/build@4.5.8` ## @trigger.dev/react-hooks@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/redis-worker@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/rsc@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/schema-to-json@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/sdk@4.5.8 ### Patch Changes - Preserve the partial assistant message when a chat turn's model stream fails mid-response. `chat.agent` now passes the recovered partial to `onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it before rethrowing, instead of dropping the streamed-so-far output. ([#4348](https://github.com/triggerdotdev/trigger.dev/pull/4348)) - Allow additional environment API keys to create scoped public access tokens through the Trigger.dev API. Use server-issued public access tokens for batch operations so environment-scoped API keys can read batch results. ([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387)) - Updated dependencies: - `@trigger.dev/core@4.5.8` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-07-27 16:40:31 +01:00
James Ritchie	d30ee6e570	feat(webapp): favorite pages and sidebar customization (#4375) ... ## Summary Favorite any dashboard page and it appears in a new "Favorites" section at the top of the side menu. The star next to the page title (or Option+F) saves the exact view, filters and tabs included, with a name derived from the URL ("Runs: Completed successfully, last 7d", "Run: 05hrqq9n") that you can rename inline from each item's hover menu. The sidebar is customizable too: "Customize sidebar" (on section header menus and in each "More" menu) opens a modal where you can reorder sections, drag items into a new order, hide items behind a per-section "More" popover, and rename or remove favorites. Changes apply on Confirm, Reset restores the default layout without touching favorites, and everything is stored per user in dashboard preferences. ## Screenshots | Favorites in the side menu | Customize sidebar modal | | --- | --- | |  |  |  ## Design notes - Favorite links carry a small marker search param so the favorite, not its identical main menu item, highlights as active. Markers from shared or stale links are cleaned on load, and changing any filter hands the highlight back to the regular menu item. - Preference writes are serialized with a row lock: several writers (debounced collapse and width saves, favorite toggles, the customize modal) can land concurrently and would otherwise clobber each other's read-modify-write of the JSON column. - Option+F is matched on `event.code` with a raw listener because macOS reports Option-modified letters as symbols, which the `event.key` based shortcut hook can't capture. Verified end-to-end in the browser: star toggle and shortcut, instant section appearance, inline rename and staged modal removal, filter-aware labels and unique active states, shared-link normalization, drag reordering, and persistence across reloads.	2026-07-27 16:29:36 +01:00
James Ritchie	3e53404f40	feat(webapp): live-update the runs list on task pages (#4377) ... ## Summary The runs list on a task's page now updates live, matching the main Runs page. Run rows update their status, duration, and cost in place as runs progress, and a "N new runs" button appears in the header when newer runs come in so you can pull them into the list without a manual refresh. This applies to both standard and scheduled task pages. ## Design It reuses the Runs page's polling hook. A task page scopes its runs by the task in the URL path rather than a `tasks` query filter, so the hook now takes an optional task slug and scopes new-run detection to it. The "new runs" button sits in the header, outside the deferred runs table, so the count is lifted to the page and the click action is passed through a ref. That keeps the table streaming on first load instead of blocking the header on the runs query. When newer runs come in, a `1 new run` button appears in the task page header, to the left of the time filter. Clicking it pulls the new runs into the list.	2026-07-27 15:41:47 +01:00
Matt Aitken	269470fd87	feat(webapp): gate SSO on an entitlement instead of the Enterprise plan (#4393) ... The SSO & Directory Sync settings page decided access by comparing the organization's plan code against the literal string `"enterprise"`. The webapp now reads a `hasSso` entitlement from plan limits. ## Changes - **`settings.sso` route** — `planAllowsSso` reads `limits.hasSso` rather than the plan code; the loader and the action gate on a shared `getSsoEntitlement` helper. - **`platform.v3.server`** — new `getSsoEntitlement(orgId)` returning `entitled | not_entitled | unknown`, behind a new SWR cache namespace (60s fresh / 120s stale, memory + Redis). This replaces an uncached billing round-trip that previously ran on every settings load, so the page gets cheaper than it was. - **`directorySyncEffects`** — the entitlement is now checked before applying membership effects, per organization and memoised across a batch. - **`@trigger.dev/platform` 1.2.0 → 1.3.0** — required, see below. ## Behaviour worth reviewing **Revocation now stops SCIM.** Previously the plan check existed only on the settings page, so an org that lost access kept receiving directory-sync pushes indefinitely; only the config UI froze. Provision *and* deprovision are gated, so a revoked entitlement can't remove members either. **An unreadable entitlement throws instead of skipping.** Effects are idempotent and the worker retries, so retrying is lossless where dropping would silently lose a directory change. It's raised at `warn` level so a transient billing blip doesn't page anyone. **The login path is deliberately untouched.** A hard entitlement check there turns a billing outage into a login outage. Consequence: an org that loses the entitlement keeps its existing SSO logins working until the connection is removed. Gating sign-in is a separate decision. **Self-hosted is unaffected.** With no billing service configured the helper returns `entitled`, leaving plugin presence and the kill switch as the only gates — a self-hoster who installed the plugin isn't locked out of it. ## The dependency bump is load-bearing The `Limits` schema is a plain `z.object`, so it *strips* unknown keys. On 1.2.0 the `hasSso` field was silently discarded during parsing and read as `undefined` no matter what billing sent — a structural accessor would not have helped. Verified against both builds: ``` 1.2.0 → parsed: true | hasSso survives: false 1.3.0 → parsed: true | hasSso survives: true ``` This PR therefore cannot merge before 1.3.0 is published, which it now is. ## Testing `apps/webapp/test/directorySyncEffects.server.test.ts` — 7 tests over the gate: applies when entitled, skips provision and deprovision when not, throws a warn-level retryable error when unreadable, resolves once per org across a batch, and gates per org so one unentitled org doesn't block another. `pnpm run typecheck --filter webapp` passes (18/18), oxfmt and oxlint clean.	2026-07-27 13:08:16 +01:00
claude[bot]	d91818f198	fix(webapp): remove unawaited task list metrics promises (#4380) ... <!-- ccr-slack-attribution --> _Requested via [Slack thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785082528841609)_ `TaskListPresenter` created promises that nothing ever consumed. Two of the three deferred metrics promises it returned had no reader, no `await` and no `.catch()`, so when the query behind one of them failed the rejection had nowhere to go. ## Before / After **Before** - `TaskListPresenter.call()` returned four things: `tasks`, `activity`, `runningStats` and `durations`. Its only caller reads `tasks` and `runningStats`. - Every load of the tasks page therefore fired two ClickHouse queries whose results were thrown away. - If either of those two queries failed, the resulting promise rejection was unhandled — nothing was awaiting it and nothing had attached an error handler, so it surfaced as an unhandled rejection at the process level rather than as an error anyone could attribute to a request. **After** - `TaskListPresenter.call()` returns `tasks` and `runningStats` only. - Two fewer queries run per tasks-page load. - There is no longer an unconsumed promise that can reject without a handler. `runningStats` is awaited by its caller, so its failures continue to be handled the way they always were. Nothing changes on screen: the tasks page renders `hourlyActivity` and `runningStates`, and neither of the removed values fed either of those. ## How The removed values were verified unreferenced before deleting anything: - `TaskListPresenter` has exactly one caller, `UnifiedTaskListPresenter`, which reads `taskResult.tasks` and `taskResult.runningStats` and nothing else. - No file anywhere in the repo — app code, tests, or type re-exports — reads an `activity` or `durations` field off the presenter's result. - `UnifiedTaskListPresenter` builds its own `unifiedTaskListHourlyActivity` query for the 24h chart the page actually renders, which is what made the presenter's separate 7-day daily activity data redundant. - `getDailyTaskActivity` and `getAverageDurations` on `ClickHouseEnvironmentMetricsRepository` had no callers other than the two lines being deleted, so they and their now-orphaned helpers and types were removed too. Changes: - `apps/webapp/app/presenters/v3/TaskListPresenter.server.ts` — drop the `activity` and `durations` fields (both from the main return and from the no-current-worker early return) and the two repository calls behind them. Drop the unreferenced `TaskActivity` type alias. The "don't await this" comment on the remaining `runningStats` promise now spells out that the caller has to consume it. - `apps/webapp/app/services/environmentMetricsRepository.server.ts` — remove `getDailyTaskActivity` and `getAverageDurations` from the `EnvironmentMetricsRepository` interface and its ClickHouse implementation, along with `fillInDailyTaskActivity` and the `DailyTaskActivity` / `AverageDurations` types. `getCurrentRunningStats` is the control that shows the diagnosis is right. It throws on query failure in exactly the same way as the two removed methods — `if (queryError) throw queryError` — but it never produced an unhandled rejection, because `UnifiedTaskListPresenter` passes its promise into a `Promise.all(...).then(...)` chain that the route then awaits. Same failure mode, opposite outcome, and the only difference is whether anything consumes the promise. Follow-ups, not in this PR: - `AgentListPresenter` returns three sparkline promises in the same shape and they look similarly unconsumed. Left alone here to keep this change reviewable. - With these two callers gone, the `getTaskActivity` and `getAverageDurations` query builders in `@internal/clickhouse` have no remaining callers in this repo. Whether to remove them is a separate call for someone who owns that package. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing - `pnpm run typecheck --filter webapp` — passes. This is the meaningful check here: it proves nothing still references the removed fields, methods or types. - `pnpm run format` and `pnpm run lint:fix` — clean, no changes produced. - No test file referenced the removed symbols, so no test needed updating. --- ## Changelog Server-only change, so this carries a `.server-changes/` note rather than a changeset: `.server-changes/task-list-remove-unused-metrics-queries.md`. > The tasks page no longer runs two queries whose results were never displayed, cutting wasted work on every page load and removing a source of hidden server errors --- ## Screenshots _No visual change — the removed data was never rendered._ Co-authored-by: Claude <noreply@anthropic.com>	2026-07-27 11:39:43 +01:00
Eric Allam	d3906241a5	feat(webapp): read realtime run rows from the primary, not the replica (#4378) ... ## Summary The realtime runs feed hydrates run rows from read replicas, which means it needs a replica-lag gate to avoid serving a run's previous state right after a write. Setting `REALTIME_BACKEND_NATIVE_RUN_READS_FROM_PRIMARY=1` reads those rows from each run store's primary instead, so there is no lag to gate against: no probe, no wake delay, no stale-read retries. Off by default, so nothing changes unless you set it. ## Design The run stores already decide replica-vs-primary from the *brand* on the read client they are handed: a branded replica keeps the read on the owning store's replica, an unbranded writer escalates it to that store's own primary. So this is a one-line choice at the hydrator, and it stays correct across topologies. With the run-ops split on, each leg lands on its own writer and the caller's client is never forwarded across databases; with the split off, it is the single database's primary. ```ts const runReader = new RunHydrator({ readClient: runReadsFromPrimary ? prisma : $replica, runStore, }); ``` The same flag skips constructing the lag estimator, since probing a replica the feed no longer reads would be measuring the wrong thing. Independently, `AuroraReplicaLagSource` detected Aurora by letting `aurora_replica_status()` fail, on the assumption that the app-level catch made that free. It isn't: an unresolvable function is a query error the driver reports to the error log on every sample, so a non-Aurora replica produced a continuous stream of error events while the estimator quietly fell through to its next candidate. It now resolves the function with `to_regproc` and memoizes the answer, so the unparseable call never reaches the wire.	2026-07-26 19:43:41 +01:00
claude[bot]	109e245d56	feat(webapp): show the first and last characters of a new PAT (#4363)	2026-07-24 12:55:31 +01:00
Eric Allam	7188eecd83	perf(webapp): clamp list-endpoint page size to 100 (#4360) ... ## Summary Several list endpoints accepted an unbounded page size (`perPage` / `per_page` / `pageSize`). An unbounded page lets one request pull an arbitrarily large result set and do a proportional amount of work, which is a poor default for a shared API. This clamps the page size to 100 on every list endpoint that was uncapped, matching the existing cap on `api.v1.runs` and `api.v1.sessions`. Clamping rather than rejecting keeps existing clients working: a request for a larger page returns up to 100 items and offset pagination continues from there. ## Endpoints capped - `api.v1.schedules` (`perPage`) - `api.v1.queues` (`perPage`) - `resources.…versions` (`per_page`) - `resources.…queues` (`per_page`) - `admin.api.v1.…engine.report` (`per_page`) - `admin.api.v1.llm-models` (`pageSize`) Already capped, left as-is: `api.v1.runs`, `api.v1.sessions`, `api.v1.deployments`.	2026-07-24 11:56:58 +01:00
Eric Allam	9c85e0ecdc	perf(database): index BatchTaskRun on (runtimeEnvironmentId, createdAt, id) for the batches list (#4361) ... ## Summary The batches list page orders by `createdAt DESC, id DESC` filtered by environment and a created-at window, but the only supporting index on `BatchTaskRun` was `(runtimeEnvironmentId, id)`. That index can't satisfy the `createdAt` ordering, so on environments with a large number of batches the query fell back to a full table scan and in-memory sort, which could run long enough to hit the statement timeout. ## Fix Adds `(runtimeEnvironmentId, createdAt DESC, id DESC)` on `BatchTaskRun`. The query now reads straight from the index in order with no sort step, returning a page with only a handful of heap fetches instead of scanning the whole environment slice. The migration uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it takes no table lock and is a no-op if the index already exists.	2026-07-24 11:52:34 +01:00
nicktrn	722e240e4d	feat(supervisor): add prometheus metric for outbound http requests (#4350) ... Adds Prometheus metrics so the supervisor's outbound HTTP calls are observable - including client-side failures that previously only surfaced as a log line. - `supervisor_outbound_request_total{name, method, status, outcome}` - counts every outbound request. `outcome` separates a transport failure (`network_error`), an HTTP error response (`http_error`), a response that failed schema validation (`invalid_response`), and success (`ok`). - `supervisor_outbound_request_duration_seconds{name, outcome}` - latency histogram. Leaner labels than the counter (no `status`) to avoid bucket×label cardinality; buckets match the existing dequeue-latency histogram since these calls share the same retrying HTTP client and long-poll envelope. Coverage: - The warm-start request (a one-off `fetch`) - instrumented inline; the response status code is now also included in the failure log (it was previously dropped). - All worker API client calls (`SupervisorHttpClient`: dequeue, run attempt start/complete, heartbeats, snapshots, continue, suspend, debug-log, connect) - routed through a single instrumented `request()` helper that reports via an optional `onHttpRequestComplete` callback on the client, which the supervisor wires into the counter + histogram. Low cardinality by design: `name` is a **static per-endpoint label** (e.g. `dequeue`, `start_run_attempt`), never the interpolated URL - so no run/snapshot IDs land in labels, mirroring the templated `route` labels on the inbound HTTP server. Registered on the existing metrics registry, exposed on `/metrics` with no new wiring. Internal-only change (no package release needed), so the changelog note is a single `.server-changes` entry.	2026-07-23 18:18:58 +01:00
Eric Allam	88ca0091a9	fix(docker): stop the container entrypoint printing database connection strings in logs (#4346) ... ## Summary The container entrypoint runs under `set -x`, which echoes every command to the logs with its variables expanded. Several startup guards reference full database connection strings, so the DSN (including the password) was printed to the container logs on every boot. This turns tracing off around those lines so connection strings are never traced, while leaving migration behavior and ordinary startup logging unchanged. ## Fix The leaking lines are the `[ -n "$RUN_OPS_DATABASE_URL" ]` and `[ -n "$RUN_OPS_LEGACY_DIRECT_URL" ]` guards, and the ClickHouse block (its `[ -n "$CLICKHOUSE_URL" ]` guard plus the lines that build `GOOSE_DBSTRING` from `CLICKHOUSE_URL`). `set -x` prints each of these with the credential expanded. Tracing is now disabled around each region and restored afterward, so non-secret tracing is preserved everywhere else. The existing legacy-migration subshell already protected its own command body; this adds the missing protection for the guards and the ClickHouse block. ```sh { set +x; } 2>/dev/null if [ -n "$RUN_OPS_DATABASE_URL" ]; then set -x ... ``` ## Verification Built the webapp image and ran it with dummy sentinel connection strings whose password token is `S3NTINEL_PW_DoNotLog`, then grepped the boot logs. Before (unmodified), the token appears in the traced guards: ``` + [ -n postgresql://user:S3NTINEL_PW_DoNotLog@fake-host:6432/run-ops ] + [ -n postgresql://user:S3NTINEL_PW_DoNotLog@fake-host:5432/legacy ] + [ -n https://default:S3NTINEL_PW_DoNotLog@fake-host:8443 ] ``` After, `grep S3NTINEL_PW_DoNotLog` on the same run returns nothing, and the normal "skipping ... migrations" lines still log.	2026-07-23 11:51:34 +01:00
Katia Bulatova	23d5771d56	feat(webapp): unconfigured billing limit UX and default billing alerts (#4328) ... ## Default billing alerts + billing limit page UX - New orgs get default billing alerts: $5, $100, $500, $1000, $2500. Existing orgs are backfilled by a billing-side data migration (companion [PR](https://github.com/triggerdotdev/cloud/pull/1657)). - The billing limit form starts with nothing selected for orgs that never set a limit — the save button appears once an option is picked. - The yellow banner now also shows on the billing limits page itself, asking to configure a limit. Hidden everywhere for members who can't manage billing. - Also fixes billing limit alert preview. Tests - `apps/webapp/test/billingLimitsRoute.test.ts` — dirty logic for empty/selected mode - `apps/webapp/test/billingAlertsDefaults.test.ts` — default values - `apps/webapp/test/billingAlertsFormat.test.ts` — preview after a limit change	2026-07-22 21:23:52 +02:00
Eric Allam	a2d382b2be	feat(webapp): add emission fan-out metrics to the native realtime feed (#4341) ... ## Summary Adds two counters to the native realtime backend so we can see how much duplicate row serialization the change router does per batch. When a run changes it can match several held feeds at once (a run subscription plus one or more tag/list feeds), and today each matching feed serializes that run's wire value independently. These counters quantify that fan-out so we can decide whether a shared serialization step is worth it. ## What they measure - `realtime_native.emission_run_serializations`: total wire-value serializations performed across feeds per batch (what the current path does). - `realtime_native.emission_distinct_serializations`: distinct (columns, run) rows those serializations cover (what a serialize-once-per-batch step would do). Average feeds-per-run is `run_serializations / distinct_serializations`, and `1 - distinct / run_serializations` is the serialization work a shared step could save. Wired through a new optional `onEmissionFanout` callback on the router. No behavior change.	2026-07-22 17:48:44 +01:00
github-actions[bot]	aafc333523	chore: release v4.5.7 (#4319) ... ## Summary 5 improvements, 5 bug fixes. ## Improvements - Add `node-24` and `node-26` as supported `runtime` options in `trigger.config.ts`. The `experimental-node-24` and `experimental-node-26` names are now deprecated aliases and emit a deprecation warning; switch to `node-24` / `node-26` instead. ([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337)) ```ts import { defineConfig } from "@trigger.dev/sdk"; export default defineConfig({ runtime: "node-24", project: "<your-project-ref>", }); ``` - Avoid logging task run environment variable values at debug level ([#4336](https://github.com/triggerdotdev/trigger.dev/pull/4336)) - Custom chat agent loops get two ergonomic wins for owning the turn loop. ([#4304](https://github.com/triggerdotdev/trigger.dev/pull/4304)) `chat.writeTurnComplete()` now returns the turn boundary's resume cursors (`lastEventId` for the output stream and `sessionInEventId` for the input stream), so you can persist them straight from the task instead of round-tripping them back from the client. ```ts const { lastEventId, sessionInEventId } = await chat.writeTurnComplete(); await db.chats.update(chatId, { lastEventId, sessionInEventId }); ``` `chat.pipeAndCapture()` no longer throws when a stream is stopped or fails. It now returns a `PipeAndCaptureResult` whose `message` holds any partial output captured before the stop or failure, alongside a typed `status` (`"complete" | "aborted" | "error"`) and, on failure, the `error`. Read the message off the result: ```ts const { message, status, error } = await chat.pipeAndCapture(result, { signal, }); if (message) conversation.addResponse(message); if (status === "error") logger.error("turn failed", { error }); ``` Note: `pipeAndCapture` previously resolved to `UIMessage | undefined`. Update call sites to read `.message` from the returned result. - Suppress a build-time warning that could appear in Vite-based projects when the optional `@ai-sdk/otel` package is not installed. ([#4188](https://github.com/triggerdotdev/trigger.dev/pull/4188)) ## Bug fixes - Fixes intermittent `trigger dev` run crashes where a run could fail at boot with a cryptic `Cannot find module .../dev-run-worker.mjs` after a rebuild had cleaned up the build directory the run was launched against. Dev runs now retry cleanly instead of hard-crashing when their build directory is missing, the dev watchdog no longer removes the build tree of a still-running session, and a run assigned to a worker version that was superseded by a rebuild now fails fast with a clear message instead of silently hanging until it times out. ([#4276](https://github.com/triggerdotdev/trigger.dev/pull/4276)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Refreshed the side menu: separate organization and account menus, a new project switcher, and the menu is now resizable by dragging its edge. The account Profile page has also been redesigned. ([#4066](https://github.com/triggerdotdev/trigger.dev/pull/4066)) - Allow different organization members to use the same development branch name without sharing or colliding with each other's branch environments. ([#4323](https://github.com/triggerdotdev/trigger.dev/pull/4323)) - Limit account settings email input to 254 characters. ([#4330](https://github.com/triggerdotdev/trigger.dev/pull/4330)) - Prevent duplicate Staging and Preview environments when account setup requests overlap ([#4261](https://github.com/triggerdotdev/trigger.dev/pull/4261)) - Fix the docs link on the empty Prompts page, which pointed to a page that no longer exists. ([#4247](https://github.com/triggerdotdev/trigger.dev/pull/4247)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.7 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.7` ## trigger.dev@4.5.7 ### Patch Changes - Fixes intermittent `trigger dev` run crashes where a run could fail at boot with a cryptic `Cannot find module .../dev-run-worker.mjs` after a rebuild had cleaned up the build directory the run was launched against. Dev runs now retry cleanly instead of hard-crashing when their build directory is missing, the dev watchdog no longer removes the build tree of a still-running session, and a run assigned to a worker version that was superseded by a rebuild now fails fast with a clear message instead of silently hanging until it times out. ([#4276](https://github.com/triggerdotdev/trigger.dev/pull/4276)) - Add `node-24` and `node-26` as supported `runtime` options in `trigger.config.ts`. The `experimental-node-24` and `experimental-node-26` names are now deprecated aliases and emit a deprecation warning; switch to `node-24` / `node-26` instead. ([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337)) ```ts import { defineConfig } from "@trigger.dev/sdk"; export default defineConfig({ runtime: "node-24", project: "<your-project-ref>", }); ``` - Avoid logging task run environment variable values at debug level ([#4336](https://github.com/triggerdotdev/trigger.dev/pull/4336)) - Updated dependencies: - `@trigger.dev/core@4.5.7` - `@trigger.dev/build@4.5.7` - `@trigger.dev/schema-to-json@4.5.7` ## @trigger.dev/core@4.5.7 ### Patch Changes - Add `node-24` and `node-26` as supported `runtime` options in `trigger.config.ts`. The `experimental-node-24` and `experimental-node-26` names are now deprecated aliases and emit a deprecation warning; switch to `node-24` / `node-26` instead. ([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337)) ```ts import { defineConfig } from "@trigger.dev/sdk"; export default defineConfig({ runtime: "node-24", project: "<your-project-ref>", }); ``` ## @trigger.dev/python@4.5.7 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.7` - `@trigger.dev/core@4.5.7` - `@trigger.dev/build@4.5.7` ## @trigger.dev/react-hooks@4.5.7 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.7` ## @trigger.dev/redis-worker@4.5.7 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.7` ## @trigger.dev/rsc@4.5.7 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.7` ## @trigger.dev/schema-to-json@4.5.7 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.7` ## @trigger.dev/sdk@4.5.7 ### Patch Changes - Custom chat agent loops get two ergonomic wins for owning the turn loop. ([#4304](https://github.com/triggerdotdev/trigger.dev/pull/4304)) `chat.writeTurnComplete()` now returns the turn boundary's resume cursors (`lastEventId` for the output stream and `sessionInEventId` for the input stream), so you can persist them straight from the task instead of round-tripping them back from the client. ```ts const { lastEventId, sessionInEventId } = await chat.writeTurnComplete(); await db.chats.update(chatId, { lastEventId, sessionInEventId }); ``` `chat.pipeAndCapture()` no longer throws when a stream is stopped or fails. It now returns a `PipeAndCaptureResult` whose `message` holds any partial output captured before the stop or failure, alongside a typed `status` (`"complete" | "aborted" | "error"`) and, on failure, the `error`. Read the message off the result: ```ts const { message, status, error } = await chat.pipeAndCapture(result, { signal, }); if (message) conversation.addResponse(message); if (status === "error") logger.error("turn failed", { error }); ``` Note: `pipeAndCapture` previously resolved to `UIMessage | undefined`. Update call sites to read `.message` from the returned result. - Suppress a build-time warning that could appear in Vite-based projects when the optional `@ai-sdk/otel` package is not installed. ([#4188](https://github.com/triggerdotdev/trigger.dev/pull/4188)) - Updated dependencies: - `@trigger.dev/core@4.5.7` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-07-22 15:51:29 +01:00