v4.5.10
4642 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Eric Allam
|
7246f677db |
fix(webapp): strip null bytes from idempotency and debounce keys at trigger (#4527)
## What
A trigger request carrying a Unicode NUL (`U+0000`) in the **idempotency
key** or **debounce key** reached `prisma.taskRun.create()` and failed
the insert, so the caller got an opaque 500 and the run was never
created.
These two keys are stored in `jsonb` columns (`idempotencyKeyOptions`,
`debounce`), and Postgres rejects a NUL inside a `jsonb` value with
`SQLSTATE 22P05` ("unsupported Unicode escape sequence ... cannot be
converted to text"). This fix strips the NUL from both keys at the
single trigger-input chokepoint (`#buildEngineTriggerInput`), which
every trigger path flows through (single, batch item, mollified, and
drainer replay).
Stripping matches the existing precedent for run errors and task events.
It does not change dedup behaviour: the idempotency **dedup identity**
is the hashed key (a clean 64-char digest), computed independently of
the raw key we clean, so dedup keeps working exactly as before. For
debounce the key is used directly, so the cleaned key also becomes the
grouping key, an acceptable change for input that is already malformed.
## Why not payload / metadata / tags
Those are `text` columns fed by `JSON.stringify`, which escapes a NUL to
a safe escape sequence, so they do not hit this failure on the normal
JSON path. (A raw NUL in a `text` column throws a different code,
`22021`, and is not what triggers this issue.) The observed failures are
the `jsonb` `22P05` variant, which is only reachable via the two key
fields.
## Evidence
Red then green (containerTest, real Postgres): with the fix reverted,
triggering through the real service with a NUL in
`idempotencyKeyOptions.key` / `debounce.key` fails with the exact
`22P05` signature; with the fix, the run is created and the stored key
has the NUL removed.
Full-stack e2e (isolated stack, real HTTP): `POST
/api/v1/tasks/:taskId/trigger` with a NUL inside
`idempotencyKeyOptions.key` (`"acme<NUL>inc"`) and, separately,
`debounce.key` (`"grp<NUL>1"`):
- both returned `HTTP 200` with a created run (previously `500`)
- stored `idempotencyKeyOptions` = `{ "key": "acmeinc", "scope": "run"
}` (7 chars, NUL removed)
- stored `debounce.key` = `"grp1"` (4 chars, NUL removed)
- both runs render in the dashboard
Unit tests cover the helper (strip, no-op fast path, object-reference
reuse, null/undefined pass-through).
## Rollout / rollback
Server-only webapp change, no flag. Zero behaviour change for clean
input; only affects inputs that previously 500'd. Rollback is a straight
revert, no data migration.
## Known limitation
A raw NUL in a plain-string idempotency key (not created via
`idempotencyKeys.create()`) lands in a `text` column and throws `22021`
instead. That variant is not addressed here because stripping it would
change the dedup identity, so it warrants a separate decision. Not
observed in practice.
refs TRI-13030
|
||
|
claude[bot]
|
dc529414df | feat(webapp): add /_/* redirect route (#4523) | ||
|
Chris Arderne
|
0a44b88b39 | fix: security release 2026-07-21 (#4528) | ||
|
Eric Allam
|
db67a856fe |
perf(webapp,database): index the newest-task-version lookup (#4518)
📦 Preview packages (pkg.pr.new) / Build and publish previews (push) Has been cancelled
📚 Publish docs / publish (push) Has been cancelled
Implementing PlanetScale Insights improvement. ## Summary Validating a schedule (creating or updating one through the API or the dashboard, and deploying a project that declares schedules) looks up the newest version of a task by slug. That lookup reads *every* version of the task and sorts them to return one. A project gains a row per task on every deploy, so the work grows with the project's age: the oldest projects pay the most, and dev-mode redeploys make it worse. This was picked because it was the largest single consumer of database time on the schedules path, and the fix is a sort key with no index behind it. ## Fix `BackgroundWorkerTask` is indexed on `(projectId, slug)`, which serves the equality but not the `ORDER BY createdAt DESC`. Postgres seeks the index, then bitmap-scans and top-N sorts the whole group to produce a single row. Adding `createdAt` to the index lets it scan backward and stop at the first row. The same call site also selected all 21 columns, including five JSON blobs, to read one field (`triggerSource`), so it now selects that field alone. ## Benchmark Local Postgres 17, 997,000 seeded rows / 748 MB, group sizes chosen to match the distribution seen in production. | Group size | Before | After | | --- | --- | --- | | 15,000 versions of one task | 11.118 ms, 1,510 buffers, 15,000 rows scanned | 0.027 ms, 4 buffers, 1 row | | 2,000 versions of one task | 2.081 ms, 1,455 buffers, 2,000 rows scanned | 0.022 ms, 4 buffers, 1 row | ``` before: Limit -> Sort (top-N heapsort) -> Bitmap Heap Scan after: Limit -> Index Scan Backward using BackgroundWorkerTask_projectId_slug_createdAt_idx ``` An ascending index scanned backward is enough here, so no descending index is needed. ## Impact and risk Real-world gain lands between the two rows above and scales with how many deploys a project has accumulated. Projects with few deploys will see little change, since there is barely anything to sort. The new index costs noticeably more than the existing two-column one: 43 MB against 7.3 MB on the benchmark rig. Adding `createdAt` makes every key unique, which defeats btree deduplication, so this is a real disk and write cost rather than a rounding error. Writes to this table happen at deploy time, not on the run path, so the write amplification is acceptable. The existing `(projectId, slug)` index is now a redundant prefix and could be dropped, but this PR keeps it so index usage can be observed before removing it. Behavior is unchanged: same predicate, same ordering, same row returned. The narrowed select is the only code change, and the field it keeps is the only one the caller read. Deploy note: the migration is `20260806100000_add_background_worker_task_project_id_slug_created_at_index` and uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it can be pre-applied by hand before the deploy. |
||
|
Eric Allam
|
6c6e58e6ff |
perf(webapp): batch declarative schedule cleanup queries (#4522)
## Summary `syncDeclarativeSchedules` runs on every background-worker creation (every deploy, and every file save during `trigger dev`). It issued one instance-delete per declarative schedule the current worker no longer declares, in a loop, and the overwhelming majority of those deletes matched zero rows. This collapses the loop into at most two set-based statements and skips the instance delete entirely when the current environment owns no instance of the schedule. ## Why so many, and mostly no-op The loop runs once per entry in `missingSchedules`, which starts as every DECLARATIVE schedule for the whole project across all its environments (the query filters only by `projectId`). A schedule leaves that set only when a declared task matches it by `taskIdentifier` **and** the schedule already has an instance in the current environment. That last clause is the amplifier. When a task's schedule has no instance in the current environment, the create branch inserts a brand-new `TaskSchedule` row with an instance for this environment rather than adding an instance to the existing row. So the same scheduled task, once it has run in dev and been deployed to prod, exists as two separate schedule rows: one carrying a dev instance, one carrying a prod instance. On a dev worker sync of that project: - the dev-instance row matches the declared task and is removed from the set - the prod-instance row has the same `taskIdentifier` but no dev instance, so it stays in the set and gets `deleteMany(taskScheduleId = prodRow, environmentId = dev)`, which matches zero rows So every declarative task that has been synced in another environment contributes one guaranteed no-op delete per sync, and the count scales with (declarative tasks x environments), plus any leftover rows from renamed or removed tasks. A project does not need to have dropped a schedule to generate these; it just needs the same declarative tasks present in more than one environment, which is the normal develop-in-dev, deploy-to-prod case. ## Fix The candidate schedules are already loaded with their instances, so the branch is decided in memory: - schedules with no instances (or only current-environment instances) are removed in a single `taskSchedule.deleteMany` - schedules that still have another environment's instance have only the current environment's instance detached, in a single `taskScheduleInstance.deleteMany`, and only when such an instance actually exists Behavior is unchanged (cascade delete still removes the instances of a deleted schedule); the difference is statement count. A zero-row delete writes no WAL and creates no dead tuples, so the removed work was pure query and commit overhead. Verified with a testcontainer test (red before, green after) counting the emitted deletes across the no-op, batched-detach, and schedule-delete cases, and end to end through `trigger dev`: three declarative schedules created, surviving a re-sync, then two removed in a single batched delete with the third preserved. |
||
|
Matt Aitken
|
04f9c4e1a5 |
fix(webapp,run-engine,core): drop the hidden debounce ceiling, fail fast on an unusable maxDelay (#4521)
Debouncing with a `delay` longer than an hour did nothing at all. The engine applied a server-side ceiling on how long a debounced run could be pushed back, measured from the run's `createdAt` and defaulting to one hour. A run is only pushed back while its new execution time stays inside that ceiling, so a `delay` at or above it could never push anything: the waiting run was released, the trigger started its own run, and the next trigger repeated it. A `delay: "12h"` produced one run per trigger, each correctly delayed by 12h, with no error raised and nothing on the run to show the debounce key had been ignored. The ceiling is now unset by default. A debounce key with no `maxDelay` keeps collapsing triggers for as long as they keep arriving, which is what the docs have always described. Self-hosters who want a bound can still set `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS`. That has a consequence worth stating plainly, so the docs now carry a warning for it: with no `maxDelay`, a continuously triggered key never executes. Set `maxDelay` when the work has to happen eventually. **Failing fast on an unusable `maxDelay`.** A caller who sets `maxDelay` no longer than their `delay` hits exactly the dead end described above, so that pair is now rejected at trigger time instead of silently behaving as if no debounce were set: ``` debounce.maxDelay (1h) must be longer than debounce.delay (12h). A debounced run is only pushed back while it stays inside maxDelay, so with these values every trigger would create its own run. ``` An unparseable `maxDelay` is rejected too, rather than quietly falling back to no bound at all, and so is a `delay` given as a date rather than a duration, which could never work because the value is re-applied on every push. The same check runs against a configured server ceiling, so a self-hosted deployment that sets `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS` gets the error rather than the silent failure this PR is about. With no `maxDelay` and no configured ceiling, which is the default, there is nothing to conflict with and nothing is rejected. The docs, the `TriggerOptions` JSDoc and the engine option all now state that the room available to push is the gap between `delay` and `maxDelay`. The run engine suite gains the case that motivated this: four triggers on one key with a 12h delay now collapse to a single run. |
||
|
Chris Arderne
|
088f68b373 |
feat(webapp): share rate limit bucket across additional API keys per environment (#4508)
## What
Rate-limit the API by **environment** rather than per API key.
Previously the limiter keyed its bucket on the hash of the full
`Authorization` header — one bucket per key. With additional environment
API keys (`tr_*_sk_*`), an environment can mint many keys and each got
its own full bucket, so more keys = higher effective rate limit. This
collapses all of an environment's keys onto a single shared
per-environment bucket, so the ceiling is exactly the configured limit
regardless of key mix.
## How
- `authorizationRateLimitMiddleware` now lets the override return `{
config?, identifier? }`. `identifier`, when present, is the rate limit
bucket key; otherwise it falls back to the hashed `Authorization` header
(unchanged legacy behavior, still used by `engineRateLimiter` and any
unauthenticated fallthrough).
- `apiRateLimiter`'s override resolves the environment id and uses it as
the identifier:
- **Additional keys** (`isAdditionalApiKey`) resolve via a new
`resolveAdditionalApiKeyRateLimitScope()` — a **scope-agnostic** keyHash
→ (environmentId, org limiter config) lookup. It is deliberately
permissive (restricted keys resolve too) because it's used **only for
bucketing, never as an auth decision** — request auth still goes through
the RBAC bearer controller, which enforces scopes. Revoked/expired keys
are excluded so they can't hold a bucket warm.
- **Root/legacy keys** reuse the environment already resolved by
`authenticateAuthorizationHeader` and key on `environment.id` too.
- The identifier is always the stable environment id, never the secret
key (which can rotate and would split the bucket).
- The whole override result is cached per key by the existing SWR cache,
so **no extra per-request lookup and no separate Redis mapping** is
added.
## Behavior notes
- Root + additional keys of the same environment now share one bucket
(ceiling = configured limit, not a multiple of it). Restricted
additional keys are included — they were the biggest gap, since they
authenticate via the RBAC controller and previously fell back to per-key
buckets.
- **Public JWTs** keep their existing fixed-window, per-token bucketing.
- One-time bucket reset on deploy (bucket keys change); harmless.
## Tests
- New: two tokens resolving to the same identifier share one bucket.
- New: with no identifier, bucketing stays per-key (legacy behavior
preserved).
- Updated existing override tests to the new `{ config }` return shape.
Base: `feat/multi-keys-surface`. Closes TRI-12888.
|
||
|
Chris Arderne
|
9409ddf9bc |
feat(webapp): add multiple environment API key management (#4390)
## Summary Projects can create, inspect, expire, and revoke multiple API keys for each environment. Plaintext values are shown only at creation; stored credentials are hashed and the API keys page displays only an obfuscated suffix afterward. Self-hosted installations support full-access additional keys by default. Authorization extensions can provide additional access presets and optional task selection. Additional keys can also mint scoped public access tokens through the Trigger.dev API without receiving the environment signing key. ## Feature notes - Only admin+ can create API keys (Developer can make in Development branch). - JWT self-signing will be a server call when used with new `_ak_` keys. - JWTs with long expiry can keep working even with api key deleted (gets priveleges from api key, signed with root key) - Unfiltered session listings intentionally preserve the existing broad task-read behavior. Filtered listings enforce task-level scopes for every requested task. - Buffered runs without a task identifier are not safely authorizable, so cancel/replay requests fail closed rather than resolving an unscoped run. - Batch and waitpoint endpoints intentionally return server-minted, narrowly scoped public tokens to all callers. These tokens have bounded lifetimes and may remain valid until expiry after API-key revocation. ## Deployment notes Deploy the management UI and public-token endpoint with new key creation disabled. Enable creation for selected organizations after the authentication path and released SDK have been verified, then expand availability gradually. Revoking an API key prevents new bearer requests and new token minting. Public tokens already minted by that key remain valid until their own expiration because they are signed by the environment signing key. ## TODO - [x] Add "Created by" to the key table - [x] Document that streamed batch ingestion is non-atomic and may partially accept items before a validation or authorization error. ## Follow-ups - [x] Add an organization-level feature flag for the API key management UI and creation action. - [x] Document rollout ordering: enable additional-key lookup before enabling issuance. - [x] Add a system-wide gate that can stop new key issuance without disabling authentication for existing keys. - [x] Replace the generic SDK compatibility warning with the first published compatible version. Old SDK will mint an unusable token if given an `_ak_` key. - [x] Add public documentation covering creation, storage, expiration, revocation, SDK compatibility, and public-token lifetime behavior. - [x] Add observability for key creation, revocation, policy preparation failures, and public-token mint failures. - [ ] Exercise create, copy-once display, authenticate, mint, expire, and revoke flows end to end before broad enablement. |
||
|
Katia Bulatova
|
337dda1e97 |
feat(webapp): name of the page in tab titles (#4517)
Adds a shared `pageMeta()` helper and 74 route declarations, so a title reads `run_abc | Runs | Trigger.dev` — the specific thing first, then the page. Org pages also carry the organization: `Team | Acme | Trigger.dev`. Inside a project no scope is added, because the dashboard switches projects in every tab at once. Page names are unchanged; what's new is that a page says which one it is at all. Three wording changes on purpose: the queue page now names the queue, the model page names the model, and entity pages carry their section. |
||
|
Eric Allam
|
58bf4e2833 |
feat(webapp): per-client database pool and connect timeout overrides (#4515)
## Summary Follow-on to #4513. The database connect timeout is now honored, but a single global value has to serve three separate databases at once (control-plane, legacy run-ops, and run-ops). This adds optional per-client overrides for the Prisma pool and connect timeouts, one pair for the writer and one for the read replica of each of the three databases, each falling back to the shared `DATABASE_POOL_TIMEOUT` / `DATABASE_CONNECTION_TIMEOUT` when unset. That lets one database's clients run a fail-fast connect timeout (with a bounded pool wait) while another keeps more headroom, without a single knob forcing the same tradeoff everywhere. No behavior change until an override is set. It also tags each client's queries with its specific datasource (`control-plane` / `legacy-run-ops` / `run-ops`, writer or replica) via the `db.datasource` span attribute, so telemetry can attribute connection behavior to a specific database instead of just writer-vs-replica. |
||
|
Eric Allam
|
771937adf5 |
fix(webapp): clamp run priority so a large value can't fail run creation (#4512)
## Summary Triggering a run with a very large `priority` could fail run creation outright with an opaque database error. `priority` is multiplied by 1000 and stored in a 32-bit integer column, with nothing bounding it, so a big enough value overflowed the column and the create failed. The trigger now caps the value to the highest supported priority instead of erroring, so the run is still created. ## Fix `priorityMs` (the stored `priority * 1000`) now goes through a `clampPriorityMs` helper before the write. It rounds to a whole number and clamps into the column range at both ends, so only a valid integer ever reaches the column and an out-of-range priority caps rather than failing. Single and batch triggers share the write path, so both are covered. |
||
|
Eric Allam
|
3039bc14d6 |
fix(webapp): honor the configured database connect timeout (#4513)
## Summary Every Prisma client built its connection URL with a `connection_timeout` query param, but the Postgres connector's parameter is `connect_timeout`. The misspelled param is silently ignored, so all clients fell back to Prisma's 5s default instead of the configured timeout. When establishing a new connection briefly took longer than 5s (for example during connection spikes), it failed with `Can't reach database server` even though the database was healthy. ## Fix All four client builders now construct their connection URL through one shared helper (`buildPrismaConnectionUrl`) that sets `connect_timeout`, so the configured value actually applies, and the parameter name lives in exactly one place. Covered by a unit test. |
||
|
Chris Arderne
|
85f5b37c68 |
chore: upgrade to TypeScript 7 (#4318)
## Summary Upgrade the monorepo to TypeScript 7.0.2 and update package build tooling for compatibility with the native compiler. ## Design Package builds now use `tshy` 4, while the packages still using `tsup` move to `tsdown`. The few scripts that depend on the legacy TypeScript compiler API use an explicit TypeScript 6 alias; declaration portability coverage invokes the TypeScript 7 CLI directly. Turbo is updated so workspace tasks can read the regenerated pnpm lockfile. --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
nicktrn
|
c01a4f18f4 |
feat(supervisor): cancel a resumed run's in-flight checkpoint (#4502)
A run controller must call the continue route to resume, so the supervisor already knows synchronously that any checkpoint still running for that run is pointless. It only acted on that for the compute path. The continue route now cancels it for the Kubernetes path too, matching what completion already does since #4493. Called after the reply so the runner is never delayed, and skipped when there is no checkpoint client or when the compute path owns the run. The request is bounded by a 5s timeout so a hung call cannot leave the handler pending. `checkpoint_cancel_requests_total{result}` records the outcome, using the same label names as the delete path where they overlap: `sent`, `no_client`, `not_applicable`, `http_error`. No changeset: `CheckpointClient` is a server-only internal API, same as #4493. refs TRI-12915 |
||
|
nicktrn
|
4f69c43e6b |
feat(supervisor): reclaim a run's checkpoint storage when it finishes (#4493)
When a run reaches a terminal state, ask the checkpoint service to
reclaim the storage its checkpoints occupied. Storage for finished runs
is not otherwise reclaimed, so nothing frees it today.
**Off by default** behind `DELETE_CHECKPOINTS_ON_COMPLETION`, and the
service-side handler ships separately, so merging this changes no
behaviour.
## Where the tenancy comes from
Addressing a run's checkpoints needs org, project, environment,
deployment version and run id. All five are already in hand at
`attempt.complete`, and three are **signed** by the deployment token:
| Value | Source | Trust |
| -- | -- | -- |
| org | claim `org_id` | signed |
| environment | claim `environment_id` | signed |
| deployment version | claim `deployment_version` | signed |
| project ref | `x-trigger-workload-project-ref` header |
runner-supplied |
| run | route param | runner-supplied |
`authorizeWorkloadRequest` previously returned only `environment_id`,
and only in enforce mode, so it now also returns the verified `claims`.
That difference is deliberate and documented on the method: claims are
used to address a run's **own** resources locally, never to scope the
platform, which is why `environmentId` stays enforce-only.
The two runner-supplied values are safe because the signed ones are
outermost - a runner lying about either can only name something inside
its own org and environment, and a project ref that doesn't pair with
its signed environment matches nothing. The run id is read from
`params.runFriendlyId`, the same value the platform just validated,
rather than from the body or a header. Where both a claim and a header
exist (`deployment_version`), the claim wins.
## Placement
The call sits after `reply.json(...)`, so the runner sees no added
latency - the same shape the suspend route already uses. The service
enqueues and returns 202, so it is one fast local hop.
Terminal means `RUN_FINISHED` **or `RUN_PENDING_CANCEL`** - a run
cancelled mid-execution never restores, and skipping it would leave its
storage behind. Retries are excluded deliberately: reclamation is
per-run, so a retry is covered by the final completion.
Also gated on `!snapshotService`, so it stays inert where checkpoints
aren't the kind this reclaims.
## Observability
`checkpoint_delete_requests_total{result}` counts `sent` **and every
reason we decide not to send**: `disabled`, `not_terminal`, `no_claims`,
`no_project_ref`, `http_error`.
The negative labels are the point - without them, "no requests are
happening" looks identical to the feature being switched off.
`no_claims` is reachable even under enforcement, since enforce only
rejects a *present-but-invalid* token; an absent or legacy id still
passes with no claims attached.
## Notes for review
- **No changeset**: `CheckpointClient` is `core/v3/serverOnly`, an
internal service-to-service API rather than customer-facing surface.
- **No `.server-changes/` note**: there is nothing a dashboard user
would notice here. Happy to add one if you disagree.
- `pnpm run typecheck` can't complete in my checkout -
`@trigger.dev/database` fails to build on a missing `tsc` in the pnpm
store, unrelated to this diff. Verified with `tsc --noEmit` against the
supervisor project instead: **zero errors in `apps/supervisor/src`**.
Worth noting it caught a real bug here - the completion response is
wrapped, so the status is `data.result.attemptStatus`.
refs TRI-12789
|
||
|
Katia Bulatova
|
fbd6df33b4 |
feat(webapp): Themes + contrast settings update (#4206)
Adds System Preferences, Dark and Light themes, gated by the `hasThemeSwitcher` feature flag (off by default — dark stays the default theme for everyone). Old theme is now "Classic"and set as default. "System preferences" theme has both Light and Dark modes and uses your laptop settings to use a correct one. It has less color accents (specifically less colored text), and they are the same for both modes, only grayscale values change between them. And Light/Dark themes can be used separately. New Contrast setting is available for System Preferences, Dark and Light themes - it changes the contrast for the whole app. All new visual Settings live in Account. |
||
|
Eric Allam
|
57254b57fb |
fix(webapp): make prop-types a production dependency (#4492)
## Summary The webapp's server bundle imports `prop-types` directly, but the package was declared only as a `devDependency`. A production install therefore leaves it out and the built server fails to boot: ``` Failed to start server: Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'prop-types' imported from /triggerdotdev/apps/webapp/build/server/assets/server-build-*.js ``` Moving it to `dependencies` is the whole change. ## Why the bundle imports it Nothing in the webapp's own code uses `prop-types` — there is no reference to it, or to `PropTypes`, anywhere under `apps/webapp/app`. It arrives through `recharts`, whose `react-smooth` dependency still declares `propTypes` on its components. That was invisible until recently. While `recharts` was resolved at runtime, its `prop-types` import was satisfied inside `recharts`' own dependency tree, which is production all the way down. #4486 added `recharts` and `victory-vendor` to `ssr.noExternal` to fix a hydration mismatch on every server-rendered chart; that inlines `react-smooth` into the server bundle, which moves its `prop-types` import into the webapp's own resolution scope — where the package was not available in production. So the bundling change was correct about *which* d3-shape build both sides resolve, and wrong about what the production runtime would be able to find. ## Verification `docker/Dockerfile` builds the runtime dependencies with `pnpm install --prod` against a `turbo prune --scope=webapp --docker` output, so I reproduced exactly that: pruned the workspace, installed with `--prod`, and imported `prop-types` from `apps/webapp`. | | result | | -- | -- | | `main` as it stands (devDependency only) | `FAILS: ERR_MODULE_NOT_FOUND` | | with this change | `prop-types resolves OK` | It resolves both as a CommonJS `require` and as an ESM `import`, which is the form the bundle uses. I also checked this is not one symptom of a wider problem: of the 169 bare specifier roots the server bundle imports, `prop-types` is the **only** one that is a devDependency and not a production dependency. The rest are node builtins or production dependencies. The hydration fix from #4486 is unaffected — the rebuilt bundle still carries the rounding d3-path build. ## Notes `prop-types` is inert in production (its entry point swaps in `factoryWithThrowingShims`), so this adds a 124 KB package that does no work at runtime. It has to be resolvable regardless, because the import is real. An alternative would be adding `prop-types` to `ssr.noExternal` so it is inlined and needs no runtime resolution. That keeps the dependency list honest about the fact that the webapp itself does not use it, at the cost of bundling a CommonJS package into the ESM server output. This route is the smaller, better-understood change. Worth following up separately: a check that every bare import in the server bundle resolves from a production install would have caught this before it landed. Local development installs every devDependency, so the gap is invisible when the built server is run from a working tree. |
||
|
nicktrn
|
3fba04573d |
fix(supervisor): hold the last backpressure verdict when a read fails (#4444)
The dequeue brake released the moment its signal became unreadable. `refresh()` caught any error from `source.read()` and set the verdict to `null`, which `computeEngaged()` treats as not-engaged — so a few failed reads dropped an engaged brake, silently, with no log and no metric. That handling was symmetric while the risk is not. A source that has stopped answering correlates with the pressure the brake exists for, so releasing on read failure gives up protection at exactly the wrong moment; holding too long only costs throughput. Now a failed read keeps the last verdict instead of discarding it. The verdict then ages normally, so the existing `maxVerdictAgeMs` check becomes the grace window and still bounds how long a dead source can hold the brake — a permanently unreachable source releases it rather than pinning dequeuing forever. Because `computeEngaged()` only consults staleness for an *engaged* verdict, a released one is unaffected and stays released. The default grace moves from 15s to 120s, comparable to how long the brake normally stays engaged. One guard worth calling out: holding is only safe when something bounds it, so when `maxVerdictAgeMs` is unset the previous discard behaviour is kept. Otherwise an unbounded hold could pin the brake indefinitely. Read failures were previously invisible — the catch block neither logged nor counted. Adds a `read_failures_total` counter, plus an error log on the transition into failure rather than once per tick, since the refresh loop runs every second. The post-release ramp needs no change: it anchors off the engaged-to-released transition, so a grace-window release still ramps back up instead of snapping to full rate, which is what you want after a blind period. Tests cover holding while reads fail, releasing past the max age, and the existing unbounded-config paths are unchanged. |
||
|
nicktrn
|
8f9db53350 |
feat(supervisor): configurable tolerations for run pods (#4491)
## Summary
Self-hosted Kubernetes deployments can now add tolerations to run pods,
so runs
can schedule onto tainted nodes. Previously the only way to do this was
to patch
the supervisor.
`KUBERNETES_RUNNER_TOLERATIONS` takes a comma separated list of
`key=value:effect`, or `key:effect` to tolerate any value. It applies to
every
run pod, and for runs from a schedule tree it merges with the existing
`KUBERNETES_SCHEDULED_RUN_TOLERATIONS`. Left unset, nothing changes: no
tolerations are added and the pod spec leaves the field off entirely.
The Helm chart takes it as a list:
```yaml
supervisor:
config:
kubernetes:
runnerTolerations:
- dedicated=runs:NoSchedule
- spot:NoExecute
```
## Naming
The issue proposed `KUBERNETES_WORKER_TOLERATIONS`. This ships as
`KUBERNETES_RUNNER_TOLERATIONS` instead, because `RUNNER_*` is already
the prefix
for run pod settings (`RUNNER_HEARTBEAT_INTERVAL_SECONDS`,
`RUNNER_ADDITIONAL_ENV_VARS`, and `DOCKER_RUNNER_NETWORKS` for the
Docker
equivalent), whereas "worker" refers to the supervisor itself throughout
this app.
## Validation
Keys and values are checked against the Kubernetes naming rules when the
supervisor starts, so `dedicated=prod runs:NoSchedule` fails immediately
with a
message naming the offending entry. Without that check a bad value is
accepted at
startup and then rejected by the API server on every pod create, which
stops all
runs with the cause buried in an API error.
`KUBERNETES_WORKER_NODETYPE_LABEL` is
trimmed and validated for the same reason: surrounding whitespace is not
valid in
a label value, so a padded value fails every pod create today.
## Node selector off switch
`KUBERNETES_WORKER_NODETYPE_LABEL` accepts an empty string to skip the
node
selector entirely, so runs schedule on any node. This already worked and
the Helm
chart has always shipped it empty, but it was not documented. It is now.
The issue also asked for general node affinity configuration. That is
not
included: the node selector off switch plus tolerations covers the
reported
problem, and a free form affinity setting is a much larger config
surface to
commit to.
Fixes #4458
|
||
|
Eric Allam
|
9d57aff542 |
fix(webapp): make the Queues hero charts environment-wide (#4486)
## Summary The four charts above the queues table aggregated over **at most the 25 queues on the current page**. They reused the loader's already-paginated queue array as a ClickHouse `queue IN (...)` filter, so paging or re-sorting changed the values, and a name search matching nothing blanked the whole chart row. The stat tiles above them were already environment-wide, so the two rows disagreed. They now read `env_metrics`, the environment-level rollup that already exists for exactly this (the built-in Queues dashboard and the health report read it). That is both correct and queue-count-independent: no `GROUP BY queue` across an entire environment, and no client-side summing. Note this is not only a paging artifact: page 1 under-reported too. On the seeded environment below, page 1 read 82% saturation against a true 87%, because the environment's running total is not the sum of one page of per-queue gauges. Three related fixes ride along. **Scheduling delay and throttling sawed to zero.** Both are event-driven, so at the 10-second bucket a short range picks, most buckets hold no samples at all and were drawn as `0ms`. Measured over a 1-hour window: **232 of 349 buckets had no scheduling-delay samples**. A bucket where nothing started is not a bucket where nothing waited, so the line was both ugly and wrong. TRQL grows a `minBucketSeconds` floor, plumbed through the metric resource route, and the hero tiles set 60s. Buckets that still have no samples render as a gap instead of a dive to zero. **The floor must not feed a width-dependent headline.** Two of the four headlines are not peaks, so widening the plotted buckets moved them: - **Throttled** is a share of buckets that saw any throttling, so a single brief throttle came to mark a whole minute instead of ten seconds: the same seeded events read 17% at 10s and 85% at 60s. - **Scheduling delay p95** is a percentile, and merging quantile states over a wider bucket yields a p95 between the sub-buckets' own. Two 240s samples among twenty in one 10-second sub-bucket give a worst-of-six p95 of 240,000ms against a merged 60-second p95 of 5,000ms — a 48x understatement of a headline whose tooltip claims it is the worst in the window. Both charts keep the floor, since a readable line was the point of it. Their headlines now come from a second query at the range's natural bucket width, via an optional `readout` on the tile, so each means what its tooltip says regardless of how the plotted buckets are sized. Saturation and backlog are genuinely width-invariant (a max of maxes is the same at any width), so they are unchanged and issue no extra query. Both caught by Devin in review; I had wrongly lumped p95 in with the peaks. **Charts reported a hydration mismatch on every render.** Recharts resolved victory-vendor's CJS entry on the server and its ESM entry in the browser. Those bundle different d3-shape builds, and the CJS one predates d3-path's digit rounding, so every server-rendered curve carried full-precision coordinates while the client rounded to 3 decimals: ``` Server: M0,3C0.9305555555555555,3,1.8611111111111112,3,... Client: M0,3C0.931,3,1.861,3,... ``` Bundling recharts for SSR makes both sides resolve the same ESM build. Verified: 45 of 45 server-rendered chart curves now match the client, and the page loads with an empty console. ## Verification An isolated stack with 40 seeded queues (20 heavily loaded, 20 idle) and 90 minutes of 10-second buckets written into `queue_metrics_raw_v1`, so the real materialized views built `queue_metrics_v1`, `env_metrics_v1` and the 5m rollup. Ground truth for the environment: 260 running against a limit of 300 (**87% saturation**), 800 queued. | | before | after | | -- | -- | -- | | Saturation, page 1 | 82% peak | **87% peak** | | Saturation, page 2 | 5% peak | **87% peak** | | Backlog / delay, page 2 | "No activity" | **800 peak / 59.5s** | | Name search matching nothing | all four charts blank | charts stay environment-wide | | Metric refetches on a page change | 4, each painting a skeleton | **0, no skeleton** | | Buckets drawn as 0ms with no samples | 232 of 349 | **0** | | Throttled readout | 17% | **17%**, unchanged by the wider buckets | | Worst-p95 readout source | plotted buckets | **natural width**, so a sub-minute spike is not averaged away | | Crosshair reach, hovering one detail-page chart | 2 of 4 others | **4 of 4** | | SSR chart curves mismatching the client | 45 | **0** | The bucket floor was measured across ranges: it widens 10s to 60s at 30m and 1h, and is correctly a no-op at 12h (300s) and 7d (3600s). One extra request per page load, for the throttled readout. The built-in Queues dashboard, which reads `env_metrics` independently, agrees at 86.7% and 260 of 300. `internal-packages/tsql` suite green (612 tests), including 5 new ones for the floor that fail without it. Webapp typecheck, oxfmt and oxlint clean. Spot-checked the Run metrics dashboard and the per-queue detail page for SSR regressions from bundling recharts: both render, console clean. The queue detail page carries the same event-driven series, so its scheduling delay, throttling and per-key mean delay take the same treatment. ## Screenshots <img width="2540" height="580" alt="after-page1-charts" src="https://github.com/user-attachments/assets/6cd23f9c-e7fd-4918-bcfa-b1d3340b16d1" /> ## Rollout Already behind the per-organization `queueMetricsUiEnabled` flag, so only gated orgs see any of it. Blast radius is chart values on one page plus the SSR bundling of recharts; rollback is a revert with no data migration. ## Stated limitations - `wait_ms_count` and the quantile state both only count `wait_ms > 0`, so "nothing started in this bucket" and "everything started instantly" are indistinguishable in storage. Both render as a gap. Distinguishing them needs a schema change, which is not in this PR. - The queue name search deliberately no longer narrows the charts. It only did so incidentally and incorrectly before (first 25 matches, and blanked on zero matches). Search-scoped charts would need the full unpaginated matching set and a server-side aggregate; worth its own ticket if we want it. - Bundling recharts for SSR grows the server bundle slightly. That is the cost of both sides resolving one d3-shape build. - The plotted delay line is a smoothed 60-second view, so a sub-minute spike above the one-minute warning threshold can fail to colour the line even though the headline reports it and colours itself. - Every chart inside one synced group shares the floor, because the hover crosshair is a reference line on a category x-axis and only draws where the hovered bucket exists in the other chart's own data. That costs the queue detail page's gauges some resolution (1 minute instead of 10 seconds) in exchange for the crosshair working across the row. Separately, while taking the screenshots I found a pre-existing rendering bug unrelated to this change: a **perfectly flat** saturation series draws no line at all (the readout still shows the right percentage), which looks like the threshold gradient's offset degenerating when the series min equals its max. It reproduces on `main`, so it is not a regression here and I have left it alone; filed as its own issue. Refs TRI-12784 |
||
|
Katia Bulatova
|
859f30e224 |
fix(webapp): report message catalogs survive the production bundle (#4488)
GET /api/v1/reports/health threw `no catalog registered for report "health"` in production (fine in dev): the catalog registered itself as a side effect of a bare import, which the SSR build tree-shakes under `"sideEffects": false`. Verified on the built server bundle — main's is missing the catalog, this branch's carries it. Fix: catalogs are values on the report registry entries; the resolver reads them from there and the mutable register-at-import step is gone. |
||
|
Chris Arderne
|
763b5dc582 |
feat(webapp): enforce scopes for environment API keys (#4389)
## Summary Environment API keys backed by the additional-key table can authenticate API requests using their stored effective scopes. Revoked and expired keys are rejected, branch environments retain their existing routing behavior, and last-used timestamps are updated on a throttled best-effort basis. ## Design API route builders receive the resolved ability and reject restricted keys on routes without an authorization declaration. Existing deployment, environment variable, queue, run, task, batch, session, and waitpoint routes declare the resources they access. Trigger and batch responses return server-signed public access tokens, so additional keys never need access to the environment signing secret. Root-key rotation also keeps public tokens valid for the existing grace window. ## Feature notes - Root environment keys remain unrestricted for backward compatibility. Additional keys enforce their persisted scopes and fail closed on routes without an authorization declaration. - Machine-key requests never exchange one credential for another. Additional keys cannot retrieve the root key, and rotated root keys are not upgraded during their grace window. - Public JWT validation remains host-owned, while installed RBAC plugins continue to supply root-key abilities. - Unfiltered session and run listings preserve existing broad task-read behavior. Filtered requests enforce the supplied task identifiers. - Related-run summaries remain embedded in run retrieval for API compatibility. Retrieving or mutating a related run independently still requires permission for that run. - Queue management authorizes at collection scope, matching the queue permissions currently issued. - Batch responses deliberately include server-signed public access tokens for all clients. Selected-task credentials continue using their original credential for per-item authorization. - Two-phase batches authorize declared task identifiers before creation and authorize every streamed item. Streaming paths that cannot declare the complete task set remain fail closed. - Authentication telemetry records successful credential resolution separately from subsequent resource-authorization failures. - API keys are high-entropy random tokens. SHA-256 is intentionally used for deterministic indexed lookup, not password hashing. ## Deployment notes The schema migration must be present before this code is deployed. Because bearer resolution runs on every authenticated request, deploy the resolver with additional-key lookup disabled, verify root-key and public-token parity, then enable lookup before any additional keys can be issued. The multi-task authorization tightening changes the result for narrowly scoped tokens that request tasks outside their grants. Observe would-deny results before enforcing that check. Request-idempotency keys are also newly isolated by environment and task, so a retry crossing the deployment boundary may execute once more before old cache entries expire. ## Follow-ups - [x] Add a system-wide kill switch for additional-key lookup, defaulted off for the initial deployment. - [x] Add authentication observability by credential kind, result, latency, and lookup path without recording credential values. - [ ] ~Add would-deny observability and an independent enforcement switch for multi-task authorization.~ - [ ] ~Add an independent switch for server-issued batch tokens while root-key parity is verified.~ - [ ] Confirm every API route reachable by a restricted key has an explicit authorization declaration or intentionally fails closed. - [x] Verify root-key rotation, revoked-key grace, and public-token validation through each bearer resolver path. |
||
|
Eric Allam
|
5f29ae49ab |
feat(webapp): default the queue metrics period to 1 hour and remember it (#4438)
## Summary
The Queues list and queue detail pages opened on a 1 day window, and
went back to it every time you navigated between queues or reloaded.
They now default to the last hour, and the period you pick is remembered
across navigations and refreshes.
## Design
The last period is stored in a `queueMetricsPeriod` cookie, written
client-side whenever a `period` lands in the URL and read by both
loaders. A cookie rather than localStorage because the queues list
renders its per-queue metrics columns server-side: with localStorage the
page would paint the 1 hour default and then re-fetch, and the picker
would flash the wrong window.
Both pages resolve the window once, in one place, and pass it down:
```ts
period: resolveQueueMetricsPeriod({
period: value("period"), // a usable period in the URL wins
from: value("from"), // an absolute range means "no period"
to: value("to"),
defaultPeriod, // otherwise the remembered default from the loader
}),
```
That keeps the picker pill and every chart query on the same value, so
no call site falls back to its own default. Periods the picker could
never produce (a hand-edited `?period=garbage`, or a window past the 30
day retention) fall back to the default, and the picker renders the
resolved window rather than the raw search param so the label can't
disagree with the data. Absolute from/to ranges, including drag-to-zoom,
are not remembered, since they would pin later visits to a window that
has gone stale.
While wiring that up: the two queue-metric queries that go straight to
ClickHouse (the list table and the concurrency-keys endpoint) never
applied the org's `queryPeriodDays` limit, so a hand-typed `?period=`
read further back than the plan allows. Everything behind
`/resources/metric` is already clipped that way by `executeQuery`; both
of these now clip with the same limit, capped at the retention window,
and the plan cap is resolved once per load and handed to the page
instead of each route deriving its own copy from the client-side
subscription.
Verified on both pages: default with no cookie is 1 hr, picking 6 hrs
survives navigating away and back to a param-free URL and a hard reload,
clearing the cookie returns to 1 hr, an oversized period falls back
without being remembered, and an absolute range still renders as a
range.
|
||
|
Iss
|
8f66af6e18 |
fix(webapp): stop the sidebar feedback popover from canceling the submit (#4445)
The Help & Feedback → "Contact us" form in the sidebar intermittently failed to send. The `<Feedback>` dialog was nested inside the Help popover, so clicking **Send** closed the popover and unmounted the form mid-submit — canceling the `POST /resources/feedback` before it went out. The message was silently lost (the success toast still shows). A race, so it "worked sometimes"; the standalone "I'm stuck!" path was unaffected. **Fix:** host the Feedback dialog *outside* the popover (same pattern as `AskAIRoot`) and open it from the menu item, so closing the popover no longer tears down the form. `Feedback` gains an optional controlled `open`/`setOpen` mode; existing `button`-triggered usages are unchanged. ## Changes - `Feedback.tsx` — optional controlled `open`/`setOpen`; `button` now optional. - `HelpAndFeedbackPopover.tsx` — "Contact us…" opens a `<Feedback>` hosted outside `PopoverContent`. - `.server-changes/fix-sidebar-feedback.md` — user-facing note. ## Testing Webapp typecheck passes. Sidebar "Contact us…" now sends on every attempt (Network: `POST /resources/feedback` → `204`, never `(canceled)`); "I'm stuck!" and the `?feedbackPanel=` open path unchanged. |
||
|
James Ritchie
|
14824b0955 |
feat(webapp): fix agent overview page scroll bug + layout fixes on task and agent pages (#4454)
## Summary The task, scheduled task and agent pages now name their runs table with its own title bar, and the controls that page the table sit beside it rather than in the bar at the top of the page. The top bar keeps just the date filter. Two agent page layout bugs are fixed along the way: scrolling a wide runs table sideways dragged the charts off screen with it, and the details panel stopped short of the bottom of the window. ## Fix The charts moved because the runs table had no horizontal scroller of its own. `stickyHeader` swaps the table's `overflow-x-auto` for `overflow-visible`, so the overflow escaped up to the page scroll box, and setting only `overflow-y-auto` on that box leaves the computed `overflow-x` at `visible`, which CSS then promotes to `auto`. The chart grid is a sibling inside that box, so it scrolled too. The table now keeps its own scroller (the same rule the queues list already documents) and the page box clips x so this cannot recur. The short panel was a second `PageContainer` wrapping the agent routes. `PageContainer` is `grid-rows-[auto_1fr]`, so a lone child lands in the `auto` row and its `h-full` resolves against content height instead of the viewport. This also reverts the global tooltip `max-w-[230px]` introduced in [#4131](https://github.com/triggerdotdev/trigger.dev/pull/4131), so longer tooltips are no longer squeezed into a narrow column. ### Agent overview page showing table now scrolling <img width="3452" height="1648" alt="CleanShot 2026-08-01 at 12 04 38@2x" src="https://github.com/user-attachments/assets/ef1ac55d-8ffb-4278-983b-031ed21c1f55" /> |
||
|
Eric Allam
|
db6228dd1e | chore(webapp,core,sdk): upgrade @s2-dev/streamstore to 0.25 and migrate S2 hosts (#4349) | ||
|
Eric Allam
|
f9c8d518c7 | perf(webapp,run-engine,database): resolve the newest worker and deployment by createdAt (#4452) | ||
|
Eric Allam
|
0445b8ec27 |
fix(webapp,clickhouse): keep the rest of a ClickHouse batch when one run or span has un-ingestable JSON (#4358)
## Summary A single run output, trace span, or payload carrying JSON that ClickHouse can't ingest (for example nesting past its depth limit) used to fail the whole insert batch, so unrelated runs and spans silently disappeared from the runs list, traces, and logs. This keeps the rest of the batch and handles the offending row instead of dropping everything around it. ## Fix Recovery is per-table, matched to what each table needs: - **Runs** (`task_runs_v2`) keep their status. We follow ClickHouse's failing-row hint to strip just the un-ingestable JSON column(s) so the run still lands (its output reads from Postgres on the detail page), up to a configurable limit (`RUN_REPLICATION_MAX_POISON_STRIPS_PER_BATCH`, default `1`). Past the limit we stop and land the batch with `allow_errors` in a single pass, skipping the remainder. Cost stays a fixed handful of inserts no matter how large or poisoned a flush is. - **Trace events and payloads** (high volume, append-only) recover with a single `allow_errors` insert: the good rows land in one pass and only the un-ingestable rows are skipped. Before falling back, a lightweight sanitizer still repairs what it can losslessly (lone UTF-16 surrogates, out-of-range integers) so a repairable row lands in full. To read the failing-row hint we patch `@clickhouse/client-common`: its error parser truncates the server response and discards the `(at row N)` position, so the patch preserves the full text for the recovery path to read. |
||
|
James Ritchie
|
fc69101252 |
feat(webapp): AI agent logo experiments (#4399)
## Summary Adds an admin-only "AI agent" storybook page exploring an animated identity for the dashboard agent: a resting dot logo that animates while the agent is thinking, then settles once it is done. The lead experiment is a 5x5 dot matrix. Shapes are five-line string bitmaps, a bright head walks each shape's route on a fixed beat, and it only hands off between shapes on a dot the two share, so the rhythm never breaks. It comes with 26 faces, six gradient palettes, and light and dark treatments. Two earlier prototypes (a crisp logo that scatters into orbiting dots, and a dotted triangle on tilted 3D orbits) are kept in their own tabs for comparison. Everything is plain canvas code with no new dependencies. Also adds an `ask-ai` Button variant: secondary styling with a soft trigger-green border and padding tuned around the leading logo. The variant supplies the agent logo itself, so callers write `<Button variant="ask-ai/small">Ask AI</Button>`. Passing a `LeadingIcon` overrides it, which is how the thinking animation gets driven. No release note: the storybook is admin gated and the button variant is not used in product UI yet. |
||
|
James Ritchie
|
55e6225b0f |
fix(webapp): focus the search field when a filter sub-menu opens (#4443)
## Summary Opening a filter sub-menu that has its own search field left the cursor outside it, so you had to click into the field before you could type. The cursor now lands in the search field every time a sub-menu opens. `ComboBox` now focuses its input whenever the popover is open and the field is present, so the cursor lands there both when a menu opens normally and when a sub-menu mounts its field late. It is a no-op wherever focus already worked. Verified in the dashboard against the Tags menu: before, the field mounted with focus still on the popover container; after, it mounts focused and accepts typing straight away. |
||
|
nicktrn
|
b42e5c3771 |
fix(supervisor): count pods from a limit=1 list instead of an aggregate metric (#4442)
The pod-count backpressure source read
`apiserver_storage_objects{resource="pods"}` from an apiserver
`/metrics` scrape. That gauge is a periodically-refreshed cached count,
and it is served by whichever apiserver replica the scrape lands on —
replicas disagree with each other at the same instant, by enough to
swamp the engage/release hysteresis band. Engage and release timing was
therefore partly a function of scrape routing.
This replaces it with a single `limit=1` list of the workload namespace
and computes `remainingItemCount + items.length`. One pod object
transferred, no informer, no watch cache.
Two request-shape constraints are load-bearing and called out in the
code: passing a label or field selector makes the apiserver omit
`remainingItemCount` entirely, and setting `resourceVersion` serves a
cached count rather than a quorum read. Neither is passed.
`remainingItemCount` is only set when the list is truncated, so
`_continue` is the truncation signal — if it is absent the returned page
is the whole collection and `items.length` is already exact. If the list
*is* truncated and the count is missing or implausible, the fetcher
throws rather than guessing.
Failure semantics are unchanged: a throw lands in the monitor's existing
catch, exactly as the previous parse did. The hysteresis, verdict shape,
and gauge are untouched. RBAC is unchanged — the existing role already
grants `pods: list`.
The `/metrics` non-resource grant in the deployment role becomes unused,
and the scrape-timeout env var is now a slight misnomer. Both left alone
deliberately: the grant may be wanted again for other apiserver signals,
and renaming the var would need a coordinated config change for no
behavioural gain.
Tests cover the not-truncated, truncated, missing-count, negative-count
and timeout paths.
|
||
|
Eric Allam
|
c72ebf9084 |
fix(webapp,run-engine): stop batchTriggerAndWait hanging when item streaming never completes (#4397)
## Summary `batchTriggerAndWait()` could leave a parent run waiting forever. The 2-phase batch API blocks the parent on the batch's waitpoint as soon as the batch is created, but the batch is only sealed at the end of item streaming. If streaming never completed, nothing sealed the batch, nothing completed the waitpoint, and the parent stayed suspended with no timeout and no way to recover. Supersedes #4016, which added the reaper alone. ## Fix Admission for item streaming was being decided twice. Batch creation passes its own rate limiter, which fixes `expectedCount` and blocks the parent, and then the item stream had to pass the general API limiter as well, competing with unrelated traffic. A second limiter could therefore veto work the first had already committed the parent to. Creation now mints a bounded grant that the item stream spends, so an admitted batch can finish streaming. The grant is capped per batch rather than exempting the path, and every failure mode (no grant, spent grant, unreachable store) falls back to the normal limiter. That makes stranding much rarer but not impossible, since a request timeout or a crash can still end streaming for good. So a seal-timeout reaper aborts any batch still unsealed after `BATCH_SEAL_TIMEOUT_MS` and completes the parent's waitpoint with an error, letting `batchTriggerAndWait()` reject instead of hang. It is race-safe against a late seal, and it is only scheduled for batches that actually block a parent, so fire-and-forget batches cost nothing. Finally, the batches page used to report "Batch completion checked." for these batches while doing nothing, because the completion path returns early on an unsealed batch. It now says the batch cannot be resumed. Rate limiting is no longer the reason a batch strands, so the reaper's default stays at 30 minutes, comfortably above the SDK's worst-case stream-retry budget. ## Verification Unit and container tests cover the grant cap, the bypass ordering (it runs after the authorization check, so it can never skip authentication), and the reaper's abort, seal race, idempotency, and no-waitpoint cases. Also verified end-to-end against a running stack. With the general limit exhausted, batch creation and other API calls returned 429 while a granted batch still streamed and sealed; an ungranted batch id was rate limited rather than bypassed; and the grant cut off exactly at its configured attempt count. Reproducing the stranded state on a real parent run, the batch was aborted at the timeout, the waitpoint completed with an error, and the parent resumed and finished instead of hanging. A parentless batch left unsealed was untouched well past the reaper window. ## Verified against deployed runs The reaper was proven end to end with a real deployed run (locally-run supervisor, containerised run) and a real network fault, rather than a simulated one: toxiproxy severs the phase 2 item stream mid-flight so every SDK stream retry genuinely fails, while phase 1 still succeeds. Only the batch calls traverse the fault, so control-plane traffic is untouched. The reproduction is the shape that actually strands a parent: the task catches the `BatchTriggerError` the SDK throws and carries on, so the phase 1 block outlives the thrown error and the parent hangs at its next suspension point. With the reaper disabled, the parent sat in `EXECUTING_WITH_WAITPOINTS` for over 24 minutes holding two blockers, and stayed stuck across a full infrastructure restart: ``` type | status | has_timeout BATCH | PENDING | f <- orphan, completedAfter NULL DATETIME | COMPLETED | t <- the wait already elapsed ``` With the reaper enabled the same task under the same fault completed in about 75 seconds with zero blockers left, the batch `ABORTED`, and its waitpoint completed carrying the error. Two conditions are required to observe this at all, which is worth knowing for any future test: the run must be deployed rather than `trigger dev` (dev runs execute in process and finish while still holding blocker rows), and the wait after the caught error must exceed the checkpoint threshold, or it is served in process and never suspends. ### Why completing the batch waitpoint is sufficient `batchTriggerAndWait` runs create, then stream, then wait. A phase 2 failure throws before the wait is ever reached, and the reaper only fires on an unsealed batch, so the parent is never suspended awaiting the batch when it runs. The parent therefore does not need a synthetic result, only to stop being blocked. Note this reasoning depends on that ordering: if the wait were ever reached with an unsealed batch, completing the batch waitpoint alone would not settle the caller. ## Follow-ups - Batches stranded before this ships still need a one-off recovery; the reaper only schedules at creation time. - That same property leaves a gap if the process dies between creating the batch and scheduling the job. A periodic sweep would close it, but wants a supporting index. - When a partially streamed batch aborts, children already enqueued keep running while the parent fails. Left as-is deliberately, since cancelling triggered work is a bigger semantic call. |
||
|
Wes Mason
|
efcb89ac26 |
fix(webapp): add hasAdminDisplayAccess to the env param test mock (#4430)
`test/envParamRoute.ownership.test.ts` fails on main: 3 of its 4 tests throw ``` Error: [vitest] No "hasAdminDisplayAccess" export is defined on the "~/services/session.server" mock. Did you forget to return it from "vi.mock"? ``` #4421 added a `hasAdminDisplayAccess(user)` call to the `env.$envParam` loader, and the test's `vi.mock` of `session.server` only returns `requireUser`, so the call blows up. Both changes were green in their own PR and only conflict once merged together, which is why nobody caught it. The mock now mirrors the real implementation rather than returning a constant, so it stays correct if the test's user fixture is ever varied. No assertions were changed: the tests were right, the mock was stale. Worth flagging separately: no workflow runs on push to main, so this has been red since #4421 landed without showing up anywhere. Every PR opened since has inherited the failure. |
||
|
claude[bot]
|
debfa2b733 | feat(webapp): impersonation consent page and a view-as-user toggle (#4421) | ||
|
Oskar Otwinowski
|
4efe0a07c4 |
fix(webapp): create dev environments for SSO and Directory Sync members (#4426)
Members added by SSO just-in-time provisioning or Directory Sync never got their per-member DEVELOPMENT environments - only invite acceptance and project creation created them. `trigger dev` returned "Environment not found" for those members and the dashboard had no dev view. ensureOrgMember now queues provisioning for every membership it settles, so both paths are covered and members missing environments are repaired on their next sync. Provisioning runs as a common-worker job to keep sign-in and directory webhooks off the per-project write loop. A failed enqueue surfaces for Directory Sync, whose worker retries the idempotent effect, and is swallowed for sign-in, where the next login enqueues again. Environment creation now tolerates a concurrent creator so the project-creation loop and the job cannot collide on the unique index. Also fixes environment resolution ignoring dev-environment ownership: a member without their own dev environment could be handed a colleague's and have it persisted as their dashboard preference. |
||
|
Iss
|
d90f06ba5e |
feat(webapp): migrate Plain to @team-plain/graphql + attribute support threads to org tenant (#4368)
## What
Two changes, shipped together:
1. **SDK migration (TRI-12460).** `@team-plain/typescript-sdk` is
deprecated. Move the webapp to its successors — `@team-plain/graphql`
(client) and `@team-plain/ui-components` (`uiComponent` builder).
Behaviour-preserving: the `PlainClient` customer upsert + thread
creation move to the new `client.mutation.*({ input })` shape; the
client now throws on failure, so `sendToPlain` wraps its calls and logs,
staying best-effort.
2. **Org tenant attribution (TRI-12461).** When org context is
available, `sendToPlain` now upserts a Plain tenant keyed by `externalId
= org_id`, links the customer to it, and stamps the created thread with
that tenant — so support threads become attributable to a Trigger.dev
org. Wired into the four add-on quota requests and the plan-cancellation
feedback (which already have org context). The tenant steps are isolated
in their own try/catch and the thread's `tenantIdentifier` is gated on
their success, so a tenant failure never blocks thread creation.
## Not affected
- `customer.externalId` stays `User.id` — the customer cards +
impersonation link are unchanged.
- No ticket content leaves Plain.
- Callers without a single org (e.g. the feedback widget) are unchanged
— the org params are optional.
## Deploy prerequisite
The webapp's Plain API key needs three **new** scopes for attribution to
work (it already has `customer:create`, `customer:edit`,
`thread:create`):
- [x] `tenant:create`
- [x] `tenant:edit`
- [x] `customerTenantMembership:create`
Until granted, nothing breaks — `sendToPlain` logs the forbidden error
and creates the thread without attribution.
## Testing
- `pnpm typecheck --filter webapp` passes; oxfmt + oxlint clean.
- Ran the real `sendToPlain` end-to-end via a throwaway vitest harness
against live Plain — confirmed the code path executes; the live write is
gated only by the key scopes above.
|
||
|
Eric Allam
|
6e5f0f0fe7 |
fix(webapp,clickhouse): stop invalid customer queries alerting, and isolate Sentry scope per request (#4372)
## Summary
A query sent to the query API with a typo in it, like a column name that
does not exist, was being reported as a server error. That put customer
SQL mistakes into our error alerting, where they made up almost all of
the volume on one of our noisiest alerts, and it drowned out the
failures that are actually ours to fix. This makes the level match who
is at fault, and fixes two related problems found alongside it.
## Invalid queries are the caller's, not ours
The query API route already got this right. It checks for `QueryError`,
logs at warn, and returns a 400, with a comment saying the system
handles it gracefully and no alert is needed.
The layer underneath ignored that. `executeTSQL` logged every exception
out of its catch block at error, including the compile failures the
route was about to turn into a 400, and error-level logs are forwarded
to error reporting.
The TSQL package already draws the line we need:
```ts
export class ExposedTSQLError extends BaseTSQLError {
/** An exception that can be exposed to the user. */
}
export class InternalTSQLError extends BaseTSQLError {
/** An internal exception in the TSQL engine. */
}
```
`SyntaxError` and `QueryError` extend the first. So the catch block now
branches on `ExposedTSQLError` and logs those at warn, keeping error for
`InternalTSQLError` and anything unanticipated, which is a genuine
compiler bug.
## SQL the caller wrote is their mistake, not ours
The same asymmetry showed up one level down. A query that compiles fine
can still be rejected by ClickHouse at execution, and most of those
rejections mean the caller's SQL is wrong rather than that we generated
something bad.
This is where the volume actually is. Checking production, one error
group alone, a missing `GROUP BY` on the public query API
(`NOT_AN_AGGREGATE`), accounts for over a million events across hundreds
of users. It is by far the largest error group in the project, and
classifying only by resource limit would have left every one of those at
error level.
So rejections are split three ways in `ClickhouseClient`, which is the
only place holding the parsed `ClickHouseError` and its symbolic type.
By the time the error reaches `executeTSQL` it has been wrapped and the
type is gone, and the type never appears in the message text, so it
cannot be recovered by string matching.
- **Resource limits** (memory ceiling, timeout, row/byte caps) log at
warn. The query is valid, it just asked for more than it is allowed to
spend.
- **Invalid SQL** (`NOT_AN_AGGREGATE`, `UNKNOWN_IDENTIFIER`,
`SYNTAX_ERROR`, the type and parse families) logs at warn **only when
the caller wrote the SQL**.
- **Everything else** keeps alerting.
That gate matters. The client is shared, so the identical rejection on
TRQL *we* generated is our bug and has to stay at error. Callers opt in
with `userAuthoredQuery`:
| caller | who wrote the SQL | opts in |
| --- | --- | --- |
| public query API | the customer | yes |
| query editor | the customer | yes |
| agent charts | the agent's model | yes |
| built-in dashboard tiles | us, in code | no |
| queue metric cards | us, in code | no |
| health report | us, in code | no |
The agent is the one judgement call. Its TRQL is not typed by a person,
but it is also not something a code fix makes correct, so a query it
gets wrong is not worth waking anyone for. The same endpoint serves
built-in tiles whose TRQL we do write, so the opt-in lives with the
caller rather than the route.
Separately, when one of these queries did fail, the log recorded the
generated ClickHouse SQL but not the query the caller actually wrote,
which made the reports hard to act on. `queryWithStats` takes an
optional `logFields` that `executeTSQL` uses to attach the original
TSQL.
## Events were attributed to the wrong request
Chasing the above turned up something broader: only a tenth of the
events on that alert pointed at the query API. The rest were pinned to
unrelated requests that happened to be in flight at the same time, so
the alert looked like the trigger endpoint was failing.
`Sentry.init` runs with `skipOpenTelemetrySetup: true`, because we
register our own OTel pipeline. That skips `initOpenTelemetry`, and one
of the things it does is:
```js
api.context.setGlobalContextManager(new SentryContextManager());
```
The async-context strategy is still installed, but `withIsolationScope`
only marks the OTel context and delegates the actual fork to that
context manager:
```js
// "We depend on the otelContextManager to handle the context/hub"
return api.context.with(ctx.setValue(SENTRY_FORK_ISOLATION_SCOPE_CONTEXT_KEY, true), ...)
```
`provider.register()` installed a plain
`AsyncLocalStorageContextManager`, which does not know that key. The
lookup found no scopes on the context and fell back to the
process-global default isolation scope, so every request wrote its
request data into the same object and the last writer won.
The tracer now registers `SentryContextManager`, which subclasses
`AsyncLocalStorageContextManager`, so OTel behaviour is unchanged. It is
also registered on the path where tracing is disabled, which previously
never called `register()` at all and so had no context manager of its
own.
Tenant tags were always correct, because those come from our own async
local storage rather than the isolation scope. That is why the
attribution being wrong was not obvious.
This affects every error report the webapp sends, not just the query
API.
## Verification
`internal-packages/clickhouse`: 76 tests pass, including eight covering
each level decision against a real ClickHouse container. Three pairs pin
the gate open and shut at both layers: an invalid query, a compile
failure, and a real limit breach driven with `max_rows_to_read` each log
at warn with `userAuthoredQuery` and at error without it.
The isolation fix has a test that reproduces the leak before asserting
the fix. Two overlapping requests each tag their own isolation scope;
with the plain context manager the slower one reads back the other's
tag, and with `SentryContextManager` each reads back its own.
Measured separately against a faithful reproduction of the server's
wiring (own OTel pipeline, CommonJS entry) at 200 concurrent requests:
per-request attribution goes from 0.5% to 100%, while span nesting,
context propagation across awaits, and distinct trace IDs are identical
before and after.
|
||
|
Chris Arderne
|
2f1734c858 | fix(core,webapp): redact sensitive fields in logs by default and cap their size (#4401) | ||
|
Chris Arderne
|
8ebc8a41af | fix(webapp,redis-worker): stop logging raw metadata, alert payloads, and job items (#4403) | ||
|
Chris Arderne
|
a09817169f | fix(webapp): stop logging full batch item contents in batchTriggerV3 (#4404) | ||
|
Chris Arderne
|
ed8f5e1297 | fix(webapp): stop logging every environment on a lookup miss (#4402) | ||
|
Chris Arderne
|
a81ad4949c |
feat(database,rbac): add multiple environment API key foundations (#4388)
Adds the storage model and authorization contracts needed for multiple environment API keys. Credentials are represented by hashed values, revocation and expiration state, and persisted effective scopes. The built-in authorization fallback exposes full-access policy preparation, while optional authorization extensions can supply additional presets and task-aware scope generation. This change does not create, display, or authenticate additional keys. |
||
|
Eric Allam
|
4eb9292cbe |
feat(webapp,run-engine): queue metrics and health dashboard (#4131)
## Summary
Three related changes, each independently gated:
**Queue metrics and health.** Per-queue depth, throughput (enqueued,
started, completed), concurrency, whether a queue is throttled, and
scheduling delay (how long a run waits between becoming eligible and
actually starting), plus a per concurrency-key breakdown for keyed
queues. Collected from inside the run queue itself, stored in
ClickHouse, and surfaced on the Queues list, a new per-queue detail
page, the task pages, and the run inspector. The question it answers is
"does this queue have enough concurrency to keep up, and if not, which
key or which limit is the constraint".
**Percent-based queue concurrency limits.** A queue's concurrency
override can now be expressed as a percentage of the environment limit,
stored as the source of truth and re-materialized whenever the
environment limit changes. Absolute overrides above the environment
limit are now **rejected with a 400** instead of being silently capped,
which is a behavior change on `POST
/api/v1/queues/:queue/concurrency/override`.
**The `health` report.** A server-computed verdict on whether work is
flowing, whether the runs that do start are healthy, and whether
telemetry is fresh, rendered as text with sparklines. Available as `GET
/api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool
(plus a `report` MCP prompt, which shows up as a slash command in hosts
that support prompts).
With the flags off, the Queues page renders the pre-metrics component
verbatim, nothing is emitted, and nothing is written to ClickHouse.
## Configuration
Two independent gates, on purpose. Emission is global so data accrues
for everyone before anyone can look at it; the view is per organization
so it can be turned on for one org at a time without a deploy.
**Runtime flags (no restart)**
| Flag | Store | Gates |
| --- | --- | --- |
| `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by
default) | All emission, gauges and counters. Cached in-process for 10s
with stale-while-revalidate, warmed eagerly at boot so the first op
after a deploy is not dropped. |
| `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` |
Fraction of queue ops that emit a gauge. Counters are never sampled, so
throughput stays exact at any rate. |
| `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag`
row, per-org `Organization.featureFlags` override wins | Whether an org
sees the metrics view at all: the Queues list variant, the queue detail
route, the built-in Queues dashboard, the concurrency-keys endpoint, and
the metrics blocks on task pages and the run inspector. Off by default;
a gated org gets a 404 on the detail route rather than an empty page. |
Both Redis keys are readable and writable from `/admin/queue-metrics`
(super-admin UI, with a live per-shard stream-health table) and
`GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface
uses its own Redis client, so it works on any instance regardless of
whether that instance runs the emitter or the consumer.
**Environment variables (boot time)**
| Variable | Default | Notes |
| --- | --- | --- |
| `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and
injects it into the run engine. Without it the run queue has no emitter
at all. |
| `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on
this instance. Independent of emission, so consumers can be sized
separately from the API. |
| `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per
queue. |
| `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals
insert batch, so an ack can never outrun a write. |
| `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` |
falls back to the run-queue Redis | Set `HOST` to move the metrics
stream onto a dedicated instance so a metrics backlog cannot compete
with the run queue for memory. Self-hosters can leave it unset and get a
single-Redis deployment. |
| `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000`
dedicated | Bound on how much a stalled consumer can hold. The default
is deliberately lower when the stream shares the queue-critical Redis. |
| `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the
per-queue cumulative counter key, refreshed on every write, so only
queues idle for the whole window are purged. |
| `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue
names tracked per environment; overflow collapses into `__overflow__`. |
| `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea
one level down, per queue. |
| `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live
sample-rate key above. |
| `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics
tables in the Query page, its schema docs, the schema API and the AI
query context. Off keeps them unlisted while the feature is dark; a
query naming them still runs either way. |
| `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring |
Runs queue metrics on their own ClickHouse service: the consumer's
inserts and every queue-metrics read go through it, so a metrics-heavy
chart refresh never competes with runs-list or trace reads. Unset
reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`,
reads on the query pool). |
| `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split,
so the consumer's inserts can never land on a read endpoint. |
|
`QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}`
| `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other
per-workload ClickHouse clients. |
Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a
Postgres migration adding the nullable
`TaskQueue.concurrencyLimitOverridePercent`. Both are additive.
## How collection works
Queue operations produce two kinds of signal, and they have opposite
failure modes, so they are handled differently.
**Gauges** (queued, running, queue limit, env queued, env running, env
limit, throttled, plus keys-with-backlog and worst-key wait on keyed
queues) are read *inside* the same Redis script that performs the
enqueue or dequeue, so the reading is atomic with the operation it
describes rather than a racy follow-up read. The script returns them on
its reply and the app forwards them to the stream. Gauges are sampled
and drop-tolerant: they are aggregated with `max`, so a lost reading
costs resolution, never correctness.
**Counters** (enqueued, started, completed, plus nack and dead-lettered)
are cumulative odometers. Each event increments a per-queue key on the
metrics Redis and emits the absolute total, and ClickHouse takes the
difference across buckets at read time. This is the important property
of the design: a summed-delta counter undercounts permanently on any
lost event, while a cumulative one self-heals, because the next
surviving reading restates the whole total. Only bucket granularity can
be lost, never the total. A queue returning after its odometer TTL
expired restarts at 1 and reset detection handles it, which is safe
precisely because expiry only spans a window with no activity.
Both land on one sharded Redis stream. A consumer reads it with a
consumer group, reclaims stale pending entries on a 15s interval rather
than on every poll, maps one entry to one or two ClickHouse rows
(whole-queue and, for keyed queues, per-key), and acks only after the
insert lands. Each batch carries a dedup token derived from its
stream-entry ids, and the target tables set
`non_replicated_deduplication_window`, so a retried batch cannot
double-count either the raw rows or the aggregates that hang off them.
Consumer and emitter both emit OTel metrics
(`queue_metrics.emitter.emitted`,
`queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`);
stream depth and group lag are the two worth alerting on, and
`lag_unknown` exists because Redis can report a null lag after a trim,
which must not be read as zero.
## Storage and read path
`queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four
aggregate tiers are materialized straight from raw, never cascaded off
each other, each with a 30 day TTL:
- `queue_metrics_v1`, 10 second buckets per queue, the default read path
- `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and
cross-queue ranking
- `env_metrics_v1`, 10 second buckets per environment, queue-independent
so it stays cheap at any range
- `queue_metrics_ck_v1`, 10 second buckets per concurrency key
Every tier is an MV from raw because the counter states do not survive a
cascade: their merge is order sensitive, so a `-MergeState` chain off
the 10s table inflates the result, and the same property means an
aggregate state may only be merged inside one queue. That constraint is
now enforced by the query engine rather than by reviewer discipline: a
column can declare a `mergeGroupKey`, and any query that references it
without grouping by, or pinning to a single value of, every named key
fails to compile with an actionable message.
On the read side, TRQL gains three tables (`queue_metrics`,
`env_metrics`, and a `queue_metrics_by_key` that is hidden from the
editor, schema docs and schema API but still queryable, so per-key rows
can never silently merge into a plain per-queue query), plus
`deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level
optimizations ride along: a table can declare coarser rollups, so a
query whose bucket interval is 5 minutes or wider is routed to the 5m
table with no change to the query itself, and it can opt into the
ClickHouse query cache with time bounds floored to a fixed grid, so the
auto-refreshing dashboards actually share cache entries instead of
missing on every tick. Both are caller-side substitutions, so the
printer stays unaware of physical layout.
All of this can also live on its own ClickHouse service. A table
declares the pool its reads run on, the three queue-metrics tables name
the dedicated one, and the ingestion consumer writes through the same
client, so both directions move together with one env var and nothing
else routes differently.
The other engine change is opt-in gap filling: charts can request rows
for empty buckets, where counters zero-fill and gauges carry forward.
Grouped gauge series are densified per group and carried inside a
partition, so a quiet queue's line holds its last value without bleeding
another queue's value into it.
## Queue concurrency limits
`concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth
when an override is set as a percentage; the absolute `concurrencyLimit`
is materialized from it (floored, clamped to at least 1 so a percentage
can never act as a pause, and never above the environment limit). Every
path that changes an environment limit now recalculates the
environment's percent-based overrides afterwards, outside the
transaction, and pushes changed limits to the engine. The push is
attempted even when the stored value did not change, so a previously
failed sync self-heals rather than leaving the database and the engine
diverged; paused queues are skipped so a recalculation cannot
effectively unpause one.
The API accepts exactly one of `concurrencyLimit` or `percent`, and the
reject-instead-of-clamp change above means a request asking for more
than the environment allows now fails loudly. The percent bound (greater
than 0, at most 100) is defined once and shared by the zod schema, the
dashboard mutation handler and the service, so the three cannot drift.
The concurrency-keys table on a queue is now paginated against the
ClickHouse per-key tier, ranked by peak backlog with the total on every
row from a single scan, and only the keys on the current page are
enriched with live counts from Redis. That replaces a hard top-50 cap
with something whose cost is a function of page size rather than key
cardinality.
## The health report
`GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The
verdict is computed on the server and is deterministic, not
model-generated. Three independent analyzers run over one input
snapshot: flow (is work moving, and if not, is the cause a limit,
throttling, one bad queue, or dead-lettering), execution (are the runs
that start succeeding, and at what latency), and liveness (how fresh is
the telemetry). When telemetry is genuinely stale, the first two are
forced to unknown and every actionable field is stripped, so no surface
ever advises action off stale data.
Authorization is per query table rather than a blanket query grant: a
JWT must be scoped to every table the report reads (`runs`,
`env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull
a report that reads more than it was granted. `period` is validated as a
shorthand with a 90 day ceiling at the edge. The report catalog is a
registry of `{ load, interpret }` entries, so the next report is a new
entry and no change to the route, the view model, the renderers, the CLI
or the MCP tool.
`trigger mcp` no longer launches the install wizard when stdout is a
TTY, which fixed a real failure: hosts spawn the server over a PTY, so
the wizard would open and the client would time out waiting for a server
that never started. The wizard now needs `trigger mcp --install`.
## The part that is live regardless of every flag
The enqueue and dequeue scripts now return a 2-tuple so a gauge reading
can ride back on the reply. Every return site in the eight affected
scripts is wrapped, and a `nil` original is converted to `false` on the
way out, because a raw `nil` in the first slot would make Lua truncate
the multi-bulk reply and silently drop the gauge on the throttled and
empty-queue paths. The reply shape and the destructuring on the app side
are exercised on every queue operation whether or not metrics are
enabled, so that is the part of `run-engine` worth the closest review.
One behavior fix in the same area: the scheduling-delay anchor is set
only on a run's first entry into the queue. Anchoring it to trigger time
on re-enqueues made waitpoint and checkpoint resumes report the entire
wait as scheduling delay. Queue ordering is untouched, so a re-enqueued
run keeps its position, and nacks deliberately keep the original anchor
because a rolled-back dequeue is the same continuous wait.
A pending-version promotion still anchors to trigger time, on purpose:
that promotion is the run's first real entry into the queue, since the
trigger deliberately held it back waiting for a worker version, and the
TTL is armed at the same point for the same reason. The consequence is
worth naming, because it is a judgement call: a run that waits on a
deployment reports that wait as scheduling delay on its queue, which is
time unrelated to queue capacity.
## Verification
Unit and integration suites across the new package, the run queue, the
mapping layer, the query engine and ClickHouse (including a test that
applies migration 036 through the same splitter CI uses, and a
regression test that inserts the same batch three times to prove the
aggregates do not inflate). Beyond that, the whole path was driven end
to end against a live stack with real runs: emitter to Redis stream to
consumer to ClickHouse to the dashboards, for both the local dev path
and the deployed path where a supervisor drives the dequeue, with
assertions on exact counter reconstruction per queue and per concurrency
key, throttling, environment saturation, scheduling delay, and a
deliberate mid-stream reading drop to confirm the cumulative counters
still reconstruct the correct total. The gated-off state was checked on
every touched surface.
The dedicated ClickHouse service was verified against a second,
separately-schema'd instance: with it configured, the driven counters
reconstruct exactly on the dedicated instance, the shared instance gains
no rows for that window, a read through the query API returns the value
that exists only on the dedicated instance, and a `runs` query still
succeeds (it would fail outright if it were mis-routed to a service
without that table). With the variable unset, the full suite passes
unchanged.
---------
Co-authored-by: Katia Bulatova <katia@trigger.dev>
Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com>
Co-authored-by: James Ritchie <james@trigger.dev>
|
||
|
claude[bot]
|
639eaf6e82 |
fix(webapp): don't apply an invite's role to an existing org member (#4409)
<!-- ccr-slack-attribution --> _Requested via [Slack thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785249693523749)_ ## Summary Accepting an old invitation could change the role of someone who was already in the organization. A long-pending invite can carry a lower role than the member has since been promoted to, so accepting it was a silent demotion. When the accepting user was the organization's only Owner, the role layer refused that demotion, and the refusal (an expected, protective outcome) was logged as an error. An invitation now only sets a role on a membership the accept actually created, and people who are already in an organization are skipped when invitations are sent. ## How `acceptInvite` already skipped the `OrgMember` create when it found an existing membership, but the `rbac.setUserRole` call below it was gated only on `invite.rbacRoleId`. It now also tracks whether this accept created the membership. A create that loses the unique-constraint race counts as pre-existing, since whichever flow won it owns that membership's role. Skipping existing members outright would regress one case: a member with no RBAC role at all would never receive the invitation's role. `ensureOrgMember` handles that with `healMissingRoleAssignment`, which fills in a null role but never overwrites a real one, so `assignInviteRbacRole` takes the same gate. An established role is never touched; an absent one is filled in. `assignInviteRbacRole` branches on the result's machine-readable `code` instead of logging every refusal at `error`. `last_owner` goes to `logger.info`, matching the two directory-sync role paths; everything else, including a refusal that carries no code, goes to `logger.warn`. The helper is best-effort and never throws, so no outcome it produces warrants `error`. No string matching on the error text is involved. `inviteMembers` resolves the organization's members by email and skips those addresses before creating invites. The invite table's `@@unique([organizationId, email])` only dedupes *pending invites*, so it could never catch this. ## Invite surfaces Skipping addresses means a batch can now come back empty, and neither caller handled that: - The dashboard action built its redirect from `invites[0].organization`, so a batch where every address was skipped threw a `TypeError` that reached the admin as a raw error string. It also reported the submitted count rather than the created one. It now names what it skipped ("No invitations sent: 1 already a member of this organization") and counts what it actually created. - The invites API derived `alreadyInvited` as "everything not created", so an existing member was reported as though they had already been invited. `inviteMembers` now returns the two groups separately and the endpoint reports `alreadyMembers` alongside `alreadyInvited`. ## Testing `apps/webapp/test/member.server.test.ts` passes 16/16 locally, up from 12. Getting there needed a harness fix. The `~/db.server` mock did not export `Prisma`, so any code reaching `PrismaNamespace.PrismaClientKnownRequestError` threw before it could branch, leaving every duplicate-key path in `member.server.ts` unreachable from tests. The mock now re-exports the real `Prisma`, and there is a case covering the pending-invite skip. New cases: the invite role is applied when the accept creates the membership; it is not applied when the member already has a role; it is applied when an existing member has no role assigned; the organization is still joined when the assignment is refused with `last_owner`; and `inviteMembers` reports members separately from pending invites. Forcing the gate off fails exactly the "already has a role" case, so the coverage is load-bearing. `pnpm run typecheck --filter webapp` and `oxfmt --check` both pass. ## Changelog Accepting an old invitation could change the role of someone who was already in the organization. An invitation now leaves an existing member's role untouched, people who are already in an organization are no longer sent invitations to it, and the invite form says which addresses it skipped instead of failing with an unhelpful error. --- ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works. ## Screenshots No visual changes. The invite form's toast copy changes, as described above. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
James Ritchie
|
a11e5ffbc6 |
fix(webapp): fade overflowing side menu selector labels (#4412)
Long organization, project, and environment names in the side menu were cut off mid-character. They now fade out at the right edge like the rest of the side menu items already did. ### Example of faded long names: <img width="246" height="200" alt="CleanShot 2026-07-28 at 22 59 24" src="https://github.com/user-attachments/assets/efa60b87-286f-4ab0-9d4e-490ef2de53e5" /> |
||
|
James Ritchie
|
1e14e29d71 |
fix(webapp): restyle the leave and remove team member dialogs (#4411)
## Summary
The confirmation dialog for leaving a team or removing a teammate was
still built on the old `Alert` primitive: the entire question sat in the
title, there was no header divider or `Esc` affordance, and the footer
used small buttons pinned to the right.
It now uses the standard `Dialog` layout the rest of the dashboard uses.
The title is static ("Remove team member" / "Leave team"), the question
moves into the body with the person's name and the organization
highlighted, and the footer is a bordered row with medium Cancel and
confirm buttons. A member who has not set a name is now identified by
their email instead of "them".
Verified against a local dashboard on both dialogs. Confirming a removal
posts the member id, deletes the membership and shows the success toast.
Cancel, `Esc`, and Enter while Cancel is focused all close the dialog
without issuing a request, leaving the member in place.
No release note needed: this is a visual restyle of an existing dialog
with no behaviour change.
|
||
|
Chris Arderne
|
38bf82aebe | feat(cli,webapp): target notifications by minimum CLI version (#4407) | ||
|
Saadi Myftija
|
44eca4d166 |
feat(webapp): org-gated internal API origin in run env vars (#4366)
Adds an opt-in way for operators to route deployed runs' API traffic through a different origin than the public one, per organization. Set `INTERNAL_API_ORIGIN` on the webapp and enable the `internalApiOriginEnabled` feature flag (globally or per org, with the org override winning in both directions): deployed runs for enabled orgs then get `TRIGGER_API_URL` set to the internal origin instead of `API_ORIGIN`. Useful for gradually moving run traffic onto a private network path. ## Design The origin is resolved when an attempt starts, so flag changes take effect on the next attempt and roll back the same way, with no task redeploys. The org override is read fresh per attempt; the global default comes from the cached flags registry (a cold read fails safe to the public origin). When `INTERNAL_API_ORIGIN` is unset the flag is a no-op and no extra queries run, so existing deployments are unaffected. Dev runs always use the public origin, and `TRIGGER_STREAM_URL` remains unchanged. |
||
|
claude[bot]
|
ec562c0e68 |
fix(webapp): remove unused Electric sync trace routes (#4400)
<!-- ccr-slack-attribution --> _Requested by **Eric Allam** · [Slack thread](https://triggerdotdev.slack.com/archives/C0AU83M3136/p1785222101937829?thread_ts=1785207509.304669&cid=C0AU83M3136)_ Removes two dead Remix routes and the helpers only they used. `app/routes/sync.traces.runs.$traceId.ts` (`/sync/traces/runs/:traceId`) and `app/routes/sync.traces.$traceId.ts` (`/sync/traces/:traceId`) were added with the original ElectricSQL run page and lost their only consumers when the dashboard hooks that called them were deleted. Nothing in the repo references either route today. Also removed, because the deleted routes were their only callers: - `OtelTraceIdSchema`, `RESERVED_ELECTRIC_SHAPE_PARAMS`, `TraceScope`, `buildElectricTraceWhereClause` from `app/v3/electricShape.server.ts` (the file stays — `UNSAFE_REALTIME_TAG_CHARS` / `sanitizeRealtimeTagForSql` / `sanitizeRealtimeTagsForSql` are still used by `realtime.v1.runs.ts` and `realtimeClient.server.ts`) - the loader-specific cases in `apps/webapp/test/spanTraceRoutes.replicaLag.test.ts` and `internal-packages/run-store/src/runOpsStore.routesSpanTraceReadView.replicaLag.test.ts` `app/utils/longPollingFetch.ts` is untouched — `realtimeClient.server.ts` still uses it. `runOpsStore.ts` / `PostgresRunStore.ts` are untouched too; the unrouted-lookup mechanism there is generic and stays. As a plain code fact: the run lookup these loaders performed keyed on `TaskRun.traceId` alone, which is not an index-backed query shape. That is noted only as context for why the code is not worth keeping around unused. ### Judgement call worth a maintainer's opinion The request was specifically about `/sync/traces/runs/:traceId`, the route that looks up a run by `traceId`. This PR **also** deletes its sibling `/sync/traces/:traceId`. The reasoning: - both routes came in with the same ElectricSQL run-page work - both lost their only consumers in the same later commit - neither has any caller anywhere in the repo - they share the same helper module, so keeping one means keeping the helpers half-used If you would rather keep the sibling, reverting just that one file deletion is easy and does not affect the rest of this PR — say the word and I will restore it along with the helpers it needs. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Verification run locally from the repo root: | Command | Result | | --- | --- | | `pnpm run format` | clean, no changes produced | | `pnpm run lint:fix` | clean | | `pnpm run lint` | pass (exit 0, no findings) | | `pnpm run typecheck --filter webapp` | pass | | `pnpm run typecheck --filter @internal/run-store` | pass | A ripgrep sweep for `sync.traces`, `sync/traces`, `syncTraceRunsLoader`, `buildElectricTraceWhereClause`, `OtelTraceIdSchema` and `RESERVED_ELECTRIC_SHAPE_PARAMS` (excluding `node_modules`) returns zero hits. **Not fully verified:** both edited test files are testcontainers suites and need a Docker runtime, which was not available in my environment. I confirmed each file *collects* correctly with exactly the three intended remaining tests and no import errors — notably, dropping the `session.server` / `controlPlaneResolver.server` / `longPollingFetch` / `env.server` mocks does not break module loading for the surviving loaders. The assertions themselves then failed only on `Could not find a working container runtime strategy`. CI should be the real signal here. Per `apps/webapp/CLAUDE.md`, `pnpm run build --filter webapp` was deliberately not run. --- ## Changelog Removed two unused sync routes left over from the original ElectricSQL run page, along with the helpers and tests that existed only to serve them. No behaviour change — neither route had any caller. --- ## Screenshots _n/a — no user-visible surface changes._ 💯 --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
James Ritchie
|
73eb4c5c16 |
feat(webapp): Improve the Integrations page layout (#4379)
## Summary The project Integrations page now uses the same settings layout as the org SSO page: a centered column of titled rows with dividers, instead of headings over bordered boxes. GitHub, Vercel and build settings read as one consistent list, and the page titles itself "Integrations". Confirmations persist rather than vanishing once you move past them (`GitHub app: Installed`, `Vercel project: Connected`), plan-gated rows offer an Upgrade button instead of a dead toggle, a disabled toggle explains why in place and highlights the control that unlocks it, and warnings are rows with a hazard icon and their recovery action on the right. Copy throughout leads with the outcome instead of restating the field label. Two fixes along the way: a nested `<form>` in the Vercel panel that failed hydration and silently truncated the page, and every settings row carrying a few pixels more space above its title than below its description. ### Before <img width="1160" height="1972" alt="CleanShot 2026-07-26 at 21 56 42@2x" src="https://github.com/user-attachments/assets/ed0fd676-36d8-4eb7-a16e-827a24f007d9" /> ### After <img width="1358" height="4455" alt="CleanShot 2026-07-26 at 19 14 28@2x" src="https://github.com/user-attachments/assets/6a635e6a-c0eb-4a4c-a68f-fcde4d25e8a6" /> |