main
7928 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Wes Mason
|
ca9a74e84a |
feat(observability-map): static observability scorer for webapp route entry points (#4455)
A static observability scorer for the webapp's route entry points,
Lighthouse-style. The idea comes from evlog's `map` command, but that
tool has no Remix adapter and checks for its own logging API, so the
idea is ported rather than the tool.
It scans all 427 loader/action entry points in `apps/webapp/app/routes`
with the TypeScript compiler API and scores each against five checks:
error-classification, auth-boundary, auth-scope, request-context and
audit-trail. Current output on the real tree is **19/100** over 412
measured entry points.
```
cd internal-packages/observability-map
pnpm exec tsx src/cli.ts # terminal report
pnpm exec tsx src/cli.ts --json # machine output
pnpm exec tsx src/cli.ts api/v1/token # one entry, per-check detail
```
The two findings at the top of the fix list are real: `/auth/sso` and
`/api/v1/authorization-code` mint or exchange credentials
unauthenticated, and `/_app/orgs/:organizationSlug/settings/team`
resolves its org from a URL slug and gates each mutating branch on an
RBAC check alone, which per `apps/webapp/CLAUDE.md` is not the tenant
floor on self-hosted.
Decisions worth knowing, all with the reasoning in the README:
- The score started at 83 during development and fell to 19. Every drop
was a perverse incentive being removed, not a regression: routes were
being paid for having no error handling, two checks were reading the
same fact, suppressing a failure raised the score, and a no-op `catch
(e) { throw e }` was worth 50 points a route.
- **A mutation corpus is the tool's main defence.** 44 entries apply
semantics-preserving edits to a copy of the real route tree and assert
the score cannot rise, per route as well as globally, because a mean can
hide one route going up by taking another down. One entry runs as a live
expected failure: `try { String(0); }` with a deciding catch is a known
open hole worth 19 to 44, and it is disclosed rather than quietly
excluded.
- `audit-trail` and `request-context` are reported as headline figures
rather than one finding repeated hundreds of times. Both still count in
full where they should.
- A cohort change moves the number without anything in the codebase
getting better. Widening the sensitive cohort from 26 to 67 took the
global from 15 to 19 with no webapp change at all, so the report prints
per-check applicability and what the global would be without each one.
CI: a report-only job posts a sticky comment when a PR moves the report,
and says nothing when it does not. The package's own tests gate through
`pr_checks.yml`. The diff-scoped merge gate is still deferred until the
report has been used in anger.
524 tests plus the corpus. No runtime or dependency changes to anything
that ships.
<!-- GitButler Footer Boundary Top -->
---
This is **part 1 of 4 in a stack** made with GitButler:
- <kbd> 4 </kbd> #4485
- <kbd> 3 </kbd> #4484
- <kbd> 2 </kbd> #4483
- <kbd> 1 </kbd> #4455 👈
<!-- GitButler Footer Boundary Bottom -->
|
||
|
nicktrn
|
4f69c43e6b |
feat(supervisor): reclaim a run's checkpoint storage when it finishes (#4493)
When a run reaches a terminal state, ask the checkpoint service to
reclaim the storage its checkpoints occupied. Storage for finished runs
is not otherwise reclaimed, so nothing frees it today.
**Off by default** behind `DELETE_CHECKPOINTS_ON_COMPLETION`, and the
service-side handler ships separately, so merging this changes no
behaviour.
## Where the tenancy comes from
Addressing a run's checkpoints needs org, project, environment,
deployment version and run id. All five are already in hand at
`attempt.complete`, and three are **signed** by the deployment token:
| Value | Source | Trust |
| -- | -- | -- |
| org | claim `org_id` | signed |
| environment | claim `environment_id` | signed |
| deployment version | claim `deployment_version` | signed |
| project ref | `x-trigger-workload-project-ref` header |
runner-supplied |
| run | route param | runner-supplied |
`authorizeWorkloadRequest` previously returned only `environment_id`,
and only in enforce mode, so it now also returns the verified `claims`.
That difference is deliberate and documented on the method: claims are
used to address a run's **own** resources locally, never to scope the
platform, which is why `environmentId` stays enforce-only.
The two runner-supplied values are safe because the signed ones are
outermost - a runner lying about either can only name something inside
its own org and environment, and a project ref that doesn't pair with
its signed environment matches nothing. The run id is read from
`params.runFriendlyId`, the same value the platform just validated,
rather than from the body or a header. Where both a claim and a header
exist (`deployment_version`), the claim wins.
## Placement
The call sits after `reply.json(...)`, so the runner sees no added
latency - the same shape the suspend route already uses. The service
enqueues and returns 202, so it is one fast local hop.
Terminal means `RUN_FINISHED` **or `RUN_PENDING_CANCEL`** - a run
cancelled mid-execution never restores, and skipping it would leave its
storage behind. Retries are excluded deliberately: reclamation is
per-run, so a retry is covered by the final completion.
Also gated on `!snapshotService`, so it stays inert where checkpoints
aren't the kind this reclaims.
## Observability
`checkpoint_delete_requests_total{result}` counts `sent` **and every
reason we decide not to send**: `disabled`, `not_terminal`, `no_claims`,
`no_project_ref`, `http_error`.
The negative labels are the point - without them, "no requests are
happening" looks identical to the feature being switched off.
`no_claims` is reachable even under enforcement, since enforce only
rejects a *present-but-invalid* token; an absent or legacy id still
passes with no claims attached.
## Notes for review
- **No changeset**: `CheckpointClient` is `core/v3/serverOnly`, an
internal service-to-service API rather than customer-facing surface.
- **No `.server-changes/` note**: there is nothing a dashboard user
would notice here. Happy to add one if you disagree.
- `pnpm run typecheck` can't complete in my checkout -
`@trigger.dev/database` fails to build on a missing `tsc` in the pnpm
store, unrelated to this diff. Verified with `tsc --noEmit` against the
supervisor project instead: **zero errors in `apps/supervisor/src`**.
Worth noting it caught a real bug here - the completion response is
wrapped, so the status is `data.result.attemptStatus`.
refs TRI-12789
|
||
|
Matt Aitken
|
e8398d13be |
chore: vouch Rohan170603 (#4501)
Adds `Rohan170603` to the list of vouched outside contributors so their PRs aren't auto-closed by the vouch check. Closes #4498 |
||
|
Katia Bulatova
|
fbd6df33b4 |
feat(webapp): Themes + contrast settings update (#4206)
Adds System Preferences, Dark and Light themes, gated by the `hasThemeSwitcher` feature flag (off by default — dark stays the default theme for everyone). Old theme is now "Classic"and set as default. "System preferences" theme has both Light and Dark modes and uses your laptop settings to use a correct one. It has less color accents (specifically less colored text), and they are the same for both modes, only grayscale values change between them. And Light/Dark themes can be used separately. New Contrast setting is available for System Preferences, Dark and Light themes - it changes the contrast for the whole app. All new visual Settings live in Account. |
||
|
Eric Allam
|
57254b57fb |
fix(webapp): make prop-types a production dependency (#4492)
## Summary The webapp's server bundle imports `prop-types` directly, but the package was declared only as a `devDependency`. A production install therefore leaves it out and the built server fails to boot: ``` Failed to start server: Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'prop-types' imported from /triggerdotdev/apps/webapp/build/server/assets/server-build-*.js ``` Moving it to `dependencies` is the whole change. ## Why the bundle imports it Nothing in the webapp's own code uses `prop-types` — there is no reference to it, or to `PropTypes`, anywhere under `apps/webapp/app`. It arrives through `recharts`, whose `react-smooth` dependency still declares `propTypes` on its components. That was invisible until recently. While `recharts` was resolved at runtime, its `prop-types` import was satisfied inside `recharts`' own dependency tree, which is production all the way down. #4486 added `recharts` and `victory-vendor` to `ssr.noExternal` to fix a hydration mismatch on every server-rendered chart; that inlines `react-smooth` into the server bundle, which moves its `prop-types` import into the webapp's own resolution scope — where the package was not available in production. So the bundling change was correct about *which* d3-shape build both sides resolve, and wrong about what the production runtime would be able to find. ## Verification `docker/Dockerfile` builds the runtime dependencies with `pnpm install --prod` against a `turbo prune --scope=webapp --docker` output, so I reproduced exactly that: pruned the workspace, installed with `--prod`, and imported `prop-types` from `apps/webapp`. | | result | | -- | -- | | `main` as it stands (devDependency only) | `FAILS: ERR_MODULE_NOT_FOUND` | | with this change | `prop-types resolves OK` | It resolves both as a CommonJS `require` and as an ESM `import`, which is the form the bundle uses. I also checked this is not one symptom of a wider problem: of the 169 bare specifier roots the server bundle imports, `prop-types` is the **only** one that is a devDependency and not a production dependency. The rest are node builtins or production dependencies. The hydration fix from #4486 is unaffected — the rebuilt bundle still carries the rounding d3-path build. ## Notes `prop-types` is inert in production (its entry point swaps in `factoryWithThrowingShims`), so this adds a 124 KB package that does no work at runtime. It has to be resolvable regardless, because the import is real. An alternative would be adding `prop-types` to `ssr.noExternal` so it is inlined and needs no runtime resolution. That keeps the dependency list honest about the fact that the webapp itself does not use it, at the cost of bundling a CommonJS package into the ESM server output. This route is the smaller, better-understood change. Worth following up separately: a check that every bare import in the server bundle resolves from a production install would have caught this before it landed. Local development installs every devDependency, so the gap is invisible when the built server is run from a working tree. |
||
|
nicktrn
|
3fba04573d |
fix(supervisor): hold the last backpressure verdict when a read fails (#4444)
The dequeue brake released the moment its signal became unreadable. `refresh()` caught any error from `source.read()` and set the verdict to `null`, which `computeEngaged()` treats as not-engaged — so a few failed reads dropped an engaged brake, silently, with no log and no metric. That handling was symmetric while the risk is not. A source that has stopped answering correlates with the pressure the brake exists for, so releasing on read failure gives up protection at exactly the wrong moment; holding too long only costs throughput. Now a failed read keeps the last verdict instead of discarding it. The verdict then ages normally, so the existing `maxVerdictAgeMs` check becomes the grace window and still bounds how long a dead source can hold the brake — a permanently unreachable source releases it rather than pinning dequeuing forever. Because `computeEngaged()` only consults staleness for an *engaged* verdict, a released one is unaffected and stays released. The default grace moves from 15s to 120s, comparable to how long the brake normally stays engaged. One guard worth calling out: holding is only safe when something bounds it, so when `maxVerdictAgeMs` is unset the previous discard behaviour is kept. Otherwise an unbounded hold could pin the brake indefinitely. Read failures were previously invisible — the catch block neither logged nor counted. Adds a `read_failures_total` counter, plus an error log on the transition into failure rather than once per tick, since the refresh loop runs every second. The post-release ramp needs no change: it anchors off the engaged-to-released transition, so a grace-window release still ramps back up instead of snapping to full rate, which is what you want after a blind period. Tests cover holding while reads fail, releasing past the max age, and the existing unbounded-config paths are unchanged. |
||
|
nicktrn
|
8f9db53350 |
feat(supervisor): configurable tolerations for run pods (#4491)
## Summary
Self-hosted Kubernetes deployments can now add tolerations to run pods,
so runs
can schedule onto tainted nodes. Previously the only way to do this was
to patch
the supervisor.
`KUBERNETES_RUNNER_TOLERATIONS` takes a comma separated list of
`key=value:effect`, or `key:effect` to tolerate any value. It applies to
every
run pod, and for runs from a schedule tree it merges with the existing
`KUBERNETES_SCHEDULED_RUN_TOLERATIONS`. Left unset, nothing changes: no
tolerations are added and the pod spec leaves the field off entirely.
The Helm chart takes it as a list:
```yaml
supervisor:
config:
kubernetes:
runnerTolerations:
- dedicated=runs:NoSchedule
- spot:NoExecute
```
## Naming
The issue proposed `KUBERNETES_WORKER_TOLERATIONS`. This ships as
`KUBERNETES_RUNNER_TOLERATIONS` instead, because `RUNNER_*` is already
the prefix
for run pod settings (`RUNNER_HEARTBEAT_INTERVAL_SECONDS`,
`RUNNER_ADDITIONAL_ENV_VARS`, and `DOCKER_RUNNER_NETWORKS` for the
Docker
equivalent), whereas "worker" refers to the supervisor itself throughout
this app.
## Validation
Keys and values are checked against the Kubernetes naming rules when the
supervisor starts, so `dedicated=prod runs:NoSchedule` fails immediately
with a
message naming the offending entry. Without that check a bad value is
accepted at
startup and then rejected by the API server on every pod create, which
stops all
runs with the cause buried in an API error.
`KUBERNETES_WORKER_NODETYPE_LABEL` is
trimmed and validated for the same reason: surrounding whitespace is not
valid in
a label value, so a padded value fails every pod create today.
## Node selector off switch
`KUBERNETES_WORKER_NODETYPE_LABEL` accepts an empty string to skip the
node
selector entirely, so runs schedule on any node. This already worked and
the Helm
chart has always shipped it empty, but it was not documented. It is now.
The issue also asked for general node affinity configuration. That is
not
included: the node selector off switch plus tolerations covers the
reported
problem, and a free form affinity setting is a much larger config
surface to
commit to.
Fixes #4458
|
||
|
Eric Allam
|
9d57aff542 |
fix(webapp): make the Queues hero charts environment-wide (#4486)
## Summary The four charts above the queues table aggregated over **at most the 25 queues on the current page**. They reused the loader's already-paginated queue array as a ClickHouse `queue IN (...)` filter, so paging or re-sorting changed the values, and a name search matching nothing blanked the whole chart row. The stat tiles above them were already environment-wide, so the two rows disagreed. They now read `env_metrics`, the environment-level rollup that already exists for exactly this (the built-in Queues dashboard and the health report read it). That is both correct and queue-count-independent: no `GROUP BY queue` across an entire environment, and no client-side summing. Note this is not only a paging artifact: page 1 under-reported too. On the seeded environment below, page 1 read 82% saturation against a true 87%, because the environment's running total is not the sum of one page of per-queue gauges. Three related fixes ride along. **Scheduling delay and throttling sawed to zero.** Both are event-driven, so at the 10-second bucket a short range picks, most buckets hold no samples at all and were drawn as `0ms`. Measured over a 1-hour window: **232 of 349 buckets had no scheduling-delay samples**. A bucket where nothing started is not a bucket where nothing waited, so the line was both ugly and wrong. TRQL grows a `minBucketSeconds` floor, plumbed through the metric resource route, and the hero tiles set 60s. Buckets that still have no samples render as a gap instead of a dive to zero. **The floor must not feed a width-dependent headline.** Two of the four headlines are not peaks, so widening the plotted buckets moved them: - **Throttled** is a share of buckets that saw any throttling, so a single brief throttle came to mark a whole minute instead of ten seconds: the same seeded events read 17% at 10s and 85% at 60s. - **Scheduling delay p95** is a percentile, and merging quantile states over a wider bucket yields a p95 between the sub-buckets' own. Two 240s samples among twenty in one 10-second sub-bucket give a worst-of-six p95 of 240,000ms against a merged 60-second p95 of 5,000ms — a 48x understatement of a headline whose tooltip claims it is the worst in the window. Both charts keep the floor, since a readable line was the point of it. Their headlines now come from a second query at the range's natural bucket width, via an optional `readout` on the tile, so each means what its tooltip says regardless of how the plotted buckets are sized. Saturation and backlog are genuinely width-invariant (a max of maxes is the same at any width), so they are unchanged and issue no extra query. Both caught by Devin in review; I had wrongly lumped p95 in with the peaks. **Charts reported a hydration mismatch on every render.** Recharts resolved victory-vendor's CJS entry on the server and its ESM entry in the browser. Those bundle different d3-shape builds, and the CJS one predates d3-path's digit rounding, so every server-rendered curve carried full-precision coordinates while the client rounded to 3 decimals: ``` Server: M0,3C0.9305555555555555,3,1.8611111111111112,3,... Client: M0,3C0.931,3,1.861,3,... ``` Bundling recharts for SSR makes both sides resolve the same ESM build. Verified: 45 of 45 server-rendered chart curves now match the client, and the page loads with an empty console. ## Verification An isolated stack with 40 seeded queues (20 heavily loaded, 20 idle) and 90 minutes of 10-second buckets written into `queue_metrics_raw_v1`, so the real materialized views built `queue_metrics_v1`, `env_metrics_v1` and the 5m rollup. Ground truth for the environment: 260 running against a limit of 300 (**87% saturation**), 800 queued. | | before | after | | -- | -- | -- | | Saturation, page 1 | 82% peak | **87% peak** | | Saturation, page 2 | 5% peak | **87% peak** | | Backlog / delay, page 2 | "No activity" | **800 peak / 59.5s** | | Name search matching nothing | all four charts blank | charts stay environment-wide | | Metric refetches on a page change | 4, each painting a skeleton | **0, no skeleton** | | Buckets drawn as 0ms with no samples | 232 of 349 | **0** | | Throttled readout | 17% | **17%**, unchanged by the wider buckets | | Worst-p95 readout source | plotted buckets | **natural width**, so a sub-minute spike is not averaged away | | Crosshair reach, hovering one detail-page chart | 2 of 4 others | **4 of 4** | | SSR chart curves mismatching the client | 45 | **0** | The bucket floor was measured across ranges: it widens 10s to 60s at 30m and 1h, and is correctly a no-op at 12h (300s) and 7d (3600s). One extra request per page load, for the throttled readout. The built-in Queues dashboard, which reads `env_metrics` independently, agrees at 86.7% and 260 of 300. `internal-packages/tsql` suite green (612 tests), including 5 new ones for the floor that fail without it. Webapp typecheck, oxfmt and oxlint clean. Spot-checked the Run metrics dashboard and the per-queue detail page for SSR regressions from bundling recharts: both render, console clean. The queue detail page carries the same event-driven series, so its scheduling delay, throttling and per-key mean delay take the same treatment. ## Screenshots <img width="2540" height="580" alt="after-page1-charts" src="https://github.com/user-attachments/assets/6cd23f9c-e7fd-4918-bcfa-b1d3340b16d1" /> ## Rollout Already behind the per-organization `queueMetricsUiEnabled` flag, so only gated orgs see any of it. Blast radius is chart values on one page plus the SSR bundling of recharts; rollback is a revert with no data migration. ## Stated limitations - `wait_ms_count` and the quantile state both only count `wait_ms > 0`, so "nothing started in this bucket" and "everything started instantly" are indistinguishable in storage. Both render as a gap. Distinguishing them needs a schema change, which is not in this PR. - The queue name search deliberately no longer narrows the charts. It only did so incidentally and incorrectly before (first 25 matches, and blanked on zero matches). Search-scoped charts would need the full unpaginated matching set and a server-side aggregate; worth its own ticket if we want it. - Bundling recharts for SSR grows the server bundle slightly. That is the cost of both sides resolving one d3-shape build. - The plotted delay line is a smoothed 60-second view, so a sub-minute spike above the one-minute warning threshold can fail to colour the line even though the headline reports it and colours itself. - Every chart inside one synced group shares the floor, because the hover crosshair is a reference line on a category x-axis and only draws where the hovered bucket exists in the other chart's own data. That costs the queue detail page's gauges some resolution (1 minute instead of 10 seconds) in exchange for the crosshair working across the row. Separately, while taking the screenshots I found a pre-existing rendering bug unrelated to this change: a **perfectly flat** saturation series draws no line at all (the readout still shows the right percentage), which looks like the threshold gradient's offset degenerating when the series min equals its max. It reproduces on `main`, so it is not a regression here and I have left it alone; filed as its own issue. Refs TRI-12784 |
||
|
Matt Aitken
|
75df940e4c |
chore: vouch Leafgard (#4489)
Adds `Leafgard` to the list of vouched outside contributors so their PRs aren't auto-closed by the vouch check. Closes #4487 |
||
|
Katia Bulatova
|
859f30e224 |
fix(webapp): report message catalogs survive the production bundle (#4488)
GET /api/v1/reports/health threw `no catalog registered for report "health"` in production (fine in dev): the catalog registered itself as a side effect of a bare import, which the SSR build tree-shakes under `"sideEffects": false`. Verified on the built server bundle — main's is missing the catalog, this branch's carries it. Fix: catalogs are values on the report registry entries; the resolver reads them from there and the mutable register-at-import step is gone. |
||
|
Chris Arderne
|
763b5dc582 |
feat(webapp): enforce scopes for environment API keys (#4389)
## Summary Environment API keys backed by the additional-key table can authenticate API requests using their stored effective scopes. Revoked and expired keys are rejected, branch environments retain their existing routing behavior, and last-used timestamps are updated on a throttled best-effort basis. ## Design API route builders receive the resolved ability and reject restricted keys on routes without an authorization declaration. Existing deployment, environment variable, queue, run, task, batch, session, and waitpoint routes declare the resources they access. Trigger and batch responses return server-signed public access tokens, so additional keys never need access to the environment signing secret. Root-key rotation also keeps public tokens valid for the existing grace window. ## Feature notes - Root environment keys remain unrestricted for backward compatibility. Additional keys enforce their persisted scopes and fail closed on routes without an authorization declaration. - Machine-key requests never exchange one credential for another. Additional keys cannot retrieve the root key, and rotated root keys are not upgraded during their grace window. - Public JWT validation remains host-owned, while installed RBAC plugins continue to supply root-key abilities. - Unfiltered session and run listings preserve existing broad task-read behavior. Filtered requests enforce the supplied task identifiers. - Related-run summaries remain embedded in run retrieval for API compatibility. Retrieving or mutating a related run independently still requires permission for that run. - Queue management authorizes at collection scope, matching the queue permissions currently issued. - Batch responses deliberately include server-signed public access tokens for all clients. Selected-task credentials continue using their original credential for per-item authorization. - Two-phase batches authorize declared task identifiers before creation and authorize every streamed item. Streaming paths that cannot declare the complete task set remain fail closed. - Authentication telemetry records successful credential resolution separately from subsequent resource-authorization failures. - API keys are high-entropy random tokens. SHA-256 is intentionally used for deterministic indexed lookup, not password hashing. ## Deployment notes The schema migration must be present before this code is deployed. Because bearer resolution runs on every authenticated request, deploy the resolver with additional-key lookup disabled, verify root-key and public-token parity, then enable lookup before any additional keys can be issued. The multi-task authorization tightening changes the result for narrowly scoped tokens that request tasks outside their grants. Observe would-deny results before enforcing that check. Request-idempotency keys are also newly isolated by environment and task, so a retry crossing the deployment boundary may execute once more before old cache entries expire. ## Follow-ups - [x] Add a system-wide kill switch for additional-key lookup, defaulted off for the initial deployment. - [x] Add authentication observability by credential kind, result, latency, and lookup path without recording credential values. - [ ] ~Add would-deny observability and an independent enforcement switch for multi-task authorization.~ - [ ] ~Add an independent switch for server-issued batch tokens while root-key parity is verified.~ - [ ] Confirm every API route reachable by a restricted key has an explicit authorization declaration or intentionally fails closed. - [x] Verify root-key rotation, revoked-key grace, and public-token validation through each bearer resolver path. |
||
|
nicktrn
|
d9f4fea939 |
docs: restructure self-hosting kubernetes guide (#4481)
Restructures the Kubernetes self-hosting guide around two explicit paths - an **evaluation install** (bundled datastores, one command) and a **production install** (external datastores, your own secrets) - so every configuration decision belongs to one path or the other instead of being a flat list of options with caveats. Also in this pass: - Adds an architecture overview (component-to-`values.yaml` map) and a post-install "verify it" step. - Consolidates the previously scattered upgrade notes into a single collapsible group, and cuts implementation detail and historical asides that no longer apply. - Removes a duplicated object-storage section (two configs in two styles) and trims the Docker ClickHouse note down to what a self-hoster needs to act on. |
||
|
Eric Allam
|
5f29ae49ab |
feat(webapp): default the queue metrics period to 1 hour and remember it (#4438)
## Summary
The Queues list and queue detail pages opened on a 1 day window, and
went back to it every time you navigated between queues or reloaded.
They now default to the last hour, and the period you pick is remembered
across navigations and refreshes.
## Design
The last period is stored in a `queueMetricsPeriod` cookie, written
client-side whenever a `period` lands in the URL and read by both
loaders. A cookie rather than localStorage because the queues list
renders its per-queue metrics columns server-side: with localStorage the
page would paint the 1 hour default and then re-fetch, and the picker
would flash the wrong window.
Both pages resolve the window once, in one place, and pass it down:
```ts
period: resolveQueueMetricsPeriod({
period: value("period"), // a usable period in the URL wins
from: value("from"), // an absolute range means "no period"
to: value("to"),
defaultPeriod, // otherwise the remembered default from the loader
}),
```
That keeps the picker pill and every chart query on the same value, so
no call site falls back to its own default. Periods the picker could
never produce (a hand-edited `?period=garbage`, or a window past the 30
day retention) fall back to the default, and the picker renders the
resolved window rather than the raw search param so the label can't
disagree with the data. Absolute from/to ranges, including drag-to-zoom,
are not remembered, since they would pin later visits to a window that
has gone stale.
While wiring that up: the two queue-metric queries that go straight to
ClickHouse (the list table and the concurrency-keys endpoint) never
applied the org's `queryPeriodDays` limit, so a hand-typed `?period=`
read further back than the plan allows. Everything behind
`/resources/metric` is already clipped that way by `executeQuery`; both
of these now clip with the same limit, capped at the retention window,
and the plan cap is resolved once per load and handed to the page
instead of each route deriving its own copy from the client-side
subscription.
Verified on both pages: default with no cookie is 1 hr, picking 6 hrs
survives navigating away and back to a param-free URL and a hard reload,
clearing the cookie returns to 1 hr, an oversized period falls back
without being remembered, and an absolute range still renders as a
range.
|
||
|
Iss
|
8f66af6e18 |
fix(webapp): stop the sidebar feedback popover from canceling the submit (#4445)
The Help & Feedback → "Contact us" form in the sidebar intermittently failed to send. The `<Feedback>` dialog was nested inside the Help popover, so clicking **Send** closed the popover and unmounted the form mid-submit — canceling the `POST /resources/feedback` before it went out. The message was silently lost (the success toast still shows). A race, so it "worked sometimes"; the standalone "I'm stuck!" path was unaffected. **Fix:** host the Feedback dialog *outside* the popover (same pattern as `AskAIRoot`) and open it from the menu item, so closing the popover no longer tears down the form. `Feedback` gains an optional controlled `open`/`setOpen` mode; existing `button`-triggered usages are unchanged. ## Changes - `Feedback.tsx` — optional controlled `open`/`setOpen`; `button` now optional. - `HelpAndFeedbackPopover.tsx` — "Contact us…" opens a `<Feedback>` hosted outside `PopoverContent`. - `.server-changes/fix-sidebar-feedback.md` — user-facing note. ## Testing Webapp typecheck passes. Sidebar "Contact us…" now sends on every attempt (Network: `POST /resources/feedback` → `204`, never `(canceled)`); "I'm stuck!" and the `?feedbackPanel=` open path unchanged. |
||
|
James Ritchie
|
14824b0955 |
feat(webapp): fix agent overview page scroll bug + layout fixes on task and agent pages (#4454)
## Summary The task, scheduled task and agent pages now name their runs table with its own title bar, and the controls that page the table sit beside it rather than in the bar at the top of the page. The top bar keeps just the date filter. Two agent page layout bugs are fixed along the way: scrolling a wide runs table sideways dragged the charts off screen with it, and the details panel stopped short of the bottom of the window. ## Fix The charts moved because the runs table had no horizontal scroller of its own. `stickyHeader` swaps the table's `overflow-x-auto` for `overflow-visible`, so the overflow escaped up to the page scroll box, and setting only `overflow-y-auto` on that box leaves the computed `overflow-x` at `visible`, which CSS then promotes to `auto`. The chart grid is a sibling inside that box, so it scrolled too. The table now keeps its own scroller (the same rule the queues list already documents) and the page box clips x so this cannot recur. The short panel was a second `PageContainer` wrapping the agent routes. `PageContainer` is `grid-rows-[auto_1fr]`, so a lone child lands in the `auto` row and its `h-full` resolves against content height instead of the viewport. This also reverts the global tooltip `max-w-[230px]` introduced in [#4131](https://github.com/triggerdotdev/trigger.dev/pull/4131), so longer tooltips are no longer squeezed into a narrow column. ### Agent overview page showing table now scrolling <img width="3452" height="1648" alt="CleanShot 2026-08-01 at 12 04 38@2x" src="https://github.com/user-attachments/assets/ef1ac55d-8ffb-4278-983b-031ed21c1f55" /> |
||
|
Matt Aitken
|
cb9aefd49b |
fix(hosting): deploy ClickHouse from the official image instead of Bitnami (#4249)
## Summary Self-hosted deployments now run ClickHouse from the official [`clickhouse/clickhouse-server`](https://hub.docker.com/r/clickhouse/clickhouse-server) image instead of `bitnamilegacy/clickhouse`. Bitnami's free image catalog is EOL and the frozen legacy archive tops out at ClickHouse 25.7.5, below the 25.8 minimum the platform requires since v4.5.0, which broke every ClickHouse insert on chart-bundled deployments. Both stacks now default to 26.2, the same version the platform is developed and tested against. Existing deployments keep their ClickHouse data with no manual migration. Fixes #4197. ## Details **Docker Compose**: the `clickhouse` service uses the official image with its native env vars, plus the recommended `nofile` ulimits. It reuses the same named volume as before: a `data-paths.xml` config override points ClickHouse at the `data/` subdirectory of the volume, which is exactly the layout the Bitnami image used, so old volumes work in place (including SQL-created users) and fresh installs get the identical layout. The service follows the required-secrets model: `CLICKHOUSE_PASSWORD` must be set, matching the other services. **Helm chart**: the Bitnami ClickHouse subchart is replaced by a chart-owned single-node StatefulSet and Service running the official image (non-root, HTTP `/ping` probes, config overrides mounted into `config.d`, and the same `data-paths.xml` layout compatibility). On upgrade, the chart automatically adopts the data PVC left behind by the old subchart (`data-<release>-clickhouse-shard0-0`) via `lookup`, and `fsGroup` relabeling handles the uid change on first mount. Both the ClickHouse server and the webapp read the password from the same chart-managed datastore secret (auto-generated and retained across upgrades), so the server credential and the app's connection URL always match. Existing `clickhouse.*` values keep working: `auth` (including `existingSecret`/`existingSecretKey`), `persistence` (including `global.storageClass`), `resources`, `secure`, `external.*`, `configdFiles`, and now `nodeSelector`/`tolerations`/`affinity`. Bitnami-only keys (`shards`, `replicaCount`, `keeper`, `resourcesPreset`) are gone; default `resources` requests/limits match what the old preset applied. The docs state the 25.8 minimum for bring-your-own ClickHouse. ## Upgrade caveats An adversarial review of the upgrade path found a few cohorts that need awareness (all documented): - **GitOps tools that render with `helm template`** (no cluster access): PVC auto-detection can't run, so `clickhouse.persistence.existingClaim` must be set to the old PVC name or ClickHouse starts on a fresh empty volume. Documented in the values file and the Kubernetes self-hosting docs. Tools that run real helm installs (e.g. Flux) adopt automatically. - **A pinned `CLICKHOUSE_IMAGE_TAG`** pointing at a Bitnami tag must be updated to an official image tag; documented in the Docker self-hosting docs. - **Storage without `fsGroup` support** (NFS, hostPath): set `clickhouse.volumePermissions.enabled: true` for a one-time ownership-fixing init container. - **Rollback is not automatic**: once the official image has run, file ownership changes and the Bitnami image can no longer read the volume without a manual chown, and ClickHouse does not support downgrades across the version gap. ## Verification - Full upgrade simulation for Compose, twice (before and after rebasing onto the required-secrets release): booted the ClickHouse service from the old compose file on `main` (Bitnami), wrote thousands of rows, then brought the same project up with this branch's compose file. The official 26.2 server came up healthy on the same volume with all rows intact, SQL-created users working, and writes succeeding. - Adoption scenarios tested against real containers: old volume + root entrypoint (Compose), old volume owned by the Bitnami uid + non-root 101 with fsGroup-style group permissions (Kubernetes), and fresh volumes for both. - `helm lint`, `helm template` (default values, `existingClaim` set, external ClickHouse, volumePermissions/scheduling toggles, and the production example) and kubeconform all pass, mirroring the release CI steps. The rendered webapp Deployment and ClickHouse StatefulSet resolve to the same datastore secret key. - Inserts using `input_format_json_infer_array_of_dynamic_from_array_of_different_types` (the setting that fails on 25.7.5) succeed on the upgraded volume. ## Upgrade preflight and docs A production upgrade report on this branch surfaced two hazards that predate this PR — both landed in chart 4.5.6 (#4316) — so they are fixed here rather than left for the next person to hit. **`secrets.existingSecret` gained two required keys.** The webapp started reading `PROVIDER_SECRET` and `COORDINATOR_SECRET`, and when `existingSecret` is set the chart generates nothing, so a missing key only surfaced as a `CreateContainerConfigError` partway through the webapp rollout. The pre-install/pre-upgrade validation now looks the Secret up and fails with the complete list of missing keys, leaving the running release untouched. It is skipped under `helm template` and client-side dry-run, where `lookup` cannot read the cluster. **Bundled datastore credentials moved into the chart-managed Secret** (`<release>-clickhouse`/`admin-password` → `trigger-datastore`/`clickhouse-admin-password`). The chart wires both ends itself, but consumers outside it — maintenance CronJobs, Grafana datasources, secret syncs — have to be repointed. A new `## Upgrading` section in the Kubernetes docs carries the old→new mapping, the two new keys, and a pointer to the ClickHouse image notes. The existingSecret key list in the docs also named `OBJECT_STORE_ACCESS_KEY_ID`/`OBJECT_STORE_SECRET_ACCESS_KEY`, which are env var names rather than keys the chart reads; corrected to the real key names and the condition under which they apply. Verified on a throwaway kind cluster with `--dry-run=server`: a pre-4.5.6 Secret fails with both key names listed, the documented `kubectl patch` clears it, and default values, `existingClaim`, external ClickHouse, volumePermissions/scheduling and the production example all still render. A real `helm install` followed by an upgrade against an incomplete Secret aborts with the release still at revision 1 and `deployed`. `helm lint`, the CI render and kubeconform (59 resources, 0 invalid) pass. --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com> |
||
|
Eric Allam
|
db6228dd1e | chore(webapp,core,sdk): upgrade @s2-dev/streamstore to 0.25 and migrate S2 hosts (#4349) | ||
|
Eric Allam
|
f9c8d518c7 | perf(webapp,run-engine,database): resolve the newest worker and deployment by createdAt (#4452) | ||
|
Eric Allam
|
0445b8ec27 |
fix(webapp,clickhouse): keep the rest of a ClickHouse batch when one run or span has un-ingestable JSON (#4358)
## Summary A single run output, trace span, or payload carrying JSON that ClickHouse can't ingest (for example nesting past its depth limit) used to fail the whole insert batch, so unrelated runs and spans silently disappeared from the runs list, traces, and logs. This keeps the rest of the batch and handles the offending row instead of dropping everything around it. ## Fix Recovery is per-table, matched to what each table needs: - **Runs** (`task_runs_v2`) keep their status. We follow ClickHouse's failing-row hint to strip just the un-ingestable JSON column(s) so the run still lands (its output reads from Postgres on the detail page), up to a configurable limit (`RUN_REPLICATION_MAX_POISON_STRIPS_PER_BATCH`, default `1`). Past the limit we stop and land the batch with `allow_errors` in a single pass, skipping the remainder. Cost stays a fixed handful of inserts no matter how large or poisoned a flush is. - **Trace events and payloads** (high volume, append-only) recover with a single `allow_errors` insert: the good rows land in one pass and only the un-ingestable rows are skipped. Before falling back, a lightweight sanitizer still repairs what it can losslessly (lone UTF-16 surrogates, out-of-range integers) so a repairable row lands in full. To read the failing-row hint we patch `@clickhouse/client-common`: its error parser truncates the server response and discards the `(at row N)` position, so the patch preserves the full text for the recovery path to read. |
||
|
James Ritchie
|
fc69101252 |
feat(webapp): AI agent logo experiments (#4399)
## Summary Adds an admin-only "AI agent" storybook page exploring an animated identity for the dashboard agent: a resting dot logo that animates while the agent is thinking, then settles once it is done. The lead experiment is a 5x5 dot matrix. Shapes are five-line string bitmaps, a bright head walks each shape's route on a fixed beat, and it only hands off between shapes on a dot the two share, so the rhythm never breaks. It comes with 26 faces, six gradient palettes, and light and dark treatments. Two earlier prototypes (a crisp logo that scatters into orbiting dots, and a dotted triangle on tilted 3D orbits) are kept in their own tabs for comparison. Everything is plain canvas code with no new dependencies. Also adds an `ask-ai` Button variant: secondary styling with a soft trigger-green border and padding tuned around the leading logo. The variant supplies the agent logo itself, so callers write `<Button variant="ask-ai/small">Ask AI</Button>`. Passing a `LeadingIcon` overrides it, which is how the thinking animation gets driven. No release note: the storybook is admin gated and the button variant is not used in product UI yet. |
||
|
James Ritchie
|
55e6225b0f |
fix(webapp): focus the search field when a filter sub-menu opens (#4443)
## Summary Opening a filter sub-menu that has its own search field left the cursor outside it, so you had to click into the field before you could type. The cursor now lands in the search field every time a sub-menu opens. `ComboBox` now focuses its input whenever the popover is open and the field is present, so the cursor lands there both when a menu opens normally and when a sub-menu mounts its field late. It is a no-op wherever focus already worked. Verified in the dashboard against the Tags menu: before, the field mounted with focus still on the popover container; after, it mounts focused and accepts typing straight away. |
||
|
nicktrn
|
b42e5c3771 |
fix(supervisor): count pods from a limit=1 list instead of an aggregate metric (#4442)
The pod-count backpressure source read
`apiserver_storage_objects{resource="pods"}` from an apiserver
`/metrics` scrape. That gauge is a periodically-refreshed cached count,
and it is served by whichever apiserver replica the scrape lands on —
replicas disagree with each other at the same instant, by enough to
swamp the engage/release hysteresis band. Engage and release timing was
therefore partly a function of scrape routing.
This replaces it with a single `limit=1` list of the workload namespace
and computes `remainingItemCount + items.length`. One pod object
transferred, no informer, no watch cache.
Two request-shape constraints are load-bearing and called out in the
code: passing a label or field selector makes the apiserver omit
`remainingItemCount` entirely, and setting `resourceVersion` serves a
cached count rather than a quorum read. Neither is passed.
`remainingItemCount` is only set when the list is truncated, so
`_continue` is the truncation signal — if it is absent the returned page
is the whole collection and `items.length` is already exact. If the list
*is* truncated and the count is missing or implausible, the fetcher
throws rather than guessing.
Failure semantics are unchanged: a throw lands in the monitor's existing
catch, exactly as the previous parse did. The hysteresis, verdict shape,
and gauge are untouched. RBAC is unchanged — the existing role already
grants `pods: list`.
The `/metrics` non-resource grant in the deployment role becomes unused,
and the scrape-timeout env var is now a slight misnomer. Both left alone
deliberately: the grant may be wanted again for other apiserver signals,
and renaming the var would need a coordinated config change for no
behavioural gain.
Tests cover the not-truncated, truncated, missing-count, negative-count
and timeout paths.
|
||
|
Eric Allam
|
f10bc23785 |
perf(run-engine,run-store): one execution snapshot per triggered run (#4419)
A non-delayed run used to get two execution snapshots the moment it was triggered: `RUN_CREATED` nested in the run-create transaction, immediately followed by `QUEUED` from its own `BEGIN`/`INSERT`/`COMMIT`. It now gets a single `QUEUED` snapshot written inside the create, and the trigger path only publishes to the queue. One fewer row per run on `TaskRunExecutionSnapshot`, and one fewer round trip on the trigger hot path. `EnqueueSystem` gains a `publishRun` seam that enqueues without writing a snapshot. Every re-enqueue path (waitpoint resume, checkpoint restore, delayed enqueue, pending version, retry requeue) still calls `enqueueRun` and writes its own `QUEUED`, so only the first enqueue changes. The `QUEUED` snapshot still commits before the queue message, so a dequeue sees a dequeueable status exactly as before. Two things for reviewers. Nesting the write skips `createExecutionSnapshot`, which is what emits `executionSnapshotCreated` and therefore the run timeline's `[engine] QUEUED` entry, so the trigger path now emits it directly, the same way the dequeue and attempt-start paths already do for their nested creates. And `RUN_CREATED` is still written when a dequeued run has no background worker yet, so the status and both `statuses.ts` helpers stay live and existing rows keep reading correctly. Delayed runs are untouched: `DELAYED` then `QUEUED` are two genuinely different moments and stay two snapshots. Rollback is a revert. Create-and-enqueue happen in one request in one process, so no in-flight run needs both code paths to agree during a rollout. One note for whoever debugs this path later. The `QUEUED` snapshot now commits before the queue publish, so a failed publish leaves the run recorded as `QUEUED` with no queue message. That state was already reachable, since the publish was never part of the snapshot transaction, but it used to be recorded as `RUN_CREATED`, which was distinctive because it never otherwise persisted. `QUEUED` with no message is indistinguishable from a run waiting on a concurrency slot, so trigger-time publish failure is now one more cause of an apparently stuck queued run. |
||
|
nicktrn
|
a91c08c731 |
fix(core): retry run start-attempt on transient connection errors (#4441)
## What `startRunAttempt` — the run controller's first call when a run starts — had no retry on transient connection errors. A brief connection blip on that call would abandon the start and send the run back through the queue, delaying its first attempt. This adds a jittered backoff retry, matching the existing `continueRunExecution` path with a shorter budget, so a transient blip is ridden out in place instead of bouncing the run. ## Why a shorter budget The continue path retries generously. Start-attempt keeps a tighter budget (6 attempts, ~25-40s jittered) so it rides out a transient blip but never keeps retrying past the point the run would already have been requeued. ## Safety Retrying is safe: start-attempt is guarded server-side by the snapshot id — a retry after a start has already committed is rejected, so it can never double-start an attempt. A pure connection error (the common case) never reached the server. ## Scope One retry-options object on `startRunAttempt`; no other behavior change. Warm starts share this path and get the same resilience. |
||
|
Eric Allam
|
c72ebf9084 |
fix(webapp,run-engine): stop batchTriggerAndWait hanging when item streaming never completes (#4397)
## Summary `batchTriggerAndWait()` could leave a parent run waiting forever. The 2-phase batch API blocks the parent on the batch's waitpoint as soon as the batch is created, but the batch is only sealed at the end of item streaming. If streaming never completed, nothing sealed the batch, nothing completed the waitpoint, and the parent stayed suspended with no timeout and no way to recover. Supersedes #4016, which added the reaper alone. ## Fix Admission for item streaming was being decided twice. Batch creation passes its own rate limiter, which fixes `expectedCount` and blocks the parent, and then the item stream had to pass the general API limiter as well, competing with unrelated traffic. A second limiter could therefore veto work the first had already committed the parent to. Creation now mints a bounded grant that the item stream spends, so an admitted batch can finish streaming. The grant is capped per batch rather than exempting the path, and every failure mode (no grant, spent grant, unreachable store) falls back to the normal limiter. That makes stranding much rarer but not impossible, since a request timeout or a crash can still end streaming for good. So a seal-timeout reaper aborts any batch still unsealed after `BATCH_SEAL_TIMEOUT_MS` and completes the parent's waitpoint with an error, letting `batchTriggerAndWait()` reject instead of hang. It is race-safe against a late seal, and it is only scheduled for batches that actually block a parent, so fire-and-forget batches cost nothing. Finally, the batches page used to report "Batch completion checked." for these batches while doing nothing, because the completion path returns early on an unsealed batch. It now says the batch cannot be resumed. Rate limiting is no longer the reason a batch strands, so the reaper's default stays at 30 minutes, comfortably above the SDK's worst-case stream-retry budget. ## Verification Unit and container tests cover the grant cap, the bypass ordering (it runs after the authorization check, so it can never skip authentication), and the reaper's abort, seal race, idempotency, and no-waitpoint cases. Also verified end-to-end against a running stack. With the general limit exhausted, batch creation and other API calls returned 429 while a granted batch still streamed and sealed; an ungranted batch id was rate limited rather than bypassed; and the grant cut off exactly at its configured attempt count. Reproducing the stranded state on a real parent run, the batch was aborted at the timeout, the waitpoint completed with an error, and the parent resumed and finished instead of hanging. A parentless batch left unsealed was untouched well past the reaper window. ## Verified against deployed runs The reaper was proven end to end with a real deployed run (locally-run supervisor, containerised run) and a real network fault, rather than a simulated one: toxiproxy severs the phase 2 item stream mid-flight so every SDK stream retry genuinely fails, while phase 1 still succeeds. Only the batch calls traverse the fault, so control-plane traffic is untouched. The reproduction is the shape that actually strands a parent: the task catches the `BatchTriggerError` the SDK throws and carries on, so the phase 1 block outlives the thrown error and the parent hangs at its next suspension point. With the reaper disabled, the parent sat in `EXECUTING_WITH_WAITPOINTS` for over 24 minutes holding two blockers, and stayed stuck across a full infrastructure restart: ``` type | status | has_timeout BATCH | PENDING | f <- orphan, completedAfter NULL DATETIME | COMPLETED | t <- the wait already elapsed ``` With the reaper enabled the same task under the same fault completed in about 75 seconds with zero blockers left, the batch `ABORTED`, and its waitpoint completed carrying the error. Two conditions are required to observe this at all, which is worth knowing for any future test: the run must be deployed rather than `trigger dev` (dev runs execute in process and finish while still holding blocker rows), and the wait after the caught error must exceed the checkpoint threshold, or it is served in process and never suspends. ### Why completing the batch waitpoint is sufficient `batchTriggerAndWait` runs create, then stream, then wait. A phase 2 failure throws before the wait is ever reached, and the reaper only fires on an unsealed batch, so the parent is never suspended awaiting the batch when it runs. The parent therefore does not need a synthetic result, only to stop being blocked. Note this reasoning depends on that ordering: if the wait were ever reached with an unsealed batch, completing the batch waitpoint alone would not settle the caller. ## Follow-ups - Batches stranded before this ships still need a one-off recovery; the reaper only schedules at creation time. - That same property leaves a gap if the process dies between creating the batch and scheduling the job. A periodic sweep would close it, but wants a supporting index. - When a partially streamed batch aborts, children already enqueued keep running while the parent fails. Left as-is deliberately, since cancelling triggered work is a bigger semantic call. |
||
|
claude[bot]
|
17d849b2d6 |
feat(cli): expose region option on the MCP trigger_task tool (#4439)
<!-- ccr-slack-attribution --> _Requested by **Eric Allam** · [Slack thread](https://triggerdotdev.slack.com/archives/C0BEM9Z73TM/p1785491472104199)_ ## Checklist - [ ] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [ ] I ran and tested the code works --- ## Testing Static checks only, all clean: - `pnpm run typecheck --filter trigger.dev` - `pnpm run format` - `pnpm run lint` No live task was triggered against a running project, so the "ran and tested" box above is left unchecked. --- ## Changelog **Before:** triggering a task through the MCP server always ran it in the project's default region. There was no way to pick one. **After:** the `trigger_task` tool accepts an optional `region` option, so you can choose the region a run executes in. **How:** `region: z.string().optional()` was added to `TriggerTaskInput.options` in `packages/cli-v3/src/mcp/schemas.ts`. No call-site change was needed — `tools/tasks.ts` passes `options` through verbatim, and `TriggerTaskRequestBody.options.region` already existed. The tool description in `docs/mcp-tools.mdx` gained a matching line, and a patch changeset is included. There is no batch-trigger MCP tool, so there is no sibling tool to mirror this change on. --- ## Screenshots N/A — no UI changes. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Wes Mason
|
efcb89ac26 |
fix(webapp): add hasAdminDisplayAccess to the env param test mock (#4430)
`test/envParamRoute.ownership.test.ts` fails on main: 3 of its 4 tests throw ``` Error: [vitest] No "hasAdminDisplayAccess" export is defined on the "~/services/session.server" mock. Did you forget to return it from "vi.mock"? ``` #4421 added a `hasAdminDisplayAccess(user)` call to the `env.$envParam` loader, and the test's `vi.mock` of `session.server` only returns `requireUser`, so the call blows up. Both changes were green in their own PR and only conflict once merged together, which is why nobody caught it. The mock now mirrors the real implementation rather than returning a constant, so it stays correct if the test's user fixture is ever varied. No assertions were changed: the tests were right, the mock was stale. Worth flagging separately: no workflow runs on push to main, so this has been red since #4421 landed without showing up anywhere. Every PR opened since has inherited the failure. |
||
|
claude[bot]
|
debfa2b733 | feat(webapp): impersonation consent page and a view-as-user toggle (#4421) | ||
|
Chris Arderne
|
68ed809416 |
test(clickhouse): keep queue metrics fixtures within TTL (#4428)
## Summary Keeps the queue metrics ClickHouse tests stable as wall-clock time advances. ## Root cause The fixtures used fixed timestamps. Once those timestamps crossed the tables' 30-day retention boundary, ClickHouse immediately expired the inserted aggregate rows and all six tests read empty results. The fixtures now derive a recent minute-aligned timestamp once per test file. The second 10-second bucket and ranking window are derived from the same anchor, preserving deterministic bucket relationships while keeping rows inside both the raw and aggregate table TTLs. Verified with `pnpm --filter @internal/clickhouse exec vitest run src/queueMetrics.test.ts`. |
||
|
Oskar Otwinowski
|
4efe0a07c4 |
fix(webapp): create dev environments for SSO and Directory Sync members (#4426)
Members added by SSO just-in-time provisioning or Directory Sync never got their per-member DEVELOPMENT environments - only invite acceptance and project creation created them. `trigger dev` returned "Environment not found" for those members and the dashboard had no dev view. ensureOrgMember now queues provisioning for every membership it settles, so both paths are covered and members missing environments are repaired on their next sync. Provisioning runs as a common-worker job to keep sign-in and directory webhooks off the per-project write loop. A failed enqueue surfaces for Directory Sync, whose worker retries the idempotent effect, and is swallowed for sign-in, where the next login enqueues again. Environment creation now tolerates a concurrent creator so the project-creation loop and the job cannot collide on the unique index. Also fixes environment resolution ignoring dev-environment ownership: a member without their own dev environment could be handed a colleague's and have it persisted as their dashboard preference. |
||
|
Iss
|
d90f06ba5e |
feat(webapp): migrate Plain to @team-plain/graphql + attribute support threads to org tenant (#4368)
## What
Two changes, shipped together:
1. **SDK migration (TRI-12460).** `@team-plain/typescript-sdk` is
deprecated. Move the webapp to its successors — `@team-plain/graphql`
(client) and `@team-plain/ui-components` (`uiComponent` builder).
Behaviour-preserving: the `PlainClient` customer upsert + thread
creation move to the new `client.mutation.*({ input })` shape; the
client now throws on failure, so `sendToPlain` wraps its calls and logs,
staying best-effort.
2. **Org tenant attribution (TRI-12461).** When org context is
available, `sendToPlain` now upserts a Plain tenant keyed by `externalId
= org_id`, links the customer to it, and stamps the created thread with
that tenant — so support threads become attributable to a Trigger.dev
org. Wired into the four add-on quota requests and the plan-cancellation
feedback (which already have org context). The tenant steps are isolated
in their own try/catch and the thread's `tenantIdentifier` is gated on
their success, so a tenant failure never blocks thread creation.
## Not affected
- `customer.externalId` stays `User.id` — the customer cards +
impersonation link are unchanged.
- No ticket content leaves Plain.
- Callers without a single org (e.g. the feedback widget) are unchanged
— the org params are optional.
## Deploy prerequisite
The webapp's Plain API key needs three **new** scopes for attribution to
work (it already has `customer:create`, `customer:edit`,
`thread:create`):
- [x] `tenant:create`
- [x] `tenant:edit`
- [x] `customerTenantMembership:create`
Until granted, nothing breaks — `sendToPlain` logs the forbidden error
and creates the thread without attribution.
## Testing
- `pnpm typecheck --filter webapp` passes; oxfmt + oxlint clean.
- Ran the real `sendToPlain` end-to-end via a throwaway vitest harness
against live Plain — confirmed the code path executes; the live write is
gated only by the key scopes above.
|
||
|
github-actions[bot]
|
86b948b47a |
chore: release v4.5.9 (#4408)
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 1s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
|
||
|
Eric Allam
|
6e5f0f0fe7 |
fix(webapp,clickhouse): stop invalid customer queries alerting, and isolate Sentry scope per request (#4372)
## Summary
A query sent to the query API with a typo in it, like a column name that
does not exist, was being reported as a server error. That put customer
SQL mistakes into our error alerting, where they made up almost all of
the volume on one of our noisiest alerts, and it drowned out the
failures that are actually ours to fix. This makes the level match who
is at fault, and fixes two related problems found alongside it.
## Invalid queries are the caller's, not ours
The query API route already got this right. It checks for `QueryError`,
logs at warn, and returns a 400, with a comment saying the system
handles it gracefully and no alert is needed.
The layer underneath ignored that. `executeTSQL` logged every exception
out of its catch block at error, including the compile failures the
route was about to turn into a 400, and error-level logs are forwarded
to error reporting.
The TSQL package already draws the line we need:
```ts
export class ExposedTSQLError extends BaseTSQLError {
/** An exception that can be exposed to the user. */
}
export class InternalTSQLError extends BaseTSQLError {
/** An internal exception in the TSQL engine. */
}
```
`SyntaxError` and `QueryError` extend the first. So the catch block now
branches on `ExposedTSQLError` and logs those at warn, keeping error for
`InternalTSQLError` and anything unanticipated, which is a genuine
compiler bug.
## SQL the caller wrote is their mistake, not ours
The same asymmetry showed up one level down. A query that compiles fine
can still be rejected by ClickHouse at execution, and most of those
rejections mean the caller's SQL is wrong rather than that we generated
something bad.
This is where the volume actually is. Checking production, one error
group alone, a missing `GROUP BY` on the public query API
(`NOT_AN_AGGREGATE`), accounts for over a million events across hundreds
of users. It is by far the largest error group in the project, and
classifying only by resource limit would have left every one of those at
error level.
So rejections are split three ways in `ClickhouseClient`, which is the
only place holding the parsed `ClickHouseError` and its symbolic type.
By the time the error reaches `executeTSQL` it has been wrapped and the
type is gone, and the type never appears in the message text, so it
cannot be recovered by string matching.
- **Resource limits** (memory ceiling, timeout, row/byte caps) log at
warn. The query is valid, it just asked for more than it is allowed to
spend.
- **Invalid SQL** (`NOT_AN_AGGREGATE`, `UNKNOWN_IDENTIFIER`,
`SYNTAX_ERROR`, the type and parse families) logs at warn **only when
the caller wrote the SQL**.
- **Everything else** keeps alerting.
That gate matters. The client is shared, so the identical rejection on
TRQL *we* generated is our bug and has to stay at error. Callers opt in
with `userAuthoredQuery`:
| caller | who wrote the SQL | opts in |
| --- | --- | --- |
| public query API | the customer | yes |
| query editor | the customer | yes |
| agent charts | the agent's model | yes |
| built-in dashboard tiles | us, in code | no |
| queue metric cards | us, in code | no |
| health report | us, in code | no |
The agent is the one judgement call. Its TRQL is not typed by a person,
but it is also not something a code fix makes correct, so a query it
gets wrong is not worth waking anyone for. The same endpoint serves
built-in tiles whose TRQL we do write, so the opt-in lives with the
caller rather than the route.
Separately, when one of these queries did fail, the log recorded the
generated ClickHouse SQL but not the query the caller actually wrote,
which made the reports hard to act on. `queryWithStats` takes an
optional `logFields` that `executeTSQL` uses to attach the original
TSQL.
## Events were attributed to the wrong request
Chasing the above turned up something broader: only a tenth of the
events on that alert pointed at the query API. The rest were pinned to
unrelated requests that happened to be in flight at the same time, so
the alert looked like the trigger endpoint was failing.
`Sentry.init` runs with `skipOpenTelemetrySetup: true`, because we
register our own OTel pipeline. That skips `initOpenTelemetry`, and one
of the things it does is:
```js
api.context.setGlobalContextManager(new SentryContextManager());
```
The async-context strategy is still installed, but `withIsolationScope`
only marks the OTel context and delegates the actual fork to that
context manager:
```js
// "We depend on the otelContextManager to handle the context/hub"
return api.context.with(ctx.setValue(SENTRY_FORK_ISOLATION_SCOPE_CONTEXT_KEY, true), ...)
```
`provider.register()` installed a plain
`AsyncLocalStorageContextManager`, which does not know that key. The
lookup found no scopes on the context and fell back to the
process-global default isolation scope, so every request wrote its
request data into the same object and the last writer won.
The tracer now registers `SentryContextManager`, which subclasses
`AsyncLocalStorageContextManager`, so OTel behaviour is unchanged. It is
also registered on the path where tracing is disabled, which previously
never called `register()` at all and so had no context manager of its
own.
Tenant tags were always correct, because those come from our own async
local storage rather than the isolation scope. That is why the
attribution being wrong was not obvious.
This affects every error report the webapp sends, not just the query
API.
## Verification
`internal-packages/clickhouse`: 76 tests pass, including eight covering
each level decision against a real ClickHouse container. Three pairs pin
the gate open and shut at both layers: an invalid query, a compile
failure, and a real limit breach driven with `max_rows_to_read` each log
at warn with `userAuthoredQuery` and at error without it.
The isolation fix has a test that reproduces the leak before asserting
the fix. Two overlapping requests each tag their own isolation scope;
with the plain context manager the slower one reads back the other's
tag, and with `SentryContextManager` each reads back its own.
Measured separately against a faithful reproduction of the server's
wiring (own OTel pipeline, CommonJS entry) at 200 concurrent requests:
per-request attribution goes from 0.5% to 100%, while span nesting,
context propagation across awaits, and distinct trace IDs are identical
before and after.
|
||
|
Chris Arderne
|
2f1734c858 | fix(core,webapp): redact sensitive fields in logs by default and cap their size (#4401) | ||
|
Chris Arderne
|
8ebc8a41af | fix(webapp,redis-worker): stop logging raw metadata, alert payloads, and job items (#4403) | ||
|
Chris Arderne
|
a09817169f | fix(webapp): stop logging full batch item contents in batchTriggerV3 (#4404) | ||
|
Chris Arderne
|
ed8f5e1297 | fix(webapp): stop logging every environment on a lookup miss (#4402) | ||
|
Chris Arderne
|
878c15811a | fix(cli): redact environment values from build debug logs (#4420) | ||
|
Chris Arderne
|
a81ad4949c |
feat(database,rbac): add multiple environment API key foundations (#4388)
Adds the storage model and authorization contracts needed for multiple environment API keys. Credentials are represented by hashed values, revocation and expiration state, and persisted effective scopes. The built-in authorization fallback exposes full-access policy preparation, while optional authorization extensions can supply additional presets and task-aware scope generation. This change does not create, display, or authenticate additional keys. |
||
|
Eric Allam
|
4eb9292cbe |
feat(webapp,run-engine): queue metrics and health dashboard (#4131)
## Summary
Three related changes, each independently gated:
**Queue metrics and health.** Per-queue depth, throughput (enqueued,
started, completed), concurrency, whether a queue is throttled, and
scheduling delay (how long a run waits between becoming eligible and
actually starting), plus a per concurrency-key breakdown for keyed
queues. Collected from inside the run queue itself, stored in
ClickHouse, and surfaced on the Queues list, a new per-queue detail
page, the task pages, and the run inspector. The question it answers is
"does this queue have enough concurrency to keep up, and if not, which
key or which limit is the constraint".
**Percent-based queue concurrency limits.** A queue's concurrency
override can now be expressed as a percentage of the environment limit,
stored as the source of truth and re-materialized whenever the
environment limit changes. Absolute overrides above the environment
limit are now **rejected with a 400** instead of being silently capped,
which is a behavior change on `POST
/api/v1/queues/:queue/concurrency/override`.
**The `health` report.** A server-computed verdict on whether work is
flowing, whether the runs that do start are healthy, and whether
telemetry is fresh, rendered as text with sparklines. Available as `GET
/api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool
(plus a `report` MCP prompt, which shows up as a slash command in hosts
that support prompts).
With the flags off, the Queues page renders the pre-metrics component
verbatim, nothing is emitted, and nothing is written to ClickHouse.
## Configuration
Two independent gates, on purpose. Emission is global so data accrues
for everyone before anyone can look at it; the view is per organization
so it can be turned on for one org at a time without a deploy.
**Runtime flags (no restart)**
| Flag | Store | Gates |
| --- | --- | --- |
| `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by
default) | All emission, gauges and counters. Cached in-process for 10s
with stale-while-revalidate, warmed eagerly at boot so the first op
after a deploy is not dropped. |
| `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` |
Fraction of queue ops that emit a gauge. Counters are never sampled, so
throughput stays exact at any rate. |
| `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag`
row, per-org `Organization.featureFlags` override wins | Whether an org
sees the metrics view at all: the Queues list variant, the queue detail
route, the built-in Queues dashboard, the concurrency-keys endpoint, and
the metrics blocks on task pages and the run inspector. Off by default;
a gated org gets a 404 on the detail route rather than an empty page. |
Both Redis keys are readable and writable from `/admin/queue-metrics`
(super-admin UI, with a live per-shard stream-health table) and
`GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface
uses its own Redis client, so it works on any instance regardless of
whether that instance runs the emitter or the consumer.
**Environment variables (boot time)**
| Variable | Default | Notes |
| --- | --- | --- |
| `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and
injects it into the run engine. Without it the run queue has no emitter
at all. |
| `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on
this instance. Independent of emission, so consumers can be sized
separately from the API. |
| `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per
queue. |
| `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals
insert batch, so an ack can never outrun a write. |
| `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` |
falls back to the run-queue Redis | Set `HOST` to move the metrics
stream onto a dedicated instance so a metrics backlog cannot compete
with the run queue for memory. Self-hosters can leave it unset and get a
single-Redis deployment. |
| `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000`
dedicated | Bound on how much a stalled consumer can hold. The default
is deliberately lower when the stream shares the queue-critical Redis. |
| `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the
per-queue cumulative counter key, refreshed on every write, so only
queues idle for the whole window are purged. |
| `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue
names tracked per environment; overflow collapses into `__overflow__`. |
| `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea
one level down, per queue. |
| `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live
sample-rate key above. |
| `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics
tables in the Query page, its schema docs, the schema API and the AI
query context. Off keeps them unlisted while the feature is dark; a
query naming them still runs either way. |
| `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring |
Runs queue metrics on their own ClickHouse service: the consumer's
inserts and every queue-metrics read go through it, so a metrics-heavy
chart refresh never competes with runs-list or trace reads. Unset
reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`,
reads on the query pool). |
| `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split,
so the consumer's inserts can never land on a read endpoint. |
|
`QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}`
| `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other
per-workload ClickHouse clients. |
Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a
Postgres migration adding the nullable
`TaskQueue.concurrencyLimitOverridePercent`. Both are additive.
## How collection works
Queue operations produce two kinds of signal, and they have opposite
failure modes, so they are handled differently.
**Gauges** (queued, running, queue limit, env queued, env running, env
limit, throttled, plus keys-with-backlog and worst-key wait on keyed
queues) are read *inside* the same Redis script that performs the
enqueue or dequeue, so the reading is atomic with the operation it
describes rather than a racy follow-up read. The script returns them on
its reply and the app forwards them to the stream. Gauges are sampled
and drop-tolerant: they are aggregated with `max`, so a lost reading
costs resolution, never correctness.
**Counters** (enqueued, started, completed, plus nack and dead-lettered)
are cumulative odometers. Each event increments a per-queue key on the
metrics Redis and emits the absolute total, and ClickHouse takes the
difference across buckets at read time. This is the important property
of the design: a summed-delta counter undercounts permanently on any
lost event, while a cumulative one self-heals, because the next
surviving reading restates the whole total. Only bucket granularity can
be lost, never the total. A queue returning after its odometer TTL
expired restarts at 1 and reset detection handles it, which is safe
precisely because expiry only spans a window with no activity.
Both land on one sharded Redis stream. A consumer reads it with a
consumer group, reclaims stale pending entries on a 15s interval rather
than on every poll, maps one entry to one or two ClickHouse rows
(whole-queue and, for keyed queues, per-key), and acks only after the
insert lands. Each batch carries a dedup token derived from its
stream-entry ids, and the target tables set
`non_replicated_deduplication_window`, so a retried batch cannot
double-count either the raw rows or the aggregates that hang off them.
Consumer and emitter both emit OTel metrics
(`queue_metrics.emitter.emitted`,
`queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`);
stream depth and group lag are the two worth alerting on, and
`lag_unknown` exists because Redis can report a null lag after a trim,
which must not be read as zero.
## Storage and read path
`queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four
aggregate tiers are materialized straight from raw, never cascaded off
each other, each with a 30 day TTL:
- `queue_metrics_v1`, 10 second buckets per queue, the default read path
- `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and
cross-queue ranking
- `env_metrics_v1`, 10 second buckets per environment, queue-independent
so it stays cheap at any range
- `queue_metrics_ck_v1`, 10 second buckets per concurrency key
Every tier is an MV from raw because the counter states do not survive a
cascade: their merge is order sensitive, so a `-MergeState` chain off
the 10s table inflates the result, and the same property means an
aggregate state may only be merged inside one queue. That constraint is
now enforced by the query engine rather than by reviewer discipline: a
column can declare a `mergeGroupKey`, and any query that references it
without grouping by, or pinning to a single value of, every named key
fails to compile with an actionable message.
On the read side, TRQL gains three tables (`queue_metrics`,
`env_metrics`, and a `queue_metrics_by_key` that is hidden from the
editor, schema docs and schema API but still queryable, so per-key rows
can never silently merge into a plain per-queue query), plus
`deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level
optimizations ride along: a table can declare coarser rollups, so a
query whose bucket interval is 5 minutes or wider is routed to the 5m
table with no change to the query itself, and it can opt into the
ClickHouse query cache with time bounds floored to a fixed grid, so the
auto-refreshing dashboards actually share cache entries instead of
missing on every tick. Both are caller-side substitutions, so the
printer stays unaware of physical layout.
All of this can also live on its own ClickHouse service. A table
declares the pool its reads run on, the three queue-metrics tables name
the dedicated one, and the ingestion consumer writes through the same
client, so both directions move together with one env var and nothing
else routes differently.
The other engine change is opt-in gap filling: charts can request rows
for empty buckets, where counters zero-fill and gauges carry forward.
Grouped gauge series are densified per group and carried inside a
partition, so a quiet queue's line holds its last value without bleeding
another queue's value into it.
## Queue concurrency limits
`concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth
when an override is set as a percentage; the absolute `concurrencyLimit`
is materialized from it (floored, clamped to at least 1 so a percentage
can never act as a pause, and never above the environment limit). Every
path that changes an environment limit now recalculates the
environment's percent-based overrides afterwards, outside the
transaction, and pushes changed limits to the engine. The push is
attempted even when the stored value did not change, so a previously
failed sync self-heals rather than leaving the database and the engine
diverged; paused queues are skipped so a recalculation cannot
effectively unpause one.
The API accepts exactly one of `concurrencyLimit` or `percent`, and the
reject-instead-of-clamp change above means a request asking for more
than the environment allows now fails loudly. The percent bound (greater
than 0, at most 100) is defined once and shared by the zod schema, the
dashboard mutation handler and the service, so the three cannot drift.
The concurrency-keys table on a queue is now paginated against the
ClickHouse per-key tier, ranked by peak backlog with the total on every
row from a single scan, and only the keys on the current page are
enriched with live counts from Redis. That replaces a hard top-50 cap
with something whose cost is a function of page size rather than key
cardinality.
## The health report
`GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The
verdict is computed on the server and is deterministic, not
model-generated. Three independent analyzers run over one input
snapshot: flow (is work moving, and if not, is the cause a limit,
throttling, one bad queue, or dead-lettering), execution (are the runs
that start succeeding, and at what latency), and liveness (how fresh is
the telemetry). When telemetry is genuinely stale, the first two are
forced to unknown and every actionable field is stripped, so no surface
ever advises action off stale data.
Authorization is per query table rather than a blanket query grant: a
JWT must be scoped to every table the report reads (`runs`,
`env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull
a report that reads more than it was granted. `period` is validated as a
shorthand with a 90 day ceiling at the edge. The report catalog is a
registry of `{ load, interpret }` entries, so the next report is a new
entry and no change to the route, the view model, the renderers, the CLI
or the MCP tool.
`trigger mcp` no longer launches the install wizard when stdout is a
TTY, which fixed a real failure: hosts spawn the server over a PTY, so
the wizard would open and the client would time out waiting for a server
that never started. The wizard now needs `trigger mcp --install`.
## The part that is live regardless of every flag
The enqueue and dequeue scripts now return a 2-tuple so a gauge reading
can ride back on the reply. Every return site in the eight affected
scripts is wrapped, and a `nil` original is converted to `false` on the
way out, because a raw `nil` in the first slot would make Lua truncate
the multi-bulk reply and silently drop the gauge on the throttled and
empty-queue paths. The reply shape and the destructuring on the app side
are exercised on every queue operation whether or not metrics are
enabled, so that is the part of `run-engine` worth the closest review.
One behavior fix in the same area: the scheduling-delay anchor is set
only on a run's first entry into the queue. Anchoring it to trigger time
on re-enqueues made waitpoint and checkpoint resumes report the entire
wait as scheduling delay. Queue ordering is untouched, so a re-enqueued
run keeps its position, and nacks deliberately keep the original anchor
because a rolled-back dequeue is the same continuous wait.
A pending-version promotion still anchors to trigger time, on purpose:
that promotion is the run's first real entry into the queue, since the
trigger deliberately held it back waiting for a worker version, and the
TTL is armed at the same point for the same reason. The consequence is
worth naming, because it is a judgement call: a run that waits on a
deployment reports that wait as scheduling delay on its queue, which is
time unrelated to queue capacity.
## Verification
Unit and integration suites across the new package, the run queue, the
mapping layer, the query engine and ClickHouse (including a test that
applies migration 036 through the same splitter CI uses, and a
regression test that inserts the same batch three times to prove the
aggregates do not inflate). Beyond that, the whole path was driven end
to end against a live stack with real runs: emitter to Redis stream to
consumer to ClickHouse to the dashboards, for both the local dev path
and the deployed path where a supervisor drives the dequeue, with
assertions on exact counter reconstruction per queue and per concurrency
key, throttling, environment saturation, scheduling delay, and a
deliberate mid-stream reading drop to confirm the cumulative counters
still reconstruct the correct total. The gated-off state was checked on
every touched surface.
The dedicated ClickHouse service was verified against a second,
separately-schema'd instance: with it configured, the driven counters
reconstruct exactly on the dedicated instance, the shared instance gains
no rows for that window, a read through the query API returns the value
that exists only on the dedicated instance, and a `runs` query still
succeeds (it would fail outright if it were mis-routed to a service
without that table). With the variable unset, the full suite passes
unchanged.
---------
Co-authored-by: Katia Bulatova <katia@trigger.dev>
Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com>
Co-authored-by: James Ritchie <james@trigger.dev>
|
||
|
claude[bot]
|
639eaf6e82 |
fix(webapp): don't apply an invite's role to an existing org member (#4409)
<!-- ccr-slack-attribution --> _Requested via [Slack thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785249693523749)_ ## Summary Accepting an old invitation could change the role of someone who was already in the organization. A long-pending invite can carry a lower role than the member has since been promoted to, so accepting it was a silent demotion. When the accepting user was the organization's only Owner, the role layer refused that demotion, and the refusal (an expected, protective outcome) was logged as an error. An invitation now only sets a role on a membership the accept actually created, and people who are already in an organization are skipped when invitations are sent. ## How `acceptInvite` already skipped the `OrgMember` create when it found an existing membership, but the `rbac.setUserRole` call below it was gated only on `invite.rbacRoleId`. It now also tracks whether this accept created the membership. A create that loses the unique-constraint race counts as pre-existing, since whichever flow won it owns that membership's role. Skipping existing members outright would regress one case: a member with no RBAC role at all would never receive the invitation's role. `ensureOrgMember` handles that with `healMissingRoleAssignment`, which fills in a null role but never overwrites a real one, so `assignInviteRbacRole` takes the same gate. An established role is never touched; an absent one is filled in. `assignInviteRbacRole` branches on the result's machine-readable `code` instead of logging every refusal at `error`. `last_owner` goes to `logger.info`, matching the two directory-sync role paths; everything else, including a refusal that carries no code, goes to `logger.warn`. The helper is best-effort and never throws, so no outcome it produces warrants `error`. No string matching on the error text is involved. `inviteMembers` resolves the organization's members by email and skips those addresses before creating invites. The invite table's `@@unique([organizationId, email])` only dedupes *pending invites*, so it could never catch this. ## Invite surfaces Skipping addresses means a batch can now come back empty, and neither caller handled that: - The dashboard action built its redirect from `invites[0].organization`, so a batch where every address was skipped threw a `TypeError` that reached the admin as a raw error string. It also reported the submitted count rather than the created one. It now names what it skipped ("No invitations sent: 1 already a member of this organization") and counts what it actually created. - The invites API derived `alreadyInvited` as "everything not created", so an existing member was reported as though they had already been invited. `inviteMembers` now returns the two groups separately and the endpoint reports `alreadyMembers` alongside `alreadyInvited`. ## Testing `apps/webapp/test/member.server.test.ts` passes 16/16 locally, up from 12. Getting there needed a harness fix. The `~/db.server` mock did not export `Prisma`, so any code reaching `PrismaNamespace.PrismaClientKnownRequestError` threw before it could branch, leaving every duplicate-key path in `member.server.ts` unreachable from tests. The mock now re-exports the real `Prisma`, and there is a case covering the pending-invite skip. New cases: the invite role is applied when the accept creates the membership; it is not applied when the member already has a role; it is applied when an existing member has no role assigned; the organization is still joined when the assignment is refused with `last_owner`; and `inviteMembers` reports members separately from pending invites. Forcing the gate off fails exactly the "already has a role" case, so the coverage is load-bearing. `pnpm run typecheck --filter webapp` and `oxfmt --check` both pass. ## Changelog Accepting an old invitation could change the role of someone who was already in the organization. An invitation now leaves an existing member's role untouched, people who are already in an organization are no longer sent invitations to it, and the invite form says which addresses it skipped instead of failing with an unhelpful error. --- ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works. ## Screenshots No visual changes. The invite form's toast copy changes, as described above. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Matt Aitken <matt@mattaitken.com> |
||
|
DKP
|
8d321f8d6e |
docs: give docs pages unique title tags and redirect stale pages (#4416)
## Summary Several docs pages rendered identical `<title>` tags, which weakens search indexing and makes results ambiguous. Each affected page now has a unique, descriptive title while keeping its existing sidebar label unchanged. Alongside the retitles: - Removed two stale build-system upgrade pages that were no longer in the navigation, with redirects to the current package upgrade guide. - Redirected the build-extensions group index to its overview page so the two URLs stop sharing a title. - Dropped a leftover orphaned API reference page (its old URL already redirects to the management overview). No links break: nothing in the docs points at the removed pages, and every redirect target exists. |
||
|
Chris Arderne
|
15e160d767 | chore(ci): cache typecheck work across runs (#4415) | ||
|
James Ritchie
|
a11e5ffbc6 |
fix(webapp): fade overflowing side menu selector labels (#4412)
Long organization, project, and environment names in the side menu were cut off mid-character. They now fade out at the right edge like the rest of the side menu items already did. ### Example of faded long names: <img width="246" height="200" alt="CleanShot 2026-07-28 at 22 59 24" src="https://github.com/user-attachments/assets/efa60b87-286f-4ab0-9d4e-490ef2de53e5" /> |
||
|
James Ritchie
|
1e14e29d71 |
fix(webapp): restyle the leave and remove team member dialogs (#4411)
## Summary
The confirmation dialog for leaving a team or removing a teammate was
still built on the old `Alert` primitive: the entire question sat in the
title, there was no header divider or `Esc` affordance, and the footer
used small buttons pinned to the right.
It now uses the standard `Dialog` layout the rest of the dashboard uses.
The title is static ("Remove team member" / "Leave team"), the question
moves into the body with the person's name and the organization
highlighted, and the footer is a bordered row with medium Cancel and
confirm buttons. A member who has not set a name is now identified by
their email instead of "them".
Verified against a local dashboard on both dialogs. Confirming a removal
posts the member id, deletes the membership and shows the success toast.
Cancel, `Esc`, and Enter while Cancel is focused all close the dialog
without issuing a request, leaving the member in place.
No release note needed: this is a visual restyle of an existing dialog
with no behaviour change.
|
||
|
Matt Aitken
|
205bdc3103 |
docs(wait): separate compute billing from concurrency release (#4405)
The wait docs describe the 5 second compute-billing threshold as if it
were also the suspension threshold. It isn't, and the gap is confusing
when you're sizing a poll interval:
- **Compute** stops being charged for any wait longer than 5 seconds.
- **Concurrency** is only released once the machine has been snapshotted
and shut down. For `wait.for` and `wait.until` that happens 60 seconds
into the wait — a shorter wait stays `EXECUTING` and holds its
concurrency slot for the whole wait, even though the compute is free.
So `await wait.for({ seconds: 30 })` in a polling loop never releases
its slot, which looks like a bug if the docs told you waits over 5
seconds checkpoint.
## Changes
**`docs/snippets/paused-execution-free.mdx`** — rendered on `/wait`,
`/wait-for` and `/wait-until`. Drops "we checkpoint and" from the
billing sentence so it's purely about compute, then adds one paragraph
for the concurrency half.
**`docs/queue-concurrency.mdx`** — the "Waits and concurrency" section
states flatly that waiting runs don't consume slots. Adds a short
subsection for the time-based exception.
**`docs/how-to-reduce-your-spend.mdx`** — "Waits longer than 5 seconds
automatically checkpoint your task, meaning you don't pay for compute" →
the compute claim only. Code comments follow, plus a pointer that
waiting doesn't always free concurrency.
**`docs/how-it-works.mdx`** — the Checkpoint-Resume walkthrough used
`wait.for({ seconds: 30 })` as *the* example of a wait that suspends.
Bumped to 5 minutes and noted the sub-60s exception.
No behaviour change — docs only.
|
||
|
Chris Arderne
|
38bf82aebe | feat(cli,webapp): target notifications by minimum CLI version (#4407) | ||
|
Saadi Myftija
|
44eca4d166 |
feat(webapp): org-gated internal API origin in run env vars (#4366)
Adds an opt-in way for operators to route deployed runs' API traffic through a different origin than the public one, per organization. Set `INTERNAL_API_ORIGIN` on the webapp and enable the `internalApiOriginEnabled` feature flag (globally or per org, with the org override winning in both directions): deployed runs for enabled orgs then get `TRIGGER_API_URL` set to the internal origin instead of `API_ORIGIN`. Useful for gradually moving run traffic onto a private network path. ## Design The origin is resolved when an attempt starts, so flag changes take effect on the next attempt and roll back the same way, with no task redeploys. The org override is read fresh per attempt; the global default comes from the cached flags registry (a cold read fails safe to the public origin). When `INTERNAL_API_ORIGIN` is unset the flag is a no-op and no extra queries run, so existing deployments are unaffected. Dev runs always use the public origin, and `TRIGGER_STREAM_URL` remains unchanged. |
||
|
claude[bot]
|
ec562c0e68 |
fix(webapp): remove unused Electric sync trace routes (#4400)
<!-- ccr-slack-attribution --> _Requested by **Eric Allam** · [Slack thread](https://triggerdotdev.slack.com/archives/C0AU83M3136/p1785222101937829?thread_ts=1785207509.304669&cid=C0AU83M3136)_ Removes two dead Remix routes and the helpers only they used. `app/routes/sync.traces.runs.$traceId.ts` (`/sync/traces/runs/:traceId`) and `app/routes/sync.traces.$traceId.ts` (`/sync/traces/:traceId`) were added with the original ElectricSQL run page and lost their only consumers when the dashboard hooks that called them were deleted. Nothing in the repo references either route today. Also removed, because the deleted routes were their only callers: - `OtelTraceIdSchema`, `RESERVED_ELECTRIC_SHAPE_PARAMS`, `TraceScope`, `buildElectricTraceWhereClause` from `app/v3/electricShape.server.ts` (the file stays — `UNSAFE_REALTIME_TAG_CHARS` / `sanitizeRealtimeTagForSql` / `sanitizeRealtimeTagsForSql` are still used by `realtime.v1.runs.ts` and `realtimeClient.server.ts`) - the loader-specific cases in `apps/webapp/test/spanTraceRoutes.replicaLag.test.ts` and `internal-packages/run-store/src/runOpsStore.routesSpanTraceReadView.replicaLag.test.ts` `app/utils/longPollingFetch.ts` is untouched — `realtimeClient.server.ts` still uses it. `runOpsStore.ts` / `PostgresRunStore.ts` are untouched too; the unrouted-lookup mechanism there is generic and stays. As a plain code fact: the run lookup these loaders performed keyed on `TaskRun.traceId` alone, which is not an index-backed query shape. That is noted only as context for why the code is not worth keeping around unused. ### Judgement call worth a maintainer's opinion The request was specifically about `/sync/traces/runs/:traceId`, the route that looks up a run by `traceId`. This PR **also** deletes its sibling `/sync/traces/:traceId`. The reasoning: - both routes came in with the same ElectricSQL run-page work - both lost their only consumers in the same later commit - neither has any caller anywhere in the repo - they share the same helper module, so keeping one means keeping the helpers half-used If you would rather keep the sibling, reverting just that one file deletion is easy and does not affect the rest of this PR — say the word and I will restore it along with the helpers it needs. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Verification run locally from the repo root: | Command | Result | | --- | --- | | `pnpm run format` | clean, no changes produced | | `pnpm run lint:fix` | clean | | `pnpm run lint` | pass (exit 0, no findings) | | `pnpm run typecheck --filter webapp` | pass | | `pnpm run typecheck --filter @internal/run-store` | pass | A ripgrep sweep for `sync.traces`, `sync/traces`, `syncTraceRunsLoader`, `buildElectricTraceWhereClause`, `OtelTraceIdSchema` and `RESERVED_ELECTRIC_SHAPE_PARAMS` (excluding `node_modules`) returns zero hits. **Not fully verified:** both edited test files are testcontainers suites and need a Docker runtime, which was not available in my environment. I confirmed each file *collects* correctly with exactly the three intended remaining tests and no import errors — notably, dropping the `session.server` / `controlPlaneResolver.server` / `longPollingFetch` / `env.server` mocks does not break module loading for the surviving loaders. The assertions themselves then failed only on `Could not find a working container runtime strategy`. CI should be the real signal here. Per `apps/webapp/CLAUDE.md`, `pnpm run build --filter webapp` was deliberately not run. --- ## Changelog Removed two unused sync routes left over from the original ElectricSQL run page, along with the helpers and tests that existed only to serve them. No behaviour change — neither route had any caller. --- ## Screenshots _n/a — no user-visible surface changes._ 💯 --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Matt Aitken
|
3ed48516df |
ci: let the claude bot trigger the PR audit workflows (#4392)
🚀 Publish Trigger.dev Docker / units (push) Failing after 1s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 3s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🧭 Helm Chart Prerelease / lint-and-test (push) Has been cancelled
Workflow Checks / Actionlint (push) Has been cancelled
Workflow Checks / Zizmor (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🧭 Helm Chart Prerelease / prerelease (push) Has been cancelled
## Summary PRs opened by the claude GitHub app fail both the agent instructions audit and the REVIEW.md drift audit before Claude gets a chance to run. `claude-code-action` refuses any actor whose account type is not `User` unless the actor is listed in `allowed_bots`: ``` Workflow initiated by non-human actor: claude (type: Bot). Add bot to allowed_bots list or use '*' to allow all bots. ``` So those PRs land with two permanently red checks and no audit coverage at all. Both workflows already allowlist Devin; this adds the claude app alongside it. ## Why this does not open the workflows up to outside contributors `allowed_bots` is only consulted for non-`User` actors. Humans, contributor or maintainer, take the separate write-permission path and are unaffected by what is in the list. Beyond that, both jobs are guarded by `github.event.pull_request.head.repo.full_name == github.repository`, so a fork PR skips the job entirely, and they trigger on `pull_request` rather than `pull_request_target`, so a fork-triggered run would get no API key and a read-only token anyway. The bot is named explicitly instead of using `"*"`, which would let every bot trigger these audits, dependabot's PR stream included. |