Bumps the `@remix-run/*` family in the webapp from 2.17.4 to 2.17.5 to
keep dependencies current.
2.17.5 pulls `@remix-run/router` 1.23.2 → 1.23.3, so the local
route-matching perf patch was rebased onto 1.23.3 (regenerated via `pnpm
patch`). It is functionally identical to the previous one - the only
difference is that 1.23.3 already hoists `decodePath` out of the match
loop upstream, so that hunk is dropped; the per-route-tree branch cache
and the compiled-path cache are unchanged. Also updated the
`@remix-run/dev>tar-fs` override key to track the new dev version.
Verified locally against latest main: `typecheck --filter webapp`
passes, `--frozen-lockfile` is consistent, and the dev server boots and
server-renders pages cleanly (route matching exercised via the patched
router).
## Summary
Upgrades the dashboard form layer from `@conform-to` 0.9 to 1.x. No
behaviour change is intended; this is the conform API migration only.
conform 1.x peer-depends on `zod` `^3.21 || ^4`, so it runs on the
current zod 3 and is a prerequisite for upgrading the repo to zod 4:
conform 0.9 imports `ZodNativeEnum`, `ZodEffects`, and `ZodPipeline`,
all removed in zod 4, so the webapp cannot build against zod 4 until
conform is on 1.x. Landing this first (on zod 3) keeps the zod 4 PR
focused on zod alone.
## Testing
Did a bunch of local smoke tests and E2E playwright tests, several
rounds of different reviewers, all clean.
Invite acceptance could fail for cloud organizations with many projects
because the whole flow ran inside a single transaction and did too much
work before it completed. In larger orgs, that pushed the transaction
past its timeout and blocked the invite from being accepted.
This PR moves the expensive parts of invite acceptance out of the
transaction, excludes deleted projects from environment setup, fixes
error handling on /invites, and adds regression coverage for the failure
cases.
Fixed trace rendering for child and nested runs in large traces.
Dashboard and trace API responses now load the requested run's trace
subtree instead of depending on the run span appearing in the initial
trace slice.
## Summary
Two robustness fixes in the dashboard's error handling, found while
testing the billing-limit pause/resume flow.
## Toast cookie overflow
Toast messages are flashed into the `__message` session cookie, which
the session store rejects once the serialized cookie passes the
browser's ~4KB limit. Any call site that flashes a raw caught error (a
verbose database or validation message, for example) could turn a toast
into a failed request. `setErrorMessage` / `setSuccessMessage` now clamp
the message length, so a toast can never overflow the cookie. This
protects every toast helper at once.
## Pause/resume reporting
Resuming an environment that is paused by a billing limit is an
expected, user-actionable state, but `PauseEnvironmentService` threw it,
and the service's catch reports every throw at error level. It now
returns that case as a failure result, so callers still surface the
message to the user while genuine errors keep reporting.
## Summary
When a scheduled task fires for an organization that is out of
entitlements, the trigger can't proceed. That's an expected outcome, but
it was being logged at error level and surfaced as a failure.
## Fix
The trigger callback now classifies an out-of-entitlements result as its
own error type (`OUT_OF_ENTITLEMENTS`), and the schedule engine logs
both that and the existing queue-limit result as warnings rather than
errors. The run still doesn't fire and the `schedule_execution_failure`
metric still records the outcome (now tagged `out_of_entitlements`), so
nothing about observability or behavior changes beyond the log level.
## Summary
The billing limits page and the usage-limit banners now require a
dedicated `manage:billing-limits` permission instead of the broader
billing permission. This lets a role be granted control over billing
limits independently of subscription and payment management.
## Details
Both the loader and action of the billing limits settings page check
`manage:billing-limits`. The "Configure billing limit" and "Resolve"
actions in the limit banners (the no-limit-configured, grace, and
rejected states) gate on the same permission. The subscription and
billing pages keep using the existing billing permission.
## Summary
When replaying runs in bulk from a deployed environment, you can now
choose which region the replayed runs run in. The bulk action inspector
shows an "Override region" dropdown that defaults to "Don't override",
which keeps each run in its original region, so replaying a selection
that spans multiple regions doesn't silently re-route anything. Pick a
region and every matched run is replayed there instead.
The dropdown only appears for the replay action in a deployed
environment with more than one region available; cancel actions and
development environments don't show it.
## Design
The selected region is carried through the bulk action as a dedicated
`replayRegion` param, kept separate from the run-list selection filters
so it can't be confused with a region selection filter. When the action
runs, each replay passes it through to the existing region override on
the replay service, which already falls back to each run's original
region when no override is set. "Don't override" is a sentinel value
that the action normalizes away so the service only ever sees a real
region or nothing.
---------
Co-authored-by: Eric Allam <eric@trigger.dev>
## Summary
Improves and unifies the run-activity charts by extracting a shared set
of chart primitives and adopting them on the three task landing pages
(agent, standard, scheduled), with the density and label fixes also
carried over to the dashboard and custom query charts.
Main changes a reviewer should know about:
- **Shared primitives (DRY).** New `ChartCard` (title +
maximize/fullscreen), `ChartSyncContext` (cross-chart hover + zoom
state), `useXAxisTicks` (width-aware tick selection),
`activityTimeAxis`, and `statusColors`, plus a server-side
`activitySeries.server.ts` holding `chooseBucketSeconds`, status
grouping, and the zero-fill helpers. The three task routes and both
presenters were refactored onto these, removing roughly 3x duplicated
tick logic, status-color tables, and bucket-ladder code.
- **Denser bars on short ranges.** Server-side bucketing now uses
`chooseBucketSeconds` (nice-interval ladder, ~72 target, capped at 120
buckets) instead of the hardcoded 1h/6h/1d ladder, so a 5m or 1h range
no longer collapses into a single bar.
- **Width-aware x-axis labels.** Labels are selected to fit the measured
plot width (always first + last, evenly spaced, de-duplicated by
rendered text), stay horizontal, and reflow on panel/window resize.
Y-axis values default to compact form (8K, 1.2M) in
`ChartBar`/`ChartLine`.
- **Synced hover line.** Hovering one of the agent page's three charts
draws a dashed vertical line at the same bucket on the *other* two, and
suppresses it on the hovered chart. It is opt-in via
`ChartSyncProvider`, so single-chart pages are unaffected.
- **Maximize button.** Each chart gets a fullscreen dialog toggle
(reuses the existing dashboard-widget pattern, `v` shortcut while
hovered).
- **Drag-to-zoom on task pages.** Dragging across a task chart sets the
Time/Date filter (`from`/`to` URL params, clearing `period`/`cursor`),
with a From/To tooltip shown during the drag.
- **Custom query charts.** Long categorical x labels (run IDs, task
names) middle-truncate and auto-rotate only when needed, and label
thinning is now width-aware for both bar and line variants. Dashboard
line-chart label density is also width-aware, tuned by a
`TIME_AXIS_LABEL_SPACING_PX` constant.
- **Tests.** 46 new unit tests for the pure logic (bucket selection,
tick spacing, time-axis formatting, zoom range, truncation).
## Intentionally unchanged
- **No click/drag zoom on the dashboard or custom query charts.**
Drag-to-zoom is wired up on the task landing pages only; zooming the
dashboard and custom charts is deliberately deferred to a separate
follow-up PR. A plain click (without a drag) on a task chart is a no-op.
- **The 25 mini activity charts on the Task list (`_index`) page are
untouched.** They are hand-rolled raw-Recharts sparklines kept
deliberately lightweight and do not use these primitives.
- **Other raw-Recharts sparklines are untouched** (the usage sparkline,
errors and prompts pages).
- **No ClickHouse query semantics changed** beyond the bucket-interval
parameter (same filters, same FINAL / `_is_deleted` handling).
- **Webapp-only.** No public package (`packages/*`) changes, so there is
no changeset; the `.server-changes/` entries cover it.
---
## Testing
Added 46 unit tests covering server-side bucket selection, width-aware
tick spacing, time-axis formatting, zoom-range math, and categorical
label truncation (`pnpm --filter webapp run test`), and `pnpm run
typecheck --filter webapp` passes. Manually exercised each task landing
page (agent, standard, scheduled) plus the dashboard and custom query
charts, stepping the Date/Time filter through 5m, 1h, 24h, 7d, and 30d
to confirm dense bars on short ranges, non-overlapping labels that
reflow on resize, the synced hover line across the agent charts, the
maximize button, and drag-to-zoom updating the filter.
---
## Changelog
The activity charts on the task landing pages and the dashboard and
custom query charts now share one set of reusable primitives. X-axis
labels are width-aware so they never overlap and reflow when a panel
resizes, y-axis values are abbreviated (8K, 1.2M), and short time ranges
render dense bars instead of collapsing into a single bar. Hovering any
agent chart mirrors a vertical line on the others, every chart gains a
maximize button, and dragging across a task chart zooms the Time/Date
filter. Long categorical labels such as run IDs and task names
middle-truncate and auto-rotate only when needed.
---
https://github.com/user-attachments/assets/6be09e38-3a0e-4947-b6e3-4839daa2fbe0
## Summary
Adds a single env flag, `DEPRECATE_V3_ENABLED` (default off), that
gracefully winds down the v3 engine (`RunEngineVersion.V1`). While it's
off nothing changes, so self-hosted instances still on v3 keep working.
When it's on:
- Triggers that resolve to v3 are rejected with a clear, actionable
error pointing at the [v4 migration
guide](https://trigger.dev/docs/migrating-from-v3), instead of silently
creating runs that never execute. This covers single triggers, batches,
scheduled fires, replays, and `triggerAndWait`, which all funnel through
one place.
- The legacy `trigger dev` websocket used by v3 CLIs is closed with an
upgrade message (v4 CLIs use a different dev transport).
- The v3 shared-queue consumer refuses to start, so no deployed v3 runs
are dequeued.
- The v3 run-lifecycle background jobs (heartbeat timeout, TTL expiry,
retry, resume batch/dependency, delayed-run enqueue, and scheduled
fires) become no-ops, so abandoned v3 runs stop generating database
load.
This builds on the existing deploy deprecation flag, which already
rejects v3 CLI deploys.
## Design
Enforcement is read through one helper, `isV3Disabled()`. Every gate
combines it with a per-run or per-project engine check (`isV3Disabled()
&& engine === "V1"`), so a v4 run that happens to reach a shared service
behaves exactly as before. v4 (V2) is never affected.
The flag is a hard switch, not a drain: when it's on, in-flight v3 runs
are abandoned in place rather than failed or expired, which is the
intended behaviour for the final shutdown.
## Summary
Adds Billing Limits to the webapp.
Customers can set a monthly spend cap. When usage crosses the limit,
billable environments enter a grace period. If the limit is not resolved
before grace expires, new triggers are rejected until the organization
increases or removes the limit.
The Tasks page logged a burst of React hydration errors (#421, "this
Suspense boundary received an update before it finished hydrating") on
every load, one per task row for the Running and Activity cells. The
page still worked, but it spammed the console.
The Running and Activity (24h) cells stream in via Remix `defer()` +
`<Suspense>`/`<Await>`, two boundaries per row. A streamed boundary
stays in React's "hydrating" state until its data arrives; if the
backing queries are slow enough that the data is still in flight after
the page loads, a normal background re-render (a server-sent-events
update, a panel layout effect, a revalidation) hits the boundary and
React bails it to client rendering and throws #421. With N rows that is
2N errors. It never reproduced locally because those queries return
instantly there.
Fix: wrap the two cells in `ClientOnly` so they mount after hydration.
The stats still load asynchronously (the task list renders immediately),
but there is no longer an SSR boundary to bail. In the slow-query case
those cells already client-rendered (that was the bail); this just makes
it explicit and silent.
Verified by simulating slow stat queries against a local build: the
errors go from 2-per-row to zero, and the cells render correctly once
the data resolves.
The supervisor image build has been failing since `@trigger.dev/core`
gained
an `ai` peer dependency. `turbo prune` (2.5.4) generates a pruned
lockfile
that references the `ai@6.0.116(zod@3.25.76)` snapshot without including
the
entry itself, which causes `pnpm fetch --frozen-lockfile` to abort.
Bumping to 2.10.0 fixes the pnpm v9 peer dep snapshot pruning. Updated
both
Containerfiles for consistency.
Example failure here:
https://github.com/triggerdotdev/trigger.dev/actions/runs/28225353375/job/83618124564
Broken since:
c06005b3
<img width="2400" height="1794" alt="chat-ui-closed"
src="https://github.com/user-attachments/assets/35016a72-c6d2-4b6a-8760-b1da5b4a166c"
/>
<img width="2400" height="1794" alt="chat-ui-open"
src="https://github.com/user-attachments/assets/25b3df60-9aa2-40fd-a1ba-1447eeee4c52"
/>
## Summary
The in-dashboard agent was launched from a button pinned to the
bottom-right of every page, which floated over page controls (for
example the run inspector's action bar). It now opens from a compact
"Chat" button on the far right of the page header, and the same button
toggles to "Collapse" while the panel is open. The panel and launcher
are labelled "Chat" in the UI.
The launcher only renders on env-scoped pages where the agent is enabled
(same feature flag gating), so it stays hidden for everyone who doesn't
have it. The existing "Ask AI" support button is untouched; it stays in
place until the agent is turned on by default.
## How it works
`DashboardAgent` (env layout) shares the open/close state through a
small context, and `NavBar` renders a launcher that self-hides whenever
that context is absent. No floating overlay, and the launcher can't
appear on pages where the agent can't open.
## Summary
The in-dashboard agent button was rendered for all admins and
impersonators regardless of the `hasDashboardAgentAccess` flag, so it
appeared even where the agent is disabled (for example, floating over
the run inspector controls). It is now gated by the flag for everyone,
so it stays hidden until the flag is turned on.
## Rollout
Both levers default off, so nothing changes for users until deliberately
enabled:
- **Per-org:** set `hasDashboardAgentAccess` on an org's feature flags
to enable the agent for just that org.
- **All admins:** set `DASHBOARD_AGENT_ADMIN_PREVIEW=1` to give admins
and impersonators an everywhere-preview, independent of the per-org
flag.
Previously admins bypassed the flag unconditionally, which is why the
button showed up before the agent was ready to ship.
Closes this feature request:
[https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances)
### Feature notes:
- CLI `trigger dev` works as before
- `trigger dev --branch my-branch` to create a new branch and run
against it.
- `trigger dev archive --branch my-branch` to archive (or in webapp).
- New webapp page to manage and archive dev branches, currently feature
flagged.
### Implementation details:
- No changes to data model, no backfill. `isBranchableEnvironment`
column is ignored for dev branches, we use `parentEnvironmentId IS NULL`
instead.
- `x-trigger-branch` overloaded for preview and dev branches
- New `TRIGGER_DEV_BRANCH` env var available locally.
`TRIGGER_PREVIEW_BRANCH` overloaded for child runs.
- Lots of new glue code to sanitise the branch checks.
### Rollout
- Deploy webapp/API changes (all backwards compatible)
- Manual tests on some orgs
- Deploy docs, release CLI, flip feature flag for webapp feature
### NB
- `api.v1.projects.$projectRef.environments.ts` will return
`isBranchableEnvironment: true` for all dev environments.
### Prerequisites
- [x] Typecheck will not pass until we make a new release of
`@trigger.dev/platform` and bump it here
A deployment could be marked deployed and promoted to current without
its image ever landing in the registry. Finalize trusted the CLI: the v1
path never pushed or checked, and the v2/v3 path skips its own push when
the CLI sends `skipPushToRegistry` - which the local-build path always
does. In the happy path the CLI pushes the image itself, so this stayed
latent. But any deviation - `--no-push`/`--load`, a push that lands in a
different registry, or an old CLI - promoted a version whose image can't
be pulled, so every run failed at pull time while the deploy itself
reported success.
This adds a registry existence check after push and before finalize. If
the image isn't there, the deploy fails loudly instead of promoting a
version that can't start. The check is ECR-only (a no-op for other
registries, so self-hosted setups are unaffected) and uses
`BatchGetImage`, which the deploy role already allows. It fails open on
an ambiguous registry error so the check can't itself turn into a deploy
outage. The image reference is the platform-generated value and the
lookup is bound to the configured registry host; the CLI-supplied digest
is validated before use.
Can be turned off with `DEPLOY_IMAGE_VERIFICATION_ENABLED=0` for setups
that push images out of band (e.g. an air-gapped registry the platform
can't reach).
refs TRI-11243
SSO settings page: resolve plan before the role check. A non-Enterprise
org now renders the upsell state for every role instead of showing a
"permission denied" panel to non-Owners for a feature their org can't
use yet. manage:sso is only enforced once the org is actually entitled.
Extracts EMPTY_SSO_STATUS and uses throwPermissionDenied().
Also removes the client-side SSO session fetch guard. It monkeypatched
global window.fetch, which made it the initiator of every request and
obfuscated the real call site on any 4xx/5xx. Session revocation is
still enforced server-side on every authenticated request and surfaces
as a logout redirect on the next navigation/refresh, so the client guard
was UX-only and not worth the cross-cutting cost.
Prefixes the dashboard feedback form thread titles with `Web app:` so
support inbox threads coming from the in-app contact form are easy to
tell apart from those submitted on the marketing site, which previously
shared an identical `Contact form:` prefix.
## Summary
Adds a `RUN_ENGINE_DEQUEUE_DISABLED_WORKER_QUEUES` setting that refuses
worker dequeue requests for the listed worker queues (or base regions),
so their runs stay queued instead of being handed to workers that can't
run them. Blocked dequeues are counted via a
`run_engine.dequeue.blocked` OTel counter (labeled by `worker_queue` and
`region`).
## Summary
The `runqueue.workerQueue.length` gauge only reported a worker queue's
depth while runs were being dequeued from it. When dequeues stop, the
metric goes stale or missing, so a queue that has backed up because
nothing is draining it can't be alerted on. This adds a small observer
that refreshes the observed set of worker queues from the
`WorkerInstanceGroup` records on an interval, so every active worker
queue (and its scheduled split variant) keeps reporting its length
regardless of dequeue activity.
The observer is off by default and enabled per service via
`RUN_ENGINE_WORKER_QUEUE_OBSERVER_ENABLED`, reads from the read replica,
and skips a configurable set of cloud providers
(`RUN_ENGINE_WORKER_QUEUE_OBSERVER_EXCLUDED_CLOUD_PROVIDERS`, default
`digitalocean`). When enabled it is the source of truth for the observed
set, so the per-dequeue registration is skipped on that instance, and it
groups by worker queue so the per-instance duplicates collapse to the
true depth.
Also removes the unused `GET`/`POST /api/v1/workers` endpoints. Their
only consumer was a CLI command group that is no longer registered.
## Verification
Verified end to end against a local stack: the gauge reports each worker
queue's length with no dequeues happening, excludes the configured
providers, includes hidden groups, and the removed endpoints return as
if they never existed. Added a run-engine test
(`workerQueueObservation.test.ts`).
## Summary
Adds an in-dashboard AI agent: a chat panel, reachable from any
environment
page, that answers questions about your runs, errors, tasks, and
analytics,
diagnoses why a run failed, charts your data, reads your connected
repo's
source, and answers product and how-to questions. It is gated behind the
`hasDashboardAgentAccess` feature flag (global or per-org, default off),
so
this PR ships disabled: the launcher is hidden unless the flag is
enabled.
## Design
The agent runs as a standalone `chat.agent` Trigger task in its own
internal
package, with no access to the webapp database, Prisma, or ClickHouse.
It reads
the user's data over the public API, acting as the user via a
short-lived
delegated user-actor token minted server-side each turn (never in the
browser),
building on
[#3997](https://github.com/triggerdotdev/trigger.dev/pull/3997). The
error and analytics tools use
[#4005](https://github.com/triggerdotdev/trigger.dev/pull/4005)
and the TRQL query API.
The first turn of a new chat streams from a warm webapp route (Head
Start) while
the durable agent boots in parallel. Structured answers (a run-failure
diagnosis
card, a live chart) render through a small typed view catalog rather
than
arbitrary markup. A knowledge lane forwards product and how-to questions
to the
support assistant.
Conversation history lives in a separate Drizzle-backed store on its own
Postgres schema, kept as a display read-model so it can never corrupt
the
agent's model context.
The SDK changes add an `apiClient` option to
`chat.createStartSessionAction` and
`chat.headStart`, and keep the Head Start tool-approval tail intact
across a
custom `prepareMessages` hook so prompt caching and Head Start compose.
Adds an in-process backpressure signal that pauses dequeuing when the
Kubernetes cluster is saturated, so work overflows cheaply in the queue
instead of piling up as unschedulable pods. Saturation is read by
scraping the apiserver's total pod-object count
(`apiserver_storage_objects{resource="pods"}`) and applying an
engage/release threshold with hysteresis - a single lightweight
aggregate scrape, not a pod listing.
Backpressure sources are now evaluated independently and OR'd: each
source has its own enable and dry-run flag, and the supervisor engages
if any enabled source trips. This adds the pod-count source alongside
the existing one without changing it, and is extensible to more sources
later. Off by default.
The scrape uses the in-cluster kubeconfig over `https` so TLS verifies
against the cluster CA (the fetch-options helper attaches the CA as an
`https.Agent`, which the global `fetch` ignores - that path silently
dropped the CA). Enabling the pod-count source requires the supervisor's
service account to be granted `get` on the `/metrics` non-resource URL;
that RBAC and the per-deployment env wiring are operator-side and live
elsewhere.
New config (pod-count source):
`TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_ENABLED` (default false),
`_POD_COUNT_DRY_RUN` (default true), `_POD_COUNT_ENGAGE` /
`_POD_COUNT_RELEASE` (hysteresis thresholds), `_POD_COUNT_REFRESH_MS`
(scrape interval, default 5s). The existing source's flags are
unchanged.
Observability: a `supervisor_cluster_pod_count` gauge, and the pod-count
monitor's metrics are namespaced (`supervisor_backpressure_pod_count_*`)
so the existing backpressure metrics keep their names.
Deploy success/failure wasn't easily observable: compute template
creation only logged, and the terminal deployment statuses (deployed /
failed / timed-out) weren't traced — so deploy health couldn't be seen
without querying the database.
This adds two spans:
- `compute.template.create` around template creation at finalize,
tagging the resolved mode and per-preset outcome. `resolveMode` now
returns its decision (mode + reason) so the span can record why a mode
was chosen.
- `deployment.outcome` via a small shared helper
(`recordDeploymentOutcome`) emitted at every terminal-status write —
finalize (deployed), fail / index-failed / background-worker (failed),
and timeout (timed out) — so deploy success/failure is queryable by
status and reason.
The helper is best-effort (org/project/env enrichment where cheaply
available) and never throws, so telemetry can't break a deploy.
## Summary
Refreshes the SVG artwork for the main task icon and the cached task
variant shown on the run trace span view.
The cached icon (previously a hardcoded blue "T" in a dashed border) now
lives alongside `TaskIcon` in `TaskIcon.tsx` and is drawn with
`currentColor`, so it inherits the `text-tasks` theme color like the
other span icons instead of ignoring it. The standalone
`TaskCachedIcon.tsx` file is removed and its two import sites updated.
## Summary
Sessions started from the agent Test playground were tagged with a
`"playground"` tag that rendered in the Sessions table's Tags column.
They are now flagged with a real `Session.isTest` boolean (mirroring
`TaskRun.isTest`) and surfaced as a dedicated **Test** column with a
check icon, to the left of Tags, on both the Sessions page and the Agent
landing page, plus a matching **Test** property on the session detail
page. This mirrors how Standard and Scheduled task runs already indicate
test runs.
## Design
`isTest` is a new `Session` column (Postgres) replicated into ClickHouse
`sessions_v1` alongside the existing fields. The Sessions list reads
`isTest` from Postgres for display (ClickHouse only supplies the ordered
session IDs), so the column renders correctly without a ClickHouse
backfill.
The playground action now sets `isTest: true` on session create instead
of writing the `"playground"` tag. The triggered run still carries
`playground:true` in its own tags (unchanged). A migration backfills
existing sessions, setting `isTest = true` and stripping the
now-redundant `"playground"` tag where it is present, so the list and
detail views render consistently without read-time tag filtering.
## Summary
Updates the task icons used across the dashboard. `TaskIcon` and its
small variant now use a new burst glyph, and `TasksIcon` adopts the
previous task glyph (the rounded square). Both still render with
`currentColor`, so they inherit text color exactly as before.
Export names are unchanged, so every existing usage (side menu, task and
queue views, run filters) picks up the new artwork with no other code
changes.
## Summary
The logs search page (behind a feature flag) ran ClickHouse out of
memory when browsing back over long time ranges. This keeps it within
bounded memory and fixes a pagination bug that could skip or duplicate
rows at a page boundary.
## Fix
Memory: the list query reads in sort-key order, which opens one read
stream per part in the window, and on object storage those per-part read
buffers dominate peak memory, so it scaled with the number of parts
scanned. Two changes bound it:
- The logs ClickHouse client caps the per-part read buffers via new
env-tunable settings. The object-storage-only setting is opt-in, so it
is never sent to a ClickHouse version that lacks it.
- Recent-first window narrowing: rows come back newest first, so the
presenter probes the most recent window and only widens toward the full
requested range when a page is short. A busy environment fills a page
from a few recent parts instead of scanning the whole range; a quiet one
still returns every row in a couple of cheap reads.
Correctness: the keyset cursor ordered on (triggered_timestamp,
trace_id), which is not unique because the spans of a trace share both,
so rows at a tie could be skipped or duplicated across pages. The cursor
and ORDER BY now include span_id, and the cursor is versioned so stale
cursors reset to the first page.
Guards: the effective page size is capped, and the existing per-query
memory limit lets a pathological wide browse fail with an error instead
of taking the node down.
## ClickHouse 26.2
The memory fix relies on lazy materialization deferring the wide
attributes column to the output rows, which only holds on 26.x. Cloud
already runs 26.2, so this moves the dev stack, testcontainers, and CI
to match. The ClickHouse test suite passes on 26.2.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`,
`findRuns`) and routes every Postgres read of `TaskRun` through them,
mirroring how writes already go through the store. Behavior-preserving:
each relocated read keeps its exact query, field selection, and database
client (writer, replica, or transaction). This lets `TaskRun` reads be
retargeted to a different backing store later without touching call
sites.
Stacked on #3981 (the write adapter); that PR is the base of this one.
## Scope
In scope: the run engine, webapp services, presenters, and route
loaders. Three reads that pulled `TaskRun` in through a parent model's
relation `include` (alert delivery, batch results, attempt-dependency
cancellation) are decomposed to fetch the run(s) through the store and
stitch them back, since a relation include would not follow `TaskRun` to
a new table.
Left reading the existing table (out of scope): the legacy MarQS paths,
the legacy trigger idempotency read, and one raw-SQL recovery script
(commented for revisiting at cutover).
## Notes
Reads default to the read replica; callers pass the writer or a
transaction client wherever the original read did, so writer-vs-replica
behavior is unchanged.
Follow-up to #3992, which gated the send runner-side - but only for new
runner images. Existing runners still POST a debug log per line.
When `SEND_RUN_DEBUG_LOGS` is off (default), the route now drops the
request immediately: `skipBodyParsing` skips the body read/parse, a bare
handler returns 204, no wide event. The route stays registered so it
avoids the `No route match` error log; the only per-request log left is
the framework's `logger.debug` trace, suppressed at the default `info`
level. Still counted by request metrics, and 204 is non-retryable so no
retry storm.
Adds a `skipBodyParsing` flag to the internal HTTP server.
Runners were POSTing a debug log to the supervisor for every log line -
one request per line, unbatched and unconditional. The supervisor
already has a `SEND_RUN_DEBUG_LOGS` toggle (off by default) that
discards them on receipt, but the runner fired the request regardless,
so the traffic hit the supervisor either way.
This gates the send at the source. The runner now reads
`TRIGGER_SEND_RUN_DEBUG_LOGS` (off by default, injected by the
supervisor from its existing `SEND_RUN_DEBUG_LOGS` setting) and skips
the POST entirely when disabled. Local log output is unchanged. Dev runs
use a separate path and are unaffected.
## Summary
Adds an environment-scoped HTTP API over the Errors feature, mirroring
the runs API. Task-run failures are grouped by a fingerprint into "error
groups," and this exposes everything you can do with them in the
dashboard:
- `GET /api/v1/errors` lists error groups, with
`filter[taskIdentifier]`, `filter[version]`, `filter[status]`
(`unresolved`/`resolved`/`ignored`), `filter[search]`, a time range, and
cursor pagination.
- `GET /api/v1/errors/{errorId}` retrieves a single group (summary,
lifecycle state, affected versions).
- `POST /api/v1/errors/{errorId}/{resolve,ignore,unresolve}` changes its
state.
- `GET /api/v1/runs?filter[error]={errorId}` lists the runs behind a
group.
Request and response schemas are exported from `@trigger.dev/core/v3` so
the SDK can reuse them, and all endpoints are documented in the API
reference (OpenAPI). `errorId` is the `error_<fingerprint>` friendly id.
## Attribution
State changes record who made them. A plain environment API key has no
user, so `resolvedBy`/`ignoredByUserId` stay null. When the caller uses
an environment JWT obtained by exchanging a personal access token or a
delegated user token at `POST /api/v1/projects/:ref/:env/jwt`, that
exchange now stamps an `act` delegation claim, and the write endpoints
read `act.sub` to attribute the change to the acting user. This is the
first endpoint to consume the `act` claim, so two small pieces of
plumbing ride along: the exchange stamps `act` for personal-access-token
subjects too (it was delegated-token-only), and the public-JWT
bearer-auth path surfaces `act.sub` to the handler.
Built on the delegated-token work in #3997.
## Summary
Adds a short-lived, delegated token (`tr_uat_...`) that authenticates
against the API as a user without handing out a long-lived personal
access token. You mint one from a PAT, optionally narrow it to a set of
scopes, and give it a lifetime; the API then treats requests as that
user, subject to their role.
`trigger.dev mint-token` is the entry point (it uses your stored PAT):
```bash
UAT=$(trigger.dev mint-token --ttl 3600 --cap read:runs)
```
The token works anywhere a PAT does for user-level endpoints, and can be
exchanged for an environment JWT at `POST
/api/v1/projects/:ref/:env/jwt` to reach environment-scoped data (the
same exchange a PAT supports).
## How it works
A user-actor token is a short-lived JWT verified by a new first-class
`authenticateUserActor` method on the RBAC plugin. Self-hosters get a
built-in fallback; role-aware enforcement comes from the plugin.
Effective permissions are the intersection of the user's role and the
token's optional scope cap, so a token is only ever narrower than the
user, never broader.
Minting is restricted to personal access tokens (a token can't mint
another one, and an environment key can't mint one). Tokens default to a
1 hour lifetime (max 365 days). When exchanged for an environment JWT,
the user is stamped on it for attribution and the scope cap is carried
through.
## Summary
Replaces the multi-select popover task type filter on the Tasks page
with a single-select segmented control: **All** plus icon-only
**Agent**, **Standard**, and **Scheduled** segments. Each segment has a
tooltip showing its label and a number-key shortcut (0-3), and the
search field no longer autofocuses so the shortcuts work on page load.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
## Summary
The Personal Access Tokens page now shows each token's maximum role in a
new column, so you can see at a glance what a token is capped to. The
column only appears when an RBAC plugin is installed, and shows "-" for
tokens with no cap. Its header tooltip reuses the same explanation shown
in the create-token panel.
Pushes new organizations and users into the Attio CRM at signup time,
for Customer Success (TRI-10431).
- Orgs → Attio `workspaces`, users → Attio `users`, keyed on Attio's
built-in unique `workspace_id` / `user_id` so writes are idempotent
upserts.
- Runs on the common Redis worker (not inline), so a slow or unavailable
Attio never blocks the signup path; failures retry (3 attempts).
- Hooks: user-created (alongside the existing Loops call) and
org-created (`createOrganization`).
- Gated behind `ATTIO_API_KEY`, no key means the sync is skipped
entirely, so OSS / self-hosted installs are unaffected.
Only creation is covered here (the record "shell"); spend, runs, plan
changes, churn, and role/relationship linking are populated by the
scheduled full sync, tracked separately.
**Deploy note:** requires an Attio API key set as `ATTIO_API_KEY` in the
webapp env, with scopes **Records (read-write)** + **Object
Configuration (read)**, the assert/upsert endpoint reads object config
to resolve the matching attribute. Without the key the sync no-ops.
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Ran the webapp locally with the change applied; it compiles and serves.
The edit only swaps the chart card title string from "LLM spend" to "LLM
spend ($)" on the agent landing page.
---
## Changelog
The agent dashboard "LLM spend" chart label now includes the currency
unit, reading "LLM spend ($)".
---
## Screenshots
_[Screenshots]_
💯
## Summary
Several dashboard routes performed actions a restricted role should not
be able to do (cancel or replay runs, manage prompt versions, invite and
manage members, manage billing) without any permission check. This adds
role-based permission enforcement to those routes, and disables the
matching UI controls (with a tooltip) when the current role lacks
permission.
Covered actions:
- Runs: cancel and replay (single, bulk create, bulk abort)
- Prompts: create or edit override versions, and promote a version to
current
- Members: invite, resend invite, revoke invite
- Billing: change plan, billing alerts, and the customer portal
## How
Each affected route now goes through the `dashboardLoader` /
`dashboardAction` route builders with an `authorization` block declaring
the required permission (or a per-intent check where one route handles
several intents). Existing tenancy and data-scoping queries are
untouched; this only layers permission checks on top. The UI follows
disable-don't-hide: controls stay visible but disabled with a "You don't
have permission to ..." tooltip.
Two reusable pieces support this: `checkPermissions(ability, checks)`
turns a set of checks into a boolean map a loader returns to the client,
and `PermissionButton` / `PermissionLink` disable the underlying control
and show a tooltip when a permission flag is false.
## Behaviour
No change in the default configuration: permissions are permissive, so
every control stays enabled and every route behaves as before. The
checks only take effect when an RBAC plugin is installed. This also
makes role assignment on invite-accept non-fatal, so a failure there
cannot block joining an org.
Verified with `pnpm run typecheck --filter webapp`; `checkPermissions`
has unit tests.
The global feature flags admin page had a few rough edges.
The percentage flags are numeric (`z.coerce.number()`) but rendered as
free-text inputs, so you could type non-numeric values that only failed
validation after submitting - and the error surfaced behind the confirm
dialog. The control-type detection now recognises numbers and renders a
proper number input, with the min/max range as the placeholder so the
type is clear even when the field is unset. The save error also shows
inside the confirm dialog now, not just behind it.
The action buttons were unreachable without zooming out. The admin
layout wrapped each page in a plain block, so `h-full` page content
overran the viewport by the height of the tab bar and got clipped by the
`overflow-hidden` body. Making the layout a flex column bounds each page
to the space below the tabs, so the existing per-page scroll works and
the feature flags page scrolls like the Users/Orgs tabs. Also capped the
confirm dialog's diff list so its footer stays on screen when there are
many changes.
## Summary
Prisma infrastructure failures (P1xxx-class: database unreachable, timed
out, connection dropped, engine init/panic) carry the database hostname
in their `.message`. This captures them centrally for observability and
ensures they never reach API clients verbatim.
## Design
A `$allOperations` client extension on the writer and replica clients
logs infrastructure errors with the originating model and operation,
then rethrows the **original** error unchanged — call sites that branch
on `error.code` (unique-violation idempotency, not-found handling) and
transaction retries keep working. Only infrastructure errors are logged;
routine query/validation errors (P2xxx) are left alone.
`$allOperations` can't see the transaction boundary (`$transaction` is a
client method, not an operation), so infrastructure errors surfacing
from `$transaction()` without a Prisma code — e.g.
`PrismaClientInitializationError` — are logged separately at the
transaction wrapper, where the existing coded-error path would otherwise
miss them.
`clientSafeErrorMessage()` swaps an infrastructure error's message for
`"Internal Server Error"` at the API routes that previously returned
`error.message` raw. Status codes, headers, and every non-infrastructure
message are unchanged.
## Test plan
- [x] P2002 / P2025 rethrow with code intact and are not logged
- [x] Statement errors inside `$transaction` keep their code (retry
logic intact)
- [x] Raw queries wrapped without crashing on the undefined model
- [x] A genuine connectivity failure is logged with model/operation/code
- [x] `clientSafeErrorMessage` obfuscates infra messages, preserves all
others
- [x] `pnpm run typecheck --filter webapp` (12/12)
## Note
Overlaps with #3391 (Prisma 7 migration) on
`apps/webapp/app/db.server.ts` — coordinate rebasing.
Replicates `TaskRun.planType` into the `task_runs_v2` ClickHouse table
so run analytics can group by plan type.
Adds a `plan_type` column (goose migration `033`,
`LowCardinality(String)`), the replication insert mapping, and the
matching schema/column/type entries - same shape as the recent `region`
addition. Write-once at trigger, so it just rides along on existing
replicated rows. Internal analytics only; not exposed in the Query API.
nicktrn