## Summary
The billing limits page and the usage-limit banners now require a
dedicated `manage:billing-limits` permission instead of the broader
billing permission. This lets a role be granted control over billing
limits independently of subscription and payment management.
## Details
Both the loader and action of the billing limits settings page check
`manage:billing-limits`. The "Configure billing limit" and "Resolve"
actions in the limit banners (the no-limit-configured, grace, and
rejected states) gate on the same permission. The subscription and
billing pages keep using the existing billing permission.
## Summary
When replaying runs in bulk from a deployed environment, you can now
choose which region the replayed runs run in. The bulk action inspector
shows an "Override region" dropdown that defaults to "Don't override",
which keeps each run in its original region, so replaying a selection
that spans multiple regions doesn't silently re-route anything. Pick a
region and every matched run is replayed there instead.
The dropdown only appears for the replay action in a deployed
environment with more than one region available; cancel actions and
development environments don't show it.
## Design
The selected region is carried through the bulk action as a dedicated
`replayRegion` param, kept separate from the run-list selection filters
so it can't be confused with a region selection filter. When the action
runs, each replay passes it through to the existing region override on
the replay service, which already falls back to each run's original
region when no override is set. "Don't override" is a sentinel value
that the action normalizes away so the service only ever sees a real
region or nothing.
---------
Co-authored-by: Eric Allam <eric@trigger.dev>
## Summary
Improves and unifies the run-activity charts by extracting a shared set
of chart primitives and adopting them on the three task landing pages
(agent, standard, scheduled), with the density and label fixes also
carried over to the dashboard and custom query charts.
Main changes a reviewer should know about:
- **Shared primitives (DRY).** New `ChartCard` (title +
maximize/fullscreen), `ChartSyncContext` (cross-chart hover + zoom
state), `useXAxisTicks` (width-aware tick selection),
`activityTimeAxis`, and `statusColors`, plus a server-side
`activitySeries.server.ts` holding `chooseBucketSeconds`, status
grouping, and the zero-fill helpers. The three task routes and both
presenters were refactored onto these, removing roughly 3x duplicated
tick logic, status-color tables, and bucket-ladder code.
- **Denser bars on short ranges.** Server-side bucketing now uses
`chooseBucketSeconds` (nice-interval ladder, ~72 target, capped at 120
buckets) instead of the hardcoded 1h/6h/1d ladder, so a 5m or 1h range
no longer collapses into a single bar.
- **Width-aware x-axis labels.** Labels are selected to fit the measured
plot width (always first + last, evenly spaced, de-duplicated by
rendered text), stay horizontal, and reflow on panel/window resize.
Y-axis values default to compact form (8K, 1.2M) in
`ChartBar`/`ChartLine`.
- **Synced hover line.** Hovering one of the agent page's three charts
draws a dashed vertical line at the same bucket on the *other* two, and
suppresses it on the hovered chart. It is opt-in via
`ChartSyncProvider`, so single-chart pages are unaffected.
- **Maximize button.** Each chart gets a fullscreen dialog toggle
(reuses the existing dashboard-widget pattern, `v` shortcut while
hovered).
- **Drag-to-zoom on task pages.** Dragging across a task chart sets the
Time/Date filter (`from`/`to` URL params, clearing `period`/`cursor`),
with a From/To tooltip shown during the drag.
- **Custom query charts.** Long categorical x labels (run IDs, task
names) middle-truncate and auto-rotate only when needed, and label
thinning is now width-aware for both bar and line variants. Dashboard
line-chart label density is also width-aware, tuned by a
`TIME_AXIS_LABEL_SPACING_PX` constant.
- **Tests.** 46 new unit tests for the pure logic (bucket selection,
tick spacing, time-axis formatting, zoom range, truncation).
## Intentionally unchanged
- **No click/drag zoom on the dashboard or custom query charts.**
Drag-to-zoom is wired up on the task landing pages only; zooming the
dashboard and custom charts is deliberately deferred to a separate
follow-up PR. A plain click (without a drag) on a task chart is a no-op.
- **The 25 mini activity charts on the Task list (`_index`) page are
untouched.** They are hand-rolled raw-Recharts sparklines kept
deliberately lightweight and do not use these primitives.
- **Other raw-Recharts sparklines are untouched** (the usage sparkline,
errors and prompts pages).
- **No ClickHouse query semantics changed** beyond the bucket-interval
parameter (same filters, same FINAL / `_is_deleted` handling).
- **Webapp-only.** No public package (`packages/*`) changes, so there is
no changeset; the `.server-changes/` entries cover it.
---
## Testing
Added 46 unit tests covering server-side bucket selection, width-aware
tick spacing, time-axis formatting, zoom-range math, and categorical
label truncation (`pnpm --filter webapp run test`), and `pnpm run
typecheck --filter webapp` passes. Manually exercised each task landing
page (agent, standard, scheduled) plus the dashboard and custom query
charts, stepping the Date/Time filter through 5m, 1h, 24h, 7d, and 30d
to confirm dense bars on short ranges, non-overlapping labels that
reflow on resize, the synced hover line across the agent charts, the
maximize button, and drag-to-zoom updating the filter.
---
## Changelog
The activity charts on the task landing pages and the dashboard and
custom query charts now share one set of reusable primitives. X-axis
labels are width-aware so they never overlap and reflow when a panel
resizes, y-axis values are abbreviated (8K, 1.2M), and short time ranges
render dense bars instead of collapsing into a single bar. Hovering any
agent chart mirrors a vertical line on the others, every chart gains a
maximize button, and dragging across a task chart zooms the Time/Date
filter. Long categorical labels such as run IDs and task names
middle-truncate and auto-rotate only when needed.
---
https://github.com/user-attachments/assets/6be09e38-3a0e-4947-b6e3-4839daa2fbe0
## Summary
Adds a single env flag, `DEPRECATE_V3_ENABLED` (default off), that
gracefully winds down the v3 engine (`RunEngineVersion.V1`). While it's
off nothing changes, so self-hosted instances still on v3 keep working.
When it's on:
- Triggers that resolve to v3 are rejected with a clear, actionable
error pointing at the [v4 migration
guide](https://trigger.dev/docs/migrating-from-v3), instead of silently
creating runs that never execute. This covers single triggers, batches,
scheduled fires, replays, and `triggerAndWait`, which all funnel through
one place.
- The legacy `trigger dev` websocket used by v3 CLIs is closed with an
upgrade message (v4 CLIs use a different dev transport).
- The v3 shared-queue consumer refuses to start, so no deployed v3 runs
are dequeued.
- The v3 run-lifecycle background jobs (heartbeat timeout, TTL expiry,
retry, resume batch/dependency, delayed-run enqueue, and scheduled
fires) become no-ops, so abandoned v3 runs stop generating database
load.
This builds on the existing deploy deprecation flag, which already
rejects v3 CLI deploys.
## Design
Enforcement is read through one helper, `isV3Disabled()`. Every gate
combines it with a per-run or per-project engine check (`isV3Disabled()
&& engine === "V1"`), so a v4 run that happens to reach a shared service
behaves exactly as before. v4 (V2) is never affected.
The flag is a hard switch, not a drain: when it's on, in-flight v3 runs
are abandoned in place rather than failed or expired, which is the
intended behaviour for the final shutdown.
## Summary
The `skills` installer command (`npx trigger.dev@latest skills`) ships
in the release candidate but is not yet on the stable release, so
running it with `@latest` fails today. Adds a warning on each docs page
that shows the command, telling users to run it with the `@rc` tag until
it lands in `@latest`.
The commands themselves stay on `@latest`, so nothing needs reverting
once the next stable release ships, just the warnings.
Pages updated:
- `/skills`
- `/mcp-agent-rules`
- `/building-with-ai`
## Summary
Adds Billing Limits to the webapp.
Customers can set a monthly spend cap. When usage crosses the limit,
billable environments enter a grace period. If the limit is not resolved
before grace expires, new triggers are rejected until the organization
increases or removes the limit.
The Tasks page logged a burst of React hydration errors (#421, "this
Suspense boundary received an update before it finished hydrating") on
every load, one per task row for the Running and Activity cells. The
page still worked, but it spammed the console.
The Running and Activity (24h) cells stream in via Remix `defer()` +
`<Suspense>`/`<Await>`, two boundaries per row. A streamed boundary
stays in React's "hydrating" state until its data arrives; if the
backing queries are slow enough that the data is still in flight after
the page loads, a normal background re-render (a server-sent-events
update, a panel layout effect, a revalidation) hits the boundary and
React bails it to client rendering and throws #421. With N rows that is
2N errors. It never reproduced locally because those queries return
instantly there.
Fix: wrap the two cells in `ClientOnly` so they mount after hydration.
The stats still load asynchronously (the task list renders immediately),
but there is no longer an SSR boundary to bail. In the slow-query case
those cells already client-rendered (that was the bail); this just makes
it explicit and silent.
Verified by simulating slow stat queries against a local build: the
errors go from 2-per-row to zero, and the cells render correctly once
the data resolves.
The supervisor image build has been failing since `@trigger.dev/core`
gained
an `ai` peer dependency. `turbo prune` (2.5.4) generates a pruned
lockfile
that references the `ai@6.0.116(zod@3.25.76)` snapshot without including
the
entry itself, which causes `pnpm fetch --frozen-lockfile` to abort.
Bumping to 2.10.0 fixes the pnpm v9 peer dep snapshot pruning. Updated
both
Containerfiles for consistency.
Example failure here:
https://github.com/triggerdotdev/trigger.dev/actions/runs/28225353375/job/83618124564
Broken since:
c06005b3
## Summary
Adds a dedicated, always-on Performance section to the repo PR review
guide (`.claude/REVIEW.md`), so every review weighs new work against
table size, hot paths, and how deep or wide the data it walks can get.
It names the tables to treat as huge (the `TaskRun` family in Postgres,
`task_events_v1`/`v2` in ClickHouse), the hot paths that warrant extra
scrutiny (trigger and batch trigger, dequeue, execution-snapshot writes,
OTEL ingestion, trace and run-list reads), the deep and wide shapes that
turn one run into a large tree or batch, and five named anti-patterns
with severities: per-level re-scans, dropping the partition-pruning
predicate, unbounded `IN` lists, sequential per-level round-trips, and
losing the single-query fast path.
Co-authored-by: Matt Aitken <matt@trigger.dev>
## Summary
The in-dashboard agent's datastore now runs migrations over a direct
(non-pooler) connection. A transaction-mode pooler can't run the
migrator (no advisory locks, no multi-statement DDL), so when the
agent's database sits behind a pooler the migration step needs a
separate direct connection.
The application keeps connecting over the pooled
`DASHBOARD_AGENT_DATABASE_URL`. Only the migration entry points changed
(`drizzle.config.ts`, `migrate.mjs`, `migrate-status.mjs`); the runtime
client is untouched.
## Connection resolution (migrations)
```
DASHBOARD_AGENT_DIRECT_URL direct agent connection (used for migrations)
DASHBOARD_AGENT_DATABASE_URL pooled agent connection (preserves current behavior)
DIRECT_URL main direct connection (single-database fallback)
DATABASE_URL last resort
```
Mirrors the existing `DATABASE_URL` / `DIRECT_URL` split. Fully
backward-compatible: with nothing new set, resolution is identical to
before. The agent-specific vars take precedence over the main
`DIRECT_URL`, so a separate agent database is never migrated against the
wrong one. When the agent falls back to the main single database,
migrations now prefer its direct connection.
<img width="2400" height="1794" alt="chat-ui-closed"
src="https://github.com/user-attachments/assets/35016a72-c6d2-4b6a-8760-b1da5b4a166c"
/>
<img width="2400" height="1794" alt="chat-ui-open"
src="https://github.com/user-attachments/assets/25b3df60-9aa2-40fd-a1ba-1447eeee4c52"
/>
## Summary
The in-dashboard agent was launched from a button pinned to the
bottom-right of every page, which floated over page controls (for
example the run inspector's action bar). It now opens from a compact
"Chat" button on the far right of the page header, and the same button
toggles to "Collapse" while the panel is open. The panel and launcher
are labelled "Chat" in the UI.
The launcher only renders on env-scoped pages where the agent is enabled
(same feature flag gating), so it stays hidden for everyone who doesn't
have it. The existing "Ask AI" support button is untouched; it stays in
place until the agent is turned on by default.
## How it works
`DashboardAgent` (env layout) shares the open/close state through a
small context, and `NavBar` renders a launcher that self-hides whenever
that context is absent. No floating overlay, and the launcher can't
appear on pages where the agent can't open.
## Summary
The in-dashboard agent button was rendered for all admins and
impersonators regardless of the `hasDashboardAgentAccess` flag, so it
appeared even where the agent is disabled (for example, floating over
the run inspector controls). It is now gated by the flag for everyone,
so it stays hidden until the flag is turned on.
## Rollout
Both levers default off, so nothing changes for users until deliberately
enabled:
- **Per-org:** set `hasDashboardAgentAccess` on an org's feature flags
to enable the agent for just that org.
- **All admins:** set `DASHBOARD_AGENT_ADMIN_PREVIEW=1` to give admins
and impersonators an everywhere-preview, independent of the per-org
flag.
Previously admins bypassed the flag unconditionally, which is why the
button showed up before the agent was ready to ship.
Closes this feature request:
[https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances)
### Feature notes:
- CLI `trigger dev` works as before
- `trigger dev --branch my-branch` to create a new branch and run
against it.
- `trigger dev archive --branch my-branch` to archive (or in webapp).
- New webapp page to manage and archive dev branches, currently feature
flagged.
### Implementation details:
- No changes to data model, no backfill. `isBranchableEnvironment`
column is ignored for dev branches, we use `parentEnvironmentId IS NULL`
instead.
- `x-trigger-branch` overloaded for preview and dev branches
- New `TRIGGER_DEV_BRANCH` env var available locally.
`TRIGGER_PREVIEW_BRANCH` overloaded for child runs.
- Lots of new glue code to sanitise the branch checks.
### Rollout
- Deploy webapp/API changes (all backwards compatible)
- Manual tests on some orgs
- Deploy docs, release CLI, flip feature flag for webapp feature
### NB
- `api.v1.projects.$projectRef.environments.ts` will return
`isBranchableEnvironment: true` for all dev environments.
### Prerequisites
- [x] Typecheck will not pass until we make a new release of
`@trigger.dev/platform` and bump it here
A deployment could be marked deployed and promoted to current without
its image ever landing in the registry. Finalize trusted the CLI: the v1
path never pushed or checked, and the v2/v3 path skips its own push when
the CLI sends `skipPushToRegistry` - which the local-build path always
does. In the happy path the CLI pushes the image itself, so this stayed
latent. But any deviation - `--no-push`/`--load`, a push that lands in a
different registry, or an old CLI - promoted a version whose image can't
be pulled, so every run failed at pull time while the deploy itself
reported success.
This adds a registry existence check after push and before finalize. If
the image isn't there, the deploy fails loudly instead of promoting a
version that can't start. The check is ECR-only (a no-op for other
registries, so self-hosted setups are unaffected) and uses
`BatchGetImage`, which the deploy role already allows. It fails open on
an ambiguous registry error so the check can't itself turn into a deploy
outage. The image reference is the platform-generated value and the
lookup is bound to the configured registry host; the CLI-supplied digest
is validated before use.
Can be turned off with `DEPLOY_IMAGE_VERIFICATION_ENABLED=0` for setups
that push images out of band (e.g. an air-gapped registry the platform
can't reach).
refs TRI-11243
SSO settings page: resolve plan before the role check. A non-Enterprise
org now renders the upsell state for every role instead of showing a
"permission denied" panel to non-Owners for a feature their org can't
use yet. manage:sso is only enforced once the org is actually entitled.
Extracts EMPTY_SSO_STATUS and uses throwPermissionDenied().
Also removes the client-side SSO session fetch guard. It monkeypatched
global window.fetch, which made it the initiator of every request and
obfuscated the real call site on any 4xx/5xx. Session revocation is
still enforced server-side on every authenticated request and surfaces
as a logout redirect on the next navigation/refresh, so the client guard
was UX-only and not worth the cross-cutting cost.
The wait.for() idempotency example passed the idempotency options as a
second argument, but wait.for() takes a single options object — so the
second object was silently ignored and the key never applied. Merged the
fields into the single options object, matching the wait.until()
example.
Supersedes #4021.
Prefixes the dashboard feedback form thread titles with `Web app:` so
support inbox threads coming from the in-app contact form are easy to
tell apart from those submitted on the marketing site, which previously
shared an identical `Contact form:` prefix.
## Summary
The dashboard agent's database migrations could be silently skipped when
its database is shared with another Drizzle application, leaving the
`trigger_dashboard_agent` schema uncreated and a later migration failing
with `schema "trigger_dashboard_agent" does not exist`.
## Root cause
Drizzle's migrator decides what to run by reading the most recent row
from its journal table by `created_at`, and skipping any migration dated
at or before it. The dashboard-agent runner used Drizzle's default
journal table, `drizzle.__drizzle_migrations`, which every Drizzle app
shares by default. When the database is shared, another app's journal
row dated between two of our migrations makes the migrator treat the
earlier one (the `CREATE SCHEMA`) as already applied and run a later one
against a schema that was never created.
## Fix
- Track the dashboard-agent migrations in a dedicated journal table
(`drizzle.__dashboard_agent_migrations`), in both the deploy runner
(`migrate.mjs`) and the drizzle-kit config, so its history is
independent of any other Drizzle app sharing the database. The table
stays in the `drizzle` schema so the first migration's `CREATE SCHEMA
"trigger_dashboard_agent"` does not collide with it.
- Make the first two migrations idempotent (`CREATE SCHEMA/TABLE/INDEX
IF NOT EXISTS`) so databases that already tracked them under the old
journal table re-run cleanly after the rename instead of erroring on the
bare `CREATE SCHEMA`.
Verified against a Postgres seeded to reproduce the skip: the old
default-table path fails as above, the dedicated-table path creates the
schema and all tables, and re-running on an already-migrated database is
a clean no-op.
## Summary
The auto-generated changeset release PR (`changeset-release/main`)
builds its `## Improvements` / `## Bug fixes` summary with
`scripts/enhance-release-pr.mjs`. The script deduplicated summary
entries by PR number, so when a single PR shipped more than one
changeset, only the first entry survived and the rest were silently
dropped from the summary. The dropped entries still appeared in the raw
`<details>` block, which is how the mismatch surfaced (for example in
[#3998](https://github.com/triggerdotdev/trigger.dev/pull/3998), where
one PR's four changesets showed up as a single summary line).
## Fix
Deduplicate on the full entry text rather than the bare PR number. The
entry text embeds the PR link, so:
- the same changeset rendered once per package section still collapses
to one,
- distinct changesets from the same PR are each kept,
- identical descriptions from different PRs stay separate.
Verified against the raw changeset output from
[#3998](https://github.com/triggerdotdev/trigger.dev/pull/3998): that
PR's changesets went from 1 to all 4 in the generated summary.
## Summary
Adds a `RUN_ENGINE_DEQUEUE_DISABLED_WORKER_QUEUES` setting that refuses
worker dequeue requests for the listed worker queues (or base regions),
so their runs stay queued instead of being handed to workers that can't
run them. Blocked dequeues are counted via a
`run_engine.dequeue.blocked` OTel counter (labeled by `worker_queue` and
`region`).
## Summary
The `runqueue.workerQueue.length` gauge only reported a worker queue's
depth while runs were being dequeued from it. When dequeues stop, the
metric goes stale or missing, so a queue that has backed up because
nothing is draining it can't be alerted on. This adds a small observer
that refreshes the observed set of worker queues from the
`WorkerInstanceGroup` records on an interval, so every active worker
queue (and its scheduled split variant) keeps reporting its length
regardless of dequeue activity.
The observer is off by default and enabled per service via
`RUN_ENGINE_WORKER_QUEUE_OBSERVER_ENABLED`, reads from the read replica,
and skips a configurable set of cloud providers
(`RUN_ENGINE_WORKER_QUEUE_OBSERVER_EXCLUDED_CLOUD_PROVIDERS`, default
`digitalocean`). When enabled it is the source of truth for the observed
set, so the per-dequeue registration is skipped on that instance, and it
groups by worker queue so the per-instance duplicates collapse to the
true depth.
Also removes the unused `GET`/`POST /api/v1/workers` endpoints. Their
only consumer was a CLI command group that is no longer registered.
## Verification
Verified end to end against a local stack: the gauge reports each worker
queue's length with no dequeues happening, excludes the configured
providers, includes hidden groups, and the removed endpoints return as
if they never existed. Added a run-engine test
(`workerQueueObservation.test.ts`).
## Summary
Adds an in-dashboard AI agent: a chat panel, reachable from any
environment
page, that answers questions about your runs, errors, tasks, and
analytics,
diagnoses why a run failed, charts your data, reads your connected
repo's
source, and answers product and how-to questions. It is gated behind the
`hasDashboardAgentAccess` feature flag (global or per-org, default off),
so
this PR ships disabled: the launcher is hidden unless the flag is
enabled.
## Design
The agent runs as a standalone `chat.agent` Trigger task in its own
internal
package, with no access to the webapp database, Prisma, or ClickHouse.
It reads
the user's data over the public API, acting as the user via a
short-lived
delegated user-actor token minted server-side each turn (never in the
browser),
building on
[#3997](https://github.com/triggerdotdev/trigger.dev/pull/3997). The
error and analytics tools use
[#4005](https://github.com/triggerdotdev/trigger.dev/pull/4005)
and the TRQL query API.
The first turn of a new chat streams from a warm webapp route (Head
Start) while
the durable agent boots in parallel. Structured answers (a run-failure
diagnosis
card, a live chart) render through a small typed view catalog rather
than
arbitrary markup. A knowledge lane forwards product and how-to questions
to the
support assistant.
Conversation history lives in a separate Drizzle-backed store on its own
Postgres schema, kept as a display read-model so it can never corrupt
the
agent's model context.
The SDK changes add an `apiClient` option to
`chat.createStartSessionAction` and
`chat.headStart`, and keep the Head Start tool-approval tail intact
across a
custom `prepareMessages` hook so prompt caching and Head Start compose.
Adds an in-process backpressure signal that pauses dequeuing when the
Kubernetes cluster is saturated, so work overflows cheaply in the queue
instead of piling up as unschedulable pods. Saturation is read by
scraping the apiserver's total pod-object count
(`apiserver_storage_objects{resource="pods"}`) and applying an
engage/release threshold with hysteresis - a single lightweight
aggregate scrape, not a pod listing.
Backpressure sources are now evaluated independently and OR'd: each
source has its own enable and dry-run flag, and the supervisor engages
if any enabled source trips. This adds the pod-count source alongside
the existing one without changing it, and is extensible to more sources
later. Off by default.
The scrape uses the in-cluster kubeconfig over `https` so TLS verifies
against the cluster CA (the fetch-options helper attaches the CA as an
`https.Agent`, which the global `fetch` ignores - that path silently
dropped the CA). Enabling the pod-count source requires the supervisor's
service account to be granted `get` on the `/metrics` non-resource URL;
that RBAC and the per-deployment env wiring are operator-side and live
elsewhere.
New config (pod-count source):
`TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_ENABLED` (default false),
`_POD_COUNT_DRY_RUN` (default true), `_POD_COUNT_ENGAGE` /
`_POD_COUNT_RELEASE` (hysteresis thresholds), `_POD_COUNT_REFRESH_MS`
(scrape interval, default 5s). The existing source's flags are
unchanged.
Observability: a `supervisor_cluster_pod_count` gauge, and the pod-count
monitor's metrics are namespaced (`supervisor_backpressure_pod_count_*`)
so the existing backpressure metrics keep their names.
## Summary
Fixes the fair-queue weighted environment shuffle, which biased
environment ordering whenever fair-queue biases are enabled (the default
configuration).
## Root cause
`#weightedShuffle` in `fairQueueSelectionStrategy.ts` computed the total
weight once and drew its random pivot against that full-set total on
every iteration, but never decremented the total as items were removed
from the working set. After the first pick, the pivot frequently
overshot the sum of the remaining items, so the inner selection loop ran
off the end and clamped to the last remaining element. The result
systematically over-selected whichever environment sat at the tail of
the set.
The first slot stayed fair (the full total is correct on the first
draw), but later positions were ordered by environment iteration order
rather than by the intended concurrency-limit and available-capacity
weighting. For four equal-weight environments, the final position landed
on one env ~9% of the time and another ~42%, instead of ~25% each.
The two sibling selection paths (`#weightedRandomQueueOrder` and
`#selectTopEnvs`) already decrement the total before splicing; this
brings the env shuffle in line with them.
## Fix
```ts
result.push(items[index].envId);
totalWeight -= items[index].weight;
items.splice(index, 1);
```
Adds a regression test that runs the weighted shuffle over equal-weight
envs with biases enabled and asserts each env lands in every position
roughly uniformly. It fails on the old code (tail position ~37%) and
passes with the fix.
Reported in #4001.
Deploy success/failure wasn't easily observable: compute template
creation only logged, and the terminal deployment statuses (deployed /
failed / timed-out) weren't traced — so deploy health couldn't be seen
without querying the database.
This adds two spans:
- `compute.template.create` around template creation at finalize,
tagging the resolved mode and per-preset outcome. `resolveMode` now
returns its decision (mode + reason) so the span can record why a mode
was chosen.
- `deployment.outcome` via a small shared helper
(`recordDeploymentOutcome`) emitted at every terminal-status write —
finalize (deployed), fail / index-failed / background-worker (failed),
and timeout (timed out) — so deploy success/failure is queryable by
status and reason.
The helper is best-effort (org/project/env enrichment where cheaply
available) and never throws, so telemetry can't break a deploy.
## Summary
Refreshes the SVG artwork for the main task icon and the cached task
variant shown on the run trace span view.
The cached icon (previously a hardcoded blue "T" in a dashed border) now
lives alongside `TaskIcon` in `TaskIcon.tsx` and is drawn with
`currentColor`, so it inherits the `text-tasks` theme color like the
other span icons instead of ignoring it. The standalone
`TaskCachedIcon.tsx` file is removed and its two import sites updated.
## Summary
Sessions started from the agent Test playground were tagged with a
`"playground"` tag that rendered in the Sessions table's Tags column.
They are now flagged with a real `Session.isTest` boolean (mirroring
`TaskRun.isTest`) and surfaced as a dedicated **Test** column with a
check icon, to the left of Tags, on both the Sessions page and the Agent
landing page, plus a matching **Test** property on the session detail
page. This mirrors how Standard and Scheduled task runs already indicate
test runs.
## Design
`isTest` is a new `Session` column (Postgres) replicated into ClickHouse
`sessions_v1` alongside the existing fields. The Sessions list reads
`isTest` from Postgres for display (ClickHouse only supplies the ordered
session IDs), so the column renders correctly without a ClickHouse
backfill.
The playground action now sets `isTest: true` on session create instead
of writing the `"playground"` tag. The triggered run still carries
`playground:true` in its own tags (unchanged). A migration backfills
existing sessions, setting `isTest = true` and stripping the
now-redundant `"playground"` tag where it is present, so the list and
detail views render consistently without read-time tag filtering.
## Summary
Updates the task icons used across the dashboard. `TaskIcon` and its
small variant now use a new burst glyph, and `TasksIcon` adopts the
previous task glyph (the rounded square). Both still render with
`currentColor`, so they inherit text color exactly as before.
Export names are unchanged, so every existing usage (side menu, task and
queue views, run filters) picks up the new artwork with no other code
changes.
## Summary
The logs search page (behind a feature flag) ran ClickHouse out of
memory when browsing back over long time ranges. This keeps it within
bounded memory and fixes a pagination bug that could skip or duplicate
rows at a page boundary.
## Fix
Memory: the list query reads in sort-key order, which opens one read
stream per part in the window, and on object storage those per-part read
buffers dominate peak memory, so it scaled with the number of parts
scanned. Two changes bound it:
- The logs ClickHouse client caps the per-part read buffers via new
env-tunable settings. The object-storage-only setting is opt-in, so it
is never sent to a ClickHouse version that lacks it.
- Recent-first window narrowing: rows come back newest first, so the
presenter probes the most recent window and only widens toward the full
requested range when a page is short. A busy environment fills a page
from a few recent parts instead of scanning the whole range; a quiet one
still returns every row in a couple of cheap reads.
Correctness: the keyset cursor ordered on (triggered_timestamp,
trace_id), which is not unique because the spans of a trace share both,
so rows at a tie could be skipped or duplicated across pages. The cursor
and ORDER BY now include span_id, and the cursor is versioned so stale
cursors reset to the first page.
Guards: the effective page size is capped, and the existing per-query
memory limit lets a pathological wide browse fail with an error instead
of taking the node down.
## ClickHouse 26.2
The memory fix relies on lazy materialization deferring the wide
attributes column to the output rows, which only holds on 26.x. Cloud
already runs 26.2, so this moves the dev stack, testcontainers, and CI
to match. The ClickHouse test suite passes on 26.2.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`,
`findRuns`) and routes every Postgres read of `TaskRun` through them,
mirroring how writes already go through the store. Behavior-preserving:
each relocated read keeps its exact query, field selection, and database
client (writer, replica, or transaction). This lets `TaskRun` reads be
retargeted to a different backing store later without touching call
sites.
Stacked on #3981 (the write adapter); that PR is the base of this one.
## Scope
In scope: the run engine, webapp services, presenters, and route
loaders. Three reads that pulled `TaskRun` in through a parent model's
relation `include` (alert delivery, batch results, attempt-dependency
cancellation) are decomposed to fetch the run(s) through the store and
stitch them back, since a relation include would not follow `TaskRun` to
a new table.
Left reading the existing table (out of scope): the legacy MarQS paths,
the legacy trigger idempotency read, and one raw-SQL recovery script
(commented for revisiting at cutover).
## Notes
Reads default to the read replica; callers pass the writer or a
transaction client wherever the original read did, so writer-vs-replica
behavior is unchanged.
Follow-up to #3992, which gated the send runner-side - but only for new
runner images. Existing runners still POST a debug log per line.
When `SEND_RUN_DEBUG_LOGS` is off (default), the route now drops the
request immediately: `skipBodyParsing` skips the body read/parse, a bare
handler returns 204, no wide event. The route stays registered so it
avoids the `No route match` error log; the only per-request log left is
the framework's `logger.debug` trace, suppressed at the default `info`
level. Still counted by request metrics, and 204 is non-retryable so no
retry storm.
Adds a `skipBodyParsing` flag to the internal HTTP server.
Runners were POSTing a debug log to the supervisor for every log line -
one request per line, unbatched and unconditional. The supervisor
already has a `SEND_RUN_DEBUG_LOGS` toggle (off by default) that
discards them on receipt, but the runner fired the request regardless,
so the traffic hit the supervisor either way.
This gates the send at the source. The runner now reads
`TRIGGER_SEND_RUN_DEBUG_LOGS` (off by default, injected by the
supervisor from its existing `SEND_RUN_DEBUG_LOGS` setting) and skips
the POST entirely when disabled. Local log output is unchanged. Dev runs
use a separate path and are unaffected.
## Summary
Adds an environment-scoped HTTP API over the Errors feature, mirroring
the runs API. Task-run failures are grouped by a fingerprint into "error
groups," and this exposes everything you can do with them in the
dashboard:
- `GET /api/v1/errors` lists error groups, with
`filter[taskIdentifier]`, `filter[version]`, `filter[status]`
(`unresolved`/`resolved`/`ignored`), `filter[search]`, a time range, and
cursor pagination.
- `GET /api/v1/errors/{errorId}` retrieves a single group (summary,
lifecycle state, affected versions).
- `POST /api/v1/errors/{errorId}/{resolve,ignore,unresolve}` changes its
state.
- `GET /api/v1/runs?filter[error]={errorId}` lists the runs behind a
group.
Request and response schemas are exported from `@trigger.dev/core/v3` so
the SDK can reuse them, and all endpoints are documented in the API
reference (OpenAPI). `errorId` is the `error_<fingerprint>` friendly id.
## Attribution
State changes record who made them. A plain environment API key has no
user, so `resolvedBy`/`ignoredByUserId` stay null. When the caller uses
an environment JWT obtained by exchanging a personal access token or a
delegated user token at `POST /api/v1/projects/:ref/:env/jwt`, that
exchange now stamps an `act` delegation claim, and the write endpoints
read `act.sub` to attribute the change to the acting user. This is the
first endpoint to consume the `act` claim, so two small pieces of
plumbing ride along: the exchange stamps `act` for personal-access-token
subjects too (it was delegated-token-only), and the public-JWT
bearer-auth path surfaces `act.sub` to the handler.
Built on the delegated-token work in #3997.
## Summary
The CLAUDE.md audit job (`.github/workflows/claude-md-audit.yml`)
frequently hits its 15-turn cap before it finishes reviewing a PR, so
the job fails without posting a verdict. For example, the audit job
failed on [this
run](https://github.com/triggerdotdev/trigger.dev/actions/runs/27837408945/job/82390460772?pr=3990).
This raises `--max-turns` from 15 to 25 to give the review room to
complete, and pins `--model claude-opus-4-8` (the job previously
inherited the action default model).
## Summary
Adds a short-lived, delegated token (`tr_uat_...`) that authenticates
against the API as a user without handing out a long-lived personal
access token. You mint one from a PAT, optionally narrow it to a set of
scopes, and give it a lifetime; the API then treats requests as that
user, subject to their role.
`trigger.dev mint-token` is the entry point (it uses your stored PAT):
```bash
UAT=$(trigger.dev mint-token --ttl 3600 --cap read:runs)
```
The token works anywhere a PAT does for user-level endpoints, and can be
exchanged for an environment JWT at `POST
/api/v1/projects/:ref/:env/jwt` to reach environment-scoped data (the
same exchange a PAT supports).
## How it works
A user-actor token is a short-lived JWT verified by a new first-class
`authenticateUserActor` method on the RBAC plugin. Self-hosters get a
built-in fallback; role-aware enforcement comes from the plugin.
Effective permissions are the intersection of the user's role and the
token's optional scope cap, so a token is only ever narrower than the
user, never broader.
Minting is restricted to personal access tokens (a token can't mint
another one, and an environment key can't mint one). Tokens default to a
1 hour lifetime (max 365 days). When exchanged for an environment JWT,
the user is stamped on it for attribution and the scope cap is carried
through.
## Summary
Replaces the multi-select popover task type filter on the Tasks page
with a single-select segmented control: **All** plus icon-only
**Agent**, **Standard**, and **Scheduled** segments. Each segment has a
tooltip showing its label and a number-key shortcut (0-3), and the
search field no longer autofocuses so the shortcuts work on page load.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
## Summary
The Personal Access Tokens page now shows each token's maximum role in a
new column, so you can see at a glance what a token is capped to. The
column only appears when an RBAC plugin is installed, and shows "-" for
tokens with no cap. Its header tooltip reuses the same explanation shown
in the create-token panel.
## Summary
Adds the `4.5.0-rc.7` entry to the AI chat changelog, covering the
agent-facing changes in
[v4.5.0-rc.7](https://github.com/triggerdotdev/trigger.dev/releases/tag/v4.5.0-rc.7):
- `chat.headStart` now works with the `chat.customAgent` and
`chat.createSession` backends, not just `chat.agent`
- Opt-in Anthropic system-prompt caching via
`chat.toStreamTextOptions()`
- Three custom-agent-loop fixes: continuation replay, mid-stream
steering, and task-backed tools
- `trigger skills` follow-ups: `trigger-` namespacing, SDK-bundled docs,
and a new cost-savings skill
Generic, non-agent rc.7 items (the CLI uninitialized-project error
message, run-span cost fields) are intentionally left out to keep this
changelog scoped to AI chat agents.
Chris Arderne