The `Exec` helper in `@trigger.dev/core` logs command args at debug
level (and in its output/error metadata). For commands that take a
credential directly on the command line - `--password`, `--token`,
`--secret`, etc. - that value is logged verbatim, so turning on debug
logging can surface secrets in log sinks.
This masks the value of known credential-bearing flags (both `--flag
value` and `--flag=value` forms) before the args are logged. The
executed command is untouched - only the logged copy is redacted. Added
a small unit test for the redaction helper.
Adds a `redis_worker.queue.oldest_message_age` observable gauge (labeled
`worker_name`) and `SimpleQueue.oldestMessageAge()`, reporting the age
of the
oldest overdue message in each queue. Generic queue-stall signal: 0
while a
queue drains healthily, rising only when due work sits undrained
(blocked
dequeue, dead consumer, backpressure) — even when no items are being
processed.
## Summary
Upgrades the dashboard form layer from `@conform-to` 0.9 to 1.x. No
behaviour change is intended; this is the conform API migration only.
conform 1.x peer-depends on `zod` `^3.21 || ^4`, so it runs on the
current zod 3 and is a prerequisite for upgrading the repo to zod 4:
conform 0.9 imports `ZodNativeEnum`, `ZodEffects`, and `ZodPipeline`,
all removed in zod 4, so the webapp cannot build against zod 4 until
conform is on 1.x. Landing this first (on zod 3) keeps the zod 4 PR
focused on zod alone.
## Testing
Did a bunch of local smoke tests and E2E playwright tests, several
rounds of different reviewers, all clean.
Invite acceptance could fail for cloud organizations with many projects
because the whole flow ran inside a single transaction and did too much
work before it completed. In larger orgs, that pushed the transaction
past its timeout and blocked the invite from being accepted.
This PR moves the expensive parts of invite acceptance out of the
transaction, excludes deleted projects from environment setup, fixes
error handling on /invites, and adds regression coverage for the failure
cases.
Fixed trace rendering for child and nested runs in large traces.
Dashboard and trace API responses now load the requested run's trace
subtree instead of depending on the run span appearing in the initial
trace slice.
## Summary
Two robustness fixes in the dashboard's error handling, found while
testing the billing-limit pause/resume flow.
## Toast cookie overflow
Toast messages are flashed into the `__message` session cookie, which
the session store rejects once the serialized cookie passes the
browser's ~4KB limit. Any call site that flashes a raw caught error (a
verbose database or validation message, for example) could turn a toast
into a failed request. `setErrorMessage` / `setSuccessMessage` now clamp
the message length, so a toast can never overflow the cookie. This
protects every toast helper at once.
## Pause/resume reporting
Resuming an environment that is paused by a billing limit is an
expected, user-actionable state, but `PauseEnvironmentService` threw it,
and the service's catch reports every throw at error level. It now
returns that case as a failure result, so callers still surface the
message to the user while genuine errors keep reporting.
## Summary
When a scheduled task fires for an organization that is out of
entitlements, the trigger can't proceed. That's an expected outcome, but
it was being logged at error level and surfaced as a failure.
## Fix
The trigger callback now classifies an out-of-entitlements result as its
own error type (`OUT_OF_ENTITLEMENTS`), and the schedule engine logs
both that and the existing queue-limit result as warnings rather than
errors. The run still doesn't fire and the `schedule_execution_failure`
metric still records the outcome (now tagged `out_of_entitlements`), so
nothing about observability or behavior changes beyond the log level.
## Summary
The billing limits page and the usage-limit banners now require a
dedicated `manage:billing-limits` permission instead of the broader
billing permission. This lets a role be granted control over billing
limits independently of subscription and payment management.
## Details
Both the loader and action of the billing limits settings page check
`manage:billing-limits`. The "Configure billing limit" and "Resolve"
actions in the limit banners (the no-limit-configured, grace, and
rejected states) gate on the same permission. The subscription and
billing pages keep using the existing billing permission.
## Summary
When replaying runs in bulk from a deployed environment, you can now
choose which region the replayed runs run in. The bulk action inspector
shows an "Override region" dropdown that defaults to "Don't override",
which keeps each run in its original region, so replaying a selection
that spans multiple regions doesn't silently re-route anything. Pick a
region and every matched run is replayed there instead.
The dropdown only appears for the replay action in a deployed
environment with more than one region available; cancel actions and
development environments don't show it.
## Design
The selected region is carried through the bulk action as a dedicated
`replayRegion` param, kept separate from the run-list selection filters
so it can't be confused with a region selection filter. When the action
runs, each replay passes it through to the existing region override on
the replay service, which already falls back to each run's original
region when no override is set. "Don't override" is a sentinel value
that the action normalizes away so the service only ever sees a real
region or nothing.
---------
Co-authored-by: Eric Allam <eric@trigger.dev>
## Summary
Improves and unifies the run-activity charts by extracting a shared set
of chart primitives and adopting them on the three task landing pages
(agent, standard, scheduled), with the density and label fixes also
carried over to the dashboard and custom query charts.
Main changes a reviewer should know about:
- **Shared primitives (DRY).** New `ChartCard` (title +
maximize/fullscreen), `ChartSyncContext` (cross-chart hover + zoom
state), `useXAxisTicks` (width-aware tick selection),
`activityTimeAxis`, and `statusColors`, plus a server-side
`activitySeries.server.ts` holding `chooseBucketSeconds`, status
grouping, and the zero-fill helpers. The three task routes and both
presenters were refactored onto these, removing roughly 3x duplicated
tick logic, status-color tables, and bucket-ladder code.
- **Denser bars on short ranges.** Server-side bucketing now uses
`chooseBucketSeconds` (nice-interval ladder, ~72 target, capped at 120
buckets) instead of the hardcoded 1h/6h/1d ladder, so a 5m or 1h range
no longer collapses into a single bar.
- **Width-aware x-axis labels.** Labels are selected to fit the measured
plot width (always first + last, evenly spaced, de-duplicated by
rendered text), stay horizontal, and reflow on panel/window resize.
Y-axis values default to compact form (8K, 1.2M) in
`ChartBar`/`ChartLine`.
- **Synced hover line.** Hovering one of the agent page's three charts
draws a dashed vertical line at the same bucket on the *other* two, and
suppresses it on the hovered chart. It is opt-in via
`ChartSyncProvider`, so single-chart pages are unaffected.
- **Maximize button.** Each chart gets a fullscreen dialog toggle
(reuses the existing dashboard-widget pattern, `v` shortcut while
hovered).
- **Drag-to-zoom on task pages.** Dragging across a task chart sets the
Time/Date filter (`from`/`to` URL params, clearing `period`/`cursor`),
with a From/To tooltip shown during the drag.
- **Custom query charts.** Long categorical x labels (run IDs, task
names) middle-truncate and auto-rotate only when needed, and label
thinning is now width-aware for both bar and line variants. Dashboard
line-chart label density is also width-aware, tuned by a
`TIME_AXIS_LABEL_SPACING_PX` constant.
- **Tests.** 46 new unit tests for the pure logic (bucket selection,
tick spacing, time-axis formatting, zoom range, truncation).
## Intentionally unchanged
- **No click/drag zoom on the dashboard or custom query charts.**
Drag-to-zoom is wired up on the task landing pages only; zooming the
dashboard and custom charts is deliberately deferred to a separate
follow-up PR. A plain click (without a drag) on a task chart is a no-op.
- **The 25 mini activity charts on the Task list (`_index`) page are
untouched.** They are hand-rolled raw-Recharts sparklines kept
deliberately lightweight and do not use these primitives.
- **Other raw-Recharts sparklines are untouched** (the usage sparkline,
errors and prompts pages).
- **No ClickHouse query semantics changed** beyond the bucket-interval
parameter (same filters, same FINAL / `_is_deleted` handling).
- **Webapp-only.** No public package (`packages/*`) changes, so there is
no changeset; the `.server-changes/` entries cover it.
---
## Testing
Added 46 unit tests covering server-side bucket selection, width-aware
tick spacing, time-axis formatting, zoom-range math, and categorical
label truncation (`pnpm --filter webapp run test`), and `pnpm run
typecheck --filter webapp` passes. Manually exercised each task landing
page (agent, standard, scheduled) plus the dashboard and custom query
charts, stepping the Date/Time filter through 5m, 1h, 24h, 7d, and 30d
to confirm dense bars on short ranges, non-overlapping labels that
reflow on resize, the synced hover line across the agent charts, the
maximize button, and drag-to-zoom updating the filter.
---
## Changelog
The activity charts on the task landing pages and the dashboard and
custom query charts now share one set of reusable primitives. X-axis
labels are width-aware so they never overlap and reflow when a panel
resizes, y-axis values are abbreviated (8K, 1.2M), and short time ranges
render dense bars instead of collapsing into a single bar. Hovering any
agent chart mirrors a vertical line on the others, every chart gains a
maximize button, and dragging across a task chart zooms the Time/Date
filter. Long categorical labels such as run IDs and task names
middle-truncate and auto-rotate only when needed.
---
https://github.com/user-attachments/assets/6be09e38-3a0e-4947-b6e3-4839daa2fbe0
## Summary
Adds a single env flag, `DEPRECATE_V3_ENABLED` (default off), that
gracefully winds down the v3 engine (`RunEngineVersion.V1`). While it's
off nothing changes, so self-hosted instances still on v3 keep working.
When it's on:
- Triggers that resolve to v3 are rejected with a clear, actionable
error pointing at the [v4 migration
guide](https://trigger.dev/docs/migrating-from-v3), instead of silently
creating runs that never execute. This covers single triggers, batches,
scheduled fires, replays, and `triggerAndWait`, which all funnel through
one place.
- The legacy `trigger dev` websocket used by v3 CLIs is closed with an
upgrade message (v4 CLIs use a different dev transport).
- The v3 shared-queue consumer refuses to start, so no deployed v3 runs
are dequeued.
- The v3 run-lifecycle background jobs (heartbeat timeout, TTL expiry,
retry, resume batch/dependency, delayed-run enqueue, and scheduled
fires) become no-ops, so abandoned v3 runs stop generating database
load.
This builds on the existing deploy deprecation flag, which already
rejects v3 CLI deploys.
## Design
Enforcement is read through one helper, `isV3Disabled()`. Every gate
combines it with a per-run or per-project engine check (`isV3Disabled()
&& engine === "V1"`), so a v4 run that happens to reach a shared service
behaves exactly as before. v4 (V2) is never affected.
The flag is a hard switch, not a drain: when it's on, in-flight v3 runs
are abandoned in place rather than failed or expired, which is the
intended behaviour for the final shutdown.
## Summary
The `skills` installer command (`npx trigger.dev@latest skills`) ships
in the release candidate but is not yet on the stable release, so
running it with `@latest` fails today. Adds a warning on each docs page
that shows the command, telling users to run it with the `@rc` tag until
it lands in `@latest`.
The commands themselves stay on `@latest`, so nothing needs reverting
once the next stable release ships, just the warnings.
Pages updated:
- `/skills`
- `/mcp-agent-rules`
- `/building-with-ai`
## Summary
Adds Billing Limits to the webapp.
Customers can set a monthly spend cap. When usage crosses the limit,
billable environments enter a grace period. If the limit is not resolved
before grace expires, new triggers are rejected until the organization
increases or removes the limit.
The Tasks page logged a burst of React hydration errors (#421, "this
Suspense boundary received an update before it finished hydrating") on
every load, one per task row for the Running and Activity cells. The
page still worked, but it spammed the console.
The Running and Activity (24h) cells stream in via Remix `defer()` +
`<Suspense>`/`<Await>`, two boundaries per row. A streamed boundary
stays in React's "hydrating" state until its data arrives; if the
backing queries are slow enough that the data is still in flight after
the page loads, a normal background re-render (a server-sent-events
update, a panel layout effect, a revalidation) hits the boundary and
React bails it to client rendering and throws #421. With N rows that is
2N errors. It never reproduced locally because those queries return
instantly there.
Fix: wrap the two cells in `ClientOnly` so they mount after hydration.
The stats still load asynchronously (the task list renders immediately),
but there is no longer an SSR boundary to bail. In the slow-query case
those cells already client-rendered (that was the bail); this just makes
it explicit and silent.
Verified by simulating slow stat queries against a local build: the
errors go from 2-per-row to zero, and the cells render correctly once
the data resolves.
The supervisor image build has been failing since `@trigger.dev/core`
gained
an `ai` peer dependency. `turbo prune` (2.5.4) generates a pruned
lockfile
that references the `ai@6.0.116(zod@3.25.76)` snapshot without including
the
entry itself, which causes `pnpm fetch --frozen-lockfile` to abort.
Bumping to 2.10.0 fixes the pnpm v9 peer dep snapshot pruning. Updated
both
Containerfiles for consistency.
Example failure here:
https://github.com/triggerdotdev/trigger.dev/actions/runs/28225353375/job/83618124564
Broken since:
c06005b3
## Summary
Adds a dedicated, always-on Performance section to the repo PR review
guide (`.claude/REVIEW.md`), so every review weighs new work against
table size, hot paths, and how deep or wide the data it walks can get.
It names the tables to treat as huge (the `TaskRun` family in Postgres,
`task_events_v1`/`v2` in ClickHouse), the hot paths that warrant extra
scrutiny (trigger and batch trigger, dequeue, execution-snapshot writes,
OTEL ingestion, trace and run-list reads), the deep and wide shapes that
turn one run into a large tree or batch, and five named anti-patterns
with severities: per-level re-scans, dropping the partition-pruning
predicate, unbounded `IN` lists, sequential per-level round-trips, and
losing the single-query fast path.
Co-authored-by: Matt Aitken <matt@trigger.dev>
## Summary
The in-dashboard agent's datastore now runs migrations over a direct
(non-pooler) connection. A transaction-mode pooler can't run the
migrator (no advisory locks, no multi-statement DDL), so when the
agent's database sits behind a pooler the migration step needs a
separate direct connection.
The application keeps connecting over the pooled
`DASHBOARD_AGENT_DATABASE_URL`. Only the migration entry points changed
(`drizzle.config.ts`, `migrate.mjs`, `migrate-status.mjs`); the runtime
client is untouched.
## Connection resolution (migrations)
```
DASHBOARD_AGENT_DIRECT_URL direct agent connection (used for migrations)
DASHBOARD_AGENT_DATABASE_URL pooled agent connection (preserves current behavior)
DIRECT_URL main direct connection (single-database fallback)
DATABASE_URL last resort
```
Mirrors the existing `DATABASE_URL` / `DIRECT_URL` split. Fully
backward-compatible: with nothing new set, resolution is identical to
before. The agent-specific vars take precedence over the main
`DIRECT_URL`, so a separate agent database is never migrated against the
wrong one. When the agent falls back to the main single database,
migrations now prefer its direct connection.
<img width="2400" height="1794" alt="chat-ui-closed"
src="https://github.com/user-attachments/assets/35016a72-c6d2-4b6a-8760-b1da5b4a166c"
/>
<img width="2400" height="1794" alt="chat-ui-open"
src="https://github.com/user-attachments/assets/25b3df60-9aa2-40fd-a1ba-1447eeee4c52"
/>
## Summary
The in-dashboard agent was launched from a button pinned to the
bottom-right of every page, which floated over page controls (for
example the run inspector's action bar). It now opens from a compact
"Chat" button on the far right of the page header, and the same button
toggles to "Collapse" while the panel is open. The panel and launcher
are labelled "Chat" in the UI.
The launcher only renders on env-scoped pages where the agent is enabled
(same feature flag gating), so it stays hidden for everyone who doesn't
have it. The existing "Ask AI" support button is untouched; it stays in
place until the agent is turned on by default.
## How it works
`DashboardAgent` (env layout) shares the open/close state through a
small context, and `NavBar` renders a launcher that self-hides whenever
that context is absent. No floating overlay, and the launcher can't
appear on pages where the agent can't open.
## Summary
The in-dashboard agent button was rendered for all admins and
impersonators regardless of the `hasDashboardAgentAccess` flag, so it
appeared even where the agent is disabled (for example, floating over
the run inspector controls). It is now gated by the flag for everyone,
so it stays hidden until the flag is turned on.
## Rollout
Both levers default off, so nothing changes for users until deliberately
enabled:
- **Per-org:** set `hasDashboardAgentAccess` on an org's feature flags
to enable the agent for just that org.
- **All admins:** set `DASHBOARD_AGENT_ADMIN_PREVIEW=1` to give admins
and impersonators an everywhere-preview, independent of the per-org
flag.
Previously admins bypassed the flag unconditionally, which is why the
button showed up before the agent was ready to ship.
Closes this feature request:
[https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances)
### Feature notes:
- CLI `trigger dev` works as before
- `trigger dev --branch my-branch` to create a new branch and run
against it.
- `trigger dev archive --branch my-branch` to archive (or in webapp).
- New webapp page to manage and archive dev branches, currently feature
flagged.
### Implementation details:
- No changes to data model, no backfill. `isBranchableEnvironment`
column is ignored for dev branches, we use `parentEnvironmentId IS NULL`
instead.
- `x-trigger-branch` overloaded for preview and dev branches
- New `TRIGGER_DEV_BRANCH` env var available locally.
`TRIGGER_PREVIEW_BRANCH` overloaded for child runs.
- Lots of new glue code to sanitise the branch checks.
### Rollout
- Deploy webapp/API changes (all backwards compatible)
- Manual tests on some orgs
- Deploy docs, release CLI, flip feature flag for webapp feature
### NB
- `api.v1.projects.$projectRef.environments.ts` will return
`isBranchableEnvironment: true` for all dev environments.
### Prerequisites
- [x] Typecheck will not pass until we make a new release of
`@trigger.dev/platform` and bump it here
A deployment could be marked deployed and promoted to current without
its image ever landing in the registry. Finalize trusted the CLI: the v1
path never pushed or checked, and the v2/v3 path skips its own push when
the CLI sends `skipPushToRegistry` - which the local-build path always
does. In the happy path the CLI pushes the image itself, so this stayed
latent. But any deviation - `--no-push`/`--load`, a push that lands in a
different registry, or an old CLI - promoted a version whose image can't
be pulled, so every run failed at pull time while the deploy itself
reported success.
This adds a registry existence check after push and before finalize. If
the image isn't there, the deploy fails loudly instead of promoting a
version that can't start. The check is ECR-only (a no-op for other
registries, so self-hosted setups are unaffected) and uses
`BatchGetImage`, which the deploy role already allows. It fails open on
an ambiguous registry error so the check can't itself turn into a deploy
outage. The image reference is the platform-generated value and the
lookup is bound to the configured registry host; the CLI-supplied digest
is validated before use.
Can be turned off with `DEPLOY_IMAGE_VERIFICATION_ENABLED=0` for setups
that push images out of band (e.g. an air-gapped registry the platform
can't reach).
refs TRI-11243
SSO settings page: resolve plan before the role check. A non-Enterprise
org now renders the upsell state for every role instead of showing a
"permission denied" panel to non-Owners for a feature their org can't
use yet. manage:sso is only enforced once the org is actually entitled.
Extracts EMPTY_SSO_STATUS and uses throwPermissionDenied().
Also removes the client-side SSO session fetch guard. It monkeypatched
global window.fetch, which made it the initiator of every request and
obfuscated the real call site on any 4xx/5xx. Session revocation is
still enforced server-side on every authenticated request and surfaces
as a logout redirect on the next navigation/refresh, so the client guard
was UX-only and not worth the cross-cutting cost.
The wait.for() idempotency example passed the idempotency options as a
second argument, but wait.for() takes a single options object — so the
second object was silently ignored and the key never applied. Merged the
fields into the single options object, matching the wait.until()
example.
Supersedes #4021.
Prefixes the dashboard feedback form thread titles with `Web app:` so
support inbox threads coming from the in-app contact form are easy to
tell apart from those submitted on the marketing site, which previously
shared an identical `Contact form:` prefix.
## Summary
The dashboard agent's database migrations could be silently skipped when
its database is shared with another Drizzle application, leaving the
`trigger_dashboard_agent` schema uncreated and a later migration failing
with `schema "trigger_dashboard_agent" does not exist`.
## Root cause
Drizzle's migrator decides what to run by reading the most recent row
from its journal table by `created_at`, and skipping any migration dated
at or before it. The dashboard-agent runner used Drizzle's default
journal table, `drizzle.__drizzle_migrations`, which every Drizzle app
shares by default. When the database is shared, another app's journal
row dated between two of our migrations makes the migrator treat the
earlier one (the `CREATE SCHEMA`) as already applied and run a later one
against a schema that was never created.
## Fix
- Track the dashboard-agent migrations in a dedicated journal table
(`drizzle.__dashboard_agent_migrations`), in both the deploy runner
(`migrate.mjs`) and the drizzle-kit config, so its history is
independent of any other Drizzle app sharing the database. The table
stays in the `drizzle` schema so the first migration's `CREATE SCHEMA
"trigger_dashboard_agent"` does not collide with it.
- Make the first two migrations idempotent (`CREATE SCHEMA/TABLE/INDEX
IF NOT EXISTS`) so databases that already tracked them under the old
journal table re-run cleanly after the rename instead of erroring on the
bare `CREATE SCHEMA`.
Verified against a Postgres seeded to reproduce the skip: the old
default-table path fails as above, the dedicated-table path creates the
schema and all tables, and re-running on an already-migrated database is
a clean no-op.
## Summary
The auto-generated changeset release PR (`changeset-release/main`)
builds its `## Improvements` / `## Bug fixes` summary with
`scripts/enhance-release-pr.mjs`. The script deduplicated summary
entries by PR number, so when a single PR shipped more than one
changeset, only the first entry survived and the rest were silently
dropped from the summary. The dropped entries still appeared in the raw
`<details>` block, which is how the mismatch surfaced (for example in
[#3998](https://github.com/triggerdotdev/trigger.dev/pull/3998), where
one PR's four changesets showed up as a single summary line).
## Fix
Deduplicate on the full entry text rather than the bare PR number. The
entry text embeds the PR link, so:
- the same changeset rendered once per package section still collapses
to one,
- distinct changesets from the same PR are each kept,
- identical descriptions from different PRs stay separate.
Verified against the raw changeset output from
[#3998](https://github.com/triggerdotdev/trigger.dev/pull/3998): that
PR's changesets went from 1 to all 4 in the generated summary.
## Summary
Adds a `RUN_ENGINE_DEQUEUE_DISABLED_WORKER_QUEUES` setting that refuses
worker dequeue requests for the listed worker queues (or base regions),
so their runs stay queued instead of being handed to workers that can't
run them. Blocked dequeues are counted via a
`run_engine.dequeue.blocked` OTel counter (labeled by `worker_queue` and
`region`).
## Summary
The `runqueue.workerQueue.length` gauge only reported a worker queue's
depth while runs were being dequeued from it. When dequeues stop, the
metric goes stale or missing, so a queue that has backed up because
nothing is draining it can't be alerted on. This adds a small observer
that refreshes the observed set of worker queues from the
`WorkerInstanceGroup` records on an interval, so every active worker
queue (and its scheduled split variant) keeps reporting its length
regardless of dequeue activity.
The observer is off by default and enabled per service via
`RUN_ENGINE_WORKER_QUEUE_OBSERVER_ENABLED`, reads from the read replica,
and skips a configurable set of cloud providers
(`RUN_ENGINE_WORKER_QUEUE_OBSERVER_EXCLUDED_CLOUD_PROVIDERS`, default
`digitalocean`). When enabled it is the source of truth for the observed
set, so the per-dequeue registration is skipped on that instance, and it
groups by worker queue so the per-instance duplicates collapse to the
true depth.
Also removes the unused `GET`/`POST /api/v1/workers` endpoints. Their
only consumer was a CLI command group that is no longer registered.
## Verification
Verified end to end against a local stack: the gauge reports each worker
queue's length with no dequeues happening, excludes the configured
providers, includes hidden groups, and the removed endpoints return as
if they never existed. Added a run-engine test
(`workerQueueObservation.test.ts`).
## Summary
Adds an in-dashboard AI agent: a chat panel, reachable from any
environment
page, that answers questions about your runs, errors, tasks, and
analytics,
diagnoses why a run failed, charts your data, reads your connected
repo's
source, and answers product and how-to questions. It is gated behind the
`hasDashboardAgentAccess` feature flag (global or per-org, default off),
so
this PR ships disabled: the launcher is hidden unless the flag is
enabled.
## Design
The agent runs as a standalone `chat.agent` Trigger task in its own
internal
package, with no access to the webapp database, Prisma, or ClickHouse.
It reads
the user's data over the public API, acting as the user via a
short-lived
delegated user-actor token minted server-side each turn (never in the
browser),
building on
[#3997](https://github.com/triggerdotdev/trigger.dev/pull/3997). The
error and analytics tools use
[#4005](https://github.com/triggerdotdev/trigger.dev/pull/4005)
and the TRQL query API.
The first turn of a new chat streams from a warm webapp route (Head
Start) while
the durable agent boots in parallel. Structured answers (a run-failure
diagnosis
card, a live chart) render through a small typed view catalog rather
than
arbitrary markup. A knowledge lane forwards product and how-to questions
to the
support assistant.
Conversation history lives in a separate Drizzle-backed store on its own
Postgres schema, kept as a display read-model so it can never corrupt
the
agent's model context.
The SDK changes add an `apiClient` option to
`chat.createStartSessionAction` and
`chat.headStart`, and keep the Head Start tool-approval tail intact
across a
custom `prepareMessages` hook so prompt caching and Head Start compose.
Adds an in-process backpressure signal that pauses dequeuing when the
Kubernetes cluster is saturated, so work overflows cheaply in the queue
instead of piling up as unschedulable pods. Saturation is read by
scraping the apiserver's total pod-object count
(`apiserver_storage_objects{resource="pods"}`) and applying an
engage/release threshold with hysteresis - a single lightweight
aggregate scrape, not a pod listing.
Backpressure sources are now evaluated independently and OR'd: each
source has its own enable and dry-run flag, and the supervisor engages
if any enabled source trips. This adds the pod-count source alongside
the existing one without changing it, and is extensible to more sources
later. Off by default.
The scrape uses the in-cluster kubeconfig over `https` so TLS verifies
against the cluster CA (the fetch-options helper attaches the CA as an
`https.Agent`, which the global `fetch` ignores - that path silently
dropped the CA). Enabling the pod-count source requires the supervisor's
service account to be granted `get` on the `/metrics` non-resource URL;
that RBAC and the per-deployment env wiring are operator-side and live
elsewhere.
New config (pod-count source):
`TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_ENABLED` (default false),
`_POD_COUNT_DRY_RUN` (default true), `_POD_COUNT_ENGAGE` /
`_POD_COUNT_RELEASE` (hysteresis thresholds), `_POD_COUNT_REFRESH_MS`
(scrape interval, default 5s). The existing source's flags are
unchanged.
Observability: a `supervisor_cluster_pod_count` gauge, and the pod-count
monitor's metrics are namespaced (`supervisor_backpressure_pod_count_*`)
so the existing backpressure metrics keep their names.
## Summary
Fixes the fair-queue weighted environment shuffle, which biased
environment ordering whenever fair-queue biases are enabled (the default
configuration).
## Root cause
`#weightedShuffle` in `fairQueueSelectionStrategy.ts` computed the total
weight once and drew its random pivot against that full-set total on
every iteration, but never decremented the total as items were removed
from the working set. After the first pick, the pivot frequently
overshot the sum of the remaining items, so the inner selection loop ran
off the end and clamped to the last remaining element. The result
systematically over-selected whichever environment sat at the tail of
the set.
The first slot stayed fair (the full total is correct on the first
draw), but later positions were ordered by environment iteration order
rather than by the intended concurrency-limit and available-capacity
weighting. For four equal-weight environments, the final position landed
on one env ~9% of the time and another ~42%, instead of ~25% each.
The two sibling selection paths (`#weightedRandomQueueOrder` and
`#selectTopEnvs`) already decrement the total before splicing; this
brings the env shuffle in line with them.
## Fix
```ts
result.push(items[index].envId);
totalWeight -= items[index].weight;
items.splice(index, 1);
```
Adds a regression test that runs the weighted shuffle over equal-weight
envs with biases enabled and asserts each env lands in every position
roughly uniformly. It fails on the old code (tail position ~37%) and
passes with the fix.
Reported in #4001.
Deploy success/failure wasn't easily observable: compute template
creation only logged, and the terminal deployment statuses (deployed /
failed / timed-out) weren't traced — so deploy health couldn't be seen
without querying the database.
This adds two spans:
- `compute.template.create` around template creation at finalize,
tagging the resolved mode and per-preset outcome. `resolveMode` now
returns its decision (mode + reason) so the span can record why a mode
was chosen.
- `deployment.outcome` via a small shared helper
(`recordDeploymentOutcome`) emitted at every terminal-status write —
finalize (deployed), fail / index-failed / background-worker (failed),
and timeout (timed out) — so deploy success/failure is queryable by
status and reason.
The helper is best-effort (org/project/env enrichment where cheaply
available) and never throws, so telemetry can't break a deploy.
## Summary
Refreshes the SVG artwork for the main task icon and the cached task
variant shown on the run trace span view.
The cached icon (previously a hardcoded blue "T" in a dashed border) now
lives alongside `TaskIcon` in `TaskIcon.tsx` and is drawn with
`currentColor`, so it inherits the `text-tasks` theme color like the
other span icons instead of ignoring it. The standalone
`TaskCachedIcon.tsx` file is removed and its two import sites updated.
## Summary
Sessions started from the agent Test playground were tagged with a
`"playground"` tag that rendered in the Sessions table's Tags column.
They are now flagged with a real `Session.isTest` boolean (mirroring
`TaskRun.isTest`) and surfaced as a dedicated **Test** column with a
check icon, to the left of Tags, on both the Sessions page and the Agent
landing page, plus a matching **Test** property on the session detail
page. This mirrors how Standard and Scheduled task runs already indicate
test runs.
## Design
`isTest` is a new `Session` column (Postgres) replicated into ClickHouse
`sessions_v1` alongside the existing fields. The Sessions list reads
`isTest` from Postgres for display (ClickHouse only supplies the ordered
session IDs), so the column renders correctly without a ClickHouse
backfill.
The playground action now sets `isTest: true` on session create instead
of writing the `"playground"` tag. The triggered run still carries
`playground:true` in its own tags (unchanged). A migration backfills
existing sessions, setting `isTest = true` and stripping the
now-redundant `"playground"` tag where it is present, so the list and
detail views render consistently without read-time tag filtering.
## Summary
Updates the task icons used across the dashboard. `TaskIcon` and its
small variant now use a new burst glyph, and `TasksIcon` adopts the
previous task glyph (the rounded square). Both still render with
`currentColor`, so they inherit text color exactly as before.
Export names are unchanged, so every existing usage (side menu, task and
queue views, run filters) picks up the new artwork with no other code
changes.
Chris Arderne