Missing or legacy preferences resolve to the pinned Dark theme (not
system-resolved, so nobody is surprised by light mode) and the default
contrast is 50. Classic remains the flag-off look.
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 2s
🚀 Publish Trigger.dev Docker / units (push) Failing after 2s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
## Summary
The realtime runs feed hydrates run rows from read replicas, which means
it needs a replica-lag gate to avoid serving a run's previous state
right after a write. Setting
`REALTIME_BACKEND_NATIVE_RUN_READS_FROM_PRIMARY=1` reads those rows from
each run store's primary instead, so there is no lag to gate against: no
probe, no wake delay, no stale-read retries. Off by default, so nothing
changes unless you set it.
## Design
The run stores already decide replica-vs-primary from the *brand* on the
read client they are handed: a branded replica keeps the read on the
owning store's replica, an unbranded writer escalates it to that store's
own primary. So this is a one-line choice at the hydrator, and it stays
correct across topologies. With the run-ops split on, each leg lands on
its own writer and the caller's client is never forwarded across
databases; with the split off, it is the single database's primary.
```ts
const runReader = new RunHydrator({
readClient: runReadsFromPrimary ? prisma : $replica,
runStore,
});
```
The same flag skips constructing the lag estimator, since probing a
replica the feed no longer reads would be measuring the wrong thing.
Independently, `AuroraReplicaLagSource` detected Aurora by letting
`aurora_replica_status()` fail, on the assumption that the app-level
catch made that free. It isn't: an unresolvable function is a query
error the driver reports to the error log on every sample, so a
non-Aurora replica produced a continuous stream of error events while
the estimator quietly fell through to its next candidate. It now
resolves the function with `to_regproc` and memoizes the answer, so the
unparseable call never reaches the wire.
**Stacked on #4131** — targets `feat/queue-metrics-and-health`, not
`main`. GitHub will auto-retarget this to `main` once #4131 merges.
(Targeting `main` now would wrongly include all of #4131's own commits.)
UI polish for the Queue metrics & health pages (TRI-12068). 49 commits,
all under `apps/webapp/app/` — no engine/ClickHouse/package changes.
**Highlights**
- **Layout** — consolidated queue-detail header into one filter bar;
shared `MetricsLayout` spacing (2.5 scale); paused-queue banner
mirroring the environment-paused banner.
- **Charts** — inline legends under titles; threshold gradient coloring
(Env saturation orange only >100%; Concurrency orange only at/over the
limit); fullscreen legend spacing; result caching so tab switches don't
reload; brighter hover cursor + full x-axis labels.
- **Tables** — whole-row click via stretched link; custom Queues icon;
removed column sorting; Limit-cell spacing + hover-bright; "Limited by"
tooltip sizing.
- **Controls** — Pause moved into the filter bar (orange label +
accurate tooltip copy); restyled override-concurrency modal.
- **Copy** — plain-language pass; inline colour swatches; Throttled
reads "% of current period".
Also consolidated the three queue-metrics `.server-changes` notes into
one.
**Known follow-up (not in this PR):** the Concurrency-keys table is
capped at 50 keys (union of live Redis backlog + ClickHouse history, no
single cursor across both) and truncates silently — fine for most
queues, a real limit for high-cardinality per-tenant keys.
Verified locally against the `queue-metrics-demo` seed; `pnpm typecheck
--filter webapp` passes.
## Summary
When a `chat.agent` (or `chat.createSession`) turn's model stream fails
mid-response (e.g. a transport timeout like `UND_ERR_BODY_TIMEOUT`), the
assistant output that already streamed was dropped: `onTurnComplete`
fired with `responseMessage: undefined`, and the manual loop's
`turn.complete()` rethrew without keeping the partial. Apps that
register `hydrateMessages` are hit hardest, since boot-time tail-replay
recovery is off by design.
This preserves the streamed-so-far assistant output while still
reporting the turn as errored, so persistence and recovery keep it.
## Scope of behavior change
Only the **error path** changes. Successful turns are unaffected: the
same chunks stream to the client in the same order, and
backpressure/cancel behave as before. Everything here is a correctness
improvement on a turn that hit a source-stream failure.
## What it does
Follow-up to #4304 (`chat.pipeAndCapture`), extending the same
partial-recovery to the two loops that lacked it:
- **`chat.agent`**: taps the response stream (via a `TransformStream`,
so pass-through backpressure and cancel are preserved) to buffer chunks,
and on a source-stream failure reconstructs the partial (preferring the
`onFinish` message). It's surfaced on the error-path `onTurnComplete`
(`responseMessage`, `rawResponseMessage`, `uiMessages`, `newUIMessages`,
`newMessages`) and committed to the accumulator so the next turn and the
reboot snapshot keep it.
- **`chat.createSession` / `turn.complete()`**: the reconstructed
partial is accumulated (so `turn.uiMessages` reflects it and the caller
can persist after catching) before `turn.complete()` rethrows.
`onBeforeTurnComplete` stays skipped on the error path (it hands out a
writer for a stream that has already broken).
## Correctness properties (each covered by a regression test)
Each test below was confirmed to fail without its fix:
- The recovered partial reaches `onTurnComplete` and the next turn's
accumulated messages.
- An already-committed (possibly enriched) response is not overwritten
if a post-response hook then throws.
- Incomplete tool parts are cleaned from the recovered partial (text
kept), so the UI and model views agree and the next turn isn't poisoned.
- A prior turn's model-only compaction survives an errored turn (append
only the new tail, don't reconvert the full history).
- A reconstructed fragment that reuses an existing message id does not
clobber the complete message.
- Queued `chat.response` data parts are folded into the recovered
partial, matching the success path.
- `newMessages` (model delta) stays symmetric with `newUIMessages`.
## Tests
New `chat-agent-source-stream-error.test.ts` covers the cases above. The
full `@trigger.dev/sdk` unit suite passes and the package build is green
across all supported runtimes (Node 20 to 26, Bun, Deno, Cloudflare
Workers).
Adds an optional `TRIGGER_WARM_START_DISPATCH_URL`. The warm-start
dispatch request uses it when set, otherwise falls back to
`TRIGGER_WARM_START_URL`, so the dispatch target can differ from the
default warm-start URL. No behavior change when unset.
## Summary
Several list endpoints accepted an unbounded page size (`perPage` /
`per_page` / `pageSize`). An unbounded page lets one request pull an
arbitrarily large result set and do a proportional amount of work, which
is a poor default for a shared API.
This clamps the page size to 100 on every list endpoint that was
uncapped, matching the existing cap on `api.v1.runs` and
`api.v1.sessions`. Clamping rather than rejecting keeps existing clients
working: a request for a larger page returns up to 100 items and offset
pagination continues from there.
## Endpoints capped
- `api.v1.schedules` (`perPage`)
- `api.v1.queues` (`perPage`)
- `resources.…versions` (`per_page`)
- `resources.…queues` (`per_page`)
- `admin.api.v1.…engine.report` (`per_page`)
- `admin.api.v1.llm-models` (`pageSize`)
Already capped, left as-is: `api.v1.runs`, `api.v1.sessions`,
`api.v1.deployments`.
## Summary
The batches list page orders by `createdAt DESC, id DESC` filtered by
environment and a created-at window, but the only supporting index on
`BatchTaskRun` was `(runtimeEnvironmentId, id)`. That index can't
satisfy the `createdAt` ordering, so on environments with a large number
of batches the query fell back to a full table scan and in-memory sort,
which could run long enough to hit the statement timeout.
## Fix
Adds `(runtimeEnvironmentId, createdAt DESC, id DESC)` on
`BatchTaskRun`. The query now reads straight from the index in order
with no sort step, returning a page with only a handful of heap fetches
instead of scanning the whole environment slice.
The migration uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it
takes no table lock and is a no-op if the index already exists.
Light variants generated from the dark screenshots by inverting lightness
(hues survive, so the accent smudges stay true); optimized progressive
JPEGs at a fraction of the original size.
Login, invites and other anonymous pages resolve the System preference
when the theme switcher flag is on; Classic stays the flag-off default.
The invites heading goes monochrome in System themes.
React skips the data-theme write when its virtual DOM already matched the
SSR fallback while the inline script had flipped the real attribute; the
sync hook now writes pinned themes explicitly.
Adds Prometheus metrics so the supervisor's outbound HTTP calls are
observable - including client-side failures that previously only
surfaced as a log line.
- `supervisor_outbound_request_total{name, method, status, outcome}` -
counts every outbound request. `outcome` separates a transport failure
(`network_error`), an HTTP error response (`http_error`), a response
that failed schema validation (`invalid_response`), and success (`ok`).
- `supervisor_outbound_request_duration_seconds{name, outcome}` -
latency histogram. Leaner labels than the counter (no `status`) to avoid
bucket×label cardinality; buckets match the existing dequeue-latency
histogram since these calls share the same retrying HTTP client and
long-poll envelope.
Coverage:
- The warm-start request (a one-off `fetch`) - instrumented inline; the
response status code is now also included in the failure log (it was
previously dropped).
- All worker API client calls (`SupervisorHttpClient`: dequeue, run
attempt start/complete, heartbeats, snapshots, continue, suspend,
debug-log, connect) - routed through a single instrumented `request()`
helper that reports via an optional `onHttpRequestComplete` callback on
the client, which the supervisor wires into the counter + histogram.
Low cardinality by design: `name` is a **static per-endpoint label**
(e.g. `dequeue`, `start_run_attempt`), never the interpolated URL - so
no run/snapshot IDs land in labels, mirroring the templated `route`
labels on the inbound HTTP server.
Registered on the existing metrics registry, exposed on `/metrics` with
no new wiring. Internal-only change (no package release needed), so the
changelog note is a single `.server-changes` entry.
Shared contrast-chip marker on error/health/pill/source/connection chips
and subtle badges; a contrast+saturate filter driven by --theme-contrast
separates the tint and its colored text.
Backgrounds deepen (dark) or lift (light), surfaces and all border steps
strengthen, text weights increase - task names and dimmed labels now respond
visibly to the slider in both modes.
New contrast preference (0-100) stored in dashboard preferences, applied
as a --theme-contrast variable on <html>. The System theme blocks pull the
monochrome scale apart via color-mix - text toward the mode foreground,
borders a step up; contrast 0 stays byte-identical and Classic never reads
the variable. The account page shows the slider under Theme for non-Classic
themes with live preview.
Private connections list places Add Connection under the list; the prompt
override Remove becomes an outlined secondary next to the filled Edit; the
Select section probe no longer crashes on an empty filtered list.
Root-style badges and error status chips share the tint+color language;
text colors come from the unified accent set so they read in both System
modes.
bg-runs/bg-concurrency/bg-success/bg-warning instead of raw palette steps,
hover lightens via brightness so the fill always matches the unified accent
set.
Katia Bulatova