Customize kept the recommendation's original note across condition/threshold
changes, so the wake quoted a condition the user never watched — the note is
now restated whenever the condition or its number changes. The wake prompt
also hands the model a ready trigger:// markdown link to the watched object.
Chat code examples drop to 12px.
The appended-message hand-off wasn't scoped to a chat: a chat mounted after a
watch was created started with a fresh dedupe ref, saw the pending seq, and
adopted another chat's confirmation into its transcript. The hand-off now
carries its chatId and only the matching chat receives it.
Ask AI -> Ask Trigger; send/stop centered in the docked composer; Watch
button wears the chat's eye glyph; a delayed history reload picks up the
generated chat name after the first answer; dashboard agent temporarily
open to everyone for V1 rollout testing.
The free-text path used to POST the watch straight from the tool, so a
"yeah, set one up" created it with no card and no consent. The tool now
validates the spec and returns a `watch` intent; the panel scans
`tool-schedule_watch` results the same way it scans navigate_to and opens
the pre-filled configuration card, replay-safe. The card's submit is the
only creator, so it owns the opt-ins, the cap, dedup, and the one-shot
result — prompt wording updated to match.
A wake has no client to carry the stored transcript, so the session view can
miss host-appended blocks — a card-born chat starts with only those, and the
wholesale persistMessages write dropped the watch confirmation. The wake's
display copy is now an id-deduped single-message append.
A chat born from the watch card (0 LLM) never had a session, so the wake's
in.send 404'd and the delivery stayed pending. On 404 the wake now creates
the session (idempotent on externalId) and retries once.
An already-late queue was recommended the age SLA — already true, so every
watch one-shot with "that already happened". Late queue now recommends the
drain (the recovery); a healthy queue recommends the age SLA.
Navigating to another page drops the fullscreen takeover back to the side
panel.
'Morning after the deploy': report, chart, a two-revision investigation
(latest-wins), two watch confirmations, a one-shot result, a real wake with
backing watch rows (one fired, one still active for the live chip), and
docs citations. Seeded idempotently; --showcase re-seeds just this chat
over a running stand.
- the seeder now stages the email-sends QUEUE counter (what the watch checks
and queue pages read) alongside the env-level one — a drain watch on the
stand no longer one-shots against an empty live counter
- vite ignores seed-*.mts: editing or running a seeder was full-reloading
every open dashboard tab every few seconds
- ask-ai hover fills with the brand green; ink flips to charcoal-800 (white
on that green is ~1.4:1) with a hover-swapped dark logo — the canvas can't
repaint on :hover
- docs buttons keep the docs-blue ink on the light theme (monochrome stays
for the dark ones) — a text-bright label on white read as plain grey
- the send arrow and stop glyph are white; stop gets a theme-stable filled
neutral so the glyph has a surface on light
Each monoLight stop is now the cool grey whose contrast against a white panel
matches that stop's contrast against a dark one, so the ramp keeps mono's
shape: the middle stop stays the extreme the head glows with and the third
stays the dim tail. With mode="light" handling the ghost grid and the opacity
pair untouched, the light logo is a mirror of the dark one rather than a
different-looking icon.
The dot-matrix logo is drawn on canvas with a white-based ramp, so on the
light theme it was white ink on a white surface: the chat spinner, the
panel's hero logo and the Ask AI button's glyph all disappeared. Adds a
useThemeMode hook and an AgentMonoLogo wrapper that picks the ink from the
active theme, and routes every mono call site through it.
Adds System Preferences, Dark and Light themes, gated by the
`hasThemeSwitcher` feature flag (off by default — dark stays the default
theme for everyone).
Old theme is now "Classic"and set as default.
"System preferences" theme has both Light and Dark modes and uses your
laptop settings to use a correct one.
It has less color accents (specifically less colored text), and they are
the same for both modes, only grayscale values change between them. And
Light/Dark themes can be used separately.
New Contrast setting is available for System Preferences, Dark and Light
themes - it changes the contrast for the whole app. All new visual
Settings live in Account.
## Summary
The webapp's server bundle imports `prop-types` directly, but the
package was declared only as a `devDependency`. A production install
therefore leaves it out and the built server fails to boot:
```
Failed to start server: Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'prop-types'
imported from /triggerdotdev/apps/webapp/build/server/assets/server-build-*.js
```
Moving it to `dependencies` is the whole change.
## Why the bundle imports it
Nothing in the webapp's own code uses `prop-types` — there is no
reference to it, or to `PropTypes`, anywhere under `apps/webapp/app`. It
arrives through `recharts`, whose `react-smooth` dependency still
declares `propTypes` on its components.
That was invisible until recently. While `recharts` was resolved at
runtime, its `prop-types` import was satisfied inside `recharts`' own
dependency tree, which is production all the way down. #4486 added
`recharts` and `victory-vendor` to `ssr.noExternal` to fix a hydration
mismatch on every server-rendered chart; that inlines `react-smooth`
into the server bundle, which moves its `prop-types` import into the
webapp's own resolution scope — where the package was not available in
production.
So the bundling change was correct about *which* d3-shape build both
sides resolve, and wrong about what the production runtime would be able
to find.
## Verification
`docker/Dockerfile` builds the runtime dependencies with `pnpm install
--prod` against a `turbo prune --scope=webapp --docker` output, so I
reproduced exactly that: pruned the workspace, installed with `--prod`,
and imported `prop-types` from `apps/webapp`.
| | result |
| -- | -- |
| `main` as it stands (devDependency only) | `FAILS:
ERR_MODULE_NOT_FOUND` |
| with this change | `prop-types resolves OK` |
It resolves both as a CommonJS `require` and as an ESM `import`, which
is the form the bundle uses.
I also checked this is not one symptom of a wider problem: of the 169
bare specifier roots the server bundle imports, `prop-types` is the
**only** one that is a devDependency and not a production dependency.
The rest are node builtins or production dependencies.
The hydration fix from #4486 is unaffected — the rebuilt bundle still
carries the rounding d3-path build.
## Notes
`prop-types` is inert in production (its entry point swaps in
`factoryWithThrowingShims`), so this adds a 124 KB package that does no
work at runtime. It has to be resolvable regardless, because the import
is real.
An alternative would be adding `prop-types` to `ssr.noExternal` so it is
inlined and needs no runtime resolution. That keeps the dependency list
honest about the fact that the webapp itself does not use it, at the
cost of bundling a CommonJS package into the ESM server output. This
route is the smaller, better-understood change.
Worth following up separately: a check that every bare import in the
server bundle resolves from a production install would have caught this
before it landed. Local development installs every devDependency, so the
gap is invisible when the built server is run from a working tree.
The dequeue brake released the moment its signal became unreadable.
`refresh()` caught any error from `source.read()` and set the verdict to
`null`, which `computeEngaged()` treats as not-engaged — so a few failed
reads dropped an engaged brake, silently, with no log and no metric.
That handling was symmetric while the risk is not. A source that has
stopped answering correlates with the pressure the brake exists for, so
releasing on read failure gives up protection at exactly the wrong
moment; holding too long only costs throughput.
Now a failed read keeps the last verdict instead of discarding it. The
verdict then ages normally, so the existing `maxVerdictAgeMs` check
becomes the grace window and still bounds how long a dead source can
hold the brake — a permanently unreachable source releases it rather
than pinning dequeuing forever. Because `computeEngaged()` only consults
staleness for an *engaged* verdict, a released one is unaffected and
stays released.
The default grace moves from 15s to 120s, comparable to how long the
brake normally stays engaged.
One guard worth calling out: holding is only safe when something bounds
it, so when `maxVerdictAgeMs` is unset the previous discard behaviour is
kept. Otherwise an unbounded hold could pin the brake indefinitely.
Read failures were previously invisible — the catch block neither logged
nor counted. Adds a `read_failures_total` counter, plus an error log on
the transition into failure rather than once per tick, since the refresh
loop runs every second.
The post-release ramp needs no change: it anchors off the
engaged-to-released transition, so a grace-window release still ramps
back up instead of snapping to full rate, which is what you want after a
blind period.
Tests cover holding while reads fail, releasing past the max age, and
the existing unbounded-config paths are unchanged.
## Summary
Self-hosted Kubernetes deployments can now add tolerations to run pods,
so runs
can schedule onto tainted nodes. Previously the only way to do this was
to patch
the supervisor.
`KUBERNETES_RUNNER_TOLERATIONS` takes a comma separated list of
`key=value:effect`, or `key:effect` to tolerate any value. It applies to
every
run pod, and for runs from a schedule tree it merges with the existing
`KUBERNETES_SCHEDULED_RUN_TOLERATIONS`. Left unset, nothing changes: no
tolerations are added and the pod spec leaves the field off entirely.
The Helm chart takes it as a list:
```yaml
supervisor:
config:
kubernetes:
runnerTolerations:
- dedicated=runs:NoSchedule
- spot:NoExecute
```
## Naming
The issue proposed `KUBERNETES_WORKER_TOLERATIONS`. This ships as
`KUBERNETES_RUNNER_TOLERATIONS` instead, because `RUNNER_*` is already
the prefix
for run pod settings (`RUNNER_HEARTBEAT_INTERVAL_SECONDS`,
`RUNNER_ADDITIONAL_ENV_VARS`, and `DOCKER_RUNNER_NETWORKS` for the
Docker
equivalent), whereas "worker" refers to the supervisor itself throughout
this app.
## Validation
Keys and values are checked against the Kubernetes naming rules when the
supervisor starts, so `dedicated=prod runs:NoSchedule` fails immediately
with a
message naming the offending entry. Without that check a bad value is
accepted at
startup and then rejected by the API server on every pod create, which
stops all
runs with the cause buried in an API error.
`KUBERNETES_WORKER_NODETYPE_LABEL` is
trimmed and validated for the same reason: surrounding whitespace is not
valid in
a label value, so a padded value fails every pod create today.
## Node selector off switch
`KUBERNETES_WORKER_NODETYPE_LABEL` accepts an empty string to skip the
node
selector entirely, so runs schedule on any node. This already worked and
the Helm
chart has always shipped it empty, but it was not documented. It is now.
The issue also asked for general node affinity configuration. That is
not
included: the node selector off switch plus tolerations covers the
reported
problem, and a free form affinity setting is a much larger config
surface to
commit to.
Fixes#4458
## Summary
The four charts above the queues table aggregated over **at most the 25
queues on the current page**. They reused the loader's already-paginated
queue array as a ClickHouse `queue IN (...)` filter, so paging or
re-sorting changed the values, and a name search matching nothing
blanked the whole chart row. The stat tiles above them were already
environment-wide, so the two rows disagreed.
They now read `env_metrics`, the environment-level rollup that already
exists for exactly this (the built-in Queues dashboard and the health
report read it). That is both correct and queue-count-independent: no
`GROUP BY queue` across an entire environment, and no client-side
summing.
Note this is not only a paging artifact: page 1 under-reported too. On
the seeded environment below, page 1 read 82% saturation against a true
87%, because the environment's running total is not the sum of one page
of per-queue gauges.
Three related fixes ride along.
**Scheduling delay and throttling sawed to zero.** Both are
event-driven, so at the 10-second bucket a short range picks, most
buckets hold no samples at all and were drawn as `0ms`. Measured over a
1-hour window: **232 of 349 buckets had no scheduling-delay samples**. A
bucket where nothing started is not a bucket where nothing waited, so
the line was both ugly and wrong. TRQL grows a `minBucketSeconds` floor,
plumbed through the metric resource route, and the hero tiles set 60s.
Buckets that still have no samples render as a gap instead of a dive to
zero.
**The floor must not feed a width-dependent headline.** Two of the four
headlines are not peaks, so widening the plotted buckets moved them:
- **Throttled** is a share of buckets that saw any throttling, so a
single brief throttle came to mark a whole minute instead of ten
seconds: the same seeded events read 17% at 10s and 85% at 60s.
- **Scheduling delay p95** is a percentile, and merging quantile states
over a wider bucket yields a p95 between the sub-buckets' own. Two 240s
samples among twenty in one 10-second sub-bucket give a worst-of-six p95
of 240,000ms against a merged 60-second p95 of 5,000ms — a 48x
understatement of a headline whose tooltip claims it is the worst in the
window.
Both charts keep the floor, since a readable line was the point of it.
Their headlines now come from a second query at the range's natural
bucket width, via an optional `readout` on the tile, so each means what
its tooltip says regardless of how the plotted buckets are sized.
Saturation and backlog are genuinely width-invariant (a max of maxes is
the same at any width), so they are unchanged and issue no extra query.
Both caught by Devin in review; I had wrongly lumped p95 in with the
peaks.
**Charts reported a hydration mismatch on every render.** Recharts
resolved victory-vendor's CJS entry on the server and its ESM entry in
the browser. Those bundle different d3-shape builds, and the CJS one
predates d3-path's digit rounding, so every server-rendered curve
carried full-precision coordinates while the client rounded to 3
decimals:
```
Server: M0,3C0.9305555555555555,3,1.8611111111111112,3,...
Client: M0,3C0.931,3,1.861,3,...
```
Bundling recharts for SSR makes both sides resolve the same ESM build.
Verified: 45 of 45 server-rendered chart curves now match the client,
and the page loads with an empty console.
## Verification
An isolated stack with 40 seeded queues (20 heavily loaded, 20 idle) and
90 minutes of 10-second buckets written into `queue_metrics_raw_v1`, so
the real materialized views built `queue_metrics_v1`, `env_metrics_v1`
and the 5m rollup. Ground truth for the environment: 260 running against
a limit of 300 (**87% saturation**), 800 queued.
| | before | after |
| -- | -- | -- |
| Saturation, page 1 | 82% peak | **87% peak** |
| Saturation, page 2 | 5% peak | **87% peak** |
| Backlog / delay, page 2 | "No activity" | **800 peak / 59.5s** |
| Name search matching nothing | all four charts blank | charts stay
environment-wide |
| Metric refetches on a page change | 4, each painting a skeleton | **0,
no skeleton** |
| Buckets drawn as 0ms with no samples | 232 of 349 | **0** |
| Throttled readout | 17% | **17%**, unchanged by the wider buckets |
| Worst-p95 readout source | plotted buckets | **natural width**, so a
sub-minute spike is not averaged away |
| Crosshair reach, hovering one detail-page chart | 2 of 4 others | **4
of 4** |
| SSR chart curves mismatching the client | 45 | **0** |
The bucket floor was measured across ranges: it widens 10s to 60s at 30m
and 1h, and is correctly a no-op at 12h (300s) and 7d (3600s). One extra
request per page load, for the throttled readout.
The built-in Queues dashboard, which reads `env_metrics` independently,
agrees at 86.7% and 260 of 300.
`internal-packages/tsql` suite green (612 tests), including 5 new ones
for the floor that fail without it. Webapp typecheck, oxfmt and oxlint
clean. Spot-checked the Run metrics dashboard and the per-queue detail
page for SSR regressions from bundling recharts: both render, console
clean.
The queue detail page carries the same event-driven series, so its
scheduling delay, throttling and per-key mean delay take the same
treatment.
## Screenshots
<img width="2540" height="580" alt="after-page1-charts"
src="https://github.com/user-attachments/assets/6cd23f9c-e7fd-4918-bcfa-b1d3340b16d1"
/>
## Rollout
Already behind the per-organization `queueMetricsUiEnabled` flag, so
only gated orgs see any of it. Blast radius is chart values on one page
plus the SSR bundling of recharts; rollback is a revert with no data
migration.
## Stated limitations
- `wait_ms_count` and the quantile state both only count `wait_ms > 0`,
so "nothing started in this bucket" and "everything started instantly"
are indistinguishable in storage. Both render as a gap. Distinguishing
them needs a schema change, which is not in this PR.
- The queue name search deliberately no longer narrows the charts. It
only did so incidentally and incorrectly before (first 25 matches, and
blanked on zero matches). Search-scoped charts would need the full
unpaginated matching set and a server-side aggregate; worth its own
ticket if we want it.
- Bundling recharts for SSR grows the server bundle slightly. That is
the cost of both sides resolving one d3-shape build.
- The plotted delay line is a smoothed 60-second view, so a sub-minute
spike above the one-minute warning threshold can fail to colour the line
even though the headline reports it and colours itself.
- Every chart inside one synced group shares the floor, because the
hover crosshair is a reference line on a category x-axis and only draws
where the hovered bucket exists in the other chart's own data. That
costs the queue detail page's gauges some resolution (1 minute instead
of 10 seconds) in exchange for the crosshair working across the row.
Separately, while taking the screenshots I found a pre-existing
rendering bug unrelated to this change: a **perfectly flat** saturation
series draws no line at all (the readout still shows the right
percentage), which looks like the threshold gradient's offset
degenerating when the series min equals its max. It reproduces on
`main`, so it is not a regression here and I have left it alone; filed
as its own issue.
Refs TRI-12784
Only the runs, errors, queues and deployments pages described themselves
to the dashboard agent; everything else fell to "other" and offered the
generic chips. Add handle mappers for the remaining 37 env-scoped routes
and 24 page kinds in the contracts, so each page offers an explain and a
docs question about what it actually shows.
Investigate and status chips stay gated on loader data: a scheduled task
with no schedule attached, all its schedules disabled, a paused queue, a
batch whose runs failed, a wait token past its timeout, a bulk action
still running, a spent quota, a prompt pinned to an override, a session
whose run failed. Loader data only, no added queries, no new signals.
GET /api/v1/reports/health threw `no catalog registered for report
"health"` in production (fine in dev): the catalog registered itself as
a side effect of a bare import, which the SSR build tree-shakes under
`"sideEffects": false`. Verified on the built server bundle — main's is
missing the catalog, this branch's carries it.
Fix: catalogs are values on the report registry entries; the resolver
reads them from there and the mutable register-at-import step is gone.
Only the runs list and run detail described themselves to the dashboard
agent, so every other page fell to "other" and offered generic chips.
Add handle mappers for the errors list, an error group, the queues list,
a queue, the deployments list and a deployment — loader data only, no
added queries — plus list page kinds in the contracts and an optional
deployment status. Investigate chips now appear for an unhealthy queue
and a deploy that didn't land.
## Summary
Environment API keys backed by the additional-key table can authenticate
API requests using their stored effective scopes. Revoked and expired
keys are rejected, branch environments retain their existing routing
behavior, and last-used timestamps are updated on a throttled
best-effort basis.
## Design
API route builders receive the resolved ability and reject restricted
keys on routes without an authorization declaration. Existing
deployment, environment variable, queue, run, task, batch, session, and
waitpoint routes declare the resources they access.
Trigger and batch responses return server-signed public access tokens,
so additional keys never need access to the environment signing secret.
Root-key rotation also keeps public tokens valid for the existing grace
window.
## Feature notes
- Root environment keys remain unrestricted for backward compatibility.
Additional keys enforce their persisted scopes and fail closed on routes
without an authorization declaration.
- Machine-key requests never exchange one credential for another.
Additional keys cannot retrieve the root key, and rotated root keys are
not upgraded
during their grace window.
- Public JWT validation remains host-owned, while installed RBAC plugins
continue to supply root-key abilities.
- Unfiltered session and run listings preserve existing broad task-read
behavior. Filtered requests enforce the supplied task identifiers.
- Related-run summaries remain embedded in run retrieval for API
compatibility. Retrieving or mutating a related run independently still
requires
permission for that run.
- Queue management authorizes at collection scope, matching the queue
permissions currently issued.
- Batch responses deliberately include server-signed public access
tokens for all clients. Selected-task credentials continue using their
original
credential for per-item authorization.
- Two-phase batches authorize declared task identifiers before creation
and authorize every streamed item. Streaming paths that cannot declare
the
complete task set remain fail closed.
- Authentication telemetry records successful credential resolution
separately from subsequent resource-authorization failures.
- API keys are high-entropy random tokens. SHA-256 is intentionally used
for deterministic indexed lookup, not password hashing.
## Deployment notes
The schema migration must be present before this code is deployed.
Because bearer resolution runs on every authenticated request, deploy
the resolver with additional-key lookup disabled, verify root-key and
public-token parity, then enable lookup before any additional keys can
be issued.
The multi-task authorization tightening changes the result for narrowly
scoped tokens that request tasks outside their grants. Observe
would-deny results before enforcing that check. Request-idempotency keys
are also newly isolated by environment and task, so a retry crossing the
deployment boundary may execute once more before old cache entries
expire.
## Follow-ups
- [x] Add a system-wide kill switch for additional-key lookup, defaulted
off for the initial deployment.
- [x] Add authentication observability by credential kind, result,
latency, and lookup path without recording credential values.
- [ ] ~Add would-deny observability and an independent enforcement
switch for multi-task authorization.~
- [ ] ~Add an independent switch for server-issued batch tokens while
root-key parity is verified.~
- [ ] Confirm every API route reachable by a restricted key has an
explicit authorization declaration or intentionally fails closed.
- [x] Verify root-key rotation, revoked-key grace, and public-token
validation through each bearer resolver path.
The pending tool line, the generic activity row and the investigation card's
own progress collapse into a single ChatProgress mounted once at the end of
the live turn: phases only swap its label (card phrase > tool phrase >
activity), decided in the pure progress-line module. ChatPendingTool is gone;
the card renders no spinner of its own; AgentSpinner has exactly one live
render site.
Restructures the Kubernetes self-hosting guide around two explicit paths
- an **evaluation install** (bundled datastores, one command) and a
**production install** (external datastores, your own secrets) - so
every configuration decision belongs to one path or the other instead of
being a flat list of options with caveats.
Also in this pass:
- Adds an architecture overview (component-to-`values.yaml` map) and a
post-install "verify it" step.
- Consolidates the previously scattered upgrade notes into a single
collapsible group, and cuts implementation detail and historical asides
that no longer apply.
- Removes a duplicated object-storage section (two configs in two
styles) and trims the Docker ClickHouse note down to what a self-hoster
needs to act on.
- AgentSpinner rests on the playlist's first shape, so mounting shows no
logo-head flash — a spinner is born spinning
- the pending indicator keeps one stable element across tool changes: the
label swaps, the animation never restarts
One component in the spinner primitives; chat progress, pending tools, the
history thinking/watching markers, panel loading, chart loading and testing
hypotheses all use it, so agent activity reads as the agent rather than
generic loading.
- cmd+J is contextual: closed opens the panel, open starts a new chat;
closing is Esc or the header's x — the New chat tooltip now shows cmd+J
(displayed once, registered once)
- in-flight tool work renders as a bare spinner line, not a bordered pill —
chips are for artifacts that stay, progress is transient
- error evidence and navigate targets normalize the API's friendly id to
the raw fingerprint, so View similar failures opens the error page
instead of 'Error not found'
- the blank state's field shows the top suggested prompt as its placeholder;
Tab drops it into the field as editable text, never sending it
- the send and stop buttons share identical square geometry
## Summary
The Queues list and queue detail pages opened on a 1 day window, and
went back to it every time you navigated between queues or reloaded.
They now default to the last hour, and the period you pick is remembered
across navigations and refreshes.
## Design
The last period is stored in a `queueMetricsPeriod` cookie, written
client-side whenever a `period` lands in the URL and read by both
loaders. A cookie rather than localStorage because the queues list
renders its per-queue metrics columns server-side: with localStorage the
page would paint the 1 hour default and then re-fetch, and the picker
would flash the wrong window.
Both pages resolve the window once, in one place, and pass it down:
```ts
period: resolveQueueMetricsPeriod({
period: value("period"), // a usable period in the URL wins
from: value("from"), // an absolute range means "no period"
to: value("to"),
defaultPeriod, // otherwise the remembered default from the loader
}),
```
That keeps the picker pill and every chart query on the same value, so
no call site falls back to its own default. Periods the picker could
never produce (a hand-edited `?period=garbage`, or a window past the 30
day retention) fall back to the default, and the picker renders the
resolved window rather than the raw search param so the label can't
disagree with the data. Absolute from/to ranges, including drag-to-zoom,
are not remembered, since they would pin later visits to a window that
has gone stale.
While wiring that up: the two queue-metric queries that go straight to
ClickHouse (the list table and the concurrency-keys endpoint) never
applied the org's `queryPeriodDays` limit, so a hand-typed `?period=`
read further back than the plan allows. Everything behind
`/resources/metric` is already clipped that way by `executeQuery`; both
of these now clip with the same limit, capped at the retention window,
and the plan cap is resolved once per load and handed to the page
instead of each route deriving its own copy from the client-side
subscription.
Verified on both pages: default with no cookie is 1 hr, picking 6 hrs
survives navigating away and back to a param-free URL and a hard reload,
clearing the cookie returns to 1 hr, an oversized period falls back
without being remembered, and an absolute range still renders as a
range.
Katia Bulatova