triggerdotdev--trigger.dev

github-featured/triggerdotdev--trigger.dev

Author	SHA1	Message	Date
Katia Bulatova	0b750d00dd	feat(webapp): dashboard agent — Watch (#4525 ) Watch is the agent noticing something later: you ask it to tell you when a condition holds, and it answers when it does — or when it can't any more. A watch is a durable one-shot promise. The condition is checked on a schedule by deterministic code (no LLM in the checks), the answer lands in the chat once, and then the watch is over. Ten kinds: three on a run, five on a queue, error recurrence, health recovery. ## Stack Stacked on #4529 (UI), which is stacked on #4418 (chat, reports, investigate). Merge those first. #4516 (storybook gallery) sits on top of this branch. ## How to review [GUIDEBOOK.md](https://github.com/triggerdotdev/trigger.dev/blob/feat/dashboard-agent-flows-watch/internal-packages/dashboard-agent/GUIDEBOOK.md) on this branch is the behaviour reference — it states the conditions rather than the code, so you can predict what happens without running anything. "The ten watch kinds, and what makes each fire" and "Creating a watch" describe exactly this PR, and the tables there are the spec the code is written against. ## What's inside - Ten watch kinds, one deterministic check each (`dashboardAgentWatchChecks.ts`), with the spec union in `dashboard-agent-contracts/src/watch.ts`. - Scheduling* — each watch schedules its own next check; due watches of one `(environment, cadence)` group can be checked together in one batch pass, with a sweep as the backstop for expiry, redelivery and retention. - Delivery — the in-chat wake and card, an optional email alert (new `DASHBOARD_AGENT_WATCH` alert channel, so it shows on the project's Alerts page with one-click unsubscribe), and an optional investigation when the outcome needs attention. - Submission ledger — `watch_submissions`, keyed `(chat_id, client_request_id)`, so a retried card submission replays the recorded outcome instead of creating a second watch. - Watch token — a delayed-execution credential accepted only by the watch endpoints, re-checked against the user's live access on every tick. - Unread work — the panel polls for wakes that landed while it was closed, so a chat can go unread and light the launcher dot. ## Key decisions A check result is a 4-way, and only two of them are verdicts. `satisfied` / `terminal_unsatisfied` are answers; `pending` and `unavailable` are not. Any exception inside any check is caught in one place and becomes `unavailable` with an unverified observation — a check that failed is never evidence. A completed window is an answer, and whether it is good or bad news is declared per kind, never inferred. There is a table for that in the guidebook: `run_failed` completing its window is good news ("hasn't failed"), `backlog_drain` completing it is not. One rule overrides the table: a window that completed on an unverified observation is neutral and says only that the watch ended without a confirmed answer. An unreadable source is never a negative answer — and, because investigations only open on `attention`, it never starts one either. Identity is `(chat, project, environment)` plus the condition, enforced by a partial unique index over active rows (`watches_chat_active_identity_key`), not by the read-then-insert check. Cadence, window, note and `ticks` are deliberately not part of it. Two different chats may watch the same thing — a watch is a promise to a chat. The server resolves the target's name, whatever the model calls it. The model can't tell a task queue (`task/<id>`) from a custom queue, so both spellings are tried and the stored one wins — and the rewrite happens before identity and before the row is written, so the identity, the checks, the link and the wording all see one spelling. Freshness fences. Depth falls back from the live counter to the newest 60 s ClickHouse bucket, which only counts as current within 60 s of now. A non-current reading at or below the quiet line is refused as `unavailable` rather than believed, so a stale empty bucket is never read as "drained". The stall streak is the one piece of carried state: it lives in the previous check's facts and freezes on an unreadable reading rather than breaking. Chain reliability. There is no shared cron — each watch (or batch group) schedules its own next tick, so the failure mode to review is the chain dying. A failed batch check is caught, the next tick is scheduled anyway and the run resolves rather than failing, so the chain survives a check that couldn't run; the sweep re-arms groups and finalizes anything still active past its deadline, even when delivery isn't configured. Wake redelivery is id-deduped rather than conditional, because the sweep can't know whether the user was already told. Access is re-authorized on every check against the primary — replica lag would extend access the user has already lost. Wording lives in one place. `watch-wording.ts` is read by the card, banner, toast, email and the agent's own narration, and the numbers come from the frozen observation rather than a fresh read, so a retry produces the same sentence. Replay reproduces the recorded decision instead of deciding again — the transcript is append-once, so a second decision would contradict it forever. Cancellation is the ending without an answer — no resolution, no wake. One exception, decided during testing: a watch the user cancelled leaves a single neutral transcript line ("Stopped watching …"), keyed off the watch id so a retry can't repeat it. The other four reasons stay silent. Email is opt-in and only a fired watch emails. An expiry is narrated in the chat and nowhere else. Both gates (agent access, a configured email transport) are checked at subscribe time and again at delivery, and the subscription outcome is frozen on the ledger row so a retry replays it. Neither gate is a plan check. One watch offer per turn. The prompt and the renderer guard this independently — if the turn already proposed a watch card, the action button is dropped, because the card is the better affordance. Two eval cases pin the prompt side: exactly one offer with the line last and the button after it, and zero offers when the rendered card already carries one — deterministic assertions, over a real-model run. ## Testing Unit tests (vitest, testcontainers, no mocks) under `apps/webapp/test/dashboardAgentWatch.test.ts` and `internal-packages/dashboard-agent/src/watch-.test.ts` cover the invariants above: the 4-way check results and the freshness fences, identity/dedup and the submission ledger, queue-name resolution, the batch chain surviving a failed check, sweep boundaries and alert-once, tenancy and the watch token's scope, and the wording snapshot. The load-bearing ones were verified by control-breaking the guard first and checking the test goes red. Live-tested end to end against a local stack, following the guidebook: all ten watch kinds firing and expiring, cancellation, the email pair (a fired watch mails, an expired one does not), and watch recovery from a health report.	2026-08-12 09:51:40 +02:00
Katia Bulatova	9a3bee0288	feat(webapp): dashboard agent — UI (#4529 ) Stacked on #4418. Merge that first. The UI slice of the dashboard agent: the side panel, the chat transport wiring, message and card rendering, suggested prompts, and chat history. #4418 works without this — the system is simply invisible. The diff is mostly components, so the notes below cover only the three decisions you can't read off the markup. Behavior and a hands-on walkthrough live in GUIDEBOOK.md, which lands with #4525. ## Decisions worth knowing - Action rows always render at the end of a turn. The model's emission order isn't trusted for layout, so action blocks are split out of the stream and appended last. Display only — `answered` stays keyed on the emission index. - The last-chat memory is org-true. It's keyed by the chat's own organization, and a foreign or deleted chat comes back as a 404 the client treats as gone, rather than an empty chat it keeps around. - A dead stream self-heals from the settled transcript. Terminal records are written to the chat row after the client's stream closes, so the panel re-reads it. The poll gate is any unfinished turn — a dangling tool part, not just an open investigation. ## Notes - Gated by `canAccessDashboardAgent`; no behavior change with the flag off. - Page marks: `handle.agentPageContext` on 47 routes, ~20 lines each. - Entry points: Ask Trigger button, ⌘J, Help & Feedback. The old ⌘I and `?aiHelp=` links keep working. ## Screenshots <img width="1440" height="788" alt="Screenshot 2026-08-07 at 15 14 29" src="https://github.com/user-attachments/assets/f4e89e8d-13ed-4be3-a88d-d5cca3ece0fa" />	2026-08-12 08:38:59 +02:00
Katia Bulatova	4569657923	feat(webapp): dashboard agent — chat, reports, investigate (#4418 ) ## What & why This is the system behind the Dashboard Agent — an assistant that answers questions about a project's runs, errors, queues, deploys and health, and can investigate failures end to end. The agent runs as a chat.agent task in its own Trigger project. It has no access to the main database or ClickHouse; all platform data is read through the public API using a delegated, read-only user token. Everything here is behind `canAccessDashboardAgent` and inert with the flag off. The UI that mounts the panel lands in #4529. ## Stack `#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook gallery. The scenario/contract reference for the whole stack is `internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch branch): it states, per feature, what makes each thing happen and where that is decided. ## What's inside Agent runtime and tools — `internal-packages/dashboard-agent`: prompt, tool set (API reads, TRQL query, docs, navigation, evidence/investigations, repo source), conversation compaction, a prompt-prefix token budget pinned by snapshot test, and sampled LLM-judged turn evals. The package cannot import webapp server code, which is what makes the "no DB access" claim structural rather than a convention. Contracts — `internal-packages/dashboard-agent-contracts`: `trigger://` URIs, intents, and the block envelope every rendered card travels in. Conversation store — `internal-packages/dashboard-agent-db`: drizzle over postgres-js in its own `trigger_dashboard_agent` Postgres schema, plus one additive migration. Auth boundary — the user-actor token gains an optional environment claim; one guard (`userActorEnvironment.server.ts`) enforces it so routes don't each re-derive the rule. Token minting, cap ceiling, and the RBAC fallback path for self-hosted. Transport — webapp resource routes that mint the token and proxy each turn, and SDK-side mid-turn reconnect. Public API the agent reads through — orgs, projects, environments, runs, queue metrics, workers, a run's commit metadata, repo snapshot, reports, and `POST /api/v1/query`. Reports — the health report's layout is declared once and shared by the card, the markdown surface and the JSON/MCP surface, so the same report reads the same in the dashboard, the terminal and an editor. Block renderers — the report and investigation cards the flows above already emit (`app/components/dashboard-agent/`). The panel that hosts them, and the rest of the chat UI, is #4529. Query safety and CSP — see below. ## Key decisions - The agent is a separate Trigger project, not webapp code. It reads platform data over the public API with a delegated user-actor token whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no webapp imports. - The PAT-only auth helper now refuses user-actor tokens. This is an intentional behavioral change: its callers consume only a bare userId and do not enforce delegated-token capabilities. Actor-aware routes continue through the scoped route builders instead. - RBAC fallback builds a delegated token's ability from its own cap, never the blanket ability a PAT gets (read-only when the token declares none). Without this, the agent's read-only cap would buy a write JWT on self-hosted. - Org creation checks RBAC only for user-actor tokens, and only after the env gate, so an install with `ORG_CREATION_API_ENABLED` off returns 404 rather than 403, and an ordinary PAT never consults an ability the route has no org to scope. Both orderings are pinned by test. - The query path is read-only in depth. TRQL rejects write statements at the grammar level (they don't parse, rather than being filtered), ClickHouse runs with `readonly=1`, and the org/project/env filters are injected server-side from the credential — the request body cannot widen scope. An unparseable query denies instead of falling through to the permissive resource. - Document-wide img-src CSP. Remote images are an outbound-request/exfiltration surface, so the policy permits only own-origin/data/blob, the required SSO avatar hosts, and the favicon endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST; wildcard hosts and bare schemes are intentionally not allowed. - The chat transport reconnects on a mid-turn EOF (`@trigger.dev/sdk`). A body that ends without a turn-complete is terminal only when the server says `X-Session-Settled: true`; otherwise the transport resubscribes from `lastEventId` with bounded backoff, and any record re-earns the budget. Previously a closed long-poll window or a proxy restart left the reply stuck as if still generating. - Conversations live in their own datastore, schema-scoped and foreign-key-free (it references `organizationId`/`userId` by id, because in cloud it is a different database). It is a display read-model for the History tab and transport resume; `chat.agent`'s object-store snapshot remains the model's source of truth. - Deterministic first. Reports and health checks contain no LLM — they are computed from the same data the dashboard shows, and the model only narrates and links them. That is what makes a number in an answer auditable. ## Testing - 63 new test files, run with `pnpm run test --filter webapp` and per-package vitest. Heaviest coverage on the auth boundary (`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`, `contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the report layout, and the SDK reconnect. - The agent package has a separate eval lane (`pnpm run test:evals`, `vitest.eval.config.ts`) that hits the real model, so it never runs in `pnpm test`. - Live-tested against a local stack scenario by scenario; the GUIDEBOOK lists the condition each behaviour is expected under, which is what those runs were checked against. ## Changelog `.server-changes/dashboard-agent.md`, plus changesets for `@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat reconnect) and the CLI's `mint-token` help text.	2026-08-11 18:56:14 +02:00
Katia Bulatova	fbd6df33b4	feat(webapp): Themes + contrast settings update (#4206 ) Adds System Preferences, Dark and Light themes, gated by the `hasThemeSwitcher` feature flag (off by default — dark stays the default theme for everyone). Old theme is now "Classic"and set as default. "System preferences" theme has both Light and Dark modes and uses your laptop settings to use a correct one. It has less color accents (specifically less colored text), and they are the same for both modes, only grayscale values change between them. And Light/Dark themes can be used separately. New Contrast setting is available for System Preferences, Dark and Light themes - it changes the contrast for the whole app. All new visual Settings live in Account.	2026-08-03 19:29:33 +02:00
Eric Allam	9d57aff542	fix(webapp): make the Queues hero charts environment-wide (#4486 ) ## Summary The four charts above the queues table aggregated over at most the 25 queues on the current page. They reused the loader's already-paginated queue array as a ClickHouse `queue IN (...)` filter, so paging or re-sorting changed the values, and a name search matching nothing blanked the whole chart row. The stat tiles above them were already environment-wide, so the two rows disagreed. They now read `env_metrics`, the environment-level rollup that already exists for exactly this (the built-in Queues dashboard and the health report read it). That is both correct and queue-count-independent: no `GROUP BY queue` across an entire environment, and no client-side summing. Note this is not only a paging artifact: page 1 under-reported too. On the seeded environment below, page 1 read 82% saturation against a true 87%, because the environment's running total is not the sum of one page of per-queue gauges. Three related fixes ride along. Scheduling delay and throttling sawed to zero. Both are event-driven, so at the 10-second bucket a short range picks, most buckets hold no samples at all and were drawn as `0ms`. Measured over a 1-hour window: 232 of 349 buckets had no scheduling-delay samples. A bucket where nothing started is not a bucket where nothing waited, so the line was both ugly and wrong. TRQL grows a `minBucketSeconds` floor, plumbed through the metric resource route, and the hero tiles set 60s. Buckets that still have no samples render as a gap instead of a dive to zero. The floor must not feed a width-dependent headline. Two of the four headlines are not peaks, so widening the plotted buckets moved them: - Throttled is a share of buckets that saw any throttling, so a single brief throttle came to mark a whole minute instead of ten seconds: the same seeded events read 17% at 10s and 85% at 60s. - Scheduling delay p95 is a percentile, and merging quantile states over a wider bucket yields a p95 between the sub-buckets' own. Two 240s samples among twenty in one 10-second sub-bucket give a worst-of-six p95 of 240,000ms against a merged 60-second p95 of 5,000ms — a 48x understatement of a headline whose tooltip claims it is the worst in the window. Both charts keep the floor, since a readable line was the point of it. Their headlines now come from a second query at the range's natural bucket width, via an optional `readout` on the tile, so each means what its tooltip says regardless of how the plotted buckets are sized. Saturation and backlog are genuinely width-invariant (a max of maxes is the same at any width), so they are unchanged and issue no extra query. Both caught by Devin in review; I had wrongly lumped p95 in with the peaks. Charts reported a hydration mismatch on every render. Recharts resolved victory-vendor's CJS entry on the server and its ESM entry in the browser. Those bundle different d3-shape builds, and the CJS one predates d3-path's digit rounding, so every server-rendered curve carried full-precision coordinates while the client rounded to 3 decimals: ``` Server: M0,3C0.9305555555555555,3,1.8611111111111112,3,... Client: M0,3C0.931,3,1.861,3,... ``` Bundling recharts for SSR makes both sides resolve the same ESM build. Verified: 45 of 45 server-rendered chart curves now match the client, and the page loads with an empty console. ## Verification An isolated stack with 40 seeded queues (20 heavily loaded, 20 idle) and 90 minutes of 10-second buckets written into `queue_metrics_raw_v1`, so the real materialized views built `queue_metrics_v1`, `env_metrics_v1` and the 5m rollup. Ground truth for the environment: 260 running against a limit of 300 (87% saturation), 800 queued. \| \| before \| after \| \| -- \| -- \| -- \| \| Saturation, page 1 \| 82% peak \| 87% peak \| \| Saturation, page 2 \| 5% peak \| 87% peak \| \| Backlog / delay, page 2 \| "No activity" \| 800 peak / 59.5s \| \| Name search matching nothing \| all four charts blank \| charts stay environment-wide \| \| Metric refetches on a page change \| 4, each painting a skeleton \| 0, no skeleton \| \| Buckets drawn as 0ms with no samples \| 232 of 349 \| 0 \| \| Throttled readout \| 17% \| 17%, unchanged by the wider buckets \| \| Worst-p95 readout source \| plotted buckets \| natural width, so a sub-minute spike is not averaged away \| \| Crosshair reach, hovering one detail-page chart \| 2 of 4 others \| 4 of 4 \| \| SSR chart curves mismatching the client \| 45 \| 0 \| The bucket floor was measured across ranges: it widens 10s to 60s at 30m and 1h, and is correctly a no-op at 12h (300s) and 7d (3600s). One extra request per page load, for the throttled readout. The built-in Queues dashboard, which reads `env_metrics` independently, agrees at 86.7% and 260 of 300. `internal-packages/tsql` suite green (612 tests), including 5 new ones for the floor that fail without it. Webapp typecheck, oxfmt and oxlint clean. Spot-checked the Run metrics dashboard and the per-queue detail page for SSR regressions from bundling recharts: both render, console clean. The queue detail page carries the same event-driven series, so its scheduling delay, throttling and per-key mean delay take the same treatment. ## Screenshots <img width="2540" height="580" alt="after-page1-charts" src="https://github.com/user-attachments/assets/6cd23f9c-e7fd-4918-bcfa-b1d3340b16d1" /> ## Rollout Already behind the per-organization `queueMetricsUiEnabled` flag, so only gated orgs see any of it. Blast radius is chart values on one page plus the SSR bundling of recharts; rollback is a revert with no data migration. ## Stated limitations - `wait_ms_count` and the quantile state both only count `wait_ms > 0`, so "nothing started in this bucket" and "everything started instantly" are indistinguishable in storage. Both render as a gap. Distinguishing them needs a schema change, which is not in this PR. - The queue name search deliberately no longer narrows the charts. It only did so incidentally and incorrectly before (first 25 matches, and blanked on zero matches). Search-scoped charts would need the full unpaginated matching set and a server-side aggregate; worth its own ticket if we want it. - Bundling recharts for SSR grows the server bundle slightly. That is the cost of both sides resolving one d3-shape build. - The plotted delay line is a smoothed 60-second view, so a sub-minute spike above the one-minute warning threshold can fail to colour the line even though the headline reports it and colours itself. - Every chart inside one synced group shares the floor, because the hover crosshair is a reference line on a category x-axis and only draws where the hovered bucket exists in the other chart's own data. That costs the queue detail page's gauges some resolution (1 minute instead of 10 seconds) in exchange for the crosshair working across the row. Separately, while taking the screenshots I found a pre-existing rendering bug unrelated to this change: a perfectly flat saturation series draws no line at all (the readout still shows the right percentage), which looks like the threshold gradient's offset degenerating when the series min equals its max. It reproduces on `main`, so it is not a regression here and I have left it alone; filed as its own issue. Refs TRI-12784	2026-08-03 16:19:50 +01:00
Eric Allam	5f29ae49ab	feat(webapp): default the queue metrics period to 1 hour and remember it (#4438 ) ## Summary The Queues list and queue detail pages opened on a 1 day window, and went back to it every time you navigated between queues or reloaded. They now default to the last hour, and the period you pick is remembered across navigations and refreshes. ## Design The last period is stored in a `queueMetricsPeriod` cookie, written client-side whenever a `period` lands in the URL and read by both loaders. A cookie rather than localStorage because the queues list renders its per-queue metrics columns server-side: with localStorage the page would paint the 1 hour default and then re-fetch, and the picker would flash the wrong window. Both pages resolve the window once, in one place, and pass it down: ```ts period: resolveQueueMetricsPeriod({ period: value("period"), // a usable period in the URL wins from: value("from"), // an absolute range means "no period" to: value("to"), defaultPeriod, // otherwise the remembered default from the loader }), ``` That keeps the picker pill and every chart query on the same value, so no call site falls back to its own default. Periods the picker could never produce (a hand-edited `?period=garbage`, or a window past the 30 day retention) fall back to the default, and the picker renders the resolved window rather than the raw search param so the label can't disagree with the data. Absolute from/to ranges, including drag-to-zoom, are not remembered, since they would pin later visits to a window that has gone stale. While wiring that up: the two queue-metric queries that go straight to ClickHouse (the list table and the concurrency-keys endpoint) never applied the org's `queryPeriodDays` limit, so a hand-typed `?period=` read further back than the plan allows. Everything behind `/resources/metric` is already clipped that way by `executeQuery`; both of these now clip with the same limit, capped at the retention window, and the plan cap is resolved once per load and handed to the page instead of each route deriving its own copy from the client-side subscription. Verified on both pages: default with no cookie is 1 hr, picking 6 hrs survives navigating away and back to a param-free URL and a hard reload, clearing the cookie returns to 1 hr, an oversized period falls back without being remembered, and an absolute range still renders as a range.	2026-08-03 10:09:34 +01:00
Eric Allam	4eb9292cbe	feat(webapp,run-engine): queue metrics and health dashboard (#4131 ) ## Summary Three related changes, each independently gated: Queue metrics and health. Per-queue depth, throughput (enqueued, started, completed), concurrency, whether a queue is throttled, and scheduling delay (how long a run waits between becoming eligible and actually starting), plus a per concurrency-key breakdown for keyed queues. Collected from inside the run queue itself, stored in ClickHouse, and surfaced on the Queues list, a new per-queue detail page, the task pages, and the run inspector. The question it answers is "does this queue have enough concurrency to keep up, and if not, which key or which limit is the constraint". Percent-based queue concurrency limits. A queue's concurrency override can now be expressed as a percentage of the environment limit, stored as the source of truth and re-materialized whenever the environment limit changes. Absolute overrides above the environment limit are now rejected with a 400 instead of being silently capped, which is a behavior change on `POST /api/v1/queues/:queue/concurrency/override`. The `health` report. A server-computed verdict on whether work is flowing, whether the runs that do start are healthy, and whether telemetry is fresh, rendered as text with sparklines. Available as `GET /api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool (plus a `report` MCP prompt, which shows up as a slash command in hosts that support prompts). With the flags off, the Queues page renders the pre-metrics component verbatim, nothing is emitted, and nothing is written to ClickHouse. ## Configuration Two independent gates, on purpose. Emission is global so data accrues for everyone before anyone can look at it; the view is per organization so it can be turned on for one org at a time without a deploy. Runtime flags (no restart) \| Flag \| Store \| Gates \| \| --- \| --- \| --- \| \| `queue_metrics:enabled` \| run-queue Redis key (`"1"`/`"0"`, off by default) \| All emission, gauges and counters. Cached in-process for 10s with stale-while-revalidate, warmed eagerly at boot so the first op after a deploy is not dropped. \| \| `queue_metrics:gauge_sample_rate` \| run-queue Redis key, `0..1` \| Fraction of queue ops that emit a gauge. Counters are never sampled, so throughput stays exact at any rate. \| \| `queueMetricsUiEnabled` \| feature-flag catalog: global `FeatureFlag` row, per-org `Organization.featureFlags` override wins \| Whether an org sees the metrics view at all: the Queues list variant, the queue detail route, the built-in Queues dashboard, the concurrency-keys endpoint, and the metrics blocks on task pages and the run inspector. Off by default; a gated org gets a 404 on the detail route rather than an empty page. \| Both Redis keys are readable and writable from `/admin/queue-metrics` (super-admin UI, with a live per-shard stream-health table) and `GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface uses its own Redis client, so it works on any instance regardless of whether that instance runs the emitter or the consumer. Environment variables (boot time) \| Variable \| Default \| Notes \| \| --- \| --- \| --- \| \| `QUEUE_METRICS_EMIT_ENABLED` \| `0` \| Constructs the emitter and injects it into the run engine. Without it the run queue has no emitter at all. \| \| `QUEUE_METRICS_CONSUMER_ENABLED` \| `0` \| Boots the stream consumer on this instance. Independent of emission, so consumers can be sized separately from the API. \| \| `QUEUE_METRICS_STREAM_SHARD_COUNT` \| `4` \| Stream shards, hashed per queue. \| \| `QUEUE_METRICS_CONSUMER_BATCH_SIZE` \| `1000` \| Poll batch equals insert batch, so an ack can never outrun a write. \| \| `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` \| falls back to the run-queue Redis \| Set `HOST` to move the metrics stream onto a dedicated instance so a metrics backlog cannot compete with the run queue for memory. Self-hosters can leave it unset and get a single-Redis deployment. \| \| `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` \| `2000000` shared, `8000000` dedicated \| Bound on how much a stalled consumer can hold. The default is deliberately lower when the stream shares the queue-critical Redis. \| \| `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` \| `604800` \| TTL on the per-queue cumulative counter key, refreshed on every write, so only queues idle for the whole window are purged. \| \| `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` \| `1000` \| Distinct queue names tracked per environment; overflow collapses into `__overflow__`. \| \| `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` \| `10000` \| Same idea one level down, per queue. \| \| `QUEUE_METRICS_GAUGE_SAMPLE_RATE` \| `1` \| Default for the live sample-rate key above. \| \| `QUEUE_METRICS_QUERY_TABLES_VISIBLE` \| `0` \| Lists the queue-metrics tables in the Query page, its schema docs, the schema API and the AI query context. Off keeps them unlisted while the feature is dark; a query naming them still runs either way. \| \| `QUEUE_METRICS_CLICKHOUSE_URL` \| falls back to the shared wiring \| Runs queue metrics on their own ClickHouse service: the consumer's inserts and every queue-metrics read go through it, so a metrics-heavy chart refresh never competes with runs-list or trace reads. Unset reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`, reads on the query pool). \| \| `QUEUE_METRICS_CLICKHOUSE_READER_URL` \| the write URL \| Reader split, so the consumer's inserts can never land on a read endpoint. \| \| `QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}` \| `1`, unset, `10`, `info`, `1` \| Pool tuning, matching the other per-workload ClickHouse clients. \| Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a Postgres migration adding the nullable `TaskQueue.concurrencyLimitOverridePercent`. Both are additive. ## How collection works Queue operations produce two kinds of signal, and they have opposite failure modes, so they are handled differently. Gauges (queued, running, queue limit, env queued, env running, env limit, throttled, plus keys-with-backlog and worst-key wait on keyed queues) are read inside the same Redis script that performs the enqueue or dequeue, so the reading is atomic with the operation it describes rather than a racy follow-up read. The script returns them on its reply and the app forwards them to the stream. Gauges are sampled and drop-tolerant: they are aggregated with `max`, so a lost reading costs resolution, never correctness. Counters (enqueued, started, completed, plus nack and dead-lettered) are cumulative odometers. Each event increments a per-queue key on the metrics Redis and emits the absolute total, and ClickHouse takes the difference across buckets at read time. This is the important property of the design: a summed-delta counter undercounts permanently on any lost event, while a cumulative one self-heals, because the next surviving reading restates the whole total. Only bucket granularity can be lost, never the total. A queue returning after its odometer TTL expired restarts at 1 and reset detection handles it, which is safe precisely because expiry only spans a window with no activity. Both land on one sharded Redis stream. A consumer reads it with a consumer group, reclaims stale pending entries on a 15s interval rather than on every poll, maps one entry to one or two ClickHouse rows (whole-queue and, for keyed queues, per-key), and acks only after the insert lands. Each batch carries a dedup token derived from its stream-entry ids, and the target tables set `non_replicated_deduplication_window`, so a retried batch cannot double-count either the raw rows or the aggregates that hang off them. Consumer and emitter both emit OTel metrics (`queue_metrics.emitter.emitted`, `queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`); stream depth and group lag are the two worth alerting on, and `lag_unknown` exists because Redis can report a null lag after a trim, which must not be read as zero. ## Storage and read path `queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four aggregate tiers are materialized straight from raw, never cascaded off each other, each with a 30 day TTL: - `queue_metrics_v1`, 10 second buckets per queue, the default read path - `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and cross-queue ranking - `env_metrics_v1`, 10 second buckets per environment, queue-independent so it stays cheap at any range - `queue_metrics_ck_v1`, 10 second buckets per concurrency key Every tier is an MV from raw because the counter states do not survive a cascade: their merge is order sensitive, so a `-MergeState` chain off the 10s table inflates the result, and the same property means an aggregate state may only be merged inside one queue. That constraint is now enforced by the query engine rather than by reviewer discipline: a column can declare a `mergeGroupKey`, and any query that references it without grouping by, or pinning to a single value of, every named key fails to compile with an actionable message. On the read side, TRQL gains three tables (`queue_metrics`, `env_metrics`, and a `queue_metrics_by_key` that is hidden from the editor, schema docs and schema API but still queryable, so per-key rows can never silently merge into a plain per-queue query), plus `deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level optimizations ride along: a table can declare coarser rollups, so a query whose bucket interval is 5 minutes or wider is routed to the 5m table with no change to the query itself, and it can opt into the ClickHouse query cache with time bounds floored to a fixed grid, so the auto-refreshing dashboards actually share cache entries instead of missing on every tick. Both are caller-side substitutions, so the printer stays unaware of physical layout. All of this can also live on its own ClickHouse service. A table declares the pool its reads run on, the three queue-metrics tables name the dedicated one, and the ingestion consumer writes through the same client, so both directions move together with one env var and nothing else routes differently. The other engine change is opt-in gap filling: charts can request rows for empty buckets, where counters zero-fill and gauges carry forward. Grouped gauge series are densified per group and carried inside a partition, so a quiet queue's line holds its last value without bleeding another queue's value into it. ## Queue concurrency limits `concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth when an override is set as a percentage; the absolute `concurrencyLimit` is materialized from it (floored, clamped to at least 1 so a percentage can never act as a pause, and never above the environment limit). Every path that changes an environment limit now recalculates the environment's percent-based overrides afterwards, outside the transaction, and pushes changed limits to the engine. The push is attempted even when the stored value did not change, so a previously failed sync self-heals rather than leaving the database and the engine diverged; paused queues are skipped so a recalculation cannot effectively unpause one. The API accepts exactly one of `concurrencyLimit` or `percent`, and the reject-instead-of-clamp change above means a request asking for more than the environment allows now fails loudly. The percent bound (greater than 0, at most 100) is defined once and shared by the zod schema, the dashboard mutation handler and the service, so the three cannot drift. The concurrency-keys table on a queue is now paginated against the ClickHouse per-key tier, ranked by peak backlog with the total on every row from a single scan, and only the keys on the current page are enriched with live counts from Redis. That replaces a hard top-50 cap with something whose cost is a function of page size rather than key cardinality. ## The health report `GET /api/v1/reports/:key?period=&format=markdown\|ansi\|json`. The verdict is computed on the server and is deterministic, not model-generated. Three independent analyzers run over one input snapshot: flow (is work moving, and if not, is the cause a limit, throttling, one bad queue, or dead-lettering), execution (are the runs that start succeeding, and at what latency), and liveness (how fresh is the telemetry). When telemetry is genuinely stale, the first two are forced to unknown and every actionable field is stripped, so no surface ever advises action off stale data. Authorization is per query table rather than a blanket query grant: a JWT must be scoped to every table the report reads (`runs`, `env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull a report that reads more than it was granted. `period` is validated as a shorthand with a 90 day ceiling at the edge. The report catalog is a registry of `{ load, interpret }` entries, so the next report is a new entry and no change to the route, the view model, the renderers, the CLI or the MCP tool. `trigger mcp` no longer launches the install wizard when stdout is a TTY, which fixed a real failure: hosts spawn the server over a PTY, so the wizard would open and the client would time out waiting for a server that never started. The wizard now needs `trigger mcp --install`. ## The part that is live regardless of every flag The enqueue and dequeue scripts now return a 2-tuple so a gauge reading can ride back on the reply. Every return site in the eight affected scripts is wrapped, and a `nil` original is converted to `false` on the way out, because a raw `nil` in the first slot would make Lua truncate the multi-bulk reply and silently drop the gauge on the throttled and empty-queue paths. The reply shape and the destructuring on the app side are exercised on every queue operation whether or not metrics are enabled, so that is the part of `run-engine` worth the closest review. One behavior fix in the same area: the scheduling-delay anchor is set only on a run's first entry into the queue. Anchoring it to trigger time on re-enqueues made waitpoint and checkpoint resumes report the entire wait as scheduling delay. Queue ordering is untouched, so a re-enqueued run keeps its position, and nacks deliberately keep the original anchor because a rolled-back dequeue is the same continuous wait. A pending-version promotion still anchors to trigger time, on purpose: that promotion is the run's first real entry into the queue, since the trigger deliberately held it back waiting for a worker version, and the TTL is armed at the same point for the same reason. The consequence is worth naming, because it is a judgement call: a run that waits on a deployment reports that wait as scheduling delay on its queue, which is time unrelated to queue capacity. ## Verification Unit and integration suites across the new package, the run queue, the mapping layer, the query engine and ClickHouse (including a test that applies migration 036 through the same splitter CI uses, and a regression test that inserts the same batch three times to prove the aggregates do not inflate). Beyond that, the whole path was driven end to end against a live stack with real runs: emitter to Redis stream to consumer to ClickHouse to the dashboards, for both the local dev path and the deployed path where a supervisor drives the dequeue, with assertions on exact counter reconstruction per queue and per concurrency key, throttling, environment saturation, scheduling delay, and a deliberate mid-stream reading drop to confirm the cumulative counters still reconstruct the correct total. The gated-off state was checked on every touched surface. The dedicated ClickHouse service was verified against a second, separately-schema'd instance: with it configured, the driven counters reconstruct exactly on the dedicated instance, the shared instance gains no rows for that window, a read through the query API returns the value that exists only on the dedicated instance, and a `runs` query still succeeds (it would fail outright if it were mis-routed to a service without that table). With the variable unset, the full suite passes unchanged. --------- Co-authored-by: Katia Bulatova <katia@trigger.dev> Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com> Co-authored-by: James Ritchie <james@trigger.dev>	2026-07-29 16:45:24 +01:00

7 Commits