github-featured/triggerdotdev--trigger.dev
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity
7,951 Commits 310 Branches 4,167 Tags
main
Commit Graph

878 Commits

Author SHA1 Message Date
Daniel Sutton 920892bc11 feat(webapp,run-store): gen-2 shard arms in read-through and idempotency (#4781) 
Gives read-through and idempotency their gen-2 shard arms, so an id that
names its own shard is read there and nowhere else.

#4764 has landed, so this now targets `main` directly and no longer
depends on an unmerged branch. It builds on what that PR supplied:
`resolveShard`, `runOpsShardHandles` and the keyed router.

TRI-13431

## What changes

**Read-through routes by `resolveShard`, not by the binary residency
classifier.** A gen-2 id reads its own shard's replica once and probes
no other store. A gen-1 v1 id still reads new only.

**Callers now declare `idKind`.** A cuid gives no way to tell a run id
from a waitpoint id, and the two must route differently:

- a legacy-classified **run** id reads the legacy replica only — there
is no cuid run migration, so the new-store probe cannot find it;
- a cuid **waitpoint** keeps the new-first pair probe, which is
load-bearing because a cuid waitpoint can be co-located with its run on
the new store.

There is no default, because a default would pick one of those arms
silently. The field `runId` is renamed to `id`, since it carried both
kinds already.

**`ReadThroughResult` carries `found`.** `source` is an open-ended union
once shards exist, so a consumer testing found-ness by listing the hit
sources reads a gen-2 hit as a miss. One consumer did exactly that.
Discriminating on `found` makes that class of bug a compile error rather
than something a reviewer has to spot.

**Idempotency resolves its client through one shard-keyed map.** Both
call sites go through `clientForShardKey`, so they cannot disagree about
which store owns an id. An absent key takes an explicit logged branch to
the fallback, not a silent legacy default. The `classify` seam is
retyped to return a `ShardKey`: `Residency` (`"NEW"`) and the reserved
shard keys (`"new"`) differ only by case, and `ShardKey` collapses to
`string`, so the compiler would not have caught feeding one into the
other.

The dead `isMigrated` branch is deleted. Nothing implemented it, and the
one production comment recorded that omitting it was deliberate.

**`PostgresRunStore._residency` widens to `ShardKey`.** Still unused;
the store stays unaware of its siblings.

## Two behaviour fixes found while doing the above

**An unconfigured shard key logs and returns not-found instead of
throwing.** The waitpoint route takes the id from a URL parameter, and
any base32hex core plus `[a-z0-9]` plus `"2"` parses as gen-2. The route
turns a throw into a 500, so throwing here would let any authenticated
client generate 500s and error logs by guessing shard chars, of which
there are 36. An error-logged not-found is neither silent nor a
misroute. Throwing stays correct on the router path, where ids are
minted rather than received.

**The two cross-seam batch hydration sites were gen-2 blind.**
`hydrateRunsAcrossSeam` and `ApiBatchResultsPresenter` classified with
the binary `ownerEngine`, so a gen-2 run id joined the gen-1 `new`
group, missed there, and — classifying dedicated-family — never reached
the legacy probe either. The id was dropped from a bulk-action page and
from batch results with no error. Both now partition ids by shard key
and read each configured shard once.

Also: a gen-2 waitpoint that missed its shard replica fell back to the
gen-1 new writer, a different database, silently disabling
read-your-writes for the freshly minted token that fallback exists to
serve. It now falls back to its own shard's writer.

## Merge safety

Inert while `RUN_OPS_SHARDS` is unset: the shard maps are empty, so
every gen-2 arm is unreachable, and gen-2 minting is not live yet.

The one live change is the gen-1 run arm, and it removes work rather
than adding it. `RoutingRunStore.findRun` never forwards the caller's
client object — it routes by id and reads only the client's presence and
replica brand — so `readRunForEvent`'s "new" closure already resolved a
legacy-classified run id to the legacy store. The arm removes a
duplicated read of the legacy replica. A test pins this, because a
future caller passing a raw client and a run id would lose the
pre-cutover 27-char case, which is new-resident but classifies legacy.

## Testing

14 tests added, testcontainers throughout, no mocks. 22 affected test
files pass; typecheck, lint, format and knip are clean.

Both arms were verified by neutralising them and confirming the new
tests fail. The batch-results test needed rewriting after that check:
the first version passed with the fix neutralised, because it used one
container as both the gen-1 new client and the shard replica, so it was
not testing what it claimed.

Note for review: run testcontainer suites in small batches. Sixteen at
once starves Docker and everything times out at 60 seconds.

The run-ops legacy-guard baseline is refreshed in its own commit. The
baseline is keyed by line number, so partitioning the batch-results read
shifted four pre-existing entries and added one. Baselined violations in
that file go from four to five, all reads; the new one is the shard read
beside two gen-1 reads already there.

No changeset and no `.server-changes` entry: a user notices nothing
while the flag is unset.
 2026-08-26 16:46:48 +01:00
Oskar Otwinowski 4c16387426 fix(webapp): project integrations page — Staging gating, unreachable code, and follow-ups (#4784) 
Three bugs on the project integrations page, one commit each for the two
reported ones and four for the follow-ups found while fixing them.

## `chore`: remove unreachable code on the integrations page (TRI-12645)

Two notification panels in `VercelSettingsPanel` could never render:

1. The **"Failed to load Vercel settings"** panel was gated on a
`hasError` state whose setter is never called anywhere, so it was
permanently `false`.
2. The **"connection expired"** banner *inside* the `connectedProject`
branch was unreachable: `VercelSettingsPresenter` only populates
`connectedProject` on its success exit, which hardcodes `authInvalid:
false`, while both `authInvalid: true` exits return `connectedProject:
undefined`.

Removing them makes the surrounding `!showAuthInvalid` guards vacuous,
and the `onboardingData?.authInvalid` disjunct redundant — the loader
already folds onboarding auth state into `authInvalid` before it reaches
the component.

**No behaviour change.** An org with a connected project and an expired
token still gets the banner, from the branch below (untouched).

## `fix`: gate Staging settings on plans without a Staging environment
(TRI-12646)

The ticket's premise was inverted, and I've corrected it there. In Git
settings, **Preview** is the row that's correctly gated; **Staging** is
the one with no gate at all:

- Preview swaps its switch for an Upgrade button, and
`projectSettings.server.ts` neutralises a forged
`previewDeploymentsEnabled=on`.
- Staging was a plain always-editable `Input`, and
`validateStagingBranch` only checked the branch existed on GitHub. An
org without a staging environment could type a tracking branch, hit
Save, get a success toast, and have it silently do nothing.

Staging and Preview environments are created together for projects on a
plan that includes them, so gating one and not the other was an
oversight.

The Staging row now mirrors the Preview row. Server-side it ignores the
submitted branch when there's no staging environment, but **preserves
the stored branch rather than clearing it** — deliberately different
from the Preview handling. Forcing a boolean off is harmless; forcing a
*string* off would wipe a tracking branch the org had already configured
the first time they saved after losing the environment.

The Vercel write path had the same gap: `update-config` /
`complete-onboarding` / `update-env-mapping` never re-derived available
env slugs server-side, so `["stg","preview"]` could be persisted for a
project with neither environment, and
`createDefaultVercelIntegrationData` turned preview on unconditionally.
Both now filter against the project's actual environments, via a pure
`restrictConfigToAvailableEnvSlugs` helper that only touches keys
present on the input.

## `fix`: show build settings when the GitHub app is disabled
(TRI-13488)

The page wrapped Git settings, the Vercel section **and** build settings
in one `githubAppEnabled` guard, so with the GitHub app off it rendered
an empty container.

The Vercel section genuinely depends on GitHub — it can't sync
environment variables or link deployments without a connected repo — so
it stays gated. Build settings don't: they also apply to CLI deploys run
with `--native-build-server`, exactly as the section's own description
states. They now render regardless.

## `fix`: stop the Vercel onboarding modal spinning forever (TRI-13488)

`computeInitialState` starts in `loading-projects` whenever the org has
a Vercel integration but no onboarding data yet, and the effect that
escapes it waits for `availableProjects !== undefined`. When
`getOnboardingData` returns `null` — it does that on any thrown error,
and when the org integration row is missing — nothing ever arrives.

The empty-array case self-resolves (`[] !== undefined`), so this is
specifically the null case. The route can tell "still loading" from
"loaded nothing" because its fetcher always requests
`?vercelOnboarding=true`; it now passes that down and the modal explains
the failure with a retry and a link to check the integration's access on
Vercel.

## `fix`: match staging and preview environments consistently
(TRI-13488)

The four places that ask "does this project have a staging / preview
environment?" disagreed. `VercelSettingsPresenter` matched on type with
no parent filter, so any preview *branch* row satisfied it — branches
are `PREVIEW` rows too. `GitHubSettingsPresenter` and
`ProjectSettingsService` matched on slug instead.

Slug is the weaker key: it's derived at creation time and legacy rows
can carry something else, which is why
`memberDevelopmentEnvironmentWhere` deliberately avoids it. All four now
match on `type` plus `parentEnvironmentId: null`, which excludes
branches without depending on the slug being canonical.

## `fix`: explain when no Vercel environment can be mapped to Staging
(TRI-13488)

Reported while reviewing the branch. The Staging build settings show
*"Set a Vercel environment for Staging first."* whenever the project has
a staging environment and no mapping — but the control that sets the
mapping only rendered when the Vercel project had at least one custom
environment:

```
hint:     hasStagingEnvironment && !configValues.vercelStagingEnvironment
control:  hasStagingEnvironment && customEnvironments.length > 0
```

So a Vercel project with no custom environments, or one whose custom
environments failed to fetch (the presenter swallows that error to
`[]`), got an instruction with nothing to act on. Both conditions
predate this PR.

The mapping row now always renders alongside the hint and explains what
to do when there's nothing to choose from, and the build-settings hint
says the same thing.

## `chore`: remove the remaining dead code (TRI-13488)

- The `"installing"` `OnboardingState` is unproducible — no `setState`
call yields it — so its redirect effect, switch arm, `isLoadingState`
conjunct and the `vercelAppInstallPath` import it was the only user of
are all dead.
- `(state as string) !== "completed"` sits in a branch where TypeScript
has already narrowed `"completed"` out; the cast is what let it compile.
- `hideSectionToggles` was only ever passed alongside
`layout="settings"` but only read inside `layout="card"` blocks, so it
could never take effect. Removed the prop entirely.
- Unused bindings and the helpers only they referenced: `envSlugLabel`,
`_formatSelectedEnvs`, `_CompleteOnboardingForm`,
`_handleFinishOnboarding`, and the rest.

No behaviour change in that commit.

## Not included

The three overlapping modal-open effects in
`settings.integrations/route.tsx` are left alone — they're defensive
against a close-then-reopen race, and untangling them is a behavioural
risk with no user-visible payoff.

## Verification

`pnpm run typecheck --filter webapp`, `pnpm run lint` and `pnpm run
knip` are clean. New `apps/webapp/test/vercelIntegrationConfig.test.ts`
covers the slug restriction and the default-config seeding (both pure
functions); 39 tests pass across it and the three existing
Vercel/project-settings files.

The new `projectId` + `slug` query is served by the existing
`@@unique([projectId, slug, orgMemberId])` prefix — same access pattern
as the preview check it mirrors.

refs TRI-12645, TRI-12646, TRI-13488
 2026-08-26 13:26:44 +00:00
Daniel Sutton 1801b0e80b feat(webapp,docker): run-ops boot interlocks and migrations at N databases (#4780) 
## Summary

The run-ops boot interlocks and the migration entrypoint each assume
exactly two run-ops
databases. This generalizes them to any number, so a deployment that
configures
`RUN_OPS_SHARDS` gets the same safety guarantees it gets today with two
stores: no two stores
may point at one database, every store that owns its own database must
replicate to
ClickHouse, and every store must have its schema migrated.

With `RUN_OPS_SHARDS` unset, nothing changes. The distinctness check
over a two-element set is
the pairwise compare it replaces, replication coverage is the check it
was, and the entrypoint
runs the same two migration invocations.

A shard may declare `aliasOf: "new"`, which shares an existing store's
client by reference. An
aliased shard is not its own database, so it is exempt from the
distinctness check and needs no
replication slot of its own. Every check keys that exemption on the
declared field, never on
client object identity: two client objects can sit over one database,
which identity comparison
cannot see.

## Design

**Distinctness.** `probeDistinctDatabases` compared two URLs. It now
delegates to
`probeDistinctStores`, which reads every fingerprint in parallel and
groups them by system
identifier and database name. Any two stores under one key refuse the
boot. The old pairwise
entry point stays, so its existing container tests are the proof that
set uniqueness over one
pair gives the verdict it gave before. Fail-closed is unchanged: a probe
that cannot answer
returns not-distinct, because "distinct" is a positive claim a failed
probe cannot support.

**Co-residency.** The advisory runs once per store against the control
plane. The legacy
emission keeps its exact call shape and its untagged metric series, so
an existing dashboard
does not change. Each shard emits its own point carrying its shard key.
Every store emits
before any enforcement throw, so one offending store never costs another
store its metric.

**Replication.** `buildReplicationSources` appends one source per shard
that owns its own
database, taking the slot, publication and origin generation its
descriptor declares.
`assertReplicationCoversSplit` then requires a source per such shard.

That check also closes a hole it inherited. The descriptor parser
validates uniqueness among
shards only, so a shard could take the slot name, publication name or
origin generation of the
legacy or the new source. The replication service does validate this,
but it throws from its
constructor, and the caller reaches that constructor only after shutting
the bootstrap instance
down:

```ts
if (sources.length > 1) {
  await service.shutdown();                       // legacy stream stops here
  service = new RunsReplicationService({ ... });   // throws: duplicate slotName
}
```

The throw was not a `SplitReplicationMisconfiguredError`, so the process
stayed up with no
replication at all, legacy included, behind one logged line. That is the
silent ClickHouse
under-count the error exists to prevent. The check now runs at the boot
gate, before anything is
torn down, and raises a subclass the existing exit path already
recognizes. A correct deployment
already satisfies it, because two consumers on one WAL slot is a data
race that cannot work.

**Migrations.** Every shard runs the identical schema, so a new shard is
the existing migrations
against a new DSN. The runner image has no `jq`, so a small node script
prints one DSN per line
and the entrypoint loops over them. The loop is a `for` and not a `while
read` pipeline: a
pipeline subshell swallows a failed migration on any iteration but the
last, which would let a
broken shard boot. Tracing stays off across the capture and the loop,
because `set -x` prints an
assignment and a DSN carries credentials.

Verified end to end against real Postgres containers for the fingerprint
probes, and against the
real shell block with a stubbed migration command: an aliased shard is
skipped, `directUrl` wins
over `url`, a failing shard stops the container on the first failure,
and a malformed descriptor
stops it before it migrates anything.

Stacked on #4764.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-08-26 13:50:20 +01:00
Eric Allam 47ff76d727 feat(webapp,clickhouse): return an actionable error instead of a 500 when a runs list query is too expensive (#4773) 
## Summary

When a runs list query is too expensive to complete, it now fails with a
clear, actionable error instead of a generic 500.

Previously, a runs list query that exceeded ClickHouse resource limits
threw an opaque error. On the public `runs.list` API that surfaced as a
retryable 500, so a customer task calling it would keep retrying a query
that could never succeed. On the dashboard it rendered as a generic
error page with no hint about what to do.

## Fix

The ClickHouse client now tags resource-limit failures (memory, time,
rows, bytes) with their error type, and the runs repository maps those
to a dedicated `RunsListQueryError` (HTTP 422).

- `runs.list` API returns 422 with a message telling the user to narrow
their `created_at` range, plus an `x-should-retry: false` header so the
SDK does not retry it.
- The dashboard runs list (and the errors, scheduled, standard-task,
agents, and webhooks list views) render a shared error state with the
same guidance, so a too-broad time filter is recoverable by the user.
 2026-08-25 14:49:10 +01:00
Saadi Myftija 1eda438a41 feat(webapp): put the admin dashboard behind an env var flag (#4774) 
Adds an `ADMIN_DASHBOARD_ENABLED` env var (default: enabled) that turns
the admin dashboard and user impersonation off for an entire instance.

When disabled:
- every admin dashboard page redirects away, and the admin navigation
isn't rendered
- existing impersonation cookies are ignored, and any lingering session
is actively terminated with an audit record
- every flow that could start an impersonation responds 404, and no
impersonation tokens are minted

Stopping an impersonation always works regardless of the flag, so
nothing gets stuck. Machine-to-machine admin API endpoints are not
affected. The variable is documented for self-hosters; instances that
don't set it are unaffected.
 2026-08-25 15:37:43 +02:00
Eric Allam 11e1cd8174 feat(webapp): isolate the runs list ClickHouse read pool (#4763) 
## Summary

Improves the performance and reliability of the runs list and the
`runs.list` API, especially for large projects and filtered views.

## What changed

- **Filtered runs-list queries use `PREWHERE`.** Immutable and
additive-only filters (tags, task identifier, version, queue, region,
machine, and the rest) are applied in `PREWHERE` on the `task_runs_v2
FINAL` scan, so ClickHouse filters, and uses the tags skip index, before
it reconciles versions and materialises the wide columns. Same results,
far less memory per query. `status` stays in `WHERE`: it changes across
a run's versions, so filtering it before `FINAL` could return stale
rows.
- **The runs-list ClickHouse pool gets per-query guardrails**, all
env-configurable: a `max_execution_time` paired with the client request
timeout, a per-query `max_memory_usage`, a `max_threads` cap, and
`readonly`. Each bounds a single query to itself, so a heavy query can't
affect other queries, and they are safe as pool-level settings only
because this pool is read-only.
- **Billing and bulk count reads move to the read pool**, off the write
pool.

Defaults are conservative for self-hosters; production values are set
via env.
 2026-08-25 09:19:48 +01:00
Saadi Myftija f866210388 feat(cli): experimental --local-bundle deploy mode (#4331) 
Adds an experimental `--local-bundle` flag to native build deployments:
the project is installed and bundled on the local machine (exactly like
in the depot path) and only the resulting build context is uploaded. The
remote build then runs just the container image build.

### Design

- The uploaded artifact is the same build context classic deploys
produce: bundled output, a synthesized package.json with the resolved
externals, build.json, and the generated Containerfile. The bundle is
secret-free: build.json is deliberately scrubbed because it is copied
into the image, and build-arg values never enter the bundle at all.
- Build-arg values are sent with the deployment initialization request
instead, stored encrypted (aes-256-gcm) in a new
`WorkerDeployment.buildEnvVars` column, and cleared on every terminal
status transition. They exist at rest only for the active build window,
always encrypted.
- A dedicated `GET /api/v1/deployments/:id/build-env-vars` endpoint
returns the decrypted values to the same principals that can already
read the environment's variables. It answers with an empty record for
deployments without stored values or in a terminal state, keeping secret
access to a single auditable route.
- Size limits are enforced server side and pre-checked client side. If
the server does not acknowledge storing the values, the CLI fails fast
instead of letting the remote build run without them.
- A `--from-bundle <dir>` mode builds a deployment image straight from
such a bundle directory, skipping config loading and bundling entirely.
In attach mode it fetches the stored build-arg values through the new
endpoint.
- Env var syncing (the `syncEnvVars` extension) happens client side,
before the deployment initializes, since the remote side never sees the
unscrubbed manifest.
- Bundle artifacts use a distinct type and storage prefix so the server
can always distinguish them from source uploads.
 2026-08-25 09:50:49 +02:00
nicktrn b082e44389 fix(webapp): write-path and appearance-control fixes for the theme work (#4756) 
Fixes found while reviewing #4547, stacked on that branch so they can be
reviewed on their own and merged into it. One commit per fix.

## Write-path correctness

**Refuse account writes while impersonating.** The five
`dashboardPreferences` writers already no-op for an impersonating admin,
but the three profile writers added next to them did not, and
`requireUserId` returns the impersonated user's id. Both gates now
refuse up front and say so, rather than the preference writers silently
no-opping while the page reports success.

**Preserve unknown keys on a full-blob write.**
`mutateDashboardPreferences` parses the JSON column, hands the result to
a mutator and persists the whole object back. zod strips keys it does
not declare, so a deploy that predates a preference field drops it on
the next write through that path — and
`updateCurrentProjectEnvironmentId` sits on the navigation hot path.
`preserveUnknownKeys` re-attaches them at the write. Note this cannot
help deploys already running, so it makes this the last release able to
strip rather than retroactively protecting the fields added in #4547.

**Scope hidden-sidebar writes to what was shown.** The customize dialog
builds its hidden map from the sections it can see and the write
replaced `hiddenItems` wholesale. The profile page has no org in scope,
so it resolves sections from the most-recently-updated project's org:
confirming there dropped hidden ids belonging to sections that org's
flags exclude. The payload now carries the ids the dialog rendered and
the write only replaces those. Submissions without the list stay
authoritative.

**Consider both addresses when checking email ownership.** The check
only looked at the address the user already had; it now considers the
current and submitted address together, so an org managing either one
governs the change. Validation moved ahead of the check, and
`emailDomainOf` splits on the last `@`.

## Interaction

**Revert unsaved themes, debounce contrast saves.** The theme and
system-theme selects stamp `data-theme` before the write lands. When it
fails, the loader returns the value it always had — so
`useSystemThemeSync`'s effect deps are unchanged and React's vdom diff
sees no change either, and nothing rewrites the attribute. The page kept
rendering a theme that was never stored while the select showed the
stored one. The stored pair is now re-applied explicitly, as the side
menu's switcher already did. The contrast slider is debounced because
Radix commits on every arrow keypress, so a keyboard user crossing the
range fired one write per step.

**Tick More options for themes outside the short list.** The appearance
submenu offers System, Light and Dark; Black and White live on the
profile page. With one of those stored, every row read as unselected.

## Subtraction

**Drop the profile update rate limiter.** It covered one of four paths
that write the same column — `resources.preferences.sidemenu` and
`.favorites` take unlimited authenticated writes and go through the
locked read-modify-write, which is more expensive than the single narrow
`jsonb_set` this capped. It was also what made the contrast slider
unusable by keyboard. If preference writes want limiting, it belongs in
one place covering all of them.

**Resolve email ownership when the dialog opens.** It fans out one SSO
status lookup per organization the user belongs to and ran in the
profile loader on every page view, purely to pick which body the dialog
renders. The action re-derives it before writing either way, so the
check that guards the write now has one call site instead of two.

## Testing

`typecheck --filter webapp` and `lint` clean. New unit tests for
`preserveUnknownKeys`, `mergeHiddenItems` and `emailDomainOf`;
`themePreference`, `mergeHiddenItems` and `ssoManagedIdentity` suites
pass locally (26 tests). The rest of the webapp suite needs
testcontainers and is left to CI.

No changeset or `.server-changes` entry: everything here fixes code on
the parent branch that has not shipped. The one exception worth a
maintainer's call is `mergeHiddenItems`, which also touches the side
menu's own customize path.
 2026-08-21 19:27:52 +01:00
James Ritchie 4c5237ca4a feat(webapp): themes refinement, new black & white themes, 2 accessibility toggles (#4547) 
## What this does

Rounds out the theme work behind the existing `hasThemeSwitcher` flag.

**Two new themes.** Black and White sit alongside Dark and Light. They
inherit their neighbour's whole token set and only pin their surfaces
flat, so sections are separated by grid lines rather than layered fills.

**`System` is now configurable at both ends.** You choose which theme
the OS light setting lands on (Light or White) and which the dark
setting lands on (Dark or Black).

**Two accessibility toggles.**
- *Stronger colors* — swaps tinted status chips for solid fills, drops
decorative icon accents to monochrome, and darkens chart series that
didn't clear 3:1 on a white plot.
- *Underline links* — underlines body-text links, so an underline always
means the preference is on rather than being a hover style.

**Contrast slider.** Stores a 0–100 position within the active theme's
own range rather than a shared scale, so 35% stays 35% when you switch
themes. Each theme maps it in CSS, which keeps `system` working before
hydration.

**Appearance in the account popover.** A submenu listing the themes with
a check against the current one, plus a link through to the full set on
your profile. Picking one applies immediately rather than waiting for
the write to round-trip.

**Profile page.** Each row now saves on its own — no submit button. Name
and email show their value inline with an edit button; the email row is
read-only when an identity provider owns the address.

**A `/storybook/colors` audit page.** Renders every colour-carrying
pattern in the app once per theme plus once under Stronger colors, and
measures contrast ratios off the live DOM rather than a hard-coded
table, so it can't go stale.

---

## Demo


https://github.com/user-attachments/assets/d56cd4d8-719f-4ec5-a990-e04cdb98def1


---


## Compatibility

The stored preference shape is unchanged (`version: "1"`), and the four
new fields are all optional. The retired `classic` theme falls back to
Dark, whose palette at contrast 0 is what Classic shipped.

One deliberate change worth knowing: the default contrast moves from 50
to 0, so existing users who never touched the slider will see slightly
less contrast than before. That's what makes 0 mean "the base palette".

---

## Testing

Switched between every theme from both the account popover and the
profile page, in the expanded and collapsed rail, checking `data-theme`
follows and survives a reload. Dragged the contrast slider in each theme
and confirmed the percentage label tracks the handle and resnaps if a
save fails. Checked both accessibility toggles across the
`/storybook/colors` page, which is also where the contrast ratios were
read from. Confirmed the Appearance entry stays hidden for a non-admin
while the flag is off.

<!-- conductor-workspace-link -->

---

[Open workspace in
Conductor](https://app.conductor.build/workspace/fee50611-7623-4422-bada-ed1cba317ed1)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-21 19:27:52 +01:00
Oskar Otwinowski 910011d44e feat(vercel): automatic version skew protection at connect + atomic deployments deprecation (#4741) 
Connecting a Vercel project now writes
TRIGGER_AUTOMATIC_SKEW_VERSION_PROTECTION=1
(plain, create-if-absent only - an existing value, including "0", is
never
touched; presence is target-containment aware, branch-scoped records do
not
count, a truncated env listing skips the write). The onboarding wizard
no
longer offers automatic atomic deployments (default off); the settings
row is
labelled Deprecated and enabling it requires confirming a dialog that
points
to task version skew protection and the docs (TRI-13001).
 2026-08-21 18:09:27 +02:00
Eric Allam 32bf745c02 feat(webapp): customizable runs list with columns and smart columns (#4652) 
## Summary

Makes the runs list customizable. A new **Display** control lets you
show, hide, and reorder columns, and add **smart columns** that pull a
single value out of a run's payload, metadata, or output by JSON path
(e.g. `$.failed`, `$.order.total`). Column choices live in the page URL,
so a view can be bookmarked or shared. Applies to the global runs list
and every per-task / scheduled / agent / webhook / error list, which all
share one table.

ID, Task, and Status can be reordered but not hidden. Smart columns are
display-only (no sort or filter, which would defeat the ClickHouse sort
key and cursor).

## How it works

Columns come from a shared registry; the Postgres `select` is derived
from the visible columns, so a run's large payload/output are only
hydrated when a smart column actually references them. All JSON parsing
for smart columns happens client-side, respecting the packet content
type, parsed once per source per row. Offloaded (too-large) values and
paths that aren't present render distinct placeholders rather than
fetching per row. The live poll carries the same sources so smart-column
values update in place.

Scalar columns stay always-selected for now: the shared list presenter
has a fixed output shape consumed by several routes and the live poll,
and narrowing individual scalar fields would add no real query cost
benefit on a single-row read. The select derivation is already
column-driven, so tightening this later is a one-line change.

## Screenshots

<img width="590" height="1028" alt="CleanShot 2026-08-21 at 16 48 17@2x"
src="https://github.com/user-attachments/assets/86b39856-bfcc-47c0-85ed-ee6ccddc3590"
/>
<img width="1924" height="1528" alt="CleanShot 2026-08-21 at 16 48
27@2x"
src="https://github.com/user-attachments/assets/6c766249-6d5b-45be-9330-c6caa75af7f7"
/>


<!-- conductor-workspace-link -->

---

[Open workspace in
Conductor](https://app.conductor.build/workspace/d6911080-2140-4de1-b88a-1b0623593caa)

---------

Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-21 17:08:12 +01:00
DKP d04467018e feat(webapp,database): save platform notifications as drafts and publish later (#4743) 
## Summary

The platform notifications admin page can now save a notification as a
draft without committing to a schedule, then publish it later by
entering start and end dates. Drafts stay hidden from the webapp panel,
the CLI, and the "What's new" changelog until they are published.

## Design

A draft is an `isDraft` flag on `PlatformNotification`, not nullable
dates, so the existing index and every read query stay intact. All three
reader queries filter on the flag, so a draft can never surface
regardless of its placeholder dates. Publishing writes the real start
and end dates and clears the flag; the publish dialog validates the
range and shows inline errors. Editing a draft keeps it a draft, with
the schedule fields hidden until publish.

Also folds in a small tweak: the "Send preview to me" test button now
appears when editing a notification, not just when creating one.
 2026-08-20 22:16:32 +01:00
Eric Allam 9baebbd1a6 fix(webapp): keep the dashboard agent's tool calls on the user's instance (#4740) 
## Summary

Follow-up to #4738. Splits the dashboard agent's base URL into two: the
instance that hosts the agent project (used for sessions), and the
instance the agent acts against as the user (used by its read-tools).
#4738 only needed the first, but moved the second along with it, which
breaks the tools when the agent runs on a different instance than the
webapp.

## Root cause

The agent's read-tools call the API as the logged-in user via a
delegated user-actor token. The webapp signs that token with its own
`SESSION_SECRET`, scoped to its own `userId` and `environmentId`, so it
can only be verified by, and only resolves the user's data on, that same
instance. #4738 routed the injected `apiOrigin` those tools use to the
agent's host instance, so the token no longer verifies and the data
isn't there.

## Fix

`dashboardAgentApiOrigin()` stays the agent's host instance (sessions,
task triggers, realtime, the `in` forward). A new
`dashboardAgentUserApiOrigin()` returns the webapp's own origin
(`API_ORIGIN ?? APP_ORIGIN`) and is injected into the run metadata the
tools use. Same-instance deployments resolve both to the same host, so
behavior is unchanged there.
 2026-08-20 15:27:27 +01:00
Eric Allam 56f875680c fix(webapp): let the dashboard agent use a configurable base URL (#4738) 
## Summary

Lets the dashboard agent point at a specific Trigger instance instead of
assuming it runs on the same instance as the webapp. Adds an optional
`DASHBOARD_AGENT_BASE_URL`; when unset it falls back to the SDK default.

## Root cause

The agent's session start, token mint, head start, in-proxy and the
client transport all built the agent's base URL from the webapp's own
origin (`API_ORIGIN ?? APP_ORIGIN`). That only holds when the agent
project runs on the same instance as the webapp. When it runs elsewhere,
`DASHBOARD_AGENT_SECRET_KEY` belongs to that other instance, so the
webapp's own API rejects it with an "Invalid API key" and the chat can't
start.

## Fix

`dashboardAgentApiOrigin()` now returns `DASHBOARD_AGENT_BASE_URL` or
the SDK default, never the webapp origin. A concrete default (rather
than an unset value) keeps it independent of `TRIGGER_API_URL`, which a
webapp may point at a different host. Every server call site already
routes through that helper; the client transport reads the value from
the root loader via a new `useDashboardAgentBaseUrl` hook.
 2026-08-20 14:14:14 +01:00
Chris Arderne 06f99aeb31 fix: security release 2026-08-12 (#4735) 2026-08-20 12:34:33 +01:00
Oskar Otwinowski 8b0385c429 feat(run-engine): trigger tasks pinned to an external deployment id (#4664) 
The SDK discovers an external deployment id at runtime (explicit
TRIGGER_EXTERNAL_DEPLOYMENT_ID always; platform commit-SHA variables and
generic fallbacks when TRIGGER_AUTOMATIC_SKEW_VERSION_PROTECTION=1) and
sends it alongside lockToVersion; the server resolves precedence
(version > external id > current). An id held by a deployed deployment
pins the run to that worker; an in-flight or unknown id parks the run in
PENDING_VERSION with the id in TaskRun.annotations, wakes it pinned when
a deployment carrying the id finalizes (ClickHouse candidates, Postgres
authoritative), and expires it after a deadline that re-checks Postgres
before acting. Parking outranks delaying and preserves delayUntil. The
id is projected to ClickHouse task_runs_v2.external_deployment_id during
replication. Redis cache for id-to-worker resolution, guarded
version-aware writes.

Ids are not unique. Several deployments can hold one id - a --force
rebuild is the ordinary way to get there - so resolution always picks
the highest version among the candidates, never the newest by timestamp.
The rule is applied identically on both paths that can bind a run to a
worker: resolveExternalDeployment at trigger time, and
PendingVersionSystem when a landing deployment wakes a parked run.
Version comparison is numeric on the counter half, so 20260807.10
outranks 20260807.9.

A run whose id never lands expires at the deadline with
EXTERNAL_DEPLOYMENT_NOT_FOUND and an error naming the id it waited for,
which is what a failed build or a typo looks like from the caller.
Default deadline is one hour (EXTERNAL_DEPLOYMENT_PARK_DEADLINE_MS).

Debounce registration happens in both the parked and the delayed branch
through one helper, so a debounced run that parks still binds its
debounce key; without it every later trigger for the same key created
another parked run, and all of them executed when the deployment landed.
The two DELAYED-only status checks in DebounceSystem also accept
PENDING_VERSION, without which the lock-contention fallback would
rethrow a 5xx the SDK retries and amplifies, and the fast path would
push every trigger on a parked key through the redlock.

Resolution is skipped in development. A dev environment cannot hold a
WorkerDeployment - trigger dev registers a BackgroundWorker with nothing
behind it, and deploy --env refuses dev - so an external deployment id
there could only ever park, and the parked run then expired against the
dev TTL while a connected dev worker sat idle. The id is still annotated
so the dashboard shows what the app sent (TRI-13000).
 2026-08-19 17:43:53 +02:00
Chris Arderne 4dabfca1d5 feat(webapp,cli,core): list production project runtime updates (#4659) 2026-08-19 13:44:55 +01:00
Chris Arderne f4320937c5 chore: prefer direct iteration and function callback types (#4677) 
## Summary

Enable lint rules that prefer direct iteration and concise function
callback types.

The existing code now uses direct iteration where no index is needed,
and callback contracts use function types consistently.

Base: [#4675](https://github.com/triggerdotdev/trigger.dev/pull/4675)
 2026-08-19 08:28:58 +01:00
Chris Arderne fe1d5f6961 chore: enable additional correctness lint rules (#4672) 
## Summary

Enable additional lint rules that catch unsafe optional-chain
assertions, inherited-property iteration, anonymous symbols, and unsafe
external links.

The existing violations now use explicit values and own-property checks,
so the rules can prevent those patterns from returning.
 2026-08-19 08:28:56 +01:00
Chris Arderne b4313c8199 feat: logs search v2 (#4615) 2026-08-18 14:59:46 +01:00
Katia Bulatova e768d0a724 feat(webapp): run the dashboard agent through AWS Bedrock behind an env switch (#4609) 
## What & why

The dashboard agent can now run its model calls through AWS Bedrock
instead of the direct Anthropic API, chosen by a single env switch. It's
**off by default** (`DASHBOARD_AGENT_MODEL_PROVIDER` unset ⇒
`anthropic`), so merging changes nothing at runtime — the Bedrock path
is a dormant branch until an operator sets the switch and AWS config.
The default Anthropic path is byte-for-byte unchanged.

This also carries a related tenant-isolation hardening for the agent's
delegated token (kept together deliberately — both land the agent on
Bedrock for HIPAA readiness). Refs: TRI-13251, TRI-11032.

## What's inside

**Provider seam** —
`internal-packages/dashboard-agent/src/model-provider.ts`: the registry
now holds both `anthropic` and `bedrock`; `resolveDashboardAgentModel()`
maps the canonical `"anthropic:<id>"` strings the managed prompts carry
to the active provider, and the cache-breakpoint helpers emit the active
provider's shape — Anthropic `cacheControl` vs Bedrock `cachePoint`.
Managed prompt strings stay canonical, so stored prompts don't change
meaning. Unmapped model ids throw rather than shipping a guaranteed-404
profile. All agent, watch, compaction and title callsites route through
the resolver; the `dashboardAgentModelKey` locals override (test mock
injection) is preserved.

**Cache telemetry** — `step-cache.ts`: cache token usage is read from
the active provider (Anthropic reports it on provider metadata; Bedrock
reports the write on metadata and the read via standard usage), so
`gen_ai.usage.cache_*` is populated on both. This also fixes a latent
ordering bug where step attributes could null-overwrite the prompt-cache
read count.

**Webapp callsites** — `dashboardAgentHeadStart.server.ts` and the
head-start route resolve the model and the cache breakpoint through the
same seam, so the warm-up prefix and the following turn share one
provider. The head-start firing gate is provider-aware: on Bedrock it
gates on `AWS_REGION` and lets the SDK resolve credentials (IAM role /
static keys / session token / bearer), so a role-based deploy still
warms; on Anthropic it stays `Boolean(ANTHROPIC_API_KEY)`.
`app/env.server.ts` gains the optional AWS vars and validates
`DASHBOARD_AGENT_MODEL_PROVIDER`. `ANTHROPIC_API_KEY` is untouched and
not required on a Bedrock deploy.

**Tenant-isolation hardening** —
`internal-packages/rbac/src/fallback.ts`: for a **scoped** context, the
OSS `authenticateUserActor` now applies the same membership floor as the
session path — a delegated user-actor token whose user is not a member
of the scoped org/project is denied (403). Unscoped tokens keep their
prior behavior (no tenant claim, no lookup). The user lookup falls back
replica→primary so replication lag can't spuriously 401 a just-joined
member. Members and admins are unaffected. Previously this invariant
held only through per-route discipline; this makes it structural.

## Enabling Bedrock (later, ops)

- Set `DASHBOARD_AGENT_MODEL_PROVIDER=bedrock` **identically** in both
the webapp and the agent task container — the webapp warms the cache
prefix and the task reads it, so a split would silently miss the cache.
- Set `AWS_REGION` and provide credentials the Bedrock SDK can resolve
(IAM role preferred). For v1 this runs **without** an Anthropic API key.
Note: with no Anthropic key set, rollback is "turn the agent off", not
"unset the switch" (unsetting falls back to the Anthropic provider,
which then has no key).
- Two things to confirm before rollout: the Sonnet inference-profile id
is validated against the SDK's own model-id union but still warrants a
live smoke test; and Bedrock prompt caching for Sonnet is a 5-minute
window (not Anthropic's 1h), so input-token cost rises when flipped.

## Testing

Unit tests cover both provider paths: the provider switch and
per-provider cache shapes, a structural regex asserting Bedrock ids are
real inference profiles (not an echo of the table), the split-metadata
cache telemetry, and real-Postgres RBAC tests — member allowed, scoped
non-member denied (org-only and project-only), missing user → 401, admin
non-member exempt, unscoped success. `typecheck --filter webapp` and the
dashboard-agent + rbac suites pass.
 2026-08-18 13:14:01 +02:00
Chris Arderne b33197691b chore: enforce no unused deps or code in ci (#4654) 2026-08-18 11:35:51 +01:00
Eric Allam c0b84595a3 feat(webapp): hosted webhook ingress, delivery pipeline, and dashboard (#4344) 
## Summary

The server half of hosted webhooks: the public ingress endpoint,
signature verification, the delivery pipeline (Postgres partitioned
storage + ClickHouse for ordering), the in-app partition manager, the
HTTP API, and the dashboard (Deliveries, Endpoints, and the in-app test
console).

The public SDK and docs half is #4537. That PR carries the user-facing
API (`webhook()`, `chat.event` / `chat.channels`, the
`@trigger.dev/slack` connector) and builds on the shared
`@trigger.dev/core` schemas that ship here.

## Shipping behind a flag

A `WEBHOOK_ENABLED` env var (default off) gates the public ingress route
and the engine worker plus partition cron, so merging and deploying this
changes nothing in production until it is flipped on per environment.
The dashboard is separately gated per org by the `hasWebhooksAccess`
feature flag.

## Note on packages

This PR includes the `@trigger.dev/core` schema additions the server
compiles against, but carries no changeset. Core is not consumed
independently of the SDK, so it is released together with the SDK via
#4537. Keeping its changeset off `main` means no release cut from `main`
publishes it early.
 2026-08-16 14:33:42 +01:00
Eric Allam fe199f7f92 perf(webapp): aggregate admin notification interaction counts in the database (#4616) 
## Summary

The notifications admin list loaded every interaction row for the
notifications on the current page just to show three per-notification
counters (seen, clicked, dismissed), then counted them in memory. On
notifications with many interactions this made the page slow to load and
heavy on memory, even though only 20 notifications are shown.

## Fix

Compute the counters in a single grouped aggregate in the database
instead, returning one row per notification rather than one row per
interaction:

```sql
SELECT "notificationId",
  COUNT(*) AS seen,
  COUNT(*) FILTER (WHERE "webappClickedAt" IS NOT NULL) AS clicked,
  COUNT(*) FILTER (WHERE "webappDismissedAt" IS NOT NULL OR "cliDismissedAt" IS NOT NULL) AS dismissed
FROM "PlatformNotificationInteraction"
WHERE "notificationId" IN (...)
GROUP BY "notificationId"
```

Behavior is unchanged; notifications with no interactions report zero.
 2026-08-14 11:59:46 +01:00
Chris Arderne 3e7964e7fa feat: surface cron windows in webapp, cli, sdk (#4572) 
## Summary

Adds execution-window product surfaces for both declarative and
imperative schedules.

- Declarative schedules can set `window` through `schedules.task()`,
with support for whole-minute, hour, and percentage values.
- Imperative schedules can create, update, clear, and inspect windows
through the API and dashboard.
- Schedule API responses preserve `nextRun` as the nominal CRON time and
expose `nextRunEffectiveAt` as the stable assigned time.
- The dashboard displays configured windows alongside assigned
upcoming-run times.
- Deploy output summarizes declarative schedules and suggests adding a
wider window when the default 60-second placement range is used.

## Design

Window validation remains authoritative on the server and ensures each
window is compatible with the schedule cadence. Omitting a window uses
the default 60-second range, while explicit zero-duration windows remain
supported.

Deployment summaries are derived from the deployment's stored task
metadata, so they reflect the declarations associated with that
deployment.
 2026-08-14 10:07:14 +01:00
Katia Bulatova ee854480fe fix(webapp): dashboard agent maintenance moves into the agent project (#4599) 
## What & why

The dashboard agent's upkeep — retention deletes and the investigation
sweep — ran as cron jobs on the webapp's common worker, even though it
only touches the agent's own datastore. This moves that upkeep into the
agent's Trigger project as scheduled tasks (TRI-13182).

## What's inside

**Retention** — `internal-packages/dashboard-agent/src/maintenance.ts`,
a daily task (03:00 UTC). Deletes turn evals older than 30 days,
hard-deletes chats soft-deleted more than 30 days ago, and purges
terminal watches and submission rows older than 7 days. It used to run
every 5 minutes; nothing needs a hard delete that fast, so it is daily
now, draining in bounded batches and warning if it hits the cap. It
retries (3 attempts) because the next run is a day away. It connects
with `DASHBOARD_AGENT_DATABASE_URL`, falling back to `DATABASE_URL` like
every other task in the package (the deletes are confined to the agent's
own Postgres schema), and skips when neither is set.

**Investigation sweep** — `src/investigation-sweep.ts`, every 5 minutes,
same as before: settles investigation cards stuck `in_progress`
(30-minute window, attempt cap, force-abandon note). It keeps the fast
cadence because it fixes live state the UI is showing.

**What stays in the webapp.** The watch finalize/deliver sweep and batch
rearm: they cover a dead agent-side tick chain — a backstop can't live
inside the thing it backstops — and they need the main database and the
alerts worker. The org-deletion chat purge also stays: deletion must not
depend on the agent project being deployed. The removed cron job keeps a
cron-less tombstone entry so already-queued items drain cleanly; remove
it in a follow-up.

**Test plumbing** — the drizzle migration replayer that webapp tests
hand-rolled is now exported once from
`@internal/dashboard-agent-db/testing`; the moved tests live in the
agent package as `src/*.test.ts` against real Postgres.

## Testing

Agent package: retention passes (backlog drain, batch cap, no-op guard,
chat-delete cascade) and the sweep, on testcontainers Postgres. Webapp:
the watch/chat suites, plus a test that a settlement card stops the
dashboard spinner. Full typecheck on both.
 2026-08-13 13:13:02 +02:00
Katia Bulatova 480bede0ad feat(webapp,sdk): dashboard agent plan enforcement, component gallery — and fixes (#4516) 
Plan enforcement for the dashboard agent — message quota and watch
limits — plus the component gallery, fixes and test hardening from the
same stack (#4548, #4549, #4550, #4552, #4556 merged here).

## Plan enforcement
([TRI-12863](https://linear.app/triggerdotdev/issue/TRI-12863))

**Agent message quota.** The Free-plan allowance becomes a real
server-side limit with a durable counter. New `agent_message_usage`
table keyed `(organization_id, period)` — deliberately not joined to
chats, so deleting a chat can't free quota within the period. Both send
paths count one user message (wakes never count) and refuse at the cap
with `403 message_quota_reached`, which the client renders as an upgrade
panel, never a silent drop. The refusal code is a single shared constant
on both sides.

**Watch limits.** A watch whose window exceeds the plan's
`agentWatchMaxHours`, or that would push the org past its
`agentWatchers` count, is refused with `watch_limit_reached` (409 on the
API, an upgrade hint on the card). Plan limits only tighten the existing
code ceilings (`min(plan, 24h)`, per-chat cap of 3 still applies). A
plan limit of zero means zero, not unlimited. Questions answerable
instantly are answered before any plan refusal — a one-shot consumes no
slot and never sees an upgrade nag.

**Fails open by design.** Cloud ships the actual per-plan numbers
separately (TRI-12863 P0). Until then absent limits resolve to the
unlimited sentinel and the upgrade UI is gated on billing presence —
self-hosted sees no cap, no upsell, with tests proving the fallback.
Both quotas are nudges, not security boundaries: a failing limit read
never blocks a send.

## Component gallery

An admin-only gallery of every agent card state: five
`storybook.agent-*` pages (chat UI, view blocks, report, investigation,
watch) with their shared shell and manifest, demo fixtures, two
demo-only cards, toast examples, and the screenshot script. No LLM and
no data — every state renders from fixtures under
`dashboard-agent/demo/`, never reachable from a production path.
Designers and reviewers can look at every state, including the report
states, without seeding anything.

## And fixes

**SDK: watch-mode chat subscriptions survive quiet windows** (TRI-13065,
TRI-13070) — watch mode keeps reconnecting across empty long-poll
windows and only stops on abort or a settled session; a passive
subscriber can no longer stop a turn it doesn't own (`stopOnAbort` is
explicit, default off). Review findings fixed alongside: a superseded
stream's async teardown no longer removes the live successor's abort
controller or multi-tab claim, and stopping a generation hands the chat
back to the user's other tabs.

**Query boundary pinned end-to-end**
([TRI-11165](https://linear.app/triggerdotdev/issue/TRI-11165)) — a
route-level test drives `api.v1.query` with a real signed environment
JWT (writes refused before ClickHouse, a read passes); `readonly=1` made
non-overridable; a per-turn cap stops the model burning a turn rewriting
a query it can't fix (deterministic SQL errors only — busy/transport
rejections don't count).

**chat.agent durability regression suite**
([TRI-11166](https://linear.app/triggerdotdev/issue/TRI-11166)) —
testcontainers-backed coverage of the two audit criticals (cross-tenant
isolation, no duplicate mid-stream turn, both control-broken) plus
crash-resume, cursor-based refresh, clean rollback of a mid-write turn
failure (torn by a real constraint violation), and OOM-restart replay.

**Investigation sweep backoff** — stale investigations get an attempt
counter and backoff so a poison row can't pin the sweep queue head
(migration `0005`: `sweep_attempts`, `last_sweep_attempt_at`).

## Screenshots

<img width="1440" height="791" alt="Screenshot 2026-08-06 at 00 36 19"
src="https://github.com/user-attachments/assets/6a68cd42-8580-469d-afe7-e28d1eef18e1"
/>

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-08-12 13:59:35 +02:00
Chris Arderne ed1bb72fb8 feat: implement cron window spread backend (#4566) 
- New DB fields on Schedule and ScheduleInstance
- Use `queueTimestamp` for the "effectiveAt" delayed start time,
propagate it to Clickhouse TaskRun table
- Disable fastpath for delayed jobs
- Add schedule timing logic, API endpoints with windows, persistence
- Calculate phase for every schedule, only persist when window is
non-null
- Additional o11y for phased rollout
 2026-08-12 12:24:32 +01:00
Matt Aitken c2c6e5c705 fix(webapp): keep session runs off the legacy realtime streams backend (#4564) 
## Summary

Runs created for a Session were triggered without a realtime streams
version, so they fell through to the `realtimeStreamsVersion` column
default of `v1`. A Session's own `.in` / `.out` channels are always
`v2`, so any run-scoped `streams.append()` or `streams.pipe()` call made
inside a session run wrote to a different backend than the session it
belongs to, and stayed there for the life of the run.

The API trigger routes were never affected. They call
`determineRealtimeStreamsVersion` with the client's
`x-trigger-realtime-streams-version` header and always pass an explicit
value, so a current SDK asking for v2 gets it. Only the internal callers
that build trigger options by hand were leaning on the column default,
which no env var can influence because that path never calls the
resolver at all.

## The version resolver

Fixing the call site exposed a second problem in
`determineRealtimeStreamsVersion`. Its two paths disagreed: an explicit
`v2` was checked against the S2 configuration first, but when the caller
expressed no preference it returned `REALTIME_STREAMS_DEFAULT_VERSION`
verbatim with no check. A deployment that set the default to `v2`
without configuring S2 therefore stamped runs `v2`, nothing failed at
trigger time, and every later read or write against those runs' streams
threw `Realtime streams v2 is required for this run but S2 configuration
is missing` for the life of the run.

Both paths now resolve through one pure function that takes its
configuration rather than reading `env`:

```ts
const requested = streamVersion ?? config.defaultVersion;
if (requested !== "v2") return "v1";

const hasCredentials = Boolean(config.accessToken) || config.skipAccessTokens;
return hasCredentials && Boolean(config.basin) ? "v2" : "v1";
```

## The basin requirement

`resolveStreamBasin` resolves run, session and organization basins ahead
of the global setting, so a deployment that provisions a basin per
organization can serve v2 with no global basin at all. Gating purely on
the global setting would degrade every run there to `v1`.

`determineRealtimeStreamsVersion` therefore takes an optional
organization basin, and every caller that holds one passes it, including
the session path:

```ts
basin: organizationBasinName ?? env.REALTIME_STREAMS_S2_BASIN,
```

This is deliberately the resolved basin and not the
`REALTIME_STREAMS_PER_ORG_BASINS_ENABLED` flag. The flag says the
feature is on, not that a given organization has been provisioned, and
provisioning happens out of band. Keying off the flag would stamp `v2`
on runs for unprovisioned organizations, recreating the failure this
removes.

**This widens behaviour for explicit `v2` requests**, which previously
required the global basin: a provisioned organization on a per-org
deployment now resolves `v2` where it used to get `v1`. That is
intentional, and it makes every path agree.

## Scope

Only newly created runs change. A run already stamped `v1` keeps that
version for its lifetime by design, since readers resolve the backend
from the same column and its existing streams have to stay readable.
Scheduled runs reach the same column default through
`scheduleEngine.server.ts` and are deliberately left alone: that one is
a policy question about `REALTIME_STREAMS_DEFAULT_VERSION` rather than
an inconsistency inside a single feature.

## Verification

A full-stack e2e boots the real webapp plus Postgres, Redis and s2-lite,
creates a Session through the public API so the run comes from the real
trigger path, appends records the way `streams.append()` does, and
asserts three things at once: the version stamped on the run, that the
payload is readable from S2, and that no key exists in Redis. It appends
at a realistic record size so the route's body cap and S2's per-record
cap are both exercised. Reverting the session-path change flips all
three observations, so it fails against the old behaviour rather than
passing vacuously.

Unit tests cover the resolver matrix, including organization-basin-only
and credential-only configurations; two of them fail against the
previous resolver.

Also verified by hand against a local stack: a real `chat.agent` session
run writing 8 records of 250KB through `streams.append()` put 2,049,072
bytes into S2 with no Redis key, while the same agent with the
session-path change removed put 2,102,360 bytes into Redis and nothing
into S2.
 2026-08-12 11:01:59 +01:00
Chris Arderne 7b390e5984 feat(cli,webapp): allow deploys with environment API keys (#4561) 2026-08-12 10:11:31 +01:00
Katia Bulatova 0b750d00dd feat(webapp): dashboard agent — Watch (#4525) 
Watch is the agent noticing something later: you ask it to tell you when
a condition holds, and it answers when it does — or when it can't any
more.

A watch is a **durable one-shot promise**. The condition is checked on a
schedule by deterministic code (no LLM in the checks), the answer lands
in the chat once, and then the watch is over. Ten kinds: three on a run,
five on a queue, error recurrence, health recovery.

## Stack

Stacked on **#4529** (UI), which is stacked on **#4418** (chat, reports,
investigate). Merge those first. **#4516** (storybook gallery) sits on
top of this branch.

## How to review


[**GUIDEBOOK.md**](https://github.com/triggerdotdev/trigger.dev/blob/feat/dashboard-agent-flows-watch/internal-packages/dashboard-agent/GUIDEBOOK.md)
on this branch is the behaviour reference — it states the conditions
rather than the code, so you can predict what happens without running
anything. "The ten watch kinds, and what makes each fire" and "Creating
a watch" describe exactly this PR, and the tables there are the spec the
code is written against.

## What's inside

- **Ten watch kinds**, one deterministic check each
(`dashboardAgentWatch*Checks.ts`), with the spec union in
`dashboard-agent-contracts/src/watch.ts`.
- **Scheduling** — each watch schedules its own next check; due watches
of one `(environment, cadence)` group can be checked together in one
batch pass, with a sweep as the backstop for expiry, redelivery and
retention.
- **Delivery** — the in-chat wake and card, an optional email alert (new
`DASHBOARD_AGENT_WATCH` alert channel, so it shows on the project's
Alerts page with one-click unsubscribe), and an optional investigation
when the outcome needs attention.
- **Submission ledger** — `watch_submissions`, keyed `(chat_id,
client_request_id)`, so a retried card submission replays the recorded
outcome instead of creating a second watch.
- **Watch token** — a delayed-execution credential accepted only by the
watch endpoints, re-checked against the user's live access on every
tick.
- **Unread work** — the panel polls for wakes that landed while it was
closed, so a chat can go unread and light the launcher dot.

## Key decisions

**A check result is a 4-way, and only two of them are verdicts.**
`satisfied` / `terminal_unsatisfied` are answers; `pending` and
`unavailable` are not. Any exception inside any check is caught in one
place and becomes `unavailable` with an unverified observation — a check
that failed is never evidence.

**A completed window is an answer, and whether it is good or bad news is
declared per kind, never inferred.** There is a table for that in the
guidebook: `run_failed` completing its window is *good* news ("hasn't
failed"), `backlog_drain` completing it is not. One rule overrides the
table: a window that completed on an unverified observation is neutral
and says only that the watch ended without a confirmed answer. **An
unreadable source is never a negative answer** — and, because
investigations only open on `attention`, it never starts one either.

**Identity is `(chat, project, environment)` plus the condition,**
enforced by a partial unique index over active rows
(`watches_chat_active_identity_key`), not by the read-then-insert check.
Cadence, window, note and `ticks` are deliberately not part of it. Two
different chats may watch the same thing — a watch is a promise to a
chat.

**The server resolves the target's name, whatever the model calls it.**
The model can't tell a task queue (`task/<id>`) from a custom queue, so
both spellings are tried and the stored one wins — and the rewrite
happens **before** identity and before the row is written, so the
identity, the checks, the link and the wording all see one spelling.

**Freshness fences.** Depth falls back from the live counter to the
newest 60 s ClickHouse bucket, which only counts as current within 60 s
of now. A non-current reading at or below the *quiet line* is refused as
`unavailable` rather than believed, so a stale empty bucket is never
read as "drained". The stall streak is the one piece of carried state:
it lives in the previous check's facts and *freezes* on an unreadable
reading rather than breaking.

**Chain reliability.** There is no shared cron — each watch (or batch
group) schedules its own next tick, so the failure mode to review is the
chain dying. A failed batch check is caught, the next tick is scheduled
anyway and the run resolves rather than failing, so the chain survives a
check that couldn't run; the sweep re-arms groups and finalizes anything
still active past its deadline, even when delivery isn't configured.
Wake redelivery is id-deduped rather than conditional, because the sweep
can't know whether the user was already told. Access is re-authorized on
**every** check against the primary — replica lag would extend access
the user has already lost.

**Wording lives in one place.** `watch-wording.ts` is read by the card,
banner, toast, email and the agent's own narration, and the numbers come
from the frozen observation rather than a fresh read, so a retry
produces the same sentence. Replay reproduces the **recorded** decision
instead of deciding again — the transcript is append-once, so a second
decision would contradict it forever.

**Cancellation is the ending without an answer** — no resolution, no
wake. One exception, decided during testing: a watch the *user*
cancelled leaves a single neutral transcript line ("Stopped watching
…"), keyed off the watch id so a retry can't repeat it. The other four
reasons stay silent.

**Email is opt-in and only a fired watch emails.** An expiry is narrated
in the chat and nowhere else. Both gates (agent access, a configured
email transport) are checked at subscribe time *and* again at delivery,
and the subscription outcome is frozen on the ledger row so a retry
replays it. Neither gate is a plan check.

**One watch offer per turn.** The prompt and the renderer guard this
independently — if the turn already proposed a watch card, the action
button is dropped, because the card is the better affordance. Two eval
cases pin the prompt side: exactly one offer with the line last and the
button after it, and zero offers when the rendered card already carries
one — deterministic assertions, over a real-model run.

## Testing

Unit tests (vitest, testcontainers, no mocks) under
`apps/webapp/test/dashboardAgentWatch*.test.ts` and
`internal-packages/dashboard-agent/src/watch-*.test.ts` cover the
invariants above: the 4-way check results and the freshness fences,
identity/dedup and the submission ledger, queue-name resolution, the
batch chain surviving a failed check, sweep boundaries and alert-once,
tenancy and the watch token's scope, and the wording snapshot. The
load-bearing ones were verified by control-breaking the guard first and
checking the test goes red.

Live-tested end to end against a local stack, following the guidebook:
all ten watch kinds firing and expiring, cancellation, the email pair (a
fired watch mails, an expired one does not), and watch recovery from a
health report.
 2026-08-12 09:51:40 +02:00
Katia Bulatova 9a3bee0288 feat(webapp): dashboard agent — UI (#4529) 
Stacked on #4418. Merge that first.

The UI slice of the dashboard agent: the side panel, the chat transport
wiring, message and card rendering, suggested prompts, and chat history.
#4418 works without this — the system is simply invisible. The diff is
mostly components, so the notes below cover only the three decisions you
can't read off the markup. Behavior and a hands-on walkthrough live in
GUIDEBOOK.md, which lands with #4525.

## Decisions worth knowing

- **Action rows always render at the end of a turn.** The model's
emission order isn't trusted for layout, so action blocks are split out
of the stream and appended last. Display only — `answered` stays keyed
on the emission index.
- **The last-chat memory is org-true.** It's keyed by the chat's own
organization, and a foreign or deleted chat comes back as a 404 the
client treats as gone, rather than an empty chat it keeps around.
- **A dead stream self-heals from the settled transcript.** Terminal
records are written to the chat row after the client's stream closes, so
the panel re-reads it. The poll gate is any unfinished turn — a dangling
tool part, not just an open investigation.

## Notes

- Gated by `canAccessDashboardAgent`; no behavior change with the flag
off.
- Page marks: `handle.agentPageContext` on 47 routes, ~20 lines each.
- Entry points: Ask Trigger button, ⌘J, Help & Feedback. The old ⌘I and
`?aiHelp=` links keep working.

## Screenshots

<img width="1440" height="788" alt="Screenshot 2026-08-07 at 15 14 29"
src="https://github.com/user-attachments/assets/f4e89e8d-13ed-4be3-a88d-d5cca3ece0fa"
/>
 2026-08-12 08:38:59 +02:00
Katia Bulatova 4569657923 feat(webapp): dashboard agent — chat, reports, investigate (#4418) 
## What & why

This is the system behind the Dashboard Agent — an assistant that
answers questions about a project's runs, errors, queues, deploys and
health, and can investigate failures end to end.

The agent runs as a chat.agent task in its own Trigger project. It has
no access to the main database or ClickHouse; all platform data is read
through the public API using a delegated, read-only user token.

Everything here is behind `canAccessDashboardAgent` and inert with the
flag off. The UI that mounts the panel lands in #4529.

## Stack

`#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook
gallery. The scenario/contract reference for the whole stack is
`internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch
branch): it states, per feature, what makes each thing happen and where
that is decided.

## What's inside

**Agent runtime and tools** — `internal-packages/dashboard-agent`:
prompt, tool set (API reads, TRQL query, docs, navigation,
evidence/investigations, repo source), conversation compaction, a
prompt-prefix token budget pinned by snapshot test, and sampled
LLM-judged turn evals. The package cannot import webapp server code,
which is what makes the "no DB access" claim structural rather than a
convention.

**Contracts** — `internal-packages/dashboard-agent-contracts`:
`trigger://` URIs, intents, and the block envelope every rendered card
travels in.

**Conversation store** — `internal-packages/dashboard-agent-db`: drizzle
over postgres-js in its own `trigger_dashboard_agent` Postgres schema,
plus one additive migration.

**Auth boundary** — the user-actor token gains an optional environment
claim; one guard (`userActorEnvironment.server.ts`) enforces it so
routes don't each re-derive the rule. Token minting, cap ceiling, and
the RBAC fallback path for self-hosted.

**Transport** — webapp resource routes that mint the token and proxy
each turn, and SDK-side mid-turn reconnect.

**Public API the agent reads through** — orgs, projects, environments,
runs, queue metrics, workers, a run's commit metadata, repo snapshot,
reports, and `POST /api/v1/query`.

**Reports** — the health report's layout is declared once and shared by
the card, the markdown surface and the JSON/MCP surface, so the same
report reads the same in the dashboard, the terminal and an editor.

**Block renderers** — the report and investigation cards the flows above
already emit (`app/components/dashboard-agent/`). The panel that hosts
them, and the rest of the chat UI, is #4529.

**Query safety and CSP** — see below.

## Key decisions

- **The agent is a separate Trigger project, not webapp code.** It reads
platform data over the public API with a delegated user-actor token
whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no
webapp imports.
- **The PAT-only auth helper now refuses user-actor tokens.** This is an
intentional behavioral change: its callers consume only a bare userId
and do not enforce delegated-token capabilities. Actor-aware routes
continue through the scoped route builders instead.
- **RBAC fallback builds a delegated token's ability from its own cap**,
never the blanket ability a PAT gets (read-only when the token declares
none). Without this, the agent's read-only cap would buy a write JWT on
self-hosted.
- **Org creation checks RBAC only for user-actor tokens, and only after
the env gate**, so an install with `ORG_CREATION_API_ENABLED` off
returns 404 rather than 403, and an ordinary PAT never consults an
ability the route has no org to scope. Both orderings are pinned by
test.
- **The query path is read-only in depth.** TRQL rejects write
statements at the grammar level (they don't parse, rather than being
filtered), ClickHouse runs with `readonly=1`, and the org/project/env
filters are injected server-side from the credential — the request body
cannot widen scope. An unparseable query denies instead of falling
through to the permissive resource.
- **Document-wide img-src CSP.** Remote images are an
outbound-request/exfiltration surface, so the policy permits only
own-origin/data/blob, the required SSO avatar hosts, and the favicon
endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST;
wildcard hosts and bare schemes are intentionally not allowed.
- **The chat transport reconnects on a mid-turn EOF**
(`@trigger.dev/sdk`). A body that ends without a turn-complete is
terminal only when the server says `X-Session-Settled: true`; otherwise
the transport resubscribes from `lastEventId` with bounded backoff, and
any record re-earns the budget. Previously a closed long-poll window or
a proxy restart left the reply stuck as if still generating.
- **Conversations live in their own datastore**, schema-scoped and
foreign-key-free (it references `organizationId`/`userId` by id, because
in cloud it is a different database). It is a display read-model for the
History tab and transport resume; `chat.agent`'s object-store snapshot
remains the model's source of truth.
- **Deterministic first.** Reports and health checks contain no LLM —
they are computed from the same data the dashboard shows, and the model
only narrates and links them. That is what makes a number in an answer
auditable.

## Testing

- 63 new test files, run with `pnpm run test --filter webapp` and
per-package vitest. Heaviest coverage on the auth boundary
(`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`,
`contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the
report layout, and the SDK reconnect.
- The agent package has a separate eval lane (`pnpm run test:evals`,
`vitest.eval.config.ts`) that hits the real model, so it never runs in
`pnpm test`.
- Live-tested against a local stack scenario by scenario; the GUIDEBOOK
lists the condition each behaviour is expected under, which is what
those runs were checked against.

## Changelog

`.server-changes/dashboard-agent.md`, plus changesets for
`@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat
reconnect) and the CLI's `mint-token` help text.
 2026-08-11 18:56:14 +02:00
Saadi Myftija 02de2e693f feat(api): separate rate limit budget for deployment endpoints (#4565) 
Most deploy-flow API calls shared the general per-environment rate limit
bucket with all of that environment's runtime traffic, so an org with
heavy API usage could intermittently 429 its own deploys; the
`/api/v*/deployments` endpoints themselves were fully exempt from rate
limits as a stopgap
([#2774](https://github.com/triggerdotdev/trigger.dev/pull/2774)), which
promised a dedicated limiter as the follow-up. This is that follow-up:
the whole deploy-flow group now runs on its own budget, separate from
runtime API limits.

### Design

A new `deploymentRateLimiter` covers every endpoint the deploy flow
depends on: the `/api/v*/deployments` group, the env API key exchange
(`/api/v1/projects/:ref/:env`), build-time env var resolution and sync
(`/envvars`, `/envvars/:slug/import`), preview branches,
`/api/v1/remote-build-provider-status` and `/api/v1/artifacts`. The
general API limiter whitelists the same shared path list, so exactly one
limiter applies to each path and the two can't drift apart.

Buckets are keyed per environment for environment API keys and per token
for the PAT-authenticated phase of a CLI deploy (whoami, key exchange,
branches). The deploy budget is controlled via the
`DEPLOYMENT_RATE_LIMIT_*` env vars.
 2026-08-11 17:51:31 +02:00
Eric Allam c526528d8f feat(webapp,database): bound Prisma list filter arity (#4480)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
## Summary

Prisma expands `in` / `notIn` into one bind parameter per element, so
every distinct list
length is a separate prepared statement. Where the length tracks data
volume (a batch size,
a run-graph fan-out, a prior query's id set) one call site can mint
hundreds of them. Each
is used about once, but inserting it evicts an entry that was being
reused, so the cost
lands on unrelated queries sharing the pooler's statement cache. An
unbounded list also
risks the 65535 bind-parameter ceiling.

`boundedIn()` pads a filter list to the next power of two by repeating
its last element.
`IN` and `NOT IN` ignore duplicates, so results are unchanged, and a
call site drops from
one statement per length to at most `log2(cap)`. Applied to all existing
sites.

## Enforcement

Two oxlint rules require the helper: a list filter must be an inline
array literal or a
`boundedIn()` call.

- The first covers filters reached through `where` / `having` /
`cursor`, and deliberately
never descends into `data`, `create`, `update`, `set` or `equals`. A key
named `in` in
those positions is user data, not a predicate, and rewriting it would
corrupt what gets
  stored or compared.
- The second covers bare filter objects passed to where-building
helpers, which the first
cannot see. It found five sites in the run-graph batch loaders that were
otherwise
  invisible.

Both rules follow filters through the shapes they are actually written
in: conditional
expressions, logical-and objects, spread-conditional properties,
computed keys, and call
arguments. An array literal only counts as fixed-arity when nothing
spreads into it, since
`[...new Set(ids)]` has a runtime length. Twelve sites were hidden
behind those shapes
until the rules handled them.

Scoped to `in` and `notIn`. The scalar-list filters `hasSome` and
`hasEvery` compile to
`&& $1` and `@> $1`, passing the whole array as a single bind parameter,
so their arity never
reaches the statement text and there is nothing to bound.

Both rules are `error`, so new call sites fail CI. That ratchet has
already caught four
sites added by other PRs while this one was in review.

## Notes

`boundedIn` pads by repeating rather than with null: `x NOT IN (a, b,
NULL)` is never true,
so null-padding a `notIn` filter would silently return no rows. Lists
above 32768 are
returned unchanged so padding can never push a query past the parameter
limit.

Route modules reach the helper through `~/db.server` rather than
importing the database
barrel directly, since a value import of that barrel into a module that
also exports a React
component is only safe while dead-code elimination prunes it.

Measured on a local rig: 300 distinct list lengths produce 300 prepared
statements
unpadded, 10 padded. Verified end-to-end against a local stack with the
full task-suite
sweep, which surfaced no regressions.
 2026-08-07 16:39:58 +01:00
Chris Arderne 088f68b373 feat(webapp): share rate limit bucket across additional API keys per environment (#4508) 
## What

Rate-limit the API by **environment** rather than per API key.

Previously the limiter keyed its bucket on the hash of the full
`Authorization` header — one bucket per key. With additional environment
API keys (`tr_*_sk_*`), an environment can mint many keys and each got
its own full bucket, so more keys = higher effective rate limit. This
collapses all of an environment's keys onto a single shared
per-environment bucket, so the ceiling is exactly the configured limit
regardless of key mix.

## How

- `authorizationRateLimitMiddleware` now lets the override return `{
config?, identifier? }`. `identifier`, when present, is the rate limit
bucket key; otherwise it falls back to the hashed `Authorization` header
(unchanged legacy behavior, still used by `engineRateLimiter` and any
unauthenticated fallthrough).
- `apiRateLimiter`'s override resolves the environment id and uses it as
the identifier:
- **Additional keys** (`isAdditionalApiKey`) resolve via a new
`resolveAdditionalApiKeyRateLimitScope()` — a **scope-agnostic** keyHash
→ (environmentId, org limiter config) lookup. It is deliberately
permissive (restricted keys resolve too) because it's used **only for
bucketing, never as an auth decision** — request auth still goes through
the RBAC bearer controller, which enforces scopes. Revoked/expired keys
are excluded so they can't hold a bucket warm.
- **Root/legacy keys** reuse the environment already resolved by
`authenticateAuthorizationHeader` and key on `environment.id` too.
- The identifier is always the stable environment id, never the secret
key (which can rotate and would split the bucket).
- The whole override result is cached per key by the existing SWR cache,
so **no extra per-request lookup and no separate Redis mapping** is
added.

## Behavior notes

- Root + additional keys of the same environment now share one bucket
(ceiling = configured limit, not a multiple of it). Restricted
additional keys are included — they were the biggest gap, since they
authenticate via the RBAC controller and previously fell back to per-key
buckets.
- **Public JWTs** keep their existing fixed-window, per-token bucketing.
- One-time bucket reset on deploy (bucket keys change); harmless.

## Tests

- New: two tokens resolving to the same identifier share one bucket.
- New: with no identifier, bucketing stays per-key (legacy behavior
preserved).
- Updated existing override tests to the new `{ config }` return shape.

Base: `feat/multi-keys-surface`. Closes TRI-12888.
 2026-08-06 16:05:27 +01:00
Chris Arderne 9409ddf9bc feat(webapp): add multiple environment API key management (#4390) 
## Summary

Projects can create, inspect, expire, and revoke multiple API keys for
each environment. Plaintext values are shown only at creation; stored
credentials are hashed and the API keys page displays only an obfuscated
suffix afterward.

Self-hosted installations support full-access additional keys by
default. Authorization extensions can provide additional access presets
and optional task selection. Additional keys can also mint scoped public
access tokens through the Trigger.dev API without receiving the
environment signing key.

## Feature notes
- Only admin+ can create API keys (Developer can make in Development
branch).
- JWT self-signing will be a server call when used with new `_ak_` keys.
- JWTs with long expiry can keep working even with api key deleted (gets
priveleges from api key, signed with root key)
- Unfiltered session listings intentionally preserve the existing broad
task-read behavior. Filtered listings enforce task-level scopes for
every requested task.
- Buffered runs without a task identifier are not safely authorizable,
so cancel/replay requests fail closed rather than resolving an unscoped
run.
- Batch and waitpoint endpoints intentionally return server-minted,
narrowly scoped public tokens to all callers. These tokens have bounded
lifetimes and may remain valid until expiry after API-key revocation.

## Deployment notes

Deploy the management UI and public-token endpoint with new key creation
disabled. Enable creation for selected organizations after the
authentication path and released SDK have been verified, then expand
availability gradually.

Revoking an API key prevents new bearer requests and new token minting.
Public tokens already minted by that key remain valid until their own
expiration because they are signed by the environment signing key.

## TODO
- [x] Add "Created by" to the key table
- [x] Document that streamed batch ingestion is non-atomic and may
 partially accept items before a validation or authorization error.

## Follow-ups

- [x] Add an organization-level feature flag for the API key management
UI and creation action.
- [x] Document rollout ordering: enable additional-key lookup before
enabling issuance.
- [x] Add a system-wide gate that can stop new key issuance without
disabling authentication for existing keys.
- [x] Replace the generic SDK compatibility warning with the first
published compatible version. Old SDK will mint an unusable token if
given an `_ak_` key.
- [x] Add public documentation covering creation, storage, expiration,
revocation, SDK compatibility, and public-token lifetime behavior.
- [x] Add observability for key creation, revocation, policy preparation
failures, and public-token mint failures.
- [ ] Exercise create, copy-once display, authenticate, mint, expire,
and revoke flows end to end before broad enablement.
 2026-08-06 15:27:10 +01:00
Katia Bulatova fbd6df33b4 feat(webapp): Themes + contrast settings update (#4206) 
Adds System Preferences, Dark and Light themes, gated by the
`hasThemeSwitcher` feature flag (off by default — dark stays the default
theme for everyone).

Old theme is now "Classic"and set as default. 
"System preferences" theme has both Light and Dark modes and uses your
laptop settings to use a correct one.
It has less color accents (specifically less colored text), and they are
the same for both modes, only grayscale values change between them. And
Light/Dark themes can be used separately.

New Contrast setting is available for System Preferences, Dark and Light
themes - it changes the contrast for the whole app. All new visual
Settings live in Account.
 2026-08-03 19:29:33 +02:00
Eric Allam 9d57aff542 fix(webapp): make the Queues hero charts environment-wide (#4486) 
## Summary

The four charts above the queues table aggregated over **at most the 25
queues on the current page**. They reused the loader's already-paginated
queue array as a ClickHouse `queue IN (...)` filter, so paging or
re-sorting changed the values, and a name search matching nothing
blanked the whole chart row. The stat tiles above them were already
environment-wide, so the two rows disagreed.

They now read `env_metrics`, the environment-level rollup that already
exists for exactly this (the built-in Queues dashboard and the health
report read it). That is both correct and queue-count-independent: no
`GROUP BY queue` across an entire environment, and no client-side
summing.

Note this is not only a paging artifact: page 1 under-reported too. On
the seeded environment below, page 1 read 82% saturation against a true
87%, because the environment's running total is not the sum of one page
of per-queue gauges.

Three related fixes ride along.

**Scheduling delay and throttling sawed to zero.** Both are
event-driven, so at the 10-second bucket a short range picks, most
buckets hold no samples at all and were drawn as `0ms`. Measured over a
1-hour window: **232 of 349 buckets had no scheduling-delay samples**. A
bucket where nothing started is not a bucket where nothing waited, so
the line was both ugly and wrong. TRQL grows a `minBucketSeconds` floor,
plumbed through the metric resource route, and the hero tiles set 60s.
Buckets that still have no samples render as a gap instead of a dive to
zero.

**The floor must not feed a width-dependent headline.** Two of the four
headlines are not peaks, so widening the plotted buckets moved them:

- **Throttled** is a share of buckets that saw any throttling, so a
single brief throttle came to mark a whole minute instead of ten
seconds: the same seeded events read 17% at 10s and 85% at 60s.
- **Scheduling delay p95** is a percentile, and merging quantile states
over a wider bucket yields a p95 between the sub-buckets' own. Two 240s
samples among twenty in one 10-second sub-bucket give a worst-of-six p95
of 240,000ms against a merged 60-second p95 of 5,000ms — a 48x
understatement of a headline whose tooltip claims it is the worst in the
window.

Both charts keep the floor, since a readable line was the point of it.
Their headlines now come from a second query at the range's natural
bucket width, via an optional `readout` on the tile, so each means what
its tooltip says regardless of how the plotted buckets are sized.
Saturation and backlog are genuinely width-invariant (a max of maxes is
the same at any width), so they are unchanged and issue no extra query.
Both caught by Devin in review; I had wrongly lumped p95 in with the
peaks.

**Charts reported a hydration mismatch on every render.** Recharts
resolved victory-vendor's CJS entry on the server and its ESM entry in
the browser. Those bundle different d3-shape builds, and the CJS one
predates d3-path's digit rounding, so every server-rendered curve
carried full-precision coordinates while the client rounded to 3
decimals:

```
Server: M0,3C0.9305555555555555,3,1.8611111111111112,3,...
Client: M0,3C0.931,3,1.861,3,...
```

Bundling recharts for SSR makes both sides resolve the same ESM build.
Verified: 45 of 45 server-rendered chart curves now match the client,
and the page loads with an empty console.

## Verification

An isolated stack with 40 seeded queues (20 heavily loaded, 20 idle) and
90 minutes of 10-second buckets written into `queue_metrics_raw_v1`, so
the real materialized views built `queue_metrics_v1`, `env_metrics_v1`
and the 5m rollup. Ground truth for the environment: 260 running against
a limit of 300 (**87% saturation**), 800 queued.

| | before | after |
| -- | -- | -- |
| Saturation, page 1 | 82% peak | **87% peak** |
| Saturation, page 2 | 5% peak | **87% peak** |
| Backlog / delay, page 2 | "No activity" | **800 peak / 59.5s** |
| Name search matching nothing | all four charts blank | charts stay
environment-wide |
| Metric refetches on a page change | 4, each painting a skeleton | **0,
no skeleton** |
| Buckets drawn as 0ms with no samples | 232 of 349 | **0** |
| Throttled readout | 17% | **17%**, unchanged by the wider buckets |
| Worst-p95 readout source | plotted buckets | **natural width**, so a
sub-minute spike is not averaged away |
| Crosshair reach, hovering one detail-page chart | 2 of 4 others | **4
of 4** |
| SSR chart curves mismatching the client | 45 | **0** |

The bucket floor was measured across ranges: it widens 10s to 60s at 30m
and 1h, and is correctly a no-op at 12h (300s) and 7d (3600s). One extra
request per page load, for the throttled readout.

The built-in Queues dashboard, which reads `env_metrics` independently,
agrees at 86.7% and 260 of 300.

`internal-packages/tsql` suite green (612 tests), including 5 new ones
for the floor that fail without it. Webapp typecheck, oxfmt and oxlint
clean. Spot-checked the Run metrics dashboard and the per-queue detail
page for SSR regressions from bundling recharts: both render, console
clean.

The queue detail page carries the same event-driven series, so its
scheduling delay, throttling and per-key mean delay take the same
treatment.

## Screenshots

<img width="2540" height="580" alt="after-page1-charts"
src="https://github.com/user-attachments/assets/6cd23f9c-e7fd-4918-bcfa-b1d3340b16d1"
/>

## Rollout

Already behind the per-organization `queueMetricsUiEnabled` flag, so
only gated orgs see any of it. Blast radius is chart values on one page
plus the SSR bundling of recharts; rollback is a revert with no data
migration.

## Stated limitations

- `wait_ms_count` and the quantile state both only count `wait_ms > 0`,
so "nothing started in this bucket" and "everything started instantly"
are indistinguishable in storage. Both render as a gap. Distinguishing
them needs a schema change, which is not in this PR.
- The queue name search deliberately no longer narrows the charts. It
only did so incidentally and incorrectly before (first 25 matches, and
blanked on zero matches). Search-scoped charts would need the full
unpaginated matching set and a server-side aggregate; worth its own
ticket if we want it.
- Bundling recharts for SSR grows the server bundle slightly. That is
the cost of both sides resolving one d3-shape build.
- The plotted delay line is a smoothed 60-second view, so a sub-minute
spike above the one-minute warning threshold can fail to colour the line
even though the headline reports it and colours itself.
- Every chart inside one synced group shares the floor, because the
hover crosshair is a reference line on a category x-axis and only draws
where the hovered bucket exists in the other chart's own data. That
costs the queue detail page's gauges some resolution (1 minute instead
of 10 seconds) in exchange for the crosshair working across the row.

Separately, while taking the screenshots I found a pre-existing
rendering bug unrelated to this change: a **perfectly flat** saturation
series draws no line at all (the readout still shows the right
percentage), which looks like the threshold gradient's offset
degenerating when the series min equals its max. It reproduces on
`main`, so it is not a regression here and I have left it alone; filed
as its own issue.

Refs TRI-12784
 2026-08-03 16:19:50 +01:00
Chris Arderne 763b5dc582 feat(webapp): enforce scopes for environment API keys (#4389) 
## Summary

Environment API keys backed by the additional-key table can authenticate
API requests using their stored effective scopes. Revoked and expired
keys are rejected, branch environments retain their existing routing
behavior, and last-used timestamps are updated on a throttled
best-effort basis.

## Design

API route builders receive the resolved ability and reject restricted
keys on routes without an authorization declaration. Existing
deployment, environment variable, queue, run, task, batch, session, and
waitpoint routes declare the resources they access.

Trigger and batch responses return server-signed public access tokens,
so additional keys never need access to the environment signing secret.
Root-key rotation also keeps public tokens valid for the existing grace
window.

## Feature notes
- Root environment keys remain unrestricted for backward compatibility.
Additional keys enforce their persisted scopes and fail closed on routes
   without an authorization declaration.
- Machine-key requests never exchange one credential for another.
Additional keys cannot retrieve the root key, and rotated root keys are
not upgraded
   during their grace window.
- Public JWT validation remains host-owned, while installed RBAC plugins
continue to supply root-key abilities.
- Unfiltered session and run listings preserve existing broad task-read
behavior. Filtered requests enforce the supplied task identifiers.
- Related-run summaries remain embedded in run retrieval for API
compatibility. Retrieving or mutating a related run independently still
requires
   permission for that run.
- Queue management authorizes at collection scope, matching the queue
permissions currently issued.
- Batch responses deliberately include server-signed public access
tokens for all clients. Selected-task credentials continue using their
original
   credential for per-item authorization.
- Two-phase batches authorize declared task identifiers before creation
and authorize every streamed item. Streaming paths that cannot declare
the
   complete task set remain fail closed.
- Authentication telemetry records successful credential resolution
separately from subsequent resource-authorization failures.
- API keys are high-entropy random tokens. SHA-256 is intentionally used
for deterministic indexed lookup, not password hashing.

## Deployment notes

The schema migration must be present before this code is deployed.
Because bearer resolution runs on every authenticated request, deploy
the resolver with additional-key lookup disabled, verify root-key and
public-token parity, then enable lookup before any additional keys can
be issued.

The multi-task authorization tightening changes the result for narrowly
scoped tokens that request tasks outside their grants. Observe
would-deny results before enforcing that check. Request-idempotency keys
are also newly isolated by environment and task, so a retry crossing the
deployment boundary may execute once more before old cache entries
expire.

## Follow-ups

- [x] Add a system-wide kill switch for additional-key lookup, defaulted
off for the initial deployment.
- [x] Add authentication observability by credential kind, result,
latency, and lookup path without recording credential values.
- [ ] ~Add would-deny observability and an independent enforcement
switch for multi-task authorization.~
- [ ] ~Add an independent switch for server-issued batch tokens while
root-key parity is verified.~
- [ ] Confirm every API route reachable by a restricted key has an
explicit authorization declaration or intentionally fails closed.
- [x] Verify root-key rotation, revoked-key grace, and public-token
validation through each bearer resolver path.
 2026-08-03 14:00:29 +01:00
Eric Allam db6228dd1e chore(webapp,core,sdk): upgrade @s2-dev/streamstore to 0.25 and migrate S2 hosts (#4349) 2026-08-01 11:33:34 +01:00
Eric Allam f9c8d518c7 perf(webapp,run-engine,database): resolve the newest worker and deployment by createdAt (#4452) 2026-08-01 11:32:21 +01:00
Eric Allam 0445b8ec27 fix(webapp,clickhouse): keep the rest of a ClickHouse batch when one run or span has un-ingestable JSON (#4358) 
## Summary

A single run output, trace span, or payload carrying JSON that
ClickHouse can't ingest (for example nesting past its depth limit) used
to fail the whole insert batch, so unrelated runs and spans silently
disappeared from the runs list, traces, and logs. This keeps the rest of
the batch and handles the offending row instead of dropping everything
around it.

## Fix

Recovery is per-table, matched to what each table needs:

- **Runs** (`task_runs_v2`) keep their status. We follow ClickHouse's
failing-row hint to strip just the un-ingestable JSON column(s) so the
run still lands (its output reads from Postgres on the detail page), up
to a configurable limit (`RUN_REPLICATION_MAX_POISON_STRIPS_PER_BATCH`,
default `1`). Past the limit we stop and land the batch with
`allow_errors` in a single pass, skipping the remainder. Cost stays a
fixed handful of inserts no matter how large or poisoned a flush is.
- **Trace events and payloads** (high volume, append-only) recover with
a single `allow_errors` insert: the good rows land in one pass and only
the un-ingestable rows are skipped.

Before falling back, a lightweight sanitizer still repairs what it can
losslessly (lone UTF-16 surrogates, out-of-range integers) so a
repairable row lands in full.

To read the failing-row hint we patch `@clickhouse/client-common`: its
error parser truncates the server response and discards the `(at row N)`
position, so the patch preserves the full text for the recovery path to
read.
 2026-08-01 09:17:20 +01:00
Eric Allam c72ebf9084 fix(webapp,run-engine): stop batchTriggerAndWait hanging when item streaming never completes (#4397) 
## Summary

`batchTriggerAndWait()` could leave a parent run waiting forever. The
2-phase batch API blocks the parent on the batch's waitpoint as soon as
the batch is created, but the batch is only sealed at the end of item
streaming. If streaming never completed, nothing sealed the batch,
nothing completed the waitpoint, and the parent stayed suspended with no
timeout and no way to recover.

Supersedes #4016, which added the reaper alone.

## Fix

Admission for item streaming was being decided twice. Batch creation
passes its own rate limiter, which fixes `expectedCount` and blocks the
parent, and then the item stream had to pass the general API limiter as
well, competing with unrelated traffic. A second limiter could therefore
veto work the first had already committed the parent to. Creation now
mints a bounded grant that the item stream spends, so an admitted batch
can finish streaming. The grant is capped per batch rather than
exempting the path, and every failure mode (no grant, spent grant,
unreachable store) falls back to the normal limiter.

That makes stranding much rarer but not impossible, since a request
timeout or a crash can still end streaming for good. So a seal-timeout
reaper aborts any batch still unsealed after `BATCH_SEAL_TIMEOUT_MS` and
completes the parent's waitpoint with an error, letting
`batchTriggerAndWait()` reject instead of hang. It is race-safe against
a late seal, and it is only scheduled for batches that actually block a
parent, so fire-and-forget batches cost nothing.

Finally, the batches page used to report "Batch completion checked." for
these batches while doing nothing, because the completion path returns
early on an unsealed batch. It now says the batch cannot be resumed.

Rate limiting is no longer the reason a batch strands, so the reaper's
default stays at 30 minutes, comfortably above the SDK's worst-case
stream-retry budget.

## Verification

Unit and container tests cover the grant cap, the bypass ordering (it
runs after the authorization check, so it can never skip
authentication), and the reaper's abort, seal race, idempotency, and
no-waitpoint cases.

Also verified end-to-end against a running stack. With the general limit
exhausted, batch creation and other API calls returned 429 while a
granted batch still streamed and sealed; an ungranted batch id was rate
limited rather than bypassed; and the grant cut off exactly at its
configured attempt count. Reproducing the stranded state on a real
parent run, the batch was aborted at the timeout, the waitpoint
completed with an error, and the parent resumed and finished instead of
hanging. A parentless batch left unsealed was untouched well past the
reaper window.

## Verified against deployed runs

The reaper was proven end to end with a real deployed run (locally-run
supervisor, containerised
run) and a real network fault, rather than a simulated one: toxiproxy
severs the phase 2 item
stream mid-flight so every SDK stream retry genuinely fails, while phase
1 still succeeds. Only
the batch calls traverse the fault, so control-plane traffic is
untouched.

The reproduction is the shape that actually strands a parent: the task
catches the
`BatchTriggerError` the SDK throws and carries on, so the phase 1 block
outlives the thrown error
and the parent hangs at its next suspension point.

With the reaper disabled, the parent sat in `EXECUTING_WITH_WAITPOINTS`
for over 24 minutes holding
two blockers, and stayed stuck across a full infrastructure restart:

```
 type     | status    | has_timeout
 BATCH    | PENDING   | f            <- orphan, completedAfter NULL
 DATETIME | COMPLETED | t            <- the wait already elapsed
```

With the reaper enabled the same task under the same fault completed in
about 75 seconds with zero
blockers left, the batch `ABORTED`, and its waitpoint completed carrying
the error.

Two conditions are required to observe this at all, which is worth
knowing for any future test:
the run must be deployed rather than `trigger dev` (dev runs execute in
process and finish while
still holding blocker rows), and the wait after the caught error must
exceed the checkpoint
threshold, or it is served in process and never suspends.

### Why completing the batch waitpoint is sufficient

`batchTriggerAndWait` runs create, then stream, then wait. A phase 2
failure throws before the wait
is ever reached, and the reaper only fires on an unsealed batch, so the
parent is never suspended
awaiting the batch when it runs. The parent therefore does not need a
synthetic result, only to stop
being blocked. Note this reasoning depends on that ordering: if the wait
were ever reached with an
unsealed batch, completing the batch waitpoint alone would not settle
the caller.

## Follow-ups

- Batches stranded before this ships still need a one-off recovery; the
reaper only schedules at creation time.
- That same property leaves a gap if the process dies between creating
the batch and scheduling the job. A periodic sweep would close it, but
wants a supporting index.
- When a partially streamed batch aborts, children already enqueued keep
running while the parent fails. Left as-is deliberately, since
cancelling triggered work is a bigger semantic call.
 2026-07-31 11:55:25 +01:00
claude[bot] debfa2b733 feat(webapp): impersonation consent page and a view-as-user toggle (#4421) 2026-07-30 21:28:44 +01:00
Oskar Otwinowski 4efe0a07c4 fix(webapp): create dev environments for SSO and Directory Sync members (#4426) 
Members added by SSO just-in-time provisioning or Directory Sync never
got
their per-member DEVELOPMENT environments - only invite acceptance and
project creation created them. `trigger dev` returned "Environment not
found" for those members and the dashboard had no dev view.

ensureOrgMember now queues provisioning for every membership it settles,
so
both paths are covered and members missing environments are repaired on
their next sync. Provisioning runs as a common-worker job to keep
sign-in
and directory webhooks off the per-project write loop. A failed enqueue
surfaces for Directory Sync, whose worker retries the idempotent effect,
and is swallowed for sign-in, where the next login enqueues again.
Environment creation now tolerates a concurrent creator so the
project-creation loop and the job cannot collide on the unique index.

Also fixes environment resolution ignoring dev-environment ownership: a
member without their own dev environment could be handed a colleague's
and
have it persisted as their dashboard preference.
 2026-07-30 21:50:15 +02:00
Eric Allam 6e5f0f0fe7 fix(webapp,clickhouse): stop invalid customer queries alerting, and isolate Sentry scope per request (#4372) 
## Summary

A query sent to the query API with a typo in it, like a column name that
does not exist, was being reported as a server error. That put customer
SQL mistakes into our error alerting, where they made up almost all of
the volume on one of our noisiest alerts, and it drowned out the
failures that are actually ours to fix. This makes the level match who
is at fault, and fixes two related problems found alongside it.

## Invalid queries are the caller's, not ours

The query API route already got this right. It checks for `QueryError`,
logs at warn, and returns a 400, with a comment saying the system
handles it gracefully and no alert is needed.

The layer underneath ignored that. `executeTSQL` logged every exception
out of its catch block at error, including the compile failures the
route was about to turn into a 400, and error-level logs are forwarded
to error reporting.

The TSQL package already draws the line we need:

```ts
export class ExposedTSQLError extends BaseTSQLError {
  /** An exception that can be exposed to the user. */
}

export class InternalTSQLError extends BaseTSQLError {
  /** An internal exception in the TSQL engine. */
}
```

`SyntaxError` and `QueryError` extend the first. So the catch block now
branches on `ExposedTSQLError` and logs those at warn, keeping error for
`InternalTSQLError` and anything unanticipated, which is a genuine
compiler bug.

## SQL the caller wrote is their mistake, not ours

The same asymmetry showed up one level down. A query that compiles fine
can still be rejected by ClickHouse at execution, and most of those
rejections mean the caller's SQL is wrong rather than that we generated
something bad.

This is where the volume actually is. Checking production, one error
group alone, a missing `GROUP BY` on the public query API
(`NOT_AN_AGGREGATE`), accounts for over a million events across hundreds
of users. It is by far the largest error group in the project, and
classifying only by resource limit would have left every one of those at
error level.

So rejections are split three ways in `ClickhouseClient`, which is the
only place holding the parsed `ClickHouseError` and its symbolic type.
By the time the error reaches `executeTSQL` it has been wrapped and the
type is gone, and the type never appears in the message text, so it
cannot be recovered by string matching.

- **Resource limits** (memory ceiling, timeout, row/byte caps) log at
warn. The query is valid, it just asked for more than it is allowed to
spend.
- **Invalid SQL** (`NOT_AN_AGGREGATE`, `UNKNOWN_IDENTIFIER`,
`SYNTAX_ERROR`, the type and parse families) logs at warn **only when
the caller wrote the SQL**.
- **Everything else** keeps alerting.

That gate matters. The client is shared, so the identical rejection on
TRQL *we* generated is our bug and has to stay at error. Callers opt in
with `userAuthoredQuery`:

| caller | who wrote the SQL | opts in |
| --- | --- | --- |
| public query API | the customer | yes |
| query editor | the customer | yes |
| agent charts | the agent's model | yes |
| built-in dashboard tiles | us, in code | no |
| queue metric cards | us, in code | no |
| health report | us, in code | no |

The agent is the one judgement call. Its TRQL is not typed by a person,
but it is also not something a code fix makes correct, so a query it
gets wrong is not worth waking anyone for. The same endpoint serves
built-in tiles whose TRQL we do write, so the opt-in lives with the
caller rather than the route.

Separately, when one of these queries did fail, the log recorded the
generated ClickHouse SQL but not the query the caller actually wrote,
which made the reports hard to act on. `queryWithStats` takes an
optional `logFields` that `executeTSQL` uses to attach the original
TSQL.

## Events were attributed to the wrong request

Chasing the above turned up something broader: only a tenth of the
events on that alert pointed at the query API. The rest were pinned to
unrelated requests that happened to be in flight at the same time, so
the alert looked like the trigger endpoint was failing.

`Sentry.init` runs with `skipOpenTelemetrySetup: true`, because we
register our own OTel pipeline. That skips `initOpenTelemetry`, and one
of the things it does is:

```js
api.context.setGlobalContextManager(new SentryContextManager());
```

The async-context strategy is still installed, but `withIsolationScope`
only marks the OTel context and delegates the actual fork to that
context manager:

```js
// "We depend on the otelContextManager to handle the context/hub"
return api.context.with(ctx.setValue(SENTRY_FORK_ISOLATION_SCOPE_CONTEXT_KEY, true), ...)
```

`provider.register()` installed a plain
`AsyncLocalStorageContextManager`, which does not know that key. The
lookup found no scopes on the context and fell back to the
process-global default isolation scope, so every request wrote its
request data into the same object and the last writer won.

The tracer now registers `SentryContextManager`, which subclasses
`AsyncLocalStorageContextManager`, so OTel behaviour is unchanged. It is
also registered on the path where tracing is disabled, which previously
never called `register()` at all and so had no context manager of its
own.

Tenant tags were always correct, because those come from our own async
local storage rather than the isolation scope. That is why the
attribution being wrong was not obvious.

This affects every error report the webapp sends, not just the query
API.

## Verification

`internal-packages/clickhouse`: 76 tests pass, including eight covering
each level decision against a real ClickHouse container. Three pairs pin
the gate open and shut at both layers: an invalid query, a compile
failure, and a real limit breach driven with `max_rows_to_read` each log
at warn with `userAuthoredQuery` and at error without it.

The isolation fix has a test that reproduces the leak before asserting
the fix. Two overlapping requests each tag their own isolation scope;
with the plain context manager the slower one reads back the other's
tag, and with `SentryContextManager` each reads back its own.

Measured separately against a faithful reproduction of the server's
wiring (own OTel pipeline, CommonJS entry) at 200 concurrent requests:
per-request attribution goes from 0.5% to 100%, while span nesting,
context propagation across awaits, and distinct trace IDs are identical
before and after.
 2026-07-30 09:04:15 +01:00
Chris Arderne 2f1734c858 fix(core,webapp): redact sensitive fields in logs by default and cap their size (#4401) 2026-07-29 17:59:47 +01:00
Chris Arderne 8ebc8a41af fix(webapp,redis-worker): stop logging raw metadata, alert payloads, and job items (#4403) 2026-07-29 17:59:36 +01:00
Eric Allam 4eb9292cbe feat(webapp,run-engine): queue metrics and health dashboard (#4131) 
## Summary

Three related changes, each independently gated:

**Queue metrics and health.** Per-queue depth, throughput (enqueued,
started, completed), concurrency, whether a queue is throttled, and
scheduling delay (how long a run waits between becoming eligible and
actually starting), plus a per concurrency-key breakdown for keyed
queues. Collected from inside the run queue itself, stored in
ClickHouse, and surfaced on the Queues list, a new per-queue detail
page, the task pages, and the run inspector. The question it answers is
"does this queue have enough concurrency to keep up, and if not, which
key or which limit is the constraint".

**Percent-based queue concurrency limits.** A queue's concurrency
override can now be expressed as a percentage of the environment limit,
stored as the source of truth and re-materialized whenever the
environment limit changes. Absolute overrides above the environment
limit are now **rejected with a 400** instead of being silently capped,
which is a behavior change on `POST
/api/v1/queues/:queue/concurrency/override`.

**The `health` report.** A server-computed verdict on whether work is
flowing, whether the runs that do start are healthy, and whether
telemetry is fresh, rendered as text with sparklines. Available as `GET
/api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool
(plus a `report` MCP prompt, which shows up as a slash command in hosts
that support prompts).

With the flags off, the Queues page renders the pre-metrics component
verbatim, nothing is emitted, and nothing is written to ClickHouse.

## Configuration

Two independent gates, on purpose. Emission is global so data accrues
for everyone before anyone can look at it; the view is per organization
so it can be turned on for one org at a time without a deploy.

**Runtime flags (no restart)**

| Flag | Store | Gates |
| --- | --- | --- |
| `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by
default) | All emission, gauges and counters. Cached in-process for 10s
with stale-while-revalidate, warmed eagerly at boot so the first op
after a deploy is not dropped. |
| `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` |
Fraction of queue ops that emit a gauge. Counters are never sampled, so
throughput stays exact at any rate. |
| `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag`
row, per-org `Organization.featureFlags` override wins | Whether an org
sees the metrics view at all: the Queues list variant, the queue detail
route, the built-in Queues dashboard, the concurrency-keys endpoint, and
the metrics blocks on task pages and the run inspector. Off by default;
a gated org gets a 404 on the detail route rather than an empty page. |

Both Redis keys are readable and writable from `/admin/queue-metrics`
(super-admin UI, with a live per-shard stream-health table) and
`GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface
uses its own Redis client, so it works on any instance regardless of
whether that instance runs the emitter or the consumer.

**Environment variables (boot time)**

| Variable | Default | Notes |
| --- | --- | --- |
| `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and
injects it into the run engine. Without it the run queue has no emitter
at all. |
| `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on
this instance. Independent of emission, so consumers can be sized
separately from the API. |
| `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per
queue. |
| `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals
insert batch, so an ack can never outrun a write. |
| `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` |
falls back to the run-queue Redis | Set `HOST` to move the metrics
stream onto a dedicated instance so a metrics backlog cannot compete
with the run queue for memory. Self-hosters can leave it unset and get a
single-Redis deployment. |
| `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000`
dedicated | Bound on how much a stalled consumer can hold. The default
is deliberately lower when the stream shares the queue-critical Redis. |
| `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the
per-queue cumulative counter key, refreshed on every write, so only
queues idle for the whole window are purged. |
| `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue
names tracked per environment; overflow collapses into `__overflow__`. |
| `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea
one level down, per queue. |
| `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live
sample-rate key above. |
| `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics
tables in the Query page, its schema docs, the schema API and the AI
query context. Off keeps them unlisted while the feature is dark; a
query naming them still runs either way. |
| `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring |
Runs queue metrics on their own ClickHouse service: the consumer's
inserts and every queue-metrics read go through it, so a metrics-heavy
chart refresh never competes with runs-list or trace reads. Unset
reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`,
reads on the query pool). |
| `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split,
so the consumer's inserts can never land on a read endpoint. |
|
`QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}`
| `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other
per-workload ClickHouse clients. |

Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a
Postgres migration adding the nullable
`TaskQueue.concurrencyLimitOverridePercent`. Both are additive.

## How collection works

Queue operations produce two kinds of signal, and they have opposite
failure modes, so they are handled differently.

**Gauges** (queued, running, queue limit, env queued, env running, env
limit, throttled, plus keys-with-backlog and worst-key wait on keyed
queues) are read *inside* the same Redis script that performs the
enqueue or dequeue, so the reading is atomic with the operation it
describes rather than a racy follow-up read. The script returns them on
its reply and the app forwards them to the stream. Gauges are sampled
and drop-tolerant: they are aggregated with `max`, so a lost reading
costs resolution, never correctness.

**Counters** (enqueued, started, completed, plus nack and dead-lettered)
are cumulative odometers. Each event increments a per-queue key on the
metrics Redis and emits the absolute total, and ClickHouse takes the
difference across buckets at read time. This is the important property
of the design: a summed-delta counter undercounts permanently on any
lost event, while a cumulative one self-heals, because the next
surviving reading restates the whole total. Only bucket granularity can
be lost, never the total. A queue returning after its odometer TTL
expired restarts at 1 and reset detection handles it, which is safe
precisely because expiry only spans a window with no activity.

Both land on one sharded Redis stream. A consumer reads it with a
consumer group, reclaims stale pending entries on a 15s interval rather
than on every poll, maps one entry to one or two ClickHouse rows
(whole-queue and, for keyed queues, per-key), and acks only after the
insert lands. Each batch carries a dedup token derived from its
stream-entry ids, and the target tables set
`non_replicated_deduplication_window`, so a retried batch cannot
double-count either the raw rows or the aggregates that hang off them.
Consumer and emitter both emit OTel metrics
(`queue_metrics.emitter.emitted`,
`queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`);
stream depth and group lag are the two worth alerting on, and
`lag_unknown` exists because Redis can report a null lag after a trim,
which must not be read as zero.

## Storage and read path

`queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four
aggregate tiers are materialized straight from raw, never cascaded off
each other, each with a 30 day TTL:

- `queue_metrics_v1`, 10 second buckets per queue, the default read path
- `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and
cross-queue ranking
- `env_metrics_v1`, 10 second buckets per environment, queue-independent
so it stays cheap at any range
- `queue_metrics_ck_v1`, 10 second buckets per concurrency key

Every tier is an MV from raw because the counter states do not survive a
cascade: their merge is order sensitive, so a `-MergeState` chain off
the 10s table inflates the result, and the same property means an
aggregate state may only be merged inside one queue. That constraint is
now enforced by the query engine rather than by reviewer discipline: a
column can declare a `mergeGroupKey`, and any query that references it
without grouping by, or pinning to a single value of, every named key
fails to compile with an actionable message.

On the read side, TRQL gains three tables (`queue_metrics`,
`env_metrics`, and a `queue_metrics_by_key` that is hidden from the
editor, schema docs and schema API but still queryable, so per-key rows
can never silently merge into a plain per-queue query), plus
`deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level
optimizations ride along: a table can declare coarser rollups, so a
query whose bucket interval is 5 minutes or wider is routed to the 5m
table with no change to the query itself, and it can opt into the
ClickHouse query cache with time bounds floored to a fixed grid, so the
auto-refreshing dashboards actually share cache entries instead of
missing on every tick. Both are caller-side substitutions, so the
printer stays unaware of physical layout.

All of this can also live on its own ClickHouse service. A table
declares the pool its reads run on, the three queue-metrics tables name
the dedicated one, and the ingestion consumer writes through the same
client, so both directions move together with one env var and nothing
else routes differently.

The other engine change is opt-in gap filling: charts can request rows
for empty buckets, where counters zero-fill and gauges carry forward.
Grouped gauge series are densified per group and carried inside a
partition, so a quiet queue's line holds its last value without bleeding
another queue's value into it.

## Queue concurrency limits

`concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth
when an override is set as a percentage; the absolute `concurrencyLimit`
is materialized from it (floored, clamped to at least 1 so a percentage
can never act as a pause, and never above the environment limit). Every
path that changes an environment limit now recalculates the
environment's percent-based overrides afterwards, outside the
transaction, and pushes changed limits to the engine. The push is
attempted even when the stored value did not change, so a previously
failed sync self-heals rather than leaving the database and the engine
diverged; paused queues are skipped so a recalculation cannot
effectively unpause one.

The API accepts exactly one of `concurrencyLimit` or `percent`, and the
reject-instead-of-clamp change above means a request asking for more
than the environment allows now fails loudly. The percent bound (greater
than 0, at most 100) is defined once and shared by the zod schema, the
dashboard mutation handler and the service, so the three cannot drift.

The concurrency-keys table on a queue is now paginated against the
ClickHouse per-key tier, ranked by peak backlog with the total on every
row from a single scan, and only the keys on the current page are
enriched with live counts from Redis. That replaces a hard top-50 cap
with something whose cost is a function of page size rather than key
cardinality.

## The health report

`GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The
verdict is computed on the server and is deterministic, not
model-generated. Three independent analyzers run over one input
snapshot: flow (is work moving, and if not, is the cause a limit,
throttling, one bad queue, or dead-lettering), execution (are the runs
that start succeeding, and at what latency), and liveness (how fresh is
the telemetry). When telemetry is genuinely stale, the first two are
forced to unknown and every actionable field is stripped, so no surface
ever advises action off stale data.

Authorization is per query table rather than a blanket query grant: a
JWT must be scoped to every table the report reads (`runs`,
`env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull
a report that reads more than it was granted. `period` is validated as a
shorthand with a 90 day ceiling at the edge. The report catalog is a
registry of `{ load, interpret }` entries, so the next report is a new
entry and no change to the route, the view model, the renderers, the CLI
or the MCP tool.

`trigger mcp` no longer launches the install wizard when stdout is a
TTY, which fixed a real failure: hosts spawn the server over a PTY, so
the wizard would open and the client would time out waiting for a server
that never started. The wizard now needs `trigger mcp --install`.

## The part that is live regardless of every flag

The enqueue and dequeue scripts now return a 2-tuple so a gauge reading
can ride back on the reply. Every return site in the eight affected
scripts is wrapped, and a `nil` original is converted to `false` on the
way out, because a raw `nil` in the first slot would make Lua truncate
the multi-bulk reply and silently drop the gauge on the throttled and
empty-queue paths. The reply shape and the destructuring on the app side
are exercised on every queue operation whether or not metrics are
enabled, so that is the part of `run-engine` worth the closest review.

One behavior fix in the same area: the scheduling-delay anchor is set
only on a run's first entry into the queue. Anchoring it to trigger time
on re-enqueues made waitpoint and checkpoint resumes report the entire
wait as scheduling delay. Queue ordering is untouched, so a re-enqueued
run keeps its position, and nacks deliberately keep the original anchor
because a rolled-back dequeue is the same continuous wait.

A pending-version promotion still anchors to trigger time, on purpose:
that promotion is the run's first real entry into the queue, since the
trigger deliberately held it back waiting for a worker version, and the
TTL is armed at the same point for the same reason. The consequence is
worth naming, because it is a judgement call: a run that waits on a
deployment reports that wait as scheduling delay on its queue, which is
time unrelated to queue capacity.

## Verification

Unit and integration suites across the new package, the run queue, the
mapping layer, the query engine and ClickHouse (including a test that
applies migration 036 through the same splitter CI uses, and a
regression test that inserts the same batch three times to prove the
aggregates do not inflate). Beyond that, the whole path was driven end
to end against a live stack with real runs: emitter to Redis stream to
consumer to ClickHouse to the dashboards, for both the local dev path
and the deployed path where a supervisor drives the dequeue, with
assertions on exact counter reconstruction per queue and per concurrency
key, throttling, environment saturation, scheduling delay, and a
deliberate mid-stream reading drop to confirm the cumulative counters
still reconstruct the correct total. The gated-off state was checked on
every touched surface.

The dedicated ClickHouse service was verified against a second,
separately-schema'd instance: with it configured, the driven counters
reconstruct exactly on the dedicated instance, the shared instance gains
no rows for that window, a read through the query API returns the value
that exists only on the dedicated instance, and a `runs` query still
succeeds (it would fail outright if it were mis-routed to a service
without that table). With the variable unset, the full suite passes
unchanged.

---------

Co-authored-by: Katia Bulatova <katia@trigger.dev>
Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com>
Co-authored-by: James Ritchie <james@trigger.dev>
 2026-07-29 16:45:24 +01:00