Dropping the column was irreversible and blocked on a production row count; retiring it is not. The avatar host is an exact origin the app already knows, like the GitHub one.
The system — contracts, storage, auth, the agent package and its webapp routes — lands first; the panel, the page-context marks and the entry points follow in their own PR.
## What
A trigger request carrying a Unicode NUL (`U+0000`) in the **idempotency
key** or **debounce key** reached `prisma.taskRun.create()` and failed
the insert, so the caller got an opaque 500 and the run was never
created.
These two keys are stored in `jsonb` columns (`idempotencyKeyOptions`,
`debounce`), and Postgres rejects a NUL inside a `jsonb` value with
`SQLSTATE 22P05` ("unsupported Unicode escape sequence ... cannot be
converted to text"). This fix strips the NUL from both keys at the
single trigger-input chokepoint (`#buildEngineTriggerInput`), which
every trigger path flows through (single, batch item, mollified, and
drainer replay).
Stripping matches the existing precedent for run errors and task events.
It does not change dedup behaviour: the idempotency **dedup identity**
is the hashed key (a clean 64-char digest), computed independently of
the raw key we clean, so dedup keeps working exactly as before. For
debounce the key is used directly, so the cleaned key also becomes the
grouping key, an acceptable change for input that is already malformed.
## Why not payload / metadata / tags
Those are `text` columns fed by `JSON.stringify`, which escapes a NUL to
a safe escape sequence, so they do not hit this failure on the normal
JSON path. (A raw NUL in a `text` column throws a different code,
`22021`, and is not what triggers this issue.) The observed failures are
the `jsonb` `22P05` variant, which is only reachable via the two key
fields.
## Evidence
Red then green (containerTest, real Postgres): with the fix reverted,
triggering through the real service with a NUL in
`idempotencyKeyOptions.key` / `debounce.key` fails with the exact
`22P05` signature; with the fix, the run is created and the stored key
has the NUL removed.
Full-stack e2e (isolated stack, real HTTP): `POST
/api/v1/tasks/:taskId/trigger` with a NUL inside
`idempotencyKeyOptions.key` (`"acme<NUL>inc"`) and, separately,
`debounce.key` (`"grp<NUL>1"`):
- both returned `HTTP 200` with a created run (previously `500`)
- stored `idempotencyKeyOptions` = `{ "key": "acmeinc", "scope": "run"
}` (7 chars, NUL removed)
- stored `debounce.key` = `"grp1"` (4 chars, NUL removed)
- both runs render in the dashboard
Unit tests cover the helper (strip, no-op fast path, object-reference
reuse, null/undefined pass-through).
## Rollout / rollback
Server-only webapp change, no flag. Zero behaviour change for clean
input; only affects inputs that previously 500'd. Rollback is a straight
revert, no data migration.
## Known limitation
A raw NUL in a plain-string idempotency key (not created via
`idempotencyKeys.create()`) lands in a `text` column and throws `22021`
instead. That variant is not addressed here because stripping it would
change the dedup identity, so it warrants a separate decision. Not
observed in practice.
refs TRI-13030
Replaces https://*.googleusercontent.com (a host with public write access)
with CSP_IMG_SRC_ALLOWLIST: exact origins only, https outside development,
deduplicated, bad entries warned about instead of failing the boot.
Route conflicts were the tab-title work meeting the agent page-context handle: both
sides kept, duplicate meta exports resolved to pageMeta, duplicate imports merged
with unused bindings dropped. Lockfile regenerated from the merged manifests.
Adds a shared `pageMeta()` helper and 74 route declarations, so a title
reads `run_abc | Runs | Trigger.dev` — the specific thing first, then
the page. Org pages also carry the organization: `Team | Acme |
Trigger.dev`. Inside a project no scope is added, because the dashboard
switches projects in every tab at once.
Page names are unchanged; what's new is that a page says which one it is
at all. Three wording changes on purpose: the queue page now names the
queue, the model page names the model, and entity pages carry their
section.
The message-size checks ran after the body had been read, so a request without a
content-length was buffered and parsed in full before being refused. An ingress
cap on the agent's paths now counts the bytes as they arrive, and the chat proxy
reads its body with a ceiling instead of reading it whole first.
## Summary
Every Prisma client built its connection URL with a `connection_timeout`
query param, but the Postgres connector's parameter is
`connect_timeout`. The misspelled param is silently ignored, so all
clients fell back to Prisma's 5s default instead of the configured
timeout. When establishing a new connection briefly took longer than 5s
(for example during connection spikes), it failed with `Can't reach
database server` even though the database was healthy.
## Fix
All four client builders now construct their connection URL through one
shared helper (`buildPrismaConnectionUrl`) that sets `connect_timeout`,
so the configured value actually applies, and the parameter name lives
in exactly one place. Covered by a unit test.
## Summary
Upgrade the monorepo to TypeScript 7.0.2 and update package build
tooling for compatibility with the native compiler.
## Design
Package builds now use `tshy` 4, while the packages still using `tsup`
move to `tsdown`. The few scripts that depend on the legacy TypeScript
compiler API use an explicit TypeScript 6 alias; declaration portability
coverage invokes the TypeScript 7 CLI directly.
Turbo is updated so workspace tasks can read the regenerated pnpm
lockfile.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds System Preferences, Dark and Light themes, gated by the
`hasThemeSwitcher` feature flag (off by default — dark stays the default
theme for everyone).
Old theme is now "Classic"and set as default.
"System preferences" theme has both Light and Dark modes and uses your
laptop settings to use a correct one.
It has less color accents (specifically less colored text), and they are
the same for both modes, only grayscale values change between them. And
Light/Dark themes can be used separately.
New Contrast setting is available for System Preferences, Dark and Light
themes - it changes the contrast for the whole app. All new visual
Settings live in Account.
## Summary
Environment API keys backed by the additional-key table can authenticate
API requests using their stored effective scopes. Revoked and expired
keys are rejected, branch environments retain their existing routing
behavior, and last-used timestamps are updated on a throttled
best-effort basis.
## Design
API route builders receive the resolved ability and reject restricted
keys on routes without an authorization declaration. Existing
deployment, environment variable, queue, run, task, batch, session, and
waitpoint routes declare the resources they access.
Trigger and batch responses return server-signed public access tokens,
so additional keys never need access to the environment signing secret.
Root-key rotation also keeps public tokens valid for the existing grace
window.
## Feature notes
- Root environment keys remain unrestricted for backward compatibility.
Additional keys enforce their persisted scopes and fail closed on routes
without an authorization declaration.
- Machine-key requests never exchange one credential for another.
Additional keys cannot retrieve the root key, and rotated root keys are
not upgraded
during their grace window.
- Public JWT validation remains host-owned, while installed RBAC plugins
continue to supply root-key abilities.
- Unfiltered session and run listings preserve existing broad task-read
behavior. Filtered requests enforce the supplied task identifiers.
- Related-run summaries remain embedded in run retrieval for API
compatibility. Retrieving or mutating a related run independently still
requires
permission for that run.
- Queue management authorizes at collection scope, matching the queue
permissions currently issued.
- Batch responses deliberately include server-signed public access
tokens for all clients. Selected-task credentials continue using their
original
credential for per-item authorization.
- Two-phase batches authorize declared task identifiers before creation
and authorize every streamed item. Streaming paths that cannot declare
the
complete task set remain fail closed.
- Authentication telemetry records successful credential resolution
separately from subsequent resource-authorization failures.
- API keys are high-entropy random tokens. SHA-256 is intentionally used
for deterministic indexed lookup, not password hashing.
## Deployment notes
The schema migration must be present before this code is deployed.
Because bearer resolution runs on every authenticated request, deploy
the resolver with additional-key lookup disabled, verify root-key and
public-token parity, then enable lookup before any additional keys can
be issued.
The multi-task authorization tightening changes the result for narrowly
scoped tokens that request tasks outside their grants. Observe
would-deny results before enforcing that check. Request-idempotency keys
are also newly isolated by environment and task, so a retry crossing the
deployment boundary may execute once more before old cache entries
expire.
## Follow-ups
- [x] Add a system-wide kill switch for additional-key lookup, defaulted
off for the initial deployment.
- [x] Add authentication observability by credential kind, result,
latency, and lookup path without recording credential values.
- [ ] ~Add would-deny observability and an independent enforcement
switch for multi-task authorization.~
- [ ] ~Add an independent switch for server-issued batch tokens while
root-key parity is verified.~
- [ ] Confirm every API route reachable by a restricted key has an
explicit authorization declaration or intentionally fails closed.
- [x] Verify root-key rotation, revoked-key grace, and public-token
validation through each bearer resolver path.
Members added by SSO just-in-time provisioning or Directory Sync never
got
their per-member DEVELOPMENT environments - only invite acceptance and
project creation created them. `trigger dev` returned "Environment not
found" for those members and the dashboard had no dev view.
ensureOrgMember now queues provisioning for every membership it settles,
so
both paths are covered and members missing environments are repaired on
their next sync. Provisioning runs as a common-worker job to keep
sign-in
and directory webhooks off the per-project write loop. A failed enqueue
surfaces for Directory Sync, whose worker retries the idempotent effect,
and is swallowed for sign-in, where the next login enqueues again.
Environment creation now tolerates a concurrent creator so the
project-creation loop and the job cannot collide on the unique index.
Also fixes environment resolution ignoring dev-environment ownership: a
member without their own dev environment could be handed a colleague's
and
have it persisted as their dashboard preference.
## What
Two changes, shipped together:
1. **SDK migration (TRI-12460).** `@team-plain/typescript-sdk` is
deprecated. Move the webapp to its successors — `@team-plain/graphql`
(client) and `@team-plain/ui-components` (`uiComponent` builder).
Behaviour-preserving: the `PlainClient` customer upsert + thread
creation move to the new `client.mutation.*({ input })` shape; the
client now throws on failure, so `sendToPlain` wraps its calls and logs,
staying best-effort.
2. **Org tenant attribution (TRI-12461).** When org context is
available, `sendToPlain` now upserts a Plain tenant keyed by `externalId
= org_id`, links the customer to it, and stamps the created thread with
that tenant — so support threads become attributable to a Trigger.dev
org. Wired into the four add-on quota requests and the plan-cancellation
feedback (which already have org context). The tenant steps are isolated
in their own try/catch and the thread's `tenantIdentifier` is gated on
their success, so a tenant failure never blocks thread creation.
## Not affected
- `customer.externalId` stays `User.id` — the customer cards +
impersonation link are unchanged.
- No ticket content leaves Plain.
- Callers without a single org (e.g. the feedback widget) are unchanged
— the org params are optional.
## Deploy prerequisite
The webapp's Plain API key needs three **new** scopes for attribution to
work (it already has `customer:create`, `customer:edit`,
`thread:create`):
- [x] `tenant:create`
- [x] `tenant:edit`
- [x] `customerTenantMembership:create`
Until granted, nothing breaks — `sendToPlain` logs the forbidden error
and creates the thread without attribution.
## Testing
- `pnpm typecheck --filter webapp` passes; oxfmt + oxlint clean.
- Ran the real `sendToPlain` end-to-end via a throwaway vitest harness
against live Plain — confirmed the code path executes; the live write is
gated only by the key scopes above.
Adds the storage model and authorization contracts needed for multiple
environment API keys. Credentials are represented by hashed values,
revocation and expiration state, and persisted effective scopes.
The built-in authorization fallback exposes full-access policy
preparation, while optional authorization extensions can supply
additional presets and task-aware scope generation. This change does not
create, display, or authenticate additional keys.
## Summary
Three related changes, each independently gated:
**Queue metrics and health.** Per-queue depth, throughput (enqueued,
started, completed), concurrency, whether a queue is throttled, and
scheduling delay (how long a run waits between becoming eligible and
actually starting), plus a per concurrency-key breakdown for keyed
queues. Collected from inside the run queue itself, stored in
ClickHouse, and surfaced on the Queues list, a new per-queue detail
page, the task pages, and the run inspector. The question it answers is
"does this queue have enough concurrency to keep up, and if not, which
key or which limit is the constraint".
**Percent-based queue concurrency limits.** A queue's concurrency
override can now be expressed as a percentage of the environment limit,
stored as the source of truth and re-materialized whenever the
environment limit changes. Absolute overrides above the environment
limit are now **rejected with a 400** instead of being silently capped,
which is a behavior change on `POST
/api/v1/queues/:queue/concurrency/override`.
**The `health` report.** A server-computed verdict on whether work is
flowing, whether the runs that do start are healthy, and whether
telemetry is fresh, rendered as text with sparklines. Available as `GET
/api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool
(plus a `report` MCP prompt, which shows up as a slash command in hosts
that support prompts).
With the flags off, the Queues page renders the pre-metrics component
verbatim, nothing is emitted, and nothing is written to ClickHouse.
## Configuration
Two independent gates, on purpose. Emission is global so data accrues
for everyone before anyone can look at it; the view is per organization
so it can be turned on for one org at a time without a deploy.
**Runtime flags (no restart)**
| Flag | Store | Gates |
| --- | --- | --- |
| `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by
default) | All emission, gauges and counters. Cached in-process for 10s
with stale-while-revalidate, warmed eagerly at boot so the first op
after a deploy is not dropped. |
| `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` |
Fraction of queue ops that emit a gauge. Counters are never sampled, so
throughput stays exact at any rate. |
| `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag`
row, per-org `Organization.featureFlags` override wins | Whether an org
sees the metrics view at all: the Queues list variant, the queue detail
route, the built-in Queues dashboard, the concurrency-keys endpoint, and
the metrics blocks on task pages and the run inspector. Off by default;
a gated org gets a 404 on the detail route rather than an empty page. |
Both Redis keys are readable and writable from `/admin/queue-metrics`
(super-admin UI, with a live per-shard stream-health table) and
`GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface
uses its own Redis client, so it works on any instance regardless of
whether that instance runs the emitter or the consumer.
**Environment variables (boot time)**
| Variable | Default | Notes |
| --- | --- | --- |
| `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and
injects it into the run engine. Without it the run queue has no emitter
at all. |
| `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on
this instance. Independent of emission, so consumers can be sized
separately from the API. |
| `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per
queue. |
| `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals
insert batch, so an ack can never outrun a write. |
| `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` |
falls back to the run-queue Redis | Set `HOST` to move the metrics
stream onto a dedicated instance so a metrics backlog cannot compete
with the run queue for memory. Self-hosters can leave it unset and get a
single-Redis deployment. |
| `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000`
dedicated | Bound on how much a stalled consumer can hold. The default
is deliberately lower when the stream shares the queue-critical Redis. |
| `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the
per-queue cumulative counter key, refreshed on every write, so only
queues idle for the whole window are purged. |
| `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue
names tracked per environment; overflow collapses into `__overflow__`. |
| `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea
one level down, per queue. |
| `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live
sample-rate key above. |
| `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics
tables in the Query page, its schema docs, the schema API and the AI
query context. Off keeps them unlisted while the feature is dark; a
query naming them still runs either way. |
| `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring |
Runs queue metrics on their own ClickHouse service: the consumer's
inserts and every queue-metrics read go through it, so a metrics-heavy
chart refresh never competes with runs-list or trace reads. Unset
reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`,
reads on the query pool). |
| `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split,
so the consumer's inserts can never land on a read endpoint. |
|
`QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}`
| `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other
per-workload ClickHouse clients. |
Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a
Postgres migration adding the nullable
`TaskQueue.concurrencyLimitOverridePercent`. Both are additive.
## How collection works
Queue operations produce two kinds of signal, and they have opposite
failure modes, so they are handled differently.
**Gauges** (queued, running, queue limit, env queued, env running, env
limit, throttled, plus keys-with-backlog and worst-key wait on keyed
queues) are read *inside* the same Redis script that performs the
enqueue or dequeue, so the reading is atomic with the operation it
describes rather than a racy follow-up read. The script returns them on
its reply and the app forwards them to the stream. Gauges are sampled
and drop-tolerant: they are aggregated with `max`, so a lost reading
costs resolution, never correctness.
**Counters** (enqueued, started, completed, plus nack and dead-lettered)
are cumulative odometers. Each event increments a per-queue key on the
metrics Redis and emits the absolute total, and ClickHouse takes the
difference across buckets at read time. This is the important property
of the design: a summed-delta counter undercounts permanently on any
lost event, while a cumulative one self-heals, because the next
surviving reading restates the whole total. Only bucket granularity can
be lost, never the total. A queue returning after its odometer TTL
expired restarts at 1 and reset detection handles it, which is safe
precisely because expiry only spans a window with no activity.
Both land on one sharded Redis stream. A consumer reads it with a
consumer group, reclaims stale pending entries on a 15s interval rather
than on every poll, maps one entry to one or two ClickHouse rows
(whole-queue and, for keyed queues, per-key), and acks only after the
insert lands. Each batch carries a dedup token derived from its
stream-entry ids, and the target tables set
`non_replicated_deduplication_window`, so a retried batch cannot
double-count either the raw rows or the aggregates that hang off them.
Consumer and emitter both emit OTel metrics
(`queue_metrics.emitter.emitted`,
`queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`);
stream depth and group lag are the two worth alerting on, and
`lag_unknown` exists because Redis can report a null lag after a trim,
which must not be read as zero.
## Storage and read path
`queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four
aggregate tiers are materialized straight from raw, never cascaded off
each other, each with a 30 day TTL:
- `queue_metrics_v1`, 10 second buckets per queue, the default read path
- `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and
cross-queue ranking
- `env_metrics_v1`, 10 second buckets per environment, queue-independent
so it stays cheap at any range
- `queue_metrics_ck_v1`, 10 second buckets per concurrency key
Every tier is an MV from raw because the counter states do not survive a
cascade: their merge is order sensitive, so a `-MergeState` chain off
the 10s table inflates the result, and the same property means an
aggregate state may only be merged inside one queue. That constraint is
now enforced by the query engine rather than by reviewer discipline: a
column can declare a `mergeGroupKey`, and any query that references it
without grouping by, or pinning to a single value of, every named key
fails to compile with an actionable message.
On the read side, TRQL gains three tables (`queue_metrics`,
`env_metrics`, and a `queue_metrics_by_key` that is hidden from the
editor, schema docs and schema API but still queryable, so per-key rows
can never silently merge into a plain per-queue query), plus
`deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level
optimizations ride along: a table can declare coarser rollups, so a
query whose bucket interval is 5 minutes or wider is routed to the 5m
table with no change to the query itself, and it can opt into the
ClickHouse query cache with time bounds floored to a fixed grid, so the
auto-refreshing dashboards actually share cache entries instead of
missing on every tick. Both are caller-side substitutions, so the
printer stays unaware of physical layout.
All of this can also live on its own ClickHouse service. A table
declares the pool its reads run on, the three queue-metrics tables name
the dedicated one, and the ingestion consumer writes through the same
client, so both directions move together with one env var and nothing
else routes differently.
The other engine change is opt-in gap filling: charts can request rows
for empty buckets, where counters zero-fill and gauges carry forward.
Grouped gauge series are densified per group and carried inside a
partition, so a quiet queue's line holds its last value without bleeding
another queue's value into it.
## Queue concurrency limits
`concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth
when an override is set as a percentage; the absolute `concurrencyLimit`
is materialized from it (floored, clamped to at least 1 so a percentage
can never act as a pause, and never above the environment limit). Every
path that changes an environment limit now recalculates the
environment's percent-based overrides afterwards, outside the
transaction, and pushes changed limits to the engine. The push is
attempted even when the stored value did not change, so a previously
failed sync self-heals rather than leaving the database and the engine
diverged; paused queues are skipped so a recalculation cannot
effectively unpause one.
The API accepts exactly one of `concurrencyLimit` or `percent`, and the
reject-instead-of-clamp change above means a request asking for more
than the environment allows now fails loudly. The percent bound (greater
than 0, at most 100) is defined once and shared by the zod schema, the
dashboard mutation handler and the service, so the three cannot drift.
The concurrency-keys table on a queue is now paginated against the
ClickHouse per-key tier, ranked by peak backlog with the total on every
row from a single scan, and only the keys on the current page are
enriched with live counts from Redis. That replaces a hard top-50 cap
with something whose cost is a function of page size rather than key
cardinality.
## The health report
`GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The
verdict is computed on the server and is deterministic, not
model-generated. Three independent analyzers run over one input
snapshot: flow (is work moving, and if not, is the cause a limit,
throttling, one bad queue, or dead-lettering), execution (are the runs
that start succeeding, and at what latency), and liveness (how fresh is
the telemetry). When telemetry is genuinely stale, the first two are
forced to unknown and every actionable field is stripped, so no surface
ever advises action off stale data.
Authorization is per query table rather than a blanket query grant: a
JWT must be scoped to every table the report reads (`runs`,
`env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull
a report that reads more than it was granted. `period` is validated as a
shorthand with a 90 day ceiling at the edge. The report catalog is a
registry of `{ load, interpret }` entries, so the next report is a new
entry and no change to the route, the view model, the renderers, the CLI
or the MCP tool.
`trigger mcp` no longer launches the install wizard when stdout is a
TTY, which fixed a real failure: hosts spawn the server over a PTY, so
the wizard would open and the client would time out waiting for a server
that never started. The wizard now needs `trigger mcp --install`.
## The part that is live regardless of every flag
The enqueue and dequeue scripts now return a 2-tuple so a gauge reading
can ride back on the reply. Every return site in the eight affected
scripts is wrapped, and a `nil` original is converted to `false` on the
way out, because a raw `nil` in the first slot would make Lua truncate
the multi-bulk reply and silently drop the gauge on the throttled and
empty-queue paths. The reply shape and the destructuring on the app side
are exercised on every queue operation whether or not metrics are
enabled, so that is the part of `run-engine` worth the closest review.
One behavior fix in the same area: the scheduling-delay anchor is set
only on a run's first entry into the queue. Anchoring it to trigger time
on re-enqueues made waitpoint and checkpoint resumes report the entire
wait as scheduling delay. Queue ordering is untouched, so a re-enqueued
run keeps its position, and nacks deliberately keep the original anchor
because a rolled-back dequeue is the same continuous wait.
A pending-version promotion still anchors to trigger time, on purpose:
that promotion is the run's first real entry into the queue, since the
trigger deliberately held it back waiting for a worker version, and the
TTL is armed at the same point for the same reason. The consequence is
worth naming, because it is a judgement call: a run that waits on a
deployment reports that wait as scheduling delay on its queue, which is
time unrelated to queue capacity.
## Verification
Unit and integration suites across the new package, the run queue, the
mapping layer, the query engine and ClickHouse (including a test that
applies migration 036 through the same splitter CI uses, and a
regression test that inserts the same batch three times to prove the
aggregates do not inflate). Beyond that, the whole path was driven end
to end against a live stack with real runs: emitter to Redis stream to
consumer to ClickHouse to the dashboards, for both the local dev path
and the deployed path where a supervisor drives the dequeue, with
assertions on exact counter reconstruction per queue and per concurrency
key, throttling, environment saturation, scheduling delay, and a
deliberate mid-stream reading drop to confirm the cumulative counters
still reconstruct the correct total. The gated-off state was checked on
every touched surface.
The dedicated ClickHouse service was verified against a second,
separately-schema'd instance: with it configured, the driven counters
reconstruct exactly on the dedicated instance, the shared instance gains
no rows for that window, a read through the query API returns the value
that exists only on the dedicated instance, and a `runs` query still
succeeds (it would fail outright if it were mis-routed to a service
without that table). With the variable unset, the full suite passes
unchanged.
---------
Co-authored-by: Katia Bulatova <katia@trigger.dev>
Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com>
Co-authored-by: James Ritchie <james@trigger.dev>
- typed route handle + useAgentPageContext (useMatches mapper, defensive
fallback to {kind: other}); runs page ships the reference handle
- panel folds AgentPageContext into clientData on create and every turn
- launcher context gains openWith(text): new chat sends immediately, an
open chat gets the text in its composer
- view blocks render latest revision per (type, id) with legacy
no-envelope fallback; pure grouping utils + tests; storybook states
## Summary
Limits user account email addresses to 254 characters in profile
settings and onboarding. Oversized values are rejected before the
uniqueness lookup, and the form fields enforce the same limit in the
browser.
## Fix
Both email update flows use a shared bounded email schema. Basic
validation completes before the uniqueness lookup runs.
## Problem
Several display/grouping issues in the **Errors** feature, all rooted in
how the ClickHouse error materialized views (`errors_mv_v1`,
`error_occurrences_mv_v1`) read the stored error JSON produced by
`parseError`:
1. **Messageless errors show "Unknown error".** An empty message falls
straight through `coalesce(nullIf(message,''), 'Unknown error')` to the
literal, even though the error's class `name` is available (e.g. an
Effect tagged error `ListMessagesError` with no message).
2. **Unrelated errors collapse into one group.**
`calculateErrorFingerprint` keys on `type : message : stack`, where
`type` is always the union tag (`BUILT_IN_ERROR`, …), `message` is
empty, and the stack isn't read — so every messageless built-in error
(and every string/custom error) hashes to the same constant input → one
fingerprint.
3. **error_type shows the internal tag.** `coalesce(type, name, …)`
always resolves to `type` (always present), so the column shows
`BUILT_IN_ERROR` instead of the real class name.
4. **Stack traces never populate.** The MVs read `error.data.stack`, but
the serializer stores the trace under `stackTrace` — so the column is
always empty.
## Fix
All display changes are `ALTER TABLE … MODIFY QUERY` on the two views
(migration `035`); the fingerprint change is in the webapp.
- **Fingerprint** (`errorFingerprinting.ts`): fall back **message → name
→ raw**. Messageless errors now group by class name (or raw value for
non-Error throws); message-bearing errors are **unchanged**
(short-circuits at `message`), so existing groups don't split — only
currently-messageless errors get their own group going forward.
- **error_message**: same `message → name → raw` fallback before
`'Unknown error'`.
- **error_type**: coalesce `name → code → 'Error'` (drops the reliance
on the union tag). Built-in → class name, internal → `code`,
string/custom → `Error`.
- **stack trace**: read `error.data.stackTrace`. Bounded as before
(serializer caps 50 frames / 1024 chars per line; MV clips to 2000
chars).
## Migration notes
- `MODIFY QUERY` swaps the view query in place (no drop/recreate gap);
Down restores the previous query.
- **Existing rows are left unchanged** — changes apply only to rows
inserted after the migration. No backfill.
## Tests
`errorFingerprinting.test.ts` — 57 pass, incl. new cases for messageless
class names, string/custom raw values, and stability of message-bearing
fingerprints.
Fixes the display-derivation half of TRI-11938 (error_type + stack
trace); relates to TRI-9254 and TRI-9250.
## Summary
Magic link login could appear completely broken: submitting your email
on the login page showed a stale "This email is unauthorized" error
instead of the "we've sent you a magic link" confirmation, even when the
address was fine.
This PR reverts
[#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) (whose
diagnosis and fix turned out to be wrong) and fixes the actual bug,
which was in how login errors are stored and consumed.
## Root cause
Two session bugs compounded on the login page:
- The `/login` loader read the flashed `auth:error` without committing
the session. A Remix flash is only consumed when the session is
committed after the read, so once any attempt flashed an error (for
example an address rejected on an instance with `WHITELISTED_EMAILS`
set), it stayed in the session cookie and reappeared on every later
`/login` visit, making successful attempts look like failures.
- The `/login/magic` action stored its validation and rate limit errors
with `session.set`, which survives every later read and commit, so those
errors stuck permanently.
[#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) had
instead diagnosed a server-only module leaking into the client bundle
and crashing navigation. Checking the shipped images' client bundles via
their sourcemaps shows `.server` modules were always stubbed out, so
that change fixed nothing and is reverted here.
## Fix
- `/login` reads the flashed error and commits the session when one was
present, so an error renders once and clears. The `redirectTo` branch
now surfaces the error too instead of leaving it in the cookie.
- `/login/magic` flashes its errors instead of `set`ting them.
Verified end-to-end on a live preview environment: a rejected address
shows the error once and a reload clears it; a valid address lands on
the confirmation screen with the address named; GitHub, Google, and SSO
login paths are untouched by this diff.
## Problem
The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.
## Fix: change the encoding, not the structure
Mint a **26-char lowercase base32hex** run id:
```
run_<24-char base32hex core><region char><version char>
[ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```
- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.
DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.
## Residency discriminator: length → version char
`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.
## Scope / verification
- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.
## Open decisions (flagged, not silently chosen)
1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.
---
## ⚠️ Required migration — deploy in lockstep
This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):
1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
- key `runOpsMintKsuid` → `runOpsMintKind`
- value `"ksuid"` → `"runOpsId"`
Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
The **Run ID** and **Batch ID** filters on the runs list, batches list,
and logs view rejected valid IDs. The input showed an error and the
**Apply** button stayed disabled, so filtering by an affected run or
batch ID from the dashboard was impossible.
The filter validators hard-coded exact friendly-id character lengths.
Friendly IDs come in three generations that all still exist in the data
(`<prefix>_` plus a 21-char nanoid, a 25-char cuid, or a 27-char ksuid),
and the hard-coded lengths never covered all three at once.
## Fix
All the ID filter validators (run, batch, waitpoint, schedule) now share
one helper, `makeFriendlyIdValidator`
(`apps/webapp/app/utils/friendlyId.ts`), which validates by prefix plus
a base62 body of any known generator length (21 / 25 / 27). The cuid and
ksuid lengths are sourced from core so the helper tracks any future
change to those formats. Unit tests assert it accepts the output of the
real id generators and rejects malformed input.
Downstream was already unaffected: run/batch route params and
URL-applied filters use unconstrained validation, so only the manual
filter inputs needed the fix.
## Summary
Two related timezone bugs in the dashboard.
1. The date/time tooltip could show a UTC offset label that contradicted
the time it displayed. A viewer whose machine clock differs from their
saved timezone (or when a date falls in the other DST phase) would see
something like `Local (UTC +0)` next to a value that isn't at +0.
2. A user's timezone preference silently failed to save whenever their
browser reported a zone like `UTC`, `Etc/UTC`, or `Asia/Kolkata`,
leaving their timestamps stuck in a previously-saved timezone.
## Offset label
The "Local" row formatted its time using the viewer's configured
timezone but computed the `(UTC +n)` label from `new
Date().getTimezoneOffset()`, the browser's offset at the current moment.
Those are two independent sources, so they disagreed when the configured
timezone differed from the machine, and also when the displayed date was
in the opposite DST phase. The label is now derived from the same date
and timezone used to render the row (via `Intl.DateTimeFormat` with
`timeZoneName: "longOffset"`), so it always matches the displayed time.
## Preference persistence
`/resources/timezone` validated the incoming zone against
`Intl.supportedValuesOf("timeZone")`, which lists only canonical zone
ids. Browsers report zones that aren't in that list via
`resolvedOptions().timeZone`, notably `UTC` (and `Etc/UTC`,
`Asia/Kolkata`, `GMT`), so those requests returned 400 and the
preference was never stored. Validation now checks whether the runtime
can resolve the zone at all, which accepts every real zone and still
rejects invalid input.
Added unit tests for both.
Once this is merged, oxlint is at a pretty sensible baseline.
**Enable `no-unused-vars`, `typescript/consistent-type-imports`, and
`import/no-duplicates` lint rules**
Turns on three previously-disabled oxlint rules across the monorepo and
fixes all violations:
- **`no-unused-vars`** – enabled as an error with standard ignore
patterns: unused function arguments are ignored by default (`args:
"none"`), variables/caught errors/destructured array elements prefixed
with `_` are allowed, and rest siblings are permitted.
- **`typescript/consistent-type-imports`** – enforced as an error; all
type-only imports now use the `import type` syntax.
- **`import/no-duplicates`** – enforced as an error; duplicate import
statements from the same module have been merged.
The remaining commits clean up the violations found across the codebase:
removing unused variables/imports/type aliases, adding `_` prefixes to
intentionally unused bindings, fixing duplicate imports, and converting
value imports to `import type` where appropriate.
## Summary
Adds Billing Limits to the webapp.
Customers can set a monthly spend cap. When usage crosses the limit,
billable environments enter a grace period. If the limit is not resolved
before grace expires, new triggers are rejected until the organization
increases or removes the limit.
Closes this feature request:
[https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances)
### Feature notes:
- CLI `trigger dev` works as before
- `trigger dev --branch my-branch` to create a new branch and run
against it.
- `trigger dev archive --branch my-branch` to archive (or in webapp).
- New webapp page to manage and archive dev branches, currently feature
flagged.
### Implementation details:
- No changes to data model, no backfill. `isBranchableEnvironment`
column is ignored for dev branches, we use `parentEnvironmentId IS NULL`
instead.
- `x-trigger-branch` overloaded for preview and dev branches
- New `TRIGGER_DEV_BRANCH` env var available locally.
`TRIGGER_PREVIEW_BRANCH` overloaded for child runs.
- Lots of new glue code to sanitise the branch checks.
### Rollout
- Deploy webapp/API changes (all backwards compatible)
- Manual tests on some orgs
- Deploy docs, release CLI, flip feature flag for webapp feature
### NB
- `api.v1.projects.$projectRef.environments.ts` will return
`isBranchableEnvironment: true` for all dev environments.
### Prerequisites
- [x] Typecheck will not pass until we make a new release of
`@trigger.dev/platform` and bump it here
SSO settings page: resolve plan before the role check. A non-Enterprise
org now renders the upsell state for every role instead of showing a
"permission denied" panel to non-Owners for a feature their org can't
use yet. manage:sso is only enforced once the org is actually entitled.
Extracts EMPTY_SSO_STATUS and uses throwPermissionDenied().
Also removes the client-side SSO session fetch guard. It monkeypatched
global window.fetch, which made it the initiator of every request and
obfuscated the real call site on any 4xx/5xx. Session revocation is
still enforced server-side on every authenticated request and surfaces
as a logout redirect on the next navigation/refresh, so the client guard
was UX-only and not worth the cross-cutting cost.
## Summary
Several dashboard routes performed actions a restricted role should not
be able to do (cancel or replay runs, manage prompt versions, invite and
manage members, manage billing) without any permission check. This adds
role-based permission enforcement to those routes, and disables the
matching UI controls (with a tooltip) when the current role lacks
permission.
Covered actions:
- Runs: cancel and replay (single, bulk create, bulk abort)
- Prompts: create or edit override versions, and promote a version to
current
- Members: invite, resend invite, revoke invite
- Billing: change plan, billing alerts, and the customer portal
## How
Each affected route now goes through the `dashboardLoader` /
`dashboardAction` route builders with an `authorization` block declaring
the required permission (or a per-intent check where one route handles
several intents). Existing tenancy and data-scoping queries are
untouched; this only layers permission checks on top. The UI follows
disable-don't-hide: controls stay visible but disabled with a "You don't
have permission to ..." tooltip.
Two reusable pieces support this: `checkPermissions(ability, checks)`
turns a set of checks into a boolean map a loader returns to the client,
and `PermissionButton` / `PermissionLink` disable the underlying control
and show a tooltip when a permission flag is false.
## Behaviour
No change in the default configuration: permissions are permissive, so
every control stays enabled and every route behaves as before. The
checks only take effect when an RBAC plugin is installed. This also
makes role assignment on invite-accept non-fatal, so a failure there
cannot block joining an org.
Verified with `pnpm run typecheck --filter webapp`; `checkPermissions`
has unit tests.
## Summary
Prisma infrastructure failures (P1xxx-class: database unreachable, timed
out, connection dropped, engine init/panic) carry the database hostname
in their `.message`. This captures them centrally for observability and
ensures they never reach API clients verbatim.
## Design
A `$allOperations` client extension on the writer and replica clients
logs infrastructure errors with the originating model and operation,
then rethrows the **original** error unchanged — call sites that branch
on `error.code` (unique-violation idempotency, not-found handling) and
transaction retries keep working. Only infrastructure errors are logged;
routine query/validation errors (P2xxx) are left alone.
`$allOperations` can't see the transaction boundary (`$transaction` is a
client method, not an operation), so infrastructure errors surfacing
from `$transaction()` without a Prisma code — e.g.
`PrismaClientInitializationError` — are logged separately at the
transaction wrapper, where the existing coded-error path would otherwise
miss them.
`clientSafeErrorMessage()` swaps an infrastructure error's message for
`"Internal Server Error"` at the API routes that previously returned
`error.message` raw. Status codes, headers, and every non-infrastructure
message are unchanged.
## Test plan
- [x] P2002 / P2025 rethrow with code intact and are not logged
- [x] Statement errors inside `$transaction` keep their code (retry
logic intact)
- [x] Raw queries wrapped without crashing on the undefined model
- [x] A genuine connectivity failure is logged with model/operation/code
- [x] `clientSafeErrorMessage` obfuscates infra messages, preserves all
others
- [x] `pnpm run typecheck --filter webapp` (12/12)
## Note
Overlaps with #3391 (Prisma 7 migration) on
`apps/webapp/app/db.server.ts` — coordinate rebasing.
Adds an opt-in mechanism to route a configurable percentage of
organizations onto the compute (MicroVM) backing of their region at
trigger time, without changing their stored region settings.
Routing is gated by three global feature flags -
`computeMigrationEnabled`, `computeMigrationFreePercentage`,
`computeMigrationPaidPercentage` - plus a per-org
`computeMigrationEnabled` override that wins in both directions. A
region's compute backing is resolved from a new
`WorkerInstanceGroup.region` column: a container group and its MicroVM
group share one geo `region`, so the migration swaps the resolved worker
queue to the backing group's queue. Orgs are bucketed deterministically
by id, so ramping a percentage down keeps a strict subset rather than
reshuffling, and a region with no compute backing is never touched.
Everything is off by default - behaviour is unchanged unless the flags
are set.
The flags and the worker-region groups are read on the trigger hot path
from in-memory snapshots rather than the database: a small
`createReloadingRegistry` helper loads each at startup and refreshes
them on an interval, so no per-trigger query is added and a percentage
or kill-switch change propagates within the reload interval. A cold
replica whose snapshot hasn't loaded yet reads as not-migrated (the
container path) and self-corrects on the next load - the same cold-start
contract as the datastore / LLM-pricing registries, with a
`reloading_registry_loaded` metric so a never-loaded registry is
alertable.
The same migration decision is consulted at deploy-time template
creation so a migrated org gets a compute template built ahead of its
first run. This runs in shadow mode (best-effort, never fails the
deploy) by default, or - when the `computeMigrationRequireTemplate` flag
is on - in required mode, built synchronously at deploy so the first run
never builds on-demand and template errors surface at deploy time.
So operators keep "which runs ran where" while customers only see
geography: the run's actual worker queue is stored raw, and the geo
region is stamped separately on `TaskRun.region` (and a new ClickHouse
`region` column) at trigger time. Read surfaces - the dashboard, the
API, and the Query/Logs page - show the geo region, falling back to the
worker queue for runs written before the column existed.
Minor follow-ups left out of scope: the percentage flags render as text
inputs on the admin flags page (the catalog UI has no numeric control
type yet), and `createReloadingRegistry` could later gain pub/sub for
sub-second cross-replica propagation if the reload interval proves too
slow.
Katia Bulatova