Commit Graph

4572 Commits

Author SHA1 Message Date
Eric Allam c17b1cd4b6 feat(webapp,tsql): hide the per-key metrics table from user-facing query surfaces
TableSchema gains a hidden flag: hidden tables stay fully queryable
through the engine (the concurrency-keys tab keeps working, tenancy and
the merge guard still apply) but are excluded from the query editor
autocomplete, the Query page schema and examples panels, the AI query
generator context, and the schema API. queue_metrics_by_key is hidden
until the per-key surface is settled.
2026-07-13 20:39:15 +01:00
Eric Allam 046fcdae82 feat(webapp): move concurrency-key metrics to a tab on the queue detail page
The queue detail page splits into Overview (the existing concurrency,
backlog, delay, and throttle charts) and a Concurrency keys tab that
holds all per-key content. The tab only appears for queues with key
activity. Inside it, the grouped per-key charts now lead (backlog by
key plus a new throughput-by-key chart that makes fair-share visible),
followed by the two whole-queue health charts retitled to say what they
aggregate (keys with queued runs count, most-starved key wait), then
the key table and drill-down.
2026-07-13 20:39:15 +01:00
Eric Allam 9233032961 feat(webapp,run-engine,clickhouse,tsql): per-concurrency-key queue metrics history
Queues that shard work with concurrencyKey get a per-key history tier.
Counter events for CK runs advance a second per-key odometer and carry
both readings on ONE stream entry (cum + ck/ckcum), so per-key
attribution adds no stream volume; the consumer expands the entry into a
base row and a per-key row. A new 10s AggregatingMergeTree tier keyed by
(queue, concurrency_key) holds per-key enqueue/started/ack deltas,
backlog/running maxes, and wait sums. Rows are activity-bound: a
(queue, key, bucket) row exists only when that key had events, so
user-controlled key cardinality cannot inflate the table (benchmarked at
~19 bytes per event with reads under 100ms in the worst shapes).

The per-queue tiers stay exact: their counter and wait aggregations now
consume only base rows, so per-key odometers never merge under one
queue_name and waits are never double counted. A per-queue key limiter
(default 10k) acts as a safety valve; on overflow the per-key row is
dropped while the base row keeps per-queue counts exact. Per-key
odometers use a short TTL, which cumulative counters make loss-free.

The queue detail page gains a top-keys-by-backlog chart, a key table
merging live state (queued, running, oldest wait) with range stats
(started, peak backlog, mean delay), and click-through per-key
drill-down charts. The new queue_metrics_by_key table is also queryable
directly; its delta columns require grouping or pinning BOTH queue and
concurrency_key, enforced by the compile-time merge guard which now
supports compound keys.
2026-07-13 20:39:15 +01:00
Eric Allam fcc27bc7d6 feat(run-engine,clickhouse,webapp): concurrency-key health metrics and live per-key breakdown
CK queues now emit two extra gauge fields from the CK-path Lua scripts:
the number of concurrency keys with queued runs (ZCARD of the ckIndex)
and the head-of-line wait of the most-starved key (now minus the oldest
ckIndex score). Both flow through the existing stream into new
max-aggregated columns on the 10s and 5m tiers, and non-CK scripts keep
the 7-field gauge shape.

The queue detail page grows a concurrency-keys section for queues with
CK activity: charts for backlogged keys and most-starved wait, plus a
live per-key table (queued, running, oldest wait) read from the ckIndex
zset, most starved first. Per-key history is intentionally not stored:
key values are user-controlled, so the per-key dimension stays in Redis
where it is bounded by the live backlog.

The queue simulator gains a tenant-hotspot scenario that stages the CK
gauge columns and a live ckIndex so the per-key table and charts render
with data.
2026-07-13 20:39:14 +01:00
Eric Allam 465b36169a fix(webapp,metrics-pipeline): review hardening for overflow counters, cluster slots, and stream caps
Counter readings for names past the per-env cardinality cap are dropped
instead of merging unrelated odometers under the overflow label (gauges
still flow). The odometer key now shares the stream's shard hash tag so
the INCR plus XADD script stays in one Cluster slot. The counter stream
cap defaults lower when the stream shares the run-queue Redis. The
per-bucket counter boundary undercount is documented on the delta
columns.
2026-07-13 20:39:14 +01:00
Eric Allam 9c3b077dba refactor(clickhouse,webapp): use the stored quantile list in all merge calls
Short parameter lists on quantilesMerge and quantilesTDigestMerge do
execute (the state layout is parameter independent, verified on both
ClickHouse versions we run), but they rely on undocumented leniency and
make the result-array indexes mean different quantiles per call site.
Every merge now uses the stored four-quantile list with indexes
re-pointed accordingly; returned values are unchanged.
2026-07-13 20:39:14 +01:00
Eric Allam 2b866b8bc7 feat(tsql,webapp): reject cross-queue merges of per-queue counter states
deltaSumTimestamp states are kept per queue, and merging them across
queues silently returns wrong totals, on the dashboard and the public
query API alike. Columns can now declare a mergeGroupKey, and the
compiler rejects queries that merge such a column without grouping by
that key or pinning it to a single value. The error names the column,
explains the failure, and includes a corrected example query.
2026-07-13 20:39:13 +01:00
Eric Allam 1ed93e7fcf feat(webapp): remove auto-balance from the allocation view
The allocation view keeps manual limit edits, the review dialog, and
bulk apply. The one-shot auto-balance button is removed (and the row
locks whose only purpose was protecting queues from it); a policy-driven
approach can replace it if rebalancing returns.
2026-07-13 20:39:13 +01:00
Eric Allam 9bd77e8160 fix(webapp): keep the search filter on the ranked queue list's tail
The tail query's exclusion list overwrote the search's name filter via
object spread, so searching while sorted by activity showed unrelated
queues past the ranked head. Combine the conditions with AND instead.
2026-07-13 20:39:13 +01:00
Eric Allam 4be8159517 feat(webapp): stage fake Redis usage from the queue metrics simulator
A --usage flag stages plausible running counts in the local run-queue
Redis for the seeded queues, so the list's Running column and the
Allocation tab's usage bars have data without the run engine. Staged
state is reconciled on every run: present with --usage, cleared without.
Local Redis hosts only.
2026-07-13 20:39:13 +01:00
Eric Allam 15e0e5f733 fix(webapp): update the queue metrics simulator for cumulative counters
Counter events now emit per queue and op odometer readings with a seeded
zero baseline, matching the production emitter, so throughput and
started counts reconstruct from simulated data instead of reading zero.
Scenario switches prune the previous scenario's queues, a --project flag
seeds each scenario into its own project for side-by-side design review,
and a new many-queues scenario covers pagination and relevance ranking
with one runaway queue, a busy head, a bursty middle, and a sparse tail.
Adds --help.
2026-07-13 20:39:13 +01:00
Eric Allam 9f603ae21c fix(webapp): include rollup tables in the queue metrics simulator reset
The simulator's --reset only cleared the raw and 10s tables, leaving
stale rows in the 5m and env rollups. It also force-merges the rollups
after seeding so current-value widgets read cleanly.
2026-07-13 20:39:13 +01:00
Eric Allam 5c7f7af964 feat(clickhouse,webapp): keep 10-second resolution on the env metrics rollup
The env rollup's win comes from dropping the queue dimension, not from
coarser buckets: row count is queue-independent (~8640/day/env), so full
10-second granularity stays cheap at any range. Env header tiles and
saturation charts now resolve short-range detail exactly like the
per-queue charts, and the current-value tiles read the latest 10-second
bucket instead of a minute-wide one.
2026-07-13 20:39:13 +01:00
Eric Allam 7c770402cc feat(webapp): serve queue metrics reads from rollups and fix env totals
The built-in queues dashboard's enqueued vs started chart merged counter
states across queues, which mixes unrelated cumulative counters and
returns wrong totals; it now merges per queue and sums outside. Env
header tiles and saturation charts read the environment rollup, so their
cost no longer scales with queue count, and coarse-bucket ranges are
served from the 5m rollup automatically. Queue list ranking runs as one
query, time bounds are aligned to the bucket grid, and repeated
auto-refresh reads share ClickHouse query-cache entries.
2026-07-13 20:39:13 +01:00
Eric Allam ea0a4b2fc6 feat(webapp): queue allocation view and relevance-ordered queue list
Adds an Allocation tab to the Queues page (behind the queue metrics UI flag): overview cards, a burst-aware capacity bar showing each queue allocation and its live usage in a distinct color, an inline-editable limits table with per-queue locks, load-weighted auto-balance, and a review dialog that bulk-applies limits as overrides through the existing concurrency system.

The queue list now defaults to Busiest ordering (with Backlog and Name options). ClickHouse ranks queues by activity over the last 15 minutes and returns just the requested page of names, so the cost per page is one small aggregate regardless of environment size; idle queues follow in name order and any failure falls back to name ordering. The classic page keeps plain name order.
2026-07-13 20:39:13 +01:00
Eric Allam 5c372a1cb4 feat(webapp): house style hero charts on the queues list
The queues list header tiles now render the same line chart, grid, and tooltip as the rest of the metrics charts instead of a row sparkline, with the headline value in the tile header. The env saturation tile draws the environment concurrency limit and burst limit as labeled reference lines. Chart tooltips keep a gap between the series label and the value, and the shared line chart gains showDots and referenceLines options.
2026-07-13 20:39:12 +01:00
Eric Allam b1dded7c2b chore(webapp): use shared primitives on the admin queue metrics page 2026-07-13 20:39:12 +01:00
Eric Allam 69600b32d3 feat(webapp,clickhouse): standard time filter for queue metrics pages
The queues list and queue detail pages now use the shared TimeFilter (any preset period or a custom date range) and everything on the page follows it: header tiles, per queue metric columns, charts, and stats. The custom period buttons, hand rolled chart cards, and duplicated metric fetch loops are replaced by the ChartCard and Chart primitives, UsageSparkline, and a shared useMetricResourceQuery hook. The ClickHouse list queries take an explicit end bound so fixed ranges query only their window.
2026-07-13 20:39:12 +01:00
Eric Allam e9259790ce fix(metrics-pipeline): widen order_key packing factor to 1e6
Packs the stream sequence with a 1e6 factor (was 1e5) so up to 1M entries per millisecond per shard fit before a seq could spill into the next millisecond's range, far above what a single Redis stream can produce. ms*1e6 stays within UInt64. Also fixes the webapp mapping test that still expected a numeric order_key after the switch to a BigInt-derived string.
2026-07-13 20:39:12 +01:00
Eric Allam 5139255a60 fix(webapp): fail open on queue metrics and honor sparkline total override
The queues list tolerates a metrics query failure by rendering without metrics and logging a warning. UsageSparkline renders its total override even when every bucket is zero. The queue detail page returns 404 and its loader skips the metrics query when the feature flag is off. The seed script validates bucket size and only writes ClickHouse against a local host.
2026-07-13 20:39:12 +01:00
Eric Allam 26e697224c chore: use import type for type-only imports 2026-07-13 20:39:12 +01:00
Eric Allam 7a26ca576e chore: apply oxfmt formatting 2026-07-13 20:39:12 +01:00
Eric Allam 2f294d3a61 feat(webapp): Queues dashboard and per-org metrics UI flag 2026-07-13 20:39:11 +01:00
Eric Allam c206e12141 feat(webapp): queue metrics ingestion, admin controls, and emission switch 2026-07-13 20:38:31 +01:00
Chris Arderne 64e5d732ad chore(webapp,core): remove the unused ResourceMonitor server logging helper (#4244)
The `ResourceMonitor` server-side logging helper is no longer used. It
periodically logged the webapp process own memory, disk, and CPU usage
behind the `RESOURCE_MONITOR_ENABLED` flag (off by default), and was
also exported from `@trigger.dev/core/v3/serverOnly` with no other
consumers.

This removes the helper, its `@trigger.dev/core` export, the webapp
wiring, and the `RESOURCE_MONITOR_ENABLED` env var. The supervisor has
its own unrelated `ResourceMonitor` class, which is left untouched.
2026-07-13 20:32:29 +01:00
Eric Allam 29598a77b8 feat(webapp): add option to disable PostgreSQL task-event writes (#4242)
## Summary

Adds `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` (default off), which
makes the task-event store skip all PostgreSQL `TaskEvent` writes. It's
for deployments that store task events in ClickHouse
(`EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2`) and no longer want the
PostgreSQL copy.

## How it works

The guard sits at the single postgres write boundary,
`TaskEventStore.create` / `createMany`, so it covers every write path
(OTLP ingestion and run-lifecycle events) with one check. Reads are
untouched (`findMany` / trace queries / streaming), so existing
PostgreSQL events remain readable.

Leave it off unless the default store is `clickhouse_v2`, otherwise task
events for any run still routed to PostgreSQL would be dropped.
2026-07-13 17:17:02 +01:00
Daniel Sutton e0b42a88d6 perf(webapp): avoid unindexed fileId scan in get-background-worker-by-version (#4245)
## What

The `GET
/api/v1/projects/:projectRef/background-workers/:envSlug/:version`
endpoint loaded each file's tasks through the nested `files.tasks`
relation. Prisma resolves that as a separate query:

```sql
SELECT id, slug, "fileId" FROM "BackgroundWorkerTask" WHERE "fileId" IN (...)
```

`BackgroundWorkerTask.fileId` is not indexed — the FK constraint exists,
but Postgres does not auto-create an index for foreign keys — so on a
large table this can only run as a sequential scan, which gets
progressively slower as the table grows and was observed taking minutes
per call in production.

The loader already loads every task for the worker via `tasks: true`,
which uses the indexed `workerId` relation, and those rows already
include `fileId`. This PR groups task slugs by `fileId` in memory from
that already-loaded data and drops the `files.tasks` include entirely.

## Behavior change (latent bug fix)

The response shape is unchanged, but there is a semantic correction for
**source files reused across worker versions** (files are de-duplicated
by `@@unique([projectId, contentHash])`, so one file row can be linked
to many workers).

- **Before:** `file.tasks` came from the `BackgroundWorkerFile.tasks`
relation, i.e. *every* `BackgroundWorkerTask` with that `fileId` —
across all workers sharing the file. So a worker's manifest could list
tasks it doesn't actually have.
- **After:** `file.tasks` is grouped from the queried worker's own
tasks, so it reflects only that worker version's tasks.

Verified on a local DB: 460 files are referenced by tasks from more than
one worker; of 6819 (worker, file) pairs, 6 differ — all one file where
the old union leaked a task slug (`cancellation-test`) into worker
versions that never had it. The new per-worker behavior is the correct
one for a worker-version manifest. (Thanks to the automated review for
flagging this.)

## Analysis

Captured the exact SQL before/after by instrumenting Prisma against real
data (a worker with 62 files):

- **Before:** 5 statements, including the `WHERE "fileId" IN (...)`
scan.
- **After:** 4 statements; the `fileId` query is gone and the other four
are identical.

EXPLAIN of the two access paths:

```
Before  WHERE "fileId" IN (...)
  Seq Scan on "BackgroundWorkerTask"
    Filter: ("fileId" = ANY (...))          -- reads the whole table, scales with table size

After   WHERE "workerId" IN (...)
  Index Scan using "BackgroundWorkerTask_workerId_slug_key"
    Index Cond: ("workerId" = ...)          -- bounded by matching rows, scale-independent
```

No new index is required: the `workerId` access path is already covered
by the existing `BackgroundWorkerTask_workerId_slug_key` unique index.

## Testing

- `pnpm run typecheck --filter webapp` passes.
- Query capture + EXPLAIN performed against a local database seeded with
real worker/file/task data.
2026-07-13 16:14:20 +01:00
Chris Arderne 703a6dcb4c chore(ci): optimise runners, distribute test shards (#4240)
- Use bigger/smaller runners as recommended by warpbuild
- Distribute test shards more evenly, move internal tests single big
shard
2026-07-13 15:58:33 +01:00
Daniel Sutton bea7e2be90 feat(webapp,run-store): route run-graph reads and writes through the run-store router (#4237)
## Summary

Run-graph data (runs, batches, waitpoints, and their related tables) can
now live in a database separate from the control plane, with every read
and write routed to the correct database by each run's residency. This
makes reading and writing run data more reliable once the two are split,
and is a no-op for single-database installs.

## Design

- Run-graph table access goes through the run-store router, which
selects the legacy or the new run-ops store per run instead of assuming
one shared client.
- The legacy run-ops client is now independently pointable, so legacy
run data can be served from its own database (and replica) rather than
the control-plane connection.
- Run-graph writes go straight to the run-graph database instead of
being forwarded through the control plane, and replication targets are
split so runs in the new database still replicate to analytics without
under-counting.
- Read-through slots refuse the control-plane client, so a missing
residency fails loudly instead of silently reading the wrong database.
- Migration `20260710120000_drop_remaining_run_graph_seam_foreign_keys`
drops the foreign keys that still crossed the run-graph / control-plane
seam, which is what lets the two live in separate databases.

The split stays off unless explicitly enabled and the two databases are
confirmed physically distinct; startup fails closed otherwise.

Verified by running the full dashboard end-to-end suite against both a
single-database configuration and a three-database configuration
(control plane, the new database, and a physically separate legacy
database), with runs on both residencies. No misrouted reads in either
configuration.
2026-07-13 13:54:54 +01:00
Eric Allam 5ba8557a51 chore(webapp,core): remove the end-of-life v3 (engine V1) execution stack (#4236)
## Summary

v3 (the engine that ran the SDK v3 era, internally
`RunEngineVersion.V1`) is end-of-life. Following the removal of the v3
execution apps
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and
the legacy dev websocket
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this
removes the remaining v3 execution stack from the server.

Clients still on v3 (an old SDK or CLI that has not upgraded) keep
getting a clear "upgrade to v4" response. Triggers, batch triggers,
reschedules, and deploys that resolve to v3 are rejected with a graceful
4xx pointing at the migration guide, never a 5xx, so a stale client
cannot affect server health. Self-hosted instances still running v3
should stay on the 4.5.x release line until they migrate.

## What is removed

- The MarQS queue and its shared/dev queue consumers.
- The v3 socket.io namespaces (coordinator, provider, shared-queue) and
the v3 run lifecycle services (attempt, checkpoint, and batch-resume).
- The graphile-worker background job system; all live jobs already run
on `@trigger.dev/redis-worker`.
- The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally,
so the flag is gone.
- Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace`
subpath and the legacy socket message catalogs) and the now-dead MarQS
environment variables.

## What stays

The v4 engine is untouched. The graceful v3 rejection boundary stays,
`determineEngineVersion` still detects a v3 project so it can reject it,
and the batch service plus batch-completion worker stay for current
clients. Live queue concurrency limits and metrics now read from the v4
run engine instead of MarQS, and a brand-new dev environment now
defaults to v4.



## Dependency cleanup

Removes webapp dependencies left unused by this change: `seedrandom` and
`semver` (only the removed v3 code used them) plus a set that was
already dead, their orphaned `@types` packages, and two dead files. Adds
a `knip:deps` script and a `knip.json` config so unused dependencies can
be found the same way going forward.
2026-07-13 11:32:06 +01:00
Chris Arderne c0f7c803b1 fix(webapp): return 415 for invalid SSO form content types (#4238)
## Summary

SSO form submissions with an unsupported content type now receive a 415
response instead of failing while parsing the request body.
2026-07-13 11:01:08 +01:00
Daniel Sutton c601739d35 perf(webapp,run-store): grouped run-ops reads + mint-kind flip grace (#4227)
## Summary

Two threads on the run-ops split path.

Read path: per-item run reads are batched into grouped queries, a
waitpoint's connected-run reads are bounded, and the dedicated-schema
relation hydrators fetch only the requested columns instead of whole
rows. Retrieve also falls back to the other database when a routed read
misses, so a run whose physical residency diverges from its id shape is
still found rather than returning a spurious not-found. Fewer and
lighter queries on the run read path, with no change to results.

Mint-kind flip safety: flipping which database new runs mint to is now a
deterministic wall-clock cutover, for both per-org and global flips. For
a grace window every process resolves the same database, so a flip
cannot route two concurrent triggers that share an idempotency key to
different databases (which would bypass the per-database unique
constraint and create a duplicate run).

Supersedes the earlier #4205 and #4208.

Draft: validation in progress.
2026-07-13 10:17:12 +01:00
Saadi Myftija 5f2541d94f feat(webapp): make native build server the default in build settings (#3980)
Switches the native build server from opt-in to opt-out in project build
settings.

- It's now enabled by default, stored as a new
\`disableNativeBuildServer\` opt-out key so previously-saved
\`useNativeBuildServer: false\` values aren't treated as deliberate
opt-outs.
- The "Use native build server" checkbox is checked by default;
unchecking it persists the opt-out.
- Brief wording: clarifies build settings apply to GitHub-triggered and
native build server deployments, and the native build server hint no
longer says "in the future".
2026-07-13 11:05:46 +02:00
Eric Allam 45527e317a feat(webapp): opt-in worker pool for OTLP ingest transform (#4232)
## Summary

Under high OTLP ingest volume, the whole decode, transform, and enrich
pipeline runs on the request event loop, so a single CPU core becomes
the ceiling while the rest sit idle. This adds an opt-in worker pool
that moves decode, transform, and LLM-cost enrichment onto worker
threads, keeping the main thread free for I/O. It is off by default
(`OTEL_TRANSFORM_WORKER_POOL_ENABLED`), so behavior is unchanged unless
enabled.

## Design

Workers do decode, filter, convert, and enrich (including LLM pricing
match). The main thread stays the single database reader: it loads the
pricing registry and broadcasts the compiled model rows to the workers
(re-broadcasting on every reload), so workers never touch the database.
The pure transform is extracted into a dependency-light module (no
Prisma/Redis/ClickHouse imports) so it can run inside a worker.

Importantly, the main thread keeps the existing single consolidated
insert path, so ClickHouse insert batching and part count are unchanged.
The parallelism buys CPU headroom, not more insert streams (which would
add merge pressure).

The worker is bundled as a standalone file at build time and ships in
the existing image with no Dockerfile change. In local load testing the
pool sustained roughly 2.6x the throughput of the single-thread path and
kept the main thread responsive under load.
2026-07-11 13:38:12 +01:00
Eric Allam 9b3a7bd7b2 fix(sdk,webapp): stop chat losing a message sent right after an action (#4234)
## Summary

Sending a chat message immediately after an action (for example an undo)
could make the message's response vanish from the UI. The transport
opened a response stream that closed on the *earlier* turn's completion
instead of waiting for the send's own turn. The agent still produced and
persisted the answer, so it reappeared on refresh. Same "disappearing
message" class as
[#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176),
different cause.

## Fix

A send's response stream had no way to tell whether a `turn-complete`
belonged to its turn. `POST /realtime/v1/sessions/:id/in/append` now
returns the appended record's sequence number, and the transport skips
any turn-complete whose `session-in-event-id` (the agent's committed
`.in` cursor) is below that seq, closing only on its own turn. Older
webapps omit the seq, in which case the transport falls back to the
previous behavior, so the SDK and server can ship independently.

Because the fix spans the SDK and the server, both a webapp deploy and
an SDK release are needed for the full effect.

Verified end to end with the ai-chat reference app:
undo-then-immediate-send loses the follow-up's answer before the fix and
streams it inline after, with a revert-the-guard run reproducing the
loss on the same script. Unit tests cover the skip and the no-seq
fallback.
2026-07-11 12:46:24 +01:00
Eric Allam 5d0e9d9dc5 feat(webapp): make the default realtime backend configurable (#4231)
## Summary

The default realtime backend was hardcoded to Electric. This adds a
`REALTIME_BACKEND_DEFAULT` env var (`electric` | `native` | `shadow`,
default `electric`) that chooses the backend for any environment whose
org has no `realtimeBackend` override. Behavior is unchanged unless you
set it; per-org overrides still win.

The default is applied at every point where the per-org flag falls
through: the initial value, the flag lookup default, and the error
fallback.
2026-07-11 09:36:09 +01:00
Matt Aitken 2cac63f13a fix: improve error labelling, grouping, and stack traces in the Errors feature (#4225)
## Problem

Several display/grouping issues in the **Errors** feature, all rooted in
how the ClickHouse error materialized views (`errors_mv_v1`,
`error_occurrences_mv_v1`) read the stored error JSON produced by
`parseError`:

1. **Messageless errors show "Unknown error".** An empty message falls
straight through `coalesce(nullIf(message,''), 'Unknown error')` to the
literal, even though the error's class `name` is available (e.g. an
Effect tagged error `ListMessagesError` with no message).
2. **Unrelated errors collapse into one group.**
`calculateErrorFingerprint` keys on `type : message : stack`, where
`type` is always the union tag (`BUILT_IN_ERROR`, …), `message` is
empty, and the stack isn't read — so every messageless built-in error
(and every string/custom error) hashes to the same constant input → one
fingerprint.
3. **error_type shows the internal tag.** `coalesce(type, name, …)`
always resolves to `type` (always present), so the column shows
`BUILT_IN_ERROR` instead of the real class name.
4. **Stack traces never populate.** The MVs read `error.data.stack`, but
the serializer stores the trace under `stackTrace` — so the column is
always empty.

## Fix

All display changes are `ALTER TABLE … MODIFY QUERY` on the two views
(migration `035`); the fingerprint change is in the webapp.

- **Fingerprint** (`errorFingerprinting.ts`): fall back **message → name
→ raw**. Messageless errors now group by class name (or raw value for
non-Error throws); message-bearing errors are **unchanged**
(short-circuits at `message`), so existing groups don't split — only
currently-messageless errors get their own group going forward.
- **error_message**: same `message → name → raw` fallback before
`'Unknown error'`.
- **error_type**: coalesce `name → code → 'Error'` (drops the reliance
on the union tag). Built-in → class name, internal → `code`,
string/custom → `Error`.
- **stack trace**: read `error.data.stackTrace`. Bounded as before
(serializer caps 50 frames / 1024 chars per line; MV clips to 2000
chars).

## Migration notes

- `MODIFY QUERY` swaps the view query in place (no drop/recreate gap);
Down restores the previous query.
- **Existing rows are left unchanged** — changes apply only to rows
inserted after the migration. No backfill.

## Tests

`errorFingerprinting.test.ts` — 57 pass, incl. new cases for messageless
class names, string/custom raw values, and stability of message-bearing
fingerprints.

Fixes the display-derivation half of TRI-11938 (error_type + stack
trace); relates to TRI-9254 and TRI-9250.
2026-07-10 18:30:02 +01:00
Oskar Otwinowski 4be32d411c fix(webapp): keep the last Owner on directory-sync role changes (#4230)
Applying a directory-sync effect that would demote the org's last Owner
(a
group remap, or a provision) previously threw and 500'd the settings
save. Now
rbac.setUserRole reports code:"last_owner" and applyEffect skips just
that
member (they keep Owner) while the rest of the batch applies.

Adds the machine-readable RoleAssignmentResult.code to the plugin
contract so
callers can tell the last-owner guard apart from a real failure.
2026-07-10 19:28:16 +02:00
Matt Aitken b64b54c74e feat(webapp): pass database writer and reader config to auth plugins (#4229)
## Summary

The RBAC and SSO auth plugins can own their own database client, but
they could only read `DATABASE_URL`, so every connection they opened
landed on the primary. The host webapp now resolves writer and
read-replica URLs from its env (the same fallback chain its own Prisma
clients use: control-plane URL first, then the default) and passes them
to the plugins at create time via a shared `PluginDatabaseConfig`, along
with separate connection limits for writes (default 2) and reads
(default 5, tunable via `RBAC_DATABASE_*_CONNECTION_LIMIT` and
`SSO_DATABASE_*_CONNECTION_LIMIT`).

A plugin can then route hot-path reads (per-request auth checks, login
routing) to the read replica and keep only rare mutations on the
primary. With no replica configured, or no plugin installed, nothing
changes: the OSS fallback ignores the new option and keeps reading
through the Prisma clients it is already given.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-07-10 17:25:42 +01:00
DKP 25eb0c71a0 fix(webapp): clarify that region only affects where runs execute (#4226)
## Summary

This adds an always-visible info tooltip on the Location column and a
note in the "set default region" confirmation dialog making it explicit.
It also removes the obsolete "V4" badge from the Regions page title.
2026-07-10 17:02:36 +01:00
Matt Aitken 48a0b83ec6 feat(webapp): promo credits — /promo signup landing, redeem at plan selection, usage display (#4138)
## What & why

Signup promo credits. A new logged-out `/promo?code=<code>` landing page
validates the code and carries it through signup via a cookie. When the
new organization is activated by selecting a plan, the code is redeemed
and its credits are applied; the usage page then shows the remaining
promo credits and their expiry.

## Notes

- The code is redeemed at **plan selection**, not org creation: the
credit grant targets the org's usage allowance, which only exists once a
plan is selected — applying at creation would have nothing to grant
onto. Redemption is best-effort and never blocks plan selection.
- Pairs with the corresponding billing-service change (promo code
validate/apply/credits + grant issuance); the two are released together.

## Testing

Verified locally end to end: `/promo` shows the offer, a new account
carries the code through signup, selecting the Free plan redeems it, and
the usage page shows the remaining credits.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-07-10 17:48:39 +02:00
claude[bot] de536622c8 Add oxlint rule to catch thrown un-awaited redirect helpers (#4222)
##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

- Verified the rule emits exactly five errors for un-awaited throws of
the known
async redirect helpers while ignoring awaited throws, returned promises,
and
  synchronous `redirect(...)`.
- Verified `--fix` inserts `await` in async functions and produces a
clean
  second lint run.
- Verified synchronous functions remain diagnostic-only so autofix
cannot
  introduce invalid syntax.
- Ran `pnpm run format`, `pnpm run lint`,
  `pnpm run typecheck --filter webapp`, and `git diff --check`.

---

## Changelog

Adds an Oxlint rule that prevents async redirect helpers from being
thrown
without awaiting their `Response`. Existing violations are fixed, the
autofix
is limited to async functions, and the plugin uses an explicit ESM
extension.

---

## Screenshots

See the test-results comment for CLI evidence.

💯


Link to Devin session:
https://app.devin.ai/sessions/e60ad7610773401da3d3040cf1252337
Requested by: @ericallam

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Eric Allam <eric@trigger.dev>
2026-07-10 13:19:13 +02:00
James Ritchie 32e5edbd04 fix(webapp): restore magic link login on the login page (#4220)
## Summary

Magic link login could appear completely broken: submitting your email
on the login page showed a stale "This email is unauthorized" error
instead of the "we've sent you a magic link" confirmation, even when the
address was fine.

This PR reverts
[#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) (whose
diagnosis and fix turned out to be wrong) and fixes the actual bug,
which was in how login errors are stored and consumed.

## Root cause

Two session bugs compounded on the login page:

- The `/login` loader read the flashed `auth:error` without committing
the session. A Remix flash is only consumed when the session is
committed after the read, so once any attempt flashed an error (for
example an address rejected on an instance with `WHITELISTED_EMAILS`
set), it stayed in the session cookie and reappeared on every later
`/login` visit, making successful attempts look like failures.
- The `/login/magic` action stored its validation and rate limit errors
with `session.set`, which survives every later read and commit, so those
errors stuck permanently.

[#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) had
instead diagnosed a server-only module leaking into the client bundle
and crashing navigation. Checking the shipped images' client bundles via
their sourcemaps shows `.server` modules were always stubbed out, so
that change fixed nothing and is reverted here.

## Fix

- `/login` reads the flashed error and commits the session when one was
present, so an error renders once and clears. The `redirectTo` branch
now surfaces the error too instead of leaving it in the cookie.
- `/login/magic` flashes its errors instead of `set`ting them.

Verified end-to-end on a live preview environment: a rejected address
shows the error once and a reload clears it; a valid address lands on
the confirmation screen with the address named; GitHub, Google, and SSO
login paths are untouched by this diff.
2026-07-10 10:59:10 +01:00
James Ritchie afc8f9e210 fix(webapp): show magic link confirmation instead of reloading login (#4215)
## Summary

Submitting your email on the login page could reload back to an empty
login form instead of showing the "we've sent you a magic link"
confirmation. The magic link email was still sent, so it looked like
nothing happened.

## Root cause

The `/login/magic` route imported a server-only cookie module
(`magicLinkEmailCookie.server.ts`) whose top-level `env.NODE_ENV` read
got bundled into the route's client JS. On the client `env` is
undefined, so the module threw a `TypeError` at module eval, which
aborted Remix's client-side navigation to the confirmation and
hard-reloaded back to `/login`. It only surfaced in production builds
(local dev auto-logs-in, and local prod builds happen to tree-shake the
module out), which is why it slipped through.

## Fix

The email-link strategy already stores the submitted address in the
session (`auth:email`), so the separate cookie was redundant. Deleted
the cookie module and read the address from the session in the loader.
With the module gone, nothing server-only can leak into the client
bundle regardless of tree-shaking.

Verified the confirmation renders with the email address, the SSO
domain-policy redirect (with the email prefilled) still works, and a
production build no longer bundles the module.
2026-07-09 20:20:06 +01:00
Oskar Otwinowski dc6c98af5e chore(webapp): trim comments in directorySyncEffects (#4207)
Condense the kept rationale comments (logLevel/warn, last-Owner dedup,
role
overwrite) and drop the obvious function-header comments that just
restated
the code. No behavior change.
2026-07-09 20:13:21 +01:00
Oskar Otwinowski e57fd9ce90 fix(webapp): downgrade retryable directory-sync effect failures to warn (#4200)
Directory-sync effects are idempotent and the accounts-webhook worker
retries
the whole event, so a single failed attempt (typically a role assignment
losing a serializable race during a backfill burst) is self-healing
rather
than alert-worthy. Tag those thrown errors with logLevel "warn" so the
worker
logs at warn instead of error, keeping them visible for triage without
paging.
2026-07-09 17:57:40 +02:00
Eric Allam 1a0198cc5e perf(webapp,clickhouse): move runs empty-state check to ClickHouse (#4202)
## Summary

The runs page's empty-state check (whether an environment has ever had a
run, which decides between the "getting started" and "no runs match your
filters" states) ran a `findFirst` against the Postgres `TaskRun` table.
This moves it to ClickHouse, the same store the runs list itself reads
from, so the check no longer queries `TaskRun`.

## Design

Only the runs list triggers the check now (via an `includeHasAnyRuns`
flag); the other presenters that reuse `NextRunListPresenter` (API,
schedule detail, waitpoint detail, error group) no longer issue it. When
the list is empty it runs `SELECT 1 FROM task_runs_v2 ... LIMIT 1`
filtered on the full `(organization_id, project_id, environment_id)`
sort-key prefix with a configurable `created_at` lower bound
(`RUN_LIST_HAS_RUNS_LOOKBACK_DAYS`, default 30), so it hits the primary
index and reads minimal granules.

Results are cached in a tiered memory + Redis SWR cache. Only positive
("has runs") results are cached, so an environment with no runs is
always re-checked and its first run shows up immediately.
2026-07-09 15:24:31 +01:00
nicktrn 0631c8373c chore: retire legacy v3 dev websocket + delete legacy self-hosting docs (#4198)
Follow-up to #4194 (v3 execution app + core-helper removal). The v3
(engine V1) is end-of-lifed and enforced off in prod, so this removes a
self-contained slice of the remaining dead v3 code while **keeping every
user-facing deprecation message** - a user still on v3 must still be
told to upgrade.

## Legacy dev websocket

`app/v3/handleWebsockets.server.ts` backs the `/ws` transport used
**only** by the legacy v3 `trigger dev` CLI (v4 dev uses a different
transport). It's now authenticate-then-close with
`V3_DEV_DEPRECATION_MESSAGE`, so an old CLI is still told what to do -
only the legacy `AuthenticatedSocketConnection` / `DevQueueConsumer`
execution behind it (which can no longer run) is removed.

- Deleted `app/v3/authenticatedSocketConnection.server.ts` (its only
consumer).
- `engineDeprecation.server.ts` and the deprecation message constants
are untouched.

## Docs

Deleted the intentionally-legacy "Docker (legacy)" self-hosting page
(`open-source-self-hosting.mdx`) and redirected
`/open-source-self-hosting` (+ the existing
`/v3/open-source-self-hosting` alias) to `/self-hosting/overview`;
repointed the two inbound links. The current `self-hosting/*` docs
already describe the v4 (single supervisor) setup.

## Deliberately out of scope

Despite the branch name, this PR does **not** touch MarQS or the
socket.io coordinator/provider namespaces. Investigation found MarQS is
entangled with **live v2** queue/metrics/concurrency/project-cleanup
code (`runQueue`, `queueSizeLimits`, `taskRunConcurrencyTracker`,
`EnvironmentQueuePresenter`, `registerProjectMetrics`, `deleteProject`),
so it needs a per-file reviewed pass, not a bulk delete. That remainder
stays on TRI-11883.

refs TRI-11883
2026-07-09 14:40:44 +01:00
Chris Arderne 34b1a181c2 fix: security release 2026-07-06 (#4199) 2026-07-09 11:58:33 +00:00
James Ritchie bb450e608d feat(webapp): SSO & Directory Sync settings UI improvements (#4196)
📚 Publish docs / publish (push) Has been cancelled
## Summary

UI/layout/copy pass over the org **SSO & Directory Sync** settings page
(formerly "Identity & Access"). No logic, gates, flags, or data flow
changed — server-side auth (`manage:sso`), Enterprise entitlement,
action validation, and data loading are all untouched.

- Renamed the nav item, page title, and meta from "Identity & Access" to
"SSO & Directory Sync".
- Added a reusable `SettingsLayout` component system (container,
section, header, row, block, actions) modeled on `/account/security`,
and refactored the SSO page onto it (section titles, dividers, left
title/subtitle + right action rows).
- Tightened all UI copy: concise, active voice, consistent labels, no
em-dashes.
- `Select` primitive: additive `wrap`, `popoverClassName`, and
`placement` props (all default to prior behavior) so role options show a
bright title with a wrapping description, right-aligned popover, and no
horizontal overflow.
- Removed the external-link arrow icon from buttons that open a modal;
kept it only on genuinely external actions (Contact us, Open in new
tab).
- Polished the admin portal link dialog: smaller description, tighter
spacing, `ClipboardField` with a permanent copy button, removed the
redundant Copy link button, and a provider-aware Open label (e.g. "Open
in WorkOS") derived from the link host with a safe fallback.

### SSO page UI
<img width="3568" height="2550" alt="CleanShot 2026-07-08 at 18 52
11@2x"
src="https://github.com/user-attachments/assets/009d2437-7552-4ff0-a457-64744a9fcd88"
/>

### Login with SSO and normal email test (local)


https://github.com/user-attachments/assets/b33a4ce9-c1fa-45c9-bd3c-077cb6fc9473



## Test plan

- [ ] Non-Enterprise org: SSO page shows the upsell state
- [ ] Enterprise org, non-Owner without `manage:sso`: 403
- [ ] Enterprise Owner: verify domains, configure SSO, connect
directory, JIT/default/group role selects, and enforcement toggle all
work
- [ ] Role select popovers: bright title + wrapping description,
right-aligned, no horizontal scroll
- [ ] Admin portal dialog: copy button works, "Open in WorkOS" opens the
portal in a new tab

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-09 12:09:45 +01:00