The account page and useSystemThemeSync call normalizeThemePreference in
client code; importing it from dashboardPreferences.server left an empty
module in the browser bundle and crashed /account.
Fixes TRI-12078
## Summary
Prevents concurrent environment setup requests from creating duplicate
Staging and Preview environments.
## Fix
Adds database-enforced uniqueness for root Staging and Preview
environments.
If two requests race, the losing request loads the environment created
by the winner and continues successfully instead of creating a duplicate
or returning an error.
## Summary
Upgrades the workspace to TypeScript 6.0.3 and applies the compiler,
type, and build configuration changes required to preserve package
layouts and existing runtime behavior, apart from correcting the HTTP
status field used for deployment connection errors.
## Compatibility
- Centralizes TypeScript 6.0.3 through the pnpm workspace catalog.
- Replaces compiler options and module resolution modes that TypeScript
6 no longer accepts.
- Restores explicit Node types where TypeScript 6 no longer includes
them transitively.
- Adds explicit declaration build roots that preserve each package's
existing output layout.
- Patches tsup to stop injecting the removed `baseUrl` option during
declaration builds.
- Uses type-only assertions for stricter typed-array and stream
definitions without changing runtime behavior.
- Reads the EventSource v3 HTTP status from `code`, so deployment
connection errors include it correctly.
- Keeps standalone CLI compatibility fixtures pinned to their existing
TypeScript version and lockfiles.
`turbo run typecheck` and the complete PR test suite are green.
## Stacked on #4284 — tests only
This PR contains **only the tests** that guard the production fixes in
#4284 (its base). Review #4284 first; this branch adds no production
code.
## What
Caller-driven replica-lag and idempotency guards for every fixed site:
- Each guard **drives the real exported caller** (route loader/action,
presenter `.call()`, service, or engine method) against a **real
Postgres** with the owning replica frozen via the shared
`laggingReplica` testcontainer primitive — never a store-seam
reimplementation.
- For a **fixed** site the guard goes **RED when the production change
is reverted**; for a **tolerated read-view** site it's a caller-driven
**GREEN** proof the miss self-heals (returns null/empty, no mutation,
row live on primary).
- The **global-scope idempotency** guard drives the real dedup + claim
path through a **real `MollifierBuffer` over a Redis testcontainer**
(real SETNX/poll/publish), and covers the cross-DB **andWait** waitpoint
wiring and the **expired/failed clear-and-recreate** reacquire cases.
Run with `vitest --no-file-parallelism` (testcontainers). Verified
GREEN, and revert→RED verified per fixed site.
## What & why
Two related correctness fixes for the run-ops DB split. Under the split,
run-store reads can route to a **lagging read replica**; a just-written
run/waitpoint/batch can then be missed, causing a wrong decision.
**1. Read-your-writes → owning primary.** Surfaced first as an
intermittent `wait.until({ idempotencyKey })` re-wait on retry. Auditing
the run-store read surface found the same class at sibling sites (some
gating mutations or returning spurious 404s, others
tolerable/self-healing). Reads that must observe their own writes now
route to the owning **primary**
(`findRun`/`findWaitpoint`/`findBatchTaskRunByFriendlyId` →
`*OnPrimary`, a primary re-read on a miss, or a retryable 404 where the
SDK polls). Read-view reads stay on the replica. All additive — the
happy path is unchanged.
**2. Global-scope idempotency across the split.** A `global`-scope key
carries no per-run salt, so the same `(env, task, key)` triggered
concurrently from parents resident on **different** run-ops DBs could
dedup-miss on each DB and create a duplicate (the per-DB unique index
can't enforce cross-DB uniqueness). Such triggers (global scope, or
scope-absent, while split is active) are serialized through the existing
Redis idempotency claim, the loser resolves the winner by id across both
DBs, and the claim is reacquired on the expired/failed
clear-and-recreate path. `run`/`attempt` scope embed the run id and
never contend.
## Stacked for review
This is the **base** of a 2-PR stack, split so review is easier:
- **This PR** — production code only (34 files).
- **Stacked tests PR →
https://github.com/triggerdotdev/trigger.dev/pull/4285** — the
caller-driven guards (55 test files) on top of this branch.
## Validation
Local run-ops split, **both 2-DB and 3-DB**, fresh boot on this branch:
SDK canary 64/71 (only the known concurrency/input-streams/s3 failures),
quarantine sweep **0 unexpected** (340 pass / 16 known / 4 local) in
each topology, dashboard e2e 0 failed. No product regressions.
## Summary
Correctness and performance fixes for deployments that split run data
across more than one database. Single-database / self-hosted deployments
are unaffected (they collapse to a single read/write path).
- **Batches list (dashboard):** for some organizations the Batches list
could hide older batches or show them out of order. It now orders and
paginates by creation time (with the id as a stable tiebreak), so every
batch appears exactly once, newest first. The pagination cursor format
changes; older in-flight cursors simply restart from the first page.
- **Reads:** waitpoint and snapshot lookups that are keyed by a single
run now read only the database that holds that run instead of querying
both, removing redundant queries on hot paths (unblock, snapshot reads).
- **Writes:** environment-scoped writes with no owning run (standalone
wait tokens, waitpoint tags, idempotency-key resets) now land in the
same database as that environment's runs, rather than defaulting to the
other one. An idempotency-key reset also falls back to the other
database when it matches nothing, so a reset still clears the key
wherever the run actually lives.
## Notes
Verified end-to-end against multi-database setups: run-keyed reads and
env-scoped writes land on the correct database with no cross-database
writes, and the batches list surfaces every batch in creation order. New
tests cover the batches ordering/reachability and the write-residency
routing.
## Summary
Adds experimental Node.js 24 and 26 task runtimes through the
`experimental-node-24` and `experimental-node-26` config values.
Existing runtime defaults and the `node`, `node-22`, and `bun` behavior
remain unchanged. The unprefixed `node-24` and `node-26` config values
remain unavailable until the runtimes are ready for general use.
## Design
Experimental config values normalize to canonical runtime identifiers
before build manifests are created, keeping deployment metadata and
execution behavior consistent. Kubernetes task pods also use the
runtime-default seccomp profile so modern Node.js versions fall back
from io_uring to checkpoint-compatible system calls.
## Summary
`batchTrigger` requests that set a per-item `idempotencyKey` failed with
a 500 when the run-store is split across databases: the per-item
idempotency lookup errored before any run was created. Batches without
per-item keys, single `trigger` idempotency, and batch-level
(`idempotency-key` header) idempotency were unaffected.
## Root cause
`findRunsByIdempotencyKeys` built its `UNION ALL` of per-key
point-lookups with `@trigger.dev/database`'s `Prisma.sql` /
`Prisma.join`, then executed it on whichever store client it was handed.
On the dedicated run-ops store that client is a *separate* generated
Prisma client, and a `Sql` object from a different generated client is
not recognized: the bare `$queryRaw(Prisma.join(...))` form dropped the
query text entirely (`Argument \`query\` is missing`). The
tagged-template form is no better here: joining nested `Prisma.sql`
fragments across the two clients mis-numbers the bound parameters
(`syntax error at or near "$1"`).
## Fix
Build the lookup as a plain parameterized string and run it via
`$queryRawUnsafe` with positional placeholders and bound values, so it
no longer depends on which generated client executes it. The query text
contains only static SQL and integer placeholders; every value
(`runtimeEnvironmentId`, `taskIdentifier`, each key) is bound, so it is
not a raw-interpolation site. Same per-key point-lookup shape as before,
no change on the single-client path.
Verified end-to-end against a bundled build with the run-store split
enabled: before the fix, `batchTrigger` with a per-item key 500s; after,
it returns the runs and dedups correctly across fresh, repeat, and mixed
batches.
## Summary
Updates the internal development, CI, and runtime-image Node version to
24.18.0. SDK compatibility coverage continues to include Node 20, 22,
24, and 26.
The Node type definitions and the package-manager lockfiles now resolve
against Node 24 types.
## Summary
Batch triggers that use per-item idempotency keys could take seconds
instead of milliseconds when the target task had a large run history.
This keeps the idempotency lookup fast regardless of how many runs a
task has accumulated.
## Root cause
The batch path checks which items already have runs by looking up their
idempotency keys with a single `WHERE runtimeEnvironmentId = ? AND
taskIdentifier = ? AND idempotencyKey IN (...)` query. On a very large
`TaskRun` table Postgres underestimates the row count of a specific
`(environment, task)` pair, so once the `IN` list grows past a handful
of keys it stops doing per-key index probes and instead scans every run
for that `(environment, task)` and filters the keys in memory. The cost
is then flat and large regardless of how many keys are being checked,
and a routine `ANALYZE` does not correct the estimate at that table
size.
## Fix
Look each idempotency key up on its own, batched into a `UNION ALL` of
point lookups (chunked, run with bounded concurrency). Each branch is an
equality on all three columns of the unique index, so the planner can
only do a per-key index probe and can never fall back to the range scan.
Same results, same columns, confined to the batch trigger path.
Main added 035_fix_error_display_derivation.sql, which collides with the
queue-metrics migration that also landed as 035. goose versions by numeric
prefix, so two 035 files break the migrator. Renumber the queue-metrics
migration to 036 (it sorts after the error-display fix on a fresh database).
Behind the per-org queue metrics UI flag: the task detail page links its
queue to the queue detail view and shows live queued/running counts, delay
p95, peak backlog, and a queue backlog chart; the run inspector links the
queue and concurrency key, and queued, delayed, and pending-version runs
get a "Waiting in queue" block with an at-limit callout and per-key counts.
Adds optional concurrency-key params to the run engine queue reads.
The wait metric measured dequeue time minus the original trigger time
even when a run re-entered the queue after a waitpoint, checkpoint, or
pending version, so the whole wait or checkpoint duration showed up as
scheduling delay. Re-enqueues now anchor to the re-enqueue time while
first enqueues keep the trigger or delay anchor; queue ordering is
unchanged, so re-enqueued runs keep their original position. Nacked
runs never left the queue stint and keep the original anchor.
Also replaces the :ck: suffix regexes on user-controlled queue names
with indexOf slicing (identical semantics) to remove a polynomial
regex flagged by code scanning.
TableSchema gains a hidden flag: hidden tables stay fully queryable
through the engine (the concurrency-keys tab keeps working, tenancy and
the merge guard still apply) but are excluded from the query editor
autocomplete, the Query page schema and examples panels, the AI query
generator context, and the schema API. queue_metrics_by_key is hidden
until the per-key surface is settled.
Queues that shard work with concurrencyKey get a per-key history tier.
Counter events for CK runs advance a second per-key odometer and carry
both readings on ONE stream entry (cum + ck/ckcum), so per-key
attribution adds no stream volume; the consumer expands the entry into a
base row and a per-key row. A new 10s AggregatingMergeTree tier keyed by
(queue, concurrency_key) holds per-key enqueue/started/ack deltas,
backlog/running maxes, and wait sums. Rows are activity-bound: a
(queue, key, bucket) row exists only when that key had events, so
user-controlled key cardinality cannot inflate the table (benchmarked at
~19 bytes per event with reads under 100ms in the worst shapes).
The per-queue tiers stay exact: their counter and wait aggregations now
consume only base rows, so per-key odometers never merge under one
queue_name and waits are never double counted. A per-queue key limiter
(default 10k) acts as a safety valve; on overflow the per-key row is
dropped while the base row keeps per-queue counts exact. Per-key
odometers use a short TTL, which cumulative counters make loss-free.
The queue detail page gains a top-keys-by-backlog chart, a key table
merging live state (queued, running, oldest wait) with range stats
(started, peak backlog, mean delay), and click-through per-key
drill-down charts. The new queue_metrics_by_key table is also queryable
directly; its delta columns require grouping or pinning BOTH queue and
concurrency_key, enforced by the compile-time merge guard which now
supports compound keys.
CK queues now emit two extra gauge fields from the CK-path Lua scripts:
the number of concurrency keys with queued runs (ZCARD of the ckIndex)
and the head-of-line wait of the most-starved key (now minus the oldest
ckIndex score). Both flow through the existing stream into new
max-aggregated columns on the 10s and 5m tiers, and non-CK scripts keep
the 7-field gauge shape.
The queue detail page grows a concurrency-keys section for queues with
CK activity: charts for backlogged keys and most-starved wait, plus a
live per-key table (queued, running, oldest wait) read from the ckIndex
zset, most starved first. Per-key history is intentionally not stored:
key values are user-controlled, so the per-key dimension stays in Redis
where it is bounded by the live backlog.
The queue simulator gains a tenant-hotspot scenario that stages the CK
gauge columns and a live ckIndex so the per-key table and charts render
with data.
Counter readings for names past the per-env cardinality cap are dropped
instead of merging unrelated odometers under the overflow label (gauges
still flow). The odometer key now shares the stream's shard hash tag so
the INCR plus XADD script stays in one Cluster slot. The counter stream
cap defaults lower when the stream shares the run-queue Redis. The
per-bucket counter boundary undercount is documented on the delta
columns.
The consumer retries a failed batch with the same insert deduplication
token, and deduplicate_blocks_in_dependent_materialized_views re-runs
the materialized views on a source-deduplicated insert, relying on each
target table's dedup window to drop the duplicate. Only the raw table
had one, so retries appended extra copies into every aggregate tier,
silently inflating sums and quantiles. All three targets now set
non_replicated_deduplication_window, with a regression test inserting
the same batch three times.
Short parameter lists on quantilesMerge and quantilesTDigestMerge do
execute (the state layout is parameter independent, verified on both
ClickHouse versions we run), but they rely on undocumented leniency and
make the result-array indexes mean different quantiles per call site.
Every merge now uses the stored four-quantile list with indexes
re-pointed accordingly; returned values are unchanged.
deltaSumTimestamp states are kept per queue, and merging them across
queues silently returns wrong totals, on the dashboard and the public
query API alike. Columns can now declare a mergeGroupKey, and the
compiler rejects queries that merge such a column without grouping by
that key or pinning it to a single value. The error names the column,
explains the failure, and includes a corrected example query.
Without a readiness guard, every fire-and-forget emit during a metrics
Redis outage queued a command in ioredis's in-memory offline queue until
rejection. Metrics are loss-tolerant by design, so drop instead;
waitUntilReady() lets embedders await the initial connect.
The env rollup's win comes from dropping the queue dimension, not from
coarser buckets: row count is queue-independent (~8640/day/env), so full
10-second granularity stays cheap at any range. Env header tiles and
saturation charts now resolve short-range detail exactly like the
per-queue charts, and the current-value tiles read the latest 10-second
bucket instead of a minute-wide one.
Adds two rollups fed from the raw landing table: a per-queue 5-minute
tier and an environment-level 1-minute tier (gauges plus TDigest wait
quantiles). Ranking now reads the 5m tier and returns the page and the
ranked total in one windowed query instead of two scans.
The 5m materialized view reads raw rather than cascading off the 10s
table: deltaSumTimestamp states hold a single first/last segment, so
merging states in an MV's hash-ordered GROUP BY double-counts bridging
spans. For the same reason the env tier carries no counter columns, and
env-wide counter totals must group by queue before summing.
The fallback WHERE injection only targeted the top-level SELECT, so a
query shaped as an outer aggregation over a FROM subquery failed to
compile: the time column only exists inside the subquery. Descend into
the subquery so the fallback lands next to the table reference.
Queries using deltaSumTimestampMerge failed with an unknown function error, which broke the queue detail stats and the started counts on the built in Queues dashboard.
The queues list and queue detail pages now use the shared TimeFilter (any preset period or a custom date range) and everything on the page follows it: header tiles, per queue metric columns, charts, and stats. The custom period buttons, hand rolled chart cards, and duplicated metric fetch loops are replaced by the ChartCard and Chart primitives, UsageSparkline, and a shared useMetricResourceQuery hook. The ClickHouse list queries take an explicit end bound so fixed ranges query only their window.
Packs the stream sequence with a 1e6 factor (was 1e5) so up to 1M entries per millisecond per shard fit before a seq could spill into the next millisecond's range, far above what a single Redis stream can produce. ms*1e6 stays within UInt64. Also fixes the webapp mapping test that still expected a numeric order_key after the switch to a BigInt-derived string.
A bucket-led ORDER BY DESC combined with fillGaps emitted an ascending WITH FILL (positive step, ascending bounds), which produces invalid or empty fills. Skip the gap-fill rewrite for descending orders and let the plain descending query stand. Adds a DESC fillGaps test.
The wait-time quantile materialized view now excludes wait_ms = 0 rows so it matches the count aggregation. order_key accepts a string or a number. Migration comments no longer contain semicolons that split the migration into invalid statements.
entryOrderKey returns a string built with BigInt math so ordering stays correct at real epoch magnitudes. Odometer keys are namespaced by definition name. The consumer reports null lag for a missing consumer group instead of 0, and empty gauge values parse as NaN rather than 0.
Gauges are read inside the enqueue/dequeue Lua and returned on the script reply
as a 2-tuple; counters are cumulative odometers. The run-queue Redis carries no
metrics stream of its own.
Routine dependency maintenance.
- Pin a few high-fanout transitive deps to current patched versions via
`pnpm.overrides`: `form-data`, `ws`, `undici`, `hono`. Lockfile-only (no
published-package dependency changes); net shrinks via dedup.
- Upgrade `nodemailer` 8 → 9 in `internal-packages/emails` (private
package). The SES transport already uses SESv2 and the
`createTransport`/`sendMail` API is unchanged, so no code changes were
needed. `@types/nodemailer` stays at 8 (no 9.x published yet; types are
compatible).
Verified locally: `pnpm i` clean; `pnpm run typecheck --filter emails`
and `--filter webapp` both pass.
## Summary
Run-graph data (runs, batches, waitpoints, and their related tables) can
now live in a database separate from the control plane, with every read
and write routed to the correct database by each run's residency. This
makes reading and writing run data more reliable once the two are split,
and is a no-op for single-database installs.
## Design
- Run-graph table access goes through the run-store router, which
selects the legacy or the new run-ops store per run instead of assuming
one shared client.
- The legacy run-ops client is now independently pointable, so legacy
run data can be served from its own database (and replica) rather than
the control-plane connection.
- Run-graph writes go straight to the run-graph database instead of
being forwarded through the control plane, and replication targets are
split so runs in the new database still replicate to analytics without
under-counting.
- Read-through slots refuse the control-plane client, so a missing
residency fails loudly instead of silently reading the wrong database.
- Migration `20260710120000_drop_remaining_run_graph_seam_foreign_keys`
drops the foreign keys that still crossed the run-graph / control-plane
seam, which is what lets the two live in separate databases.
The split stays off unless explicitly enabled and the two databases are
confirmed physically distinct; startup fails closed otherwise.
Verified by running the full dashboard end-to-end suite against both a
single-database configuration and a three-database configuration
(control plane, the new database, and a physically separate legacy
database), with runs on both residencies. No misrouted reads in either
configuration.
Katia Bulatova