The wait metric measured dequeue time minus the original trigger time
even when a run re-entered the queue after a waitpoint, checkpoint, or
pending version, so the whole wait or checkpoint duration showed up as
scheduling delay. Re-enqueues now anchor to the re-enqueue time while
first enqueues keep the trigger or delay anchor; queue ordering is
unchanged, so re-enqueued runs keep their original position. Nacked
runs never left the queue stint and keep the original anchor.
Also replaces the :ck: suffix regexes on user-controlled queue names
with indexOf slicing (identical semantics) to remove a polynomial
regex flagged by code scanning.
TableSchema gains a hidden flag: hidden tables stay fully queryable
through the engine (the concurrency-keys tab keeps working, tenancy and
the merge guard still apply) but are excluded from the query editor
autocomplete, the Query page schema and examples panels, the AI query
generator context, and the schema API. queue_metrics_by_key is hidden
until the per-key surface is settled.
The queue detail page splits into Overview (the existing concurrency,
backlog, delay, and throttle charts) and a Concurrency keys tab that
holds all per-key content. The tab only appears for queues with key
activity. Inside it, the grouped per-key charts now lead (backlog by
key plus a new throughput-by-key chart that makes fair-share visible),
followed by the two whole-queue health charts retitled to say what they
aggregate (keys with queued runs count, most-starved key wait), then
the key table and drill-down.
Queues that shard work with concurrencyKey get a per-key history tier.
Counter events for CK runs advance a second per-key odometer and carry
both readings on ONE stream entry (cum + ck/ckcum), so per-key
attribution adds no stream volume; the consumer expands the entry into a
base row and a per-key row. A new 10s AggregatingMergeTree tier keyed by
(queue, concurrency_key) holds per-key enqueue/started/ack deltas,
backlog/running maxes, and wait sums. Rows are activity-bound: a
(queue, key, bucket) row exists only when that key had events, so
user-controlled key cardinality cannot inflate the table (benchmarked at
~19 bytes per event with reads under 100ms in the worst shapes).
The per-queue tiers stay exact: their counter and wait aggregations now
consume only base rows, so per-key odometers never merge under one
queue_name and waits are never double counted. A per-queue key limiter
(default 10k) acts as a safety valve; on overflow the per-key row is
dropped while the base row keeps per-queue counts exact. Per-key
odometers use a short TTL, which cumulative counters make loss-free.
The queue detail page gains a top-keys-by-backlog chart, a key table
merging live state (queued, running, oldest wait) with range stats
(started, peak backlog, mean delay), and click-through per-key
drill-down charts. The new queue_metrics_by_key table is also queryable
directly; its delta columns require grouping or pinning BOTH queue and
concurrency_key, enforced by the compile-time merge guard which now
supports compound keys.
CK queues now emit two extra gauge fields from the CK-path Lua scripts:
the number of concurrency keys with queued runs (ZCARD of the ckIndex)
and the head-of-line wait of the most-starved key (now minus the oldest
ckIndex score). Both flow through the existing stream into new
max-aggregated columns on the 10s and 5m tiers, and non-CK scripts keep
the 7-field gauge shape.
The queue detail page grows a concurrency-keys section for queues with
CK activity: charts for backlogged keys and most-starved wait, plus a
live per-key table (queued, running, oldest wait) read from the ckIndex
zset, most starved first. Per-key history is intentionally not stored:
key values are user-controlled, so the per-key dimension stays in Redis
where it is bounded by the live backlog.
The queue simulator gains a tenant-hotspot scenario that stages the CK
gauge columns and a live ckIndex so the per-key table and charts render
with data.
Counter readings for names past the per-env cardinality cap are dropped
instead of merging unrelated odometers under the overflow label (gauges
still flow). The odometer key now shares the stream's shard hash tag so
the INCR plus XADD script stays in one Cluster slot. The counter stream
cap defaults lower when the stream shares the run-queue Redis. The
per-bucket counter boundary undercount is documented on the delta
columns.
The consumer retries a failed batch with the same insert deduplication
token, and deduplicate_blocks_in_dependent_materialized_views re-runs
the materialized views on a source-deduplicated insert, relying on each
target table's dedup window to drop the duplicate. Only the raw table
had one, so retries appended extra copies into every aggregate tier,
silently inflating sums and quantiles. All three targets now set
non_replicated_deduplication_window, with a regression test inserting
the same batch three times.
Short parameter lists on quantilesMerge and quantilesTDigestMerge do
execute (the state layout is parameter independent, verified on both
ClickHouse versions we run), but they rely on undocumented leniency and
make the result-array indexes mean different quantiles per call site.
Every merge now uses the stored four-quantile list with indexes
re-pointed accordingly; returned values are unchanged.
deltaSumTimestamp states are kept per queue, and merging them across
queues silently returns wrong totals, on the dashboard and the public
query API alike. Columns can now declare a mergeGroupKey, and the
compiler rejects queries that merge such a column without grouping by
that key or pinning it to a single value. The error names the column,
explains the failure, and includes a corrected example query.
The allocation view keeps manual limit edits, the review dialog, and
bulk apply. The one-shot auto-balance button is removed (and the row
locks whose only purpose was protecting queues from it); a policy-driven
approach can replace it if rebalancing returns.
Without a readiness guard, every fire-and-forget emit during a metrics
Redis outage queued a command in ioredis's in-memory offline queue until
rejection. Metrics are loss-tolerant by design, so drop instead;
waitUntilReady() lets embedders await the initial connect.
The tail query's exclusion list overwrote the search's name filter via
object spread, so searching while sorted by activity showed unrelated
queues past the ranked head. Combine the conditions with AND instead.
A --usage flag stages plausible running counts in the local run-queue
Redis for the seeded queues, so the list's Running column and the
Allocation tab's usage bars have data without the run engine. Staged
state is reconciled on every run: present with --usage, cleared without.
Local Redis hosts only.
Counter events now emit per queue and op odometer readings with a seeded
zero baseline, matching the production emitter, so throughput and
started counts reconstruct from simulated data instead of reading zero.
Scenario switches prune the previous scenario's queues, a --project flag
seeds each scenario into its own project for side-by-side design review,
and a new many-queues scenario covers pagination and relevance ranking
with one runaway queue, a busy head, a bursty middle, and a sparse tail.
Adds --help.
The simulator's --reset only cleared the raw and 10s tables, leaving
stale rows in the 5m and env rollups. It also force-merges the rollups
after seeding so current-value widgets read cleanly.
The env rollup's win comes from dropping the queue dimension, not from
coarser buckets: row count is queue-independent (~8640/day/env), so full
10-second granularity stays cheap at any range. Env header tiles and
saturation charts now resolve short-range detail exactly like the
per-queue charts, and the current-value tiles read the latest 10-second
bucket instead of a minute-wide one.
The built-in queues dashboard's enqueued vs started chart merged counter
states across queues, which mixes unrelated cumulative counters and
returns wrong totals; it now merges per queue and sums outside. Env
header tiles and saturation charts read the environment rollup, so their
cost no longer scales with queue count, and coarse-bucket ranges are
served from the 5m rollup automatically. Queue list ranking runs as one
query, time bounds are aligned to the bucket grid, and repeated
auto-refresh reads share ClickHouse query-cache entries.
Adds two rollups fed from the raw landing table: a per-queue 5-minute
tier and an environment-level 1-minute tier (gauges plus TDigest wait
quantiles). Ranking now reads the 5m tier and returns the page and the
ranked total in one windowed query instead of two scans.
The 5m materialized view reads raw rather than cascading off the 10s
table: deltaSumTimestamp states hold a single first/last segment, so
merging states in an MV's hash-ordered GROUP BY double-counts bridging
spans. For the same reason the env tier carries no counter columns, and
env-wide counter totals must group by queue before summing.
The fallback WHERE injection only targeted the top-level SELECT, so a
query shaped as an outer aggregation over a FROM subquery failed to
compile: the time column only exists inside the subquery. Descend into
the subquery so the fallback lands next to the table reference.
Adds an Allocation tab to the Queues page (behind the queue metrics UI flag): overview cards, a burst-aware capacity bar showing each queue allocation and its live usage in a distinct color, an inline-editable limits table with per-queue locks, load-weighted auto-balance, and a review dialog that bulk-applies limits as overrides through the existing concurrency system.
The queue list now defaults to Busiest ordering (with Backlog and Name options). ClickHouse ranks queues by activity over the last 15 minutes and returns just the requested page of names, so the cost per page is one small aggregate regardless of environment size; idle queues follow in name order and any failure falls back to name ordering. The classic page keeps plain name order.
The queues list header tiles now render the same line chart, grid, and tooltip as the rest of the metrics charts instead of a row sparkline, with the headline value in the tile header. The env saturation tile draws the environment concurrency limit and burst limit as labeled reference lines. Chart tooltips keep a gap between the series label and the value, and the shared line chart gains showDots and referenceLines options.
Queries using deltaSumTimestampMerge failed with an unknown function error, which broke the queue detail stats and the started counts on the built in Queues dashboard.
The queues list and queue detail pages now use the shared TimeFilter (any preset period or a custom date range) and everything on the page follows it: header tiles, per queue metric columns, charts, and stats. The custom period buttons, hand rolled chart cards, and duplicated metric fetch loops are replaced by the ChartCard and Chart primitives, UsageSparkline, and a shared useMetricResourceQuery hook. The ClickHouse list queries take an explicit end bound so fixed ranges query only their window.
Packs the stream sequence with a 1e6 factor (was 1e5) so up to 1M entries per millisecond per shard fit before a seq could spill into the next millisecond's range, far above what a single Redis stream can produce. ms*1e6 stays within UInt64. Also fixes the webapp mapping test that still expected a numeric order_key after the switch to a BigInt-derived string.
A bucket-led ORDER BY DESC combined with fillGaps emitted an ascending WITH FILL (positive step, ascending bounds), which produces invalid or empty fills. Skip the gap-fill rewrite for descending orders and let the plain descending query stand. Adds a DESC fillGaps test.
The queues list tolerates a metrics query failure by rendering without metrics and logging a warning. UsageSparkline renders its total override even when every bucket is zero. The queue detail page returns 404 and its loader skips the metrics query when the feature flag is off. The seed script validates bucket size and only writes ClickHouse against a local host.
The wait-time quantile materialized view now excludes wait_ms = 0 rows so it matches the count aggregation. order_key accepts a string or a number. Migration comments no longer contain semicolons that split the migration into invalid statements.
entryOrderKey returns a string built with BigInt math so ordering stays correct at real epoch magnitudes. Odometer keys are namespaced by definition name. The consumer reports null lag for a missing consumer group instead of 0, and empty gauge values parse as NaN rather than 0.
Gauges are read inside the enqueue/dequeue Lua and returned on the script reply
as a 2-tuple; counters are cumulative odometers. The run-queue Redis carries no
metrics stream of its own.
The `ResourceMonitor` server-side logging helper is no longer used. It
periodically logged the webapp process own memory, disk, and CPU usage
behind the `RESOURCE_MONITOR_ENABLED` flag (off by default), and was
also exported from `@trigger.dev/core/v3/serverOnly` with no other
consumers.
This removes the helper, its `@trigger.dev/core` export, the webapp
wiring, and the `RESOURCE_MONITOR_ENABLED` env var. The supervisor has
its own unrelated `ResourceMonitor` class, which is left untouched.
## Summary
Adds `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` (default off), which
makes the task-event store skip all PostgreSQL `TaskEvent` writes. It's
for deployments that store task events in ClickHouse
(`EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2`) and no longer want the
PostgreSQL copy.
## How it works
The guard sits at the single postgres write boundary,
`TaskEventStore.create` / `createMany`, so it covers every write path
(OTLP ingestion and run-lifecycle events) with one check. Reads are
untouched (`findMany` / trace queries / streaming), so existing
PostgreSQL events remain readable.
Leave it off unless the default store is `clickhouse_v2`, otherwise task
events for any run still routed to PostgreSQL would be dropped.
## What
The `GET
/api/v1/projects/:projectRef/background-workers/:envSlug/:version`
endpoint loaded each file's tasks through the nested `files.tasks`
relation. Prisma resolves that as a separate query:
```sql
SELECT id, slug, "fileId" FROM "BackgroundWorkerTask" WHERE "fileId" IN (...)
```
`BackgroundWorkerTask.fileId` is not indexed — the FK constraint exists,
but Postgres does not auto-create an index for foreign keys — so on a
large table this can only run as a sequential scan, which gets
progressively slower as the table grows and was observed taking minutes
per call in production.
The loader already loads every task for the worker via `tasks: true`,
which uses the indexed `workerId` relation, and those rows already
include `fileId`. This PR groups task slugs by `fileId` in memory from
that already-loaded data and drops the `files.tasks` include entirely.
## Behavior change (latent bug fix)
The response shape is unchanged, but there is a semantic correction for
**source files reused across worker versions** (files are de-duplicated
by `@@unique([projectId, contentHash])`, so one file row can be linked
to many workers).
- **Before:** `file.tasks` came from the `BackgroundWorkerFile.tasks`
relation, i.e. *every* `BackgroundWorkerTask` with that `fileId` —
across all workers sharing the file. So a worker's manifest could list
tasks it doesn't actually have.
- **After:** `file.tasks` is grouped from the queried worker's own
tasks, so it reflects only that worker version's tasks.
Verified on a local DB: 460 files are referenced by tasks from more than
one worker; of 6819 (worker, file) pairs, 6 differ — all one file where
the old union leaked a task slug (`cancellation-test`) into worker
versions that never had it. The new per-worker behavior is the correct
one for a worker-version manifest. (Thanks to the automated review for
flagging this.)
## Analysis
Captured the exact SQL before/after by instrumenting Prisma against real
data (a worker with 62 files):
- **Before:** 5 statements, including the `WHERE "fileId" IN (...)`
scan.
- **After:** 4 statements; the `fileId` query is gone and the other four
are identical.
EXPLAIN of the two access paths:
```
Before WHERE "fileId" IN (...)
Seq Scan on "BackgroundWorkerTask"
Filter: ("fileId" = ANY (...)) -- reads the whole table, scales with table size
After WHERE "workerId" IN (...)
Index Scan using "BackgroundWorkerTask_workerId_slug_key"
Index Cond: ("workerId" = ...) -- bounded by matching rows, scale-independent
```
No new index is required: the `workerId` access path is already covered
by the existing `BackgroundWorkerTask_workerId_slug_key` unique index.
## Testing
- `pnpm run typecheck --filter webapp` passes.
- Query capture + EXPLAIN performed against a local database seeded with
real worker/file/task data.
Eric Allam