v.docker.4.5.9
450 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Eric Allam
|
6e5f0f0fe7 |
fix(webapp,clickhouse): stop invalid customer queries alerting, and isolate Sentry scope per request (#4372)
## Summary
A query sent to the query API with a typo in it, like a column name that
does not exist, was being reported as a server error. That put customer
SQL mistakes into our error alerting, where they made up almost all of
the volume on one of our noisiest alerts, and it drowned out the
failures that are actually ours to fix. This makes the level match who
is at fault, and fixes two related problems found alongside it.
## Invalid queries are the caller's, not ours
The query API route already got this right. It checks for `QueryError`,
logs at warn, and returns a 400, with a comment saying the system
handles it gracefully and no alert is needed.
The layer underneath ignored that. `executeTSQL` logged every exception
out of its catch block at error, including the compile failures the
route was about to turn into a 400, and error-level logs are forwarded
to error reporting.
The TSQL package already draws the line we need:
```ts
export class ExposedTSQLError extends BaseTSQLError {
/** An exception that can be exposed to the user. */
}
export class InternalTSQLError extends BaseTSQLError {
/** An internal exception in the TSQL engine. */
}
```
`SyntaxError` and `QueryError` extend the first. So the catch block now
branches on `ExposedTSQLError` and logs those at warn, keeping error for
`InternalTSQLError` and anything unanticipated, which is a genuine
compiler bug.
## SQL the caller wrote is their mistake, not ours
The same asymmetry showed up one level down. A query that compiles fine
can still be rejected by ClickHouse at execution, and most of those
rejections mean the caller's SQL is wrong rather than that we generated
something bad.
This is where the volume actually is. Checking production, one error
group alone, a missing `GROUP BY` on the public query API
(`NOT_AN_AGGREGATE`), accounts for over a million events across hundreds
of users. It is by far the largest error group in the project, and
classifying only by resource limit would have left every one of those at
error level.
So rejections are split three ways in `ClickhouseClient`, which is the
only place holding the parsed `ClickHouseError` and its symbolic type.
By the time the error reaches `executeTSQL` it has been wrapped and the
type is gone, and the type never appears in the message text, so it
cannot be recovered by string matching.
- **Resource limits** (memory ceiling, timeout, row/byte caps) log at
warn. The query is valid, it just asked for more than it is allowed to
spend.
- **Invalid SQL** (`NOT_AN_AGGREGATE`, `UNKNOWN_IDENTIFIER`,
`SYNTAX_ERROR`, the type and parse families) logs at warn **only when
the caller wrote the SQL**.
- **Everything else** keeps alerting.
That gate matters. The client is shared, so the identical rejection on
TRQL *we* generated is our bug and has to stay at error. Callers opt in
with `userAuthoredQuery`:
| caller | who wrote the SQL | opts in |
| --- | --- | --- |
| public query API | the customer | yes |
| query editor | the customer | yes |
| agent charts | the agent's model | yes |
| built-in dashboard tiles | us, in code | no |
| queue metric cards | us, in code | no |
| health report | us, in code | no |
The agent is the one judgement call. Its TRQL is not typed by a person,
but it is also not something a code fix makes correct, so a query it
gets wrong is not worth waking anyone for. The same endpoint serves
built-in tiles whose TRQL we do write, so the opt-in lives with the
caller rather than the route.
Separately, when one of these queries did fail, the log recorded the
generated ClickHouse SQL but not the query the caller actually wrote,
which made the reports hard to act on. `queryWithStats` takes an
optional `logFields` that `executeTSQL` uses to attach the original
TSQL.
## Events were attributed to the wrong request
Chasing the above turned up something broader: only a tenth of the
events on that alert pointed at the query API. The rest were pinned to
unrelated requests that happened to be in flight at the same time, so
the alert looked like the trigger endpoint was failing.
`Sentry.init` runs with `skipOpenTelemetrySetup: true`, because we
register our own OTel pipeline. That skips `initOpenTelemetry`, and one
of the things it does is:
```js
api.context.setGlobalContextManager(new SentryContextManager());
```
The async-context strategy is still installed, but `withIsolationScope`
only marks the OTel context and delegates the actual fork to that
context manager:
```js
// "We depend on the otelContextManager to handle the context/hub"
return api.context.with(ctx.setValue(SENTRY_FORK_ISOLATION_SCOPE_CONTEXT_KEY, true), ...)
```
`provider.register()` installed a plain
`AsyncLocalStorageContextManager`, which does not know that key. The
lookup found no scopes on the context and fell back to the
process-global default isolation scope, so every request wrote its
request data into the same object and the last writer won.
The tracer now registers `SentryContextManager`, which subclasses
`AsyncLocalStorageContextManager`, so OTel behaviour is unchanged. It is
also registered on the path where tracing is disabled, which previously
never called `register()` at all and so had no context manager of its
own.
Tenant tags were always correct, because those come from our own async
local storage rather than the isolation scope. That is why the
attribution being wrong was not obvious.
This affects every error report the webapp sends, not just the query
API.
## Verification
`internal-packages/clickhouse`: 76 tests pass, including eight covering
each level decision against a real ClickHouse container. Three pairs pin
the gate open and shut at both layers: an invalid query, a compile
failure, and a real limit breach driven with `max_rows_to_read` each log
at warn with `userAuthoredQuery` and at error without it.
The isolation fix has a test that reproduces the leak before asserting
the fix. Two overlapping requests each tag their own isolation scope;
with the plain context manager the slower one reads back the other's
tag, and with `SentryContextManager` each reads back its own.
Measured separately against a faithful reproduction of the server's
wiring (own OTel pipeline, CommonJS entry) at 200 concurrent requests:
per-request attribution goes from 0.5% to 100%, while span nesting,
context propagation across awaits, and distinct trace IDs are identical
before and after.
|
||
|
Chris Arderne
|
a81ad4949c |
feat(database,rbac): add multiple environment API key foundations (#4388)
Adds the storage model and authorization contracts needed for multiple environment API keys. Credentials are represented by hashed values, revocation and expiration state, and persisted effective scopes. The built-in authorization fallback exposes full-access policy preparation, while optional authorization extensions can supply additional presets and task-aware scope generation. This change does not create, display, or authenticate additional keys. |
||
|
Eric Allam
|
4eb9292cbe |
feat(webapp,run-engine): queue metrics and health dashboard (#4131)
## Summary
Three related changes, each independently gated:
**Queue metrics and health.** Per-queue depth, throughput (enqueued,
started, completed), concurrency, whether a queue is throttled, and
scheduling delay (how long a run waits between becoming eligible and
actually starting), plus a per concurrency-key breakdown for keyed
queues. Collected from inside the run queue itself, stored in
ClickHouse, and surfaced on the Queues list, a new per-queue detail
page, the task pages, and the run inspector. The question it answers is
"does this queue have enough concurrency to keep up, and if not, which
key or which limit is the constraint".
**Percent-based queue concurrency limits.** A queue's concurrency
override can now be expressed as a percentage of the environment limit,
stored as the source of truth and re-materialized whenever the
environment limit changes. Absolute overrides above the environment
limit are now **rejected with a 400** instead of being silently capped,
which is a behavior change on `POST
/api/v1/queues/:queue/concurrency/override`.
**The `health` report.** A server-computed verdict on whether work is
flowing, whether the runs that do start are healthy, and whether
telemetry is fresh, rendered as text with sparklines. Available as `GET
/api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool
(plus a `report` MCP prompt, which shows up as a slash command in hosts
that support prompts).
With the flags off, the Queues page renders the pre-metrics component
verbatim, nothing is emitted, and nothing is written to ClickHouse.
## Configuration
Two independent gates, on purpose. Emission is global so data accrues
for everyone before anyone can look at it; the view is per organization
so it can be turned on for one org at a time without a deploy.
**Runtime flags (no restart)**
| Flag | Store | Gates |
| --- | --- | --- |
| `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by
default) | All emission, gauges and counters. Cached in-process for 10s
with stale-while-revalidate, warmed eagerly at boot so the first op
after a deploy is not dropped. |
| `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` |
Fraction of queue ops that emit a gauge. Counters are never sampled, so
throughput stays exact at any rate. |
| `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag`
row, per-org `Organization.featureFlags` override wins | Whether an org
sees the metrics view at all: the Queues list variant, the queue detail
route, the built-in Queues dashboard, the concurrency-keys endpoint, and
the metrics blocks on task pages and the run inspector. Off by default;
a gated org gets a 404 on the detail route rather than an empty page. |
Both Redis keys are readable and writable from `/admin/queue-metrics`
(super-admin UI, with a live per-shard stream-health table) and
`GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface
uses its own Redis client, so it works on any instance regardless of
whether that instance runs the emitter or the consumer.
**Environment variables (boot time)**
| Variable | Default | Notes |
| --- | --- | --- |
| `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and
injects it into the run engine. Without it the run queue has no emitter
at all. |
| `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on
this instance. Independent of emission, so consumers can be sized
separately from the API. |
| `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per
queue. |
| `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals
insert batch, so an ack can never outrun a write. |
| `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` |
falls back to the run-queue Redis | Set `HOST` to move the metrics
stream onto a dedicated instance so a metrics backlog cannot compete
with the run queue for memory. Self-hosters can leave it unset and get a
single-Redis deployment. |
| `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000`
dedicated | Bound on how much a stalled consumer can hold. The default
is deliberately lower when the stream shares the queue-critical Redis. |
| `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the
per-queue cumulative counter key, refreshed on every write, so only
queues idle for the whole window are purged. |
| `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue
names tracked per environment; overflow collapses into `__overflow__`. |
| `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea
one level down, per queue. |
| `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live
sample-rate key above. |
| `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics
tables in the Query page, its schema docs, the schema API and the AI
query context. Off keeps them unlisted while the feature is dark; a
query naming them still runs either way. |
| `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring |
Runs queue metrics on their own ClickHouse service: the consumer's
inserts and every queue-metrics read go through it, so a metrics-heavy
chart refresh never competes with runs-list or trace reads. Unset
reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`,
reads on the query pool). |
| `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split,
so the consumer's inserts can never land on a read endpoint. |
|
`QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}`
| `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other
per-workload ClickHouse clients. |
Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a
Postgres migration adding the nullable
`TaskQueue.concurrencyLimitOverridePercent`. Both are additive.
## How collection works
Queue operations produce two kinds of signal, and they have opposite
failure modes, so they are handled differently.
**Gauges** (queued, running, queue limit, env queued, env running, env
limit, throttled, plus keys-with-backlog and worst-key wait on keyed
queues) are read *inside* the same Redis script that performs the
enqueue or dequeue, so the reading is atomic with the operation it
describes rather than a racy follow-up read. The script returns them on
its reply and the app forwards them to the stream. Gauges are sampled
and drop-tolerant: they are aggregated with `max`, so a lost reading
costs resolution, never correctness.
**Counters** (enqueued, started, completed, plus nack and dead-lettered)
are cumulative odometers. Each event increments a per-queue key on the
metrics Redis and emits the absolute total, and ClickHouse takes the
difference across buckets at read time. This is the important property
of the design: a summed-delta counter undercounts permanently on any
lost event, while a cumulative one self-heals, because the next
surviving reading restates the whole total. Only bucket granularity can
be lost, never the total. A queue returning after its odometer TTL
expired restarts at 1 and reset detection handles it, which is safe
precisely because expiry only spans a window with no activity.
Both land on one sharded Redis stream. A consumer reads it with a
consumer group, reclaims stale pending entries on a 15s interval rather
than on every poll, maps one entry to one or two ClickHouse rows
(whole-queue and, for keyed queues, per-key), and acks only after the
insert lands. Each batch carries a dedup token derived from its
stream-entry ids, and the target tables set
`non_replicated_deduplication_window`, so a retried batch cannot
double-count either the raw rows or the aggregates that hang off them.
Consumer and emitter both emit OTel metrics
(`queue_metrics.emitter.emitted`,
`queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`);
stream depth and group lag are the two worth alerting on, and
`lag_unknown` exists because Redis can report a null lag after a trim,
which must not be read as zero.
## Storage and read path
`queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four
aggregate tiers are materialized straight from raw, never cascaded off
each other, each with a 30 day TTL:
- `queue_metrics_v1`, 10 second buckets per queue, the default read path
- `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and
cross-queue ranking
- `env_metrics_v1`, 10 second buckets per environment, queue-independent
so it stays cheap at any range
- `queue_metrics_ck_v1`, 10 second buckets per concurrency key
Every tier is an MV from raw because the counter states do not survive a
cascade: their merge is order sensitive, so a `-MergeState` chain off
the 10s table inflates the result, and the same property means an
aggregate state may only be merged inside one queue. That constraint is
now enforced by the query engine rather than by reviewer discipline: a
column can declare a `mergeGroupKey`, and any query that references it
without grouping by, or pinning to a single value of, every named key
fails to compile with an actionable message.
On the read side, TRQL gains three tables (`queue_metrics`,
`env_metrics`, and a `queue_metrics_by_key` that is hidden from the
editor, schema docs and schema API but still queryable, so per-key rows
can never silently merge into a plain per-queue query), plus
`deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level
optimizations ride along: a table can declare coarser rollups, so a
query whose bucket interval is 5 minutes or wider is routed to the 5m
table with no change to the query itself, and it can opt into the
ClickHouse query cache with time bounds floored to a fixed grid, so the
auto-refreshing dashboards actually share cache entries instead of
missing on every tick. Both are caller-side substitutions, so the
printer stays unaware of physical layout.
All of this can also live on its own ClickHouse service. A table
declares the pool its reads run on, the three queue-metrics tables name
the dedicated one, and the ingestion consumer writes through the same
client, so both directions move together with one env var and nothing
else routes differently.
The other engine change is opt-in gap filling: charts can request rows
for empty buckets, where counters zero-fill and gauges carry forward.
Grouped gauge series are densified per group and carried inside a
partition, so a quiet queue's line holds its last value without bleeding
another queue's value into it.
## Queue concurrency limits
`concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth
when an override is set as a percentage; the absolute `concurrencyLimit`
is materialized from it (floored, clamped to at least 1 so a percentage
can never act as a pause, and never above the environment limit). Every
path that changes an environment limit now recalculates the
environment's percent-based overrides afterwards, outside the
transaction, and pushes changed limits to the engine. The push is
attempted even when the stored value did not change, so a previously
failed sync self-heals rather than leaving the database and the engine
diverged; paused queues are skipped so a recalculation cannot
effectively unpause one.
The API accepts exactly one of `concurrencyLimit` or `percent`, and the
reject-instead-of-clamp change above means a request asking for more
than the environment allows now fails loudly. The percent bound (greater
than 0, at most 100) is defined once and shared by the zod schema, the
dashboard mutation handler and the service, so the three cannot drift.
The concurrency-keys table on a queue is now paginated against the
ClickHouse per-key tier, ranked by peak backlog with the total on every
row from a single scan, and only the keys on the current page are
enriched with live counts from Redis. That replaces a hard top-50 cap
with something whose cost is a function of page size rather than key
cardinality.
## The health report
`GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The
verdict is computed on the server and is deterministic, not
model-generated. Three independent analyzers run over one input
snapshot: flow (is work moving, and if not, is the cause a limit,
throttling, one bad queue, or dead-lettering), execution (are the runs
that start succeeding, and at what latency), and liveness (how fresh is
the telemetry). When telemetry is genuinely stale, the first two are
forced to unknown and every actionable field is stripped, so no surface
ever advises action off stale data.
Authorization is per query table rather than a blanket query grant: a
JWT must be scoped to every table the report reads (`runs`,
`env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull
a report that reads more than it was granted. `period` is validated as a
shorthand with a 90 day ceiling at the edge. The report catalog is a
registry of `{ load, interpret }` entries, so the next report is a new
entry and no change to the route, the view model, the renderers, the CLI
or the MCP tool.
`trigger mcp` no longer launches the install wizard when stdout is a
TTY, which fixed a real failure: hosts spawn the server over a PTY, so
the wizard would open and the client would time out waiting for a server
that never started. The wizard now needs `trigger mcp --install`.
## The part that is live regardless of every flag
The enqueue and dequeue scripts now return a 2-tuple so a gauge reading
can ride back on the reply. Every return site in the eight affected
scripts is wrapped, and a `nil` original is converted to `false` on the
way out, because a raw `nil` in the first slot would make Lua truncate
the multi-bulk reply and silently drop the gauge on the throttled and
empty-queue paths. The reply shape and the destructuring on the app side
are exercised on every queue operation whether or not metrics are
enabled, so that is the part of `run-engine` worth the closest review.
One behavior fix in the same area: the scheduling-delay anchor is set
only on a run's first entry into the queue. Anchoring it to trigger time
on re-enqueues made waitpoint and checkpoint resumes report the entire
wait as scheduling delay. Queue ordering is untouched, so a re-enqueued
run keeps its position, and nacks deliberately keep the original anchor
because a rolled-back dequeue is the same continuous wait.
A pending-version promotion still anchors to trigger time, on purpose:
that promotion is the run's first real entry into the queue, since the
trigger deliberately held it back waiting for a worker version, and the
TTL is armed at the same point for the same reason. The consequence is
worth naming, because it is a judgement call: a run that waits on a
deployment reports that wait as scheduling delay on its queue, which is
time unrelated to queue capacity.
## Verification
Unit and integration suites across the new package, the run queue, the
mapping layer, the query engine and ClickHouse (including a test that
applies migration 036 through the same splitter CI uses, and a
regression test that inserts the same batch three times to prove the
aggregates do not inflate). Beyond that, the whole path was driven end
to end against a live stack with real runs: emitter to Redis stream to
consumer to ClickHouse to the dashboards, for both the local dev path
and the deployed path where a supervisor drives the dequeue, with
assertions on exact counter reconstruction per queue and per concurrency
key, throttling, environment saturation, scheduling delay, and a
deliberate mid-stream reading drop to confirm the cumulative counters
still reconstruct the correct total. The gated-off state was checked on
every touched surface.
The dedicated ClickHouse service was verified against a second,
separately-schema'd instance: with it configured, the driven counters
reconstruct exactly on the dedicated instance, the shared instance gains
no rows for that window, a read through the query API returns the value
that exists only on the dedicated instance, and a `runs` query still
succeeds (it would fail outright if it were mis-routed to a service
without that table). With the variable unset, the full suite passes
unchanged.
---------
Co-authored-by: Katia Bulatova <katia@trigger.dev>
Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com>
Co-authored-by: James Ritchie <james@trigger.dev>
|
||
|
claude[bot]
|
ec562c0e68 |
fix(webapp): remove unused Electric sync trace routes (#4400)
<!-- ccr-slack-attribution --> _Requested by **Eric Allam** · [Slack thread](https://triggerdotdev.slack.com/archives/C0AU83M3136/p1785222101937829?thread_ts=1785207509.304669&cid=C0AU83M3136)_ Removes two dead Remix routes and the helpers only they used. `app/routes/sync.traces.runs.$traceId.ts` (`/sync/traces/runs/:traceId`) and `app/routes/sync.traces.$traceId.ts` (`/sync/traces/:traceId`) were added with the original ElectricSQL run page and lost their only consumers when the dashboard hooks that called them were deleted. Nothing in the repo references either route today. Also removed, because the deleted routes were their only callers: - `OtelTraceIdSchema`, `RESERVED_ELECTRIC_SHAPE_PARAMS`, `TraceScope`, `buildElectricTraceWhereClause` from `app/v3/electricShape.server.ts` (the file stays — `UNSAFE_REALTIME_TAG_CHARS` / `sanitizeRealtimeTagForSql` / `sanitizeRealtimeTagsForSql` are still used by `realtime.v1.runs.ts` and `realtimeClient.server.ts`) - the loader-specific cases in `apps/webapp/test/spanTraceRoutes.replicaLag.test.ts` and `internal-packages/run-store/src/runOpsStore.routesSpanTraceReadView.replicaLag.test.ts` `app/utils/longPollingFetch.ts` is untouched — `realtimeClient.server.ts` still uses it. `runOpsStore.ts` / `PostgresRunStore.ts` are untouched too; the unrouted-lookup mechanism there is generic and stays. As a plain code fact: the run lookup these loaders performed keyed on `TaskRun.traceId` alone, which is not an index-backed query shape. That is noted only as context for why the code is not worth keeping around unused. ### Judgement call worth a maintainer's opinion The request was specifically about `/sync/traces/runs/:traceId`, the route that looks up a run by `traceId`. This PR **also** deletes its sibling `/sync/traces/:traceId`. The reasoning: - both routes came in with the same ElectricSQL run-page work - both lost their only consumers in the same later commit - neither has any caller anywhere in the repo - they share the same helper module, so keeping one means keeping the helpers half-used If you would rather keep the sibling, reverting just that one file deletion is easy and does not affect the rest of this PR — say the word and I will restore it along with the helpers it needs. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Verification run locally from the repo root: | Command | Result | | --- | --- | | `pnpm run format` | clean, no changes produced | | `pnpm run lint:fix` | clean | | `pnpm run lint` | pass (exit 0, no findings) | | `pnpm run typecheck --filter webapp` | pass | | `pnpm run typecheck --filter @internal/run-store` | pass | A ripgrep sweep for `sync.traces`, `sync/traces`, `syncTraceRunsLoader`, `buildElectricTraceWhereClause`, `OtelTraceIdSchema` and `RESERVED_ELECTRIC_SHAPE_PARAMS` (excluding `node_modules`) returns zero hits. **Not fully verified:** both edited test files are testcontainers suites and need a Docker runtime, which was not available in my environment. I confirmed each file *collects* correctly with exactly the three intended remaining tests and no import errors — notably, dropping the `session.server` / `controlPlaneResolver.server` / `longPollingFetch` / `env.server` mocks does not break module loading for the surviving loaders. The assertions themselves then failed only on `Could not find a working container runtime strategy`. CI should be the real signal here. Per `apps/webapp/CLAUDE.md`, `pnpm run build --filter webapp` was deliberately not run. --- ## Changelog Removed two unused sync routes left over from the original ElectricSQL run page, along with the helpers and tests that existed only to serve them. No behaviour change — neither route had any caller. --- ## Screenshots _n/a — no user-visible surface changes._ 💯 --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Eric Allam
|
fc576436e2 |
perf(run-ops-database): index BatchTaskRun for the batches list on the dedicated schema (#4396)
## Summary The batches list page orders by `(createdAt DESC, id DESC)`, which is why [#4361](https://github.com/triggerdotdev/trigger.dev/pull/4361) added a matching index on `BatchTaskRun`. That index only landed in `@trigger.dev/database`. The dedicated run-ops database has its own migration history, so it never received the index. `BatchListPresenter` reads both databases and merges, so for environments whose batches live in the dedicated database the page kept falling back to a scan and in-memory sort, which is the exact behaviour #4361 set out to fix. ## Fix Adds the index to the run-ops schema with its own migration. `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it is a no-op where the index already exists and still records its ledger row. The second half is the interesting part. Because the two packages own separate migration histories, a run-graph schema change has to be authored twice, and nothing made the miss visible: the run-ops status check truthfully reports "up to date" against its own history, so the apply step just skips. `schemaParity.test.ts` compares the physical shape of every model the run-ops schema declares against its counterpart in `@trigger.dev/database`: scalar fields with their attributes, plus `@@index`, `@@unique`, `@@id` and `@@map`. Relation navigation fields are excluded, since the run-ops schema deliberately drops relations that would cross a database boundary while keeping the scalar FK column. A field counts as a relation when its type resolves to a model name, which keeps enum-typed columns in scope. Two models are listed as run-ops-only: `CompletedWaitpoint` and `WaitpointRunConnection`, both explicit FK-free replacements for a control-plane implicit many-to-many, since an implicit m2m carries a foreign key that cannot resolve across databases. The test also asserts that exception list is exhaustive, so a new unpaired model fails rather than being silently skipped. Confirmed the guard actually fails: reverting the index turns `BatchTaskRun` red with the missing `@@index` named in the diff. |
||
|
Eric Allam
|
9c85e0ecdc |
perf(database): index BatchTaskRun on (runtimeEnvironmentId, createdAt, id) for the batches list (#4361)
## Summary The batches list page orders by `createdAt DESC, id DESC` filtered by environment and a created-at window, but the only supporting index on `BatchTaskRun` was `(runtimeEnvironmentId, id)`. That index can't satisfy the `createdAt` ordering, so on environments with a large number of batches the query fell back to a full table scan and in-memory sort, which could run long enough to hit the statement timeout. ## Fix Adds `(runtimeEnvironmentId, createdAt DESC, id DESC)` on `BatchTaskRun`. The query now reads straight from the index in order with no sort step, returning a page with only a handful of heap fetches instead of scanning the whole environment slice. The migration uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it takes no table lock and is a no-op if the index already exists. |
||
|
Eric Allam
|
e9ac98b7a1 |
perf(run-store): route id-set reads to the owning store, not both DBs (#4342)
📚 Publish docs / publish (push) Has been cancelled
## Summary The split run-store's id-set read path (`#findRunsByIdSet`, used by the runs-list hydrate, the realtime hydrator, and engine sweeps) queried the new store for the entire id set and then probed the legacy store for the misses. A run's residency is a total function of its id (run-ops ids live in the new store, every other id in legacy), so each id belongs to exactly one store. Route each id to its owner and query each store only for its own ids, in parallel. Same result set, and while a split is active with most runs still on legacy it removes a wasted new-store query from every id-set read. ## Change `#findRunsByIdSet` now partitions the ids by `classifyResidency` and runs one bounded query per store (skipping an empty side), in parallel, mirroring `expireRunsBatch` and the single-run `#route`. `finalizeRows` still applies orderBy/take/skip globally over the merged set. This drops the id-set path's cross-store fallback, which existed to prefer the new-store copy when the same id was present in both stores. That collision cannot arise when each id maps to exactly one store (nothing writes a legacy-shaped id into the new store), so the fallback is dead code. The two id-set tests that asserted "new copy wins on collision" now assert the routing invariant: a legacy-shaped id resolves to the legacy store and the path never consults the new store. The open-predicate path (`#findRunsOpen`) is unchanged: an open `where` has no id to route on, so it still unions both stores and dedupes. |
||
|
Chris Arderne
|
6642c8b785 |
fix(webapp): prevent duplicate envs from provision race (#4261)
Fixes TRI-12078 ## Summary Prevents concurrent environment setup requests from creating duplicate Staging and Preview environments. ## Fix Adds database-enforced uniqueness for root Staging and Preview environments. If two requests race, the losing request loads the environment created by the winner and continues successfully instead of creating a duplicate or returning an error. |
||
|
Katia Bulatova
|
d05f1a7398 |
chore(webapp): migrate from Remix compiler to Vite (#4188)
Replaces Remix compiler with the Vite plugin. The Express server (cluster, socket.io, ws) and the Docker image contract are unchanged. |
||
|
Chris Arderne
|
dc87b884e7 |
chore: upgrade to typescript 6 (#4310)
## Summary Upgrades the workspace to TypeScript 6.0.3 and applies the compiler, type, and build configuration changes required to preserve package layouts and existing runtime behavior, apart from correcting the HTTP status field used for deployment connection errors. ## Compatibility - Centralizes TypeScript 6.0.3 through the pnpm workspace catalog. - Replaces compiler options and module resolution modes that TypeScript 6 no longer accepts. - Restores explicit Node types where TypeScript 6 no longer includes them transitively. - Adds explicit declaration build roots that preserve each package's existing output layout. - Patches tsup to stop injecting the removed `baseUrl` option during declaration builds. - Uses type-only assertions for stricter typed-array and stream definitions without changing runtime behavior. - Reads the EventSource v3 HTTP status from `code`, so deployment connection errors include it correctly. - Keeps standalone CLI compatibility fixtures pinned to their existing TypeScript version and lockfiles. `turbo run typecheck` and the complete PR test suite are green. |
||
|
Chris Arderne
|
6997aeb05e |
fix: security release 2026-07-08 (#4316)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
|
||
|
Daniel Sutton
|
a7c734c223 |
test: caller-driven replica-lag + idempotency guards (stacked on #4284) (#4285)
## Stacked on #4284 — tests only This PR contains **only the tests** that guard the production fixes in #4284 (its base). Review #4284 first; this branch adds no production code. ## What Caller-driven replica-lag and idempotency guards for every fixed site: - Each guard **drives the real exported caller** (route loader/action, presenter `.call()`, service, or engine method) against a **real Postgres** with the owning replica frozen via the shared `laggingReplica` testcontainer primitive — never a store-seam reimplementation. - For a **fixed** site the guard goes **RED when the production change is reverted**; for a **tolerated read-view** site it's a caller-driven **GREEN** proof the miss self-heals (returns null/empty, no mutation, row live on primary). - The **global-scope idempotency** guard drives the real dedup + claim path through a **real `MollifierBuffer` over a Redis testcontainer** (real SETNX/poll/publish), and covers the cross-DB **andWait** waitpoint wiring and the **expired/failed clear-and-recreate** reacquire cases. Run with `vitest --no-file-parallelism` (testcontainers). Verified GREEN, and revert→RED verified per fixed site. |
||
|
Daniel Sutton
|
ae96b6c175 |
fix: read-your-writes + global-scope idempotency correctness under the run-ops split (#4284)
## What & why
Two related correctness fixes for the run-ops DB split. Under the split,
run-store reads can route to a **lagging read replica**; a just-written
run/waitpoint/batch can then be missed, causing a wrong decision.
**1. Read-your-writes → owning primary.** Surfaced first as an
intermittent `wait.until({ idempotencyKey })` re-wait on retry. Auditing
the run-store read surface found the same class at sibling sites (some
gating mutations or returning spurious 404s, others
tolerable/self-healing). Reads that must observe their own writes now
route to the owning **primary**
(`findRun`/`findWaitpoint`/`findBatchTaskRunByFriendlyId` →
`*OnPrimary`, a primary re-read on a miss, or a retryable 404 where the
SDK polls). Read-view reads stay on the replica. All additive — the
happy path is unchanged.
**2. Global-scope idempotency across the split.** A `global`-scope key
carries no per-run salt, so the same `(env, task, key)` triggered
concurrently from parents resident on **different** run-ops DBs could
dedup-miss on each DB and create a duplicate (the per-DB unique index
can't enforce cross-DB uniqueness). Such triggers (global scope, or
scope-absent, while split is active) are serialized through the existing
Redis idempotency claim, the loser resolves the winner by id across both
DBs, and the claim is reacquired on the expired/failed
clear-and-recreate path. `run`/`attempt` scope embed the run id and
never contend.
## Stacked for review
This is the **base** of a 2-PR stack, split so review is easier:
- **This PR** — production code only (34 files).
- **Stacked tests PR →
https://github.com/triggerdotdev/trigger.dev/pull/4285** — the
caller-driven guards (55 test files) on top of this branch.
## Validation
Local run-ops split, **both 2-DB and 3-DB**, fresh boot on this branch:
SDK canary 64/71 (only the known concurrency/input-streams/s3 failures),
quarantine sweep **0 unexpected** (340 pass / 16 known / 4 local) in
each topology, dashboard e2e 0 failed. No product regressions.
|
||
|
Daniel Sutton
|
821972176d |
fix(run-store,webapp): correct split-database read routing, write residency, and batches list ordering (#4272)
## Summary Correctness and performance fixes for deployments that split run data across more than one database. Single-database / self-hosted deployments are unaffected (they collapse to a single read/write path). - **Batches list (dashboard):** for some organizations the Batches list could hide older batches or show them out of order. It now orders and paginates by creation time (with the id as a stable tiebreak), so every batch appears exactly once, newest first. The pagination cursor format changes; older in-flight cursors simply restart from the first page. - **Reads:** waitpoint and snapshot lookups that are keyed by a single run now read only the database that holds that run instead of querying both, removing redundant queries on hot paths (unblock, snapshot reads). - **Writes:** environment-scoped writes with no owning run (standalone wait tokens, waitpoint tags, idempotency-key resets) now land in the same database as that environment's runs, rather than defaulting to the other one. An idempotency-key reset also falls back to the other database when it matches nothing, so a reset still clears the key wherever the run actually lives. ## Notes Verified end-to-end against multi-database setups: run-keyed reads and env-scoped writes land on the correct database with no cross-database writes, and the batches list surfaces every batch in creation order. New tests cover the batches ordering/reachability and the write-residency routing. |
||
|
Chris Arderne
|
d7ec75d5ad |
feat(runtime): add experimental Node.js 24 and 26 task runtimes (#4085)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
## Summary Adds experimental Node.js 24 and 26 task runtimes through the `experimental-node-24` and `experimental-node-26` config values. Existing runtime defaults and the `node`, `node-22`, and `bun` behavior remain unchanged. The unprefixed `node-24` and `node-26` config values remain unavailable until the runtimes are ready for general use. ## Design Experimental config values normalize to canonical runtime identifiers before build manifests are created, keeping deployment metadata and execution behavior consistent. Kubernetes task pods also use the runtime-default seccomp profile so modern Node.js versions fall back from io_uring to checkpoint-compatible system calls. |
||
|
Eric Allam
|
43250522a5 |
fix(run-store): fix batch idempotency lookup on the dedicated run-ops store (#4271)
## Summary `batchTrigger` requests that set a per-item `idempotencyKey` failed with a 500 when the run-store is split across databases: the per-item idempotency lookup errored before any run was created. Batches without per-item keys, single `trigger` idempotency, and batch-level (`idempotency-key` header) idempotency were unaffected. ## Root cause `findRunsByIdempotencyKeys` built its `UNION ALL` of per-key point-lookups with `@trigger.dev/database`'s `Prisma.sql` / `Prisma.join`, then executed it on whichever store client it was handed. On the dedicated run-ops store that client is a *separate* generated Prisma client, and a `Sql` object from a different generated client is not recognized: the bare `$queryRaw(Prisma.join(...))` form dropped the query text entirely (`Argument \`query\` is missing`). The tagged-template form is no better here: joining nested `Prisma.sql` fragments across the two clients mis-numbers the bound parameters (`syntax error at or near "$1"`). ## Fix Build the lookup as a plain parameterized string and run it via `$queryRawUnsafe` with positional placeholders and bound values, so it no longer depends on which generated client executes it. The query text contains only static SQL and integer placeholders; every value (`runtimeEnvironmentId`, `taskIdentifier`, each key) is bound, so it is not a raw-interpolation site. Same per-key point-lookup shape as before, no change on the single-client path. Verified end-to-end against a bundled build with the run-store split enabled: before the fix, `batchTrigger` with a per-item key 500s; after, it returns the runs and dedups correctly across fresh, repeat, and mixed batches. |
||
|
Chris Arderne
|
b902e65dfb |
chore: standardise internal node on 24.18.0 (#4254)
## Summary Updates the internal development, CI, and runtime-image Node version to 24.18.0. SDK compatibility coverage continues to include Node 20, 22, 24, and 26. The Node type definitions and the package-manager lockfiles now resolve against Node 24 types. |
||
|
Eric Allam
|
1ab5066ed0 |
perf(webapp,run-store): point-lookup batch idempotency keys (#4255)
## Summary Batch triggers that use per-item idempotency keys could take seconds instead of milliseconds when the target task had a large run history. This keeps the idempotency lookup fast regardless of how many runs a task has accumulated. ## Root cause The batch path checks which items already have runs by looking up their idempotency keys with a single `WHERE runtimeEnvironmentId = ? AND taskIdentifier = ? AND idempotencyKey IN (...)` query. On a very large `TaskRun` table Postgres underestimates the row count of a specific `(environment, task)` pair, so once the `IN` list grows past a handful of keys it stops doing per-key index probes and instead scans every run for that `(environment, task)` and filters the keys in memory. The cost is then flat and large regardless of how many keys are being checked, and a routine `ANALYZE` does not correct the estimate at that table size. ## Fix Look each idempotency key up on its own, batched into a `UNION ALL` of point lookups (chunked, run with bounded concurrency). Each branch is an equality on all three columns of the unique index, so the planner can only do a per-key index probe and can never fall back to the range scan. Same results, same columns, confined to the batch trigger path. |
||
|
nicktrn
|
022e5c1ad0 |
chore(deps): pin transitive deps and upgrade nodemailer to 9 (#4243)
Routine dependency maintenance. - Pin a few high-fanout transitive deps to current patched versions via `pnpm.overrides`: `form-data`, `ws`, `undici`, `hono`. Lockfile-only (no published-package dependency changes); net shrinks via dedup. - Upgrade `nodemailer` 8 → 9 in `internal-packages/emails` (private package). The SES transport already uses SESv2 and the `createTransport`/`sendMail` API is unchanged, so no code changes were needed. `@types/nodemailer` stays at 8 (no 9.x published yet; types are compatible). Verified locally: `pnpm i` clean; `pnpm run typecheck --filter emails` and `--filter webapp` both pass. |
||
|
Daniel Sutton
|
bea7e2be90 |
feat(webapp,run-store): route run-graph reads and writes through the run-store router (#4237)
## Summary Run-graph data (runs, batches, waitpoints, and their related tables) can now live in a database separate from the control plane, with every read and write routed to the correct database by each run's residency. This makes reading and writing run data more reliable once the two are split, and is a no-op for single-database installs. ## Design - Run-graph table access goes through the run-store router, which selects the legacy or the new run-ops store per run instead of assuming one shared client. - The legacy run-ops client is now independently pointable, so legacy run data can be served from its own database (and replica) rather than the control-plane connection. - Run-graph writes go straight to the run-graph database instead of being forwarded through the control plane, and replication targets are split so runs in the new database still replicate to analytics without under-counting. - Read-through slots refuse the control-plane client, so a missing residency fails loudly instead of silently reading the wrong database. - Migration `20260710120000_drop_remaining_run_graph_seam_foreign_keys` drops the foreign keys that still crossed the run-graph / control-plane seam, which is what lets the two live in separate databases. The split stays off unless explicitly enabled and the two databases are confirmed physically distinct; startup fails closed otherwise. Verified by running the full dashboard end-to-end suite against both a single-database configuration and a three-database configuration (control plane, the new database, and a physically separate legacy database), with runs on both residencies. No misrouted reads in either configuration. |
||
|
Eric Allam
|
5ba8557a51 |
chore(webapp,core): remove the end-of-life v3 (engine V1) execution stack (#4236)
## Summary v3 (the engine that ran the SDK v3 era, internally `RunEngineVersion.V1`) is end-of-life. Following the removal of the v3 execution apps ([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and the legacy dev websocket ([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this removes the remaining v3 execution stack from the server. Clients still on v3 (an old SDK or CLI that has not upgraded) keep getting a clear "upgrade to v4" response. Triggers, batch triggers, reschedules, and deploys that resolve to v3 are rejected with a graceful 4xx pointing at the migration guide, never a 5xx, so a stale client cannot affect server health. Self-hosted instances still running v3 should stay on the 4.5.x release line until they migrate. ## What is removed - The MarQS queue and its shared/dev queue consumers. - The v3 socket.io namespaces (coordinator, provider, shared-queue) and the v3 run lifecycle services (attempt, checkpoint, and batch-resume). - The graphile-worker background job system; all live jobs already run on `@trigger.dev/redis-worker`. - The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally, so the flag is gone. - Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace` subpath and the legacy socket message catalogs) and the now-dead MarQS environment variables. ## What stays The v4 engine is untouched. The graceful v3 rejection boundary stays, `determineEngineVersion` still detects a v3 project so it can reject it, and the batch service plus batch-completion worker stay for current clients. Live queue concurrency limits and metrics now read from the v4 run engine instead of MarQS, and a brand-new dev environment now defaults to v4. ## Dependency cleanup Removes webapp dependencies left unused by this change: `seedrandom` and `semver` (only the removed v3 code used them) plus a set that was already dead, their orphaned `@types` packages, and two dead files. Adds a `knip:deps` script and a `knip.json` config so unused dependencies can be found the same way going forward. |
||
|
Daniel Sutton
|
c601739d35 |
perf(webapp,run-store): grouped run-ops reads + mint-kind flip grace (#4227)
## Summary Two threads on the run-ops split path. Read path: per-item run reads are batched into grouped queries, a waitpoint's connected-run reads are bounded, and the dedicated-schema relation hydrators fetch only the requested columns instead of whole rows. Retrieve also falls back to the other database when a routed read misses, so a run whose physical residency diverges from its id shape is still found rather than returning a spurious not-found. Fewer and lighter queries on the run read path, with no change to results. Mint-kind flip safety: flipping which database new runs mint to is now a deterministic wall-clock cutover, for both per-org and global flips. For a grace window every process resolves the same database, so a flip cannot route two concurrent triggers that share an idempotency key to different databases (which would bypass the per-database unique constraint and create a duplicate run). Supersedes the earlier #4205 and #4208. Draft: validation in progress. |
||
|
Eric Allam
|
45527e317a |
feat(webapp): opt-in worker pool for OTLP ingest transform (#4232)
## Summary Under high OTLP ingest volume, the whole decode, transform, and enrich pipeline runs on the request event loop, so a single CPU core becomes the ceiling while the rest sit idle. This adds an opt-in worker pool that moves decode, transform, and LLM-cost enrichment onto worker threads, keeping the main thread free for I/O. It is off by default (`OTEL_TRANSFORM_WORKER_POOL_ENABLED`), so behavior is unchanged unless enabled. ## Design Workers do decode, filter, convert, and enrich (including LLM pricing match). The main thread stays the single database reader: it loads the pricing registry and broadcasts the compiled model rows to the workers (re-broadcasting on every reload), so workers never touch the database. The pure transform is extracted into a dependency-light module (no Prisma/Redis/ClickHouse imports) so it can run inside a worker. Importantly, the main thread keeps the existing single consolidated insert path, so ClickHouse insert batching and part count are unchanged. The parallelism buys CPU headroom, not more insert streams (which would add merge pressure). The worker is bundled as a standalone file at build time and ships in the existing image with no Dockerfile change. In local load testing the pool sustained roughly 2.6x the throughput of the single-thread path and kept the main thread responsive under load. |
||
|
Matt Aitken
|
2cac63f13a |
fix: improve error labelling, grouping, and stack traces in the Errors feature (#4225)
## Problem Several display/grouping issues in the **Errors** feature, all rooted in how the ClickHouse error materialized views (`errors_mv_v1`, `error_occurrences_mv_v1`) read the stored error JSON produced by `parseError`: 1. **Messageless errors show "Unknown error".** An empty message falls straight through `coalesce(nullIf(message,''), 'Unknown error')` to the literal, even though the error's class `name` is available (e.g. an Effect tagged error `ListMessagesError` with no message). 2. **Unrelated errors collapse into one group.** `calculateErrorFingerprint` keys on `type : message : stack`, where `type` is always the union tag (`BUILT_IN_ERROR`, …), `message` is empty, and the stack isn't read — so every messageless built-in error (and every string/custom error) hashes to the same constant input → one fingerprint. 3. **error_type shows the internal tag.** `coalesce(type, name, …)` always resolves to `type` (always present), so the column shows `BUILT_IN_ERROR` instead of the real class name. 4. **Stack traces never populate.** The MVs read `error.data.stack`, but the serializer stores the trace under `stackTrace` — so the column is always empty. ## Fix All display changes are `ALTER TABLE … MODIFY QUERY` on the two views (migration `035`); the fingerprint change is in the webapp. - **Fingerprint** (`errorFingerprinting.ts`): fall back **message → name → raw**. Messageless errors now group by class name (or raw value for non-Error throws); message-bearing errors are **unchanged** (short-circuits at `message`), so existing groups don't split — only currently-messageless errors get their own group going forward. - **error_message**: same `message → name → raw` fallback before `'Unknown error'`. - **error_type**: coalesce `name → code → 'Error'` (drops the reliance on the union tag). Built-in → class name, internal → `code`, string/custom → `Error`. - **stack trace**: read `error.data.stackTrace`. Bounded as before (serializer caps 50 frames / 1024 chars per line; MV clips to 2000 chars). ## Migration notes - `MODIFY QUERY` swaps the view query in place (no drop/recreate gap); Down restores the previous query. - **Existing rows are left unchanged** — changes apply only to rows inserted after the migration. No backfill. ## Tests `errorFingerprinting.test.ts` — 57 pass, incl. new cases for messageless class names, string/custom raw values, and stability of message-bearing fingerprints. Fixes the display-derivation half of TRI-11938 (error_type + stack trace); relates to TRI-9254 and TRI-9250. |
||
|
Matt Aitken
|
b64b54c74e |
feat(webapp): pass database writer and reader config to auth plugins (#4229)
## Summary The RBAC and SSO auth plugins can own their own database client, but they could only read `DATABASE_URL`, so every connection they opened landed on the primary. The host webapp now resolves writer and read-replica URLs from its env (the same fallback chain its own Prisma clients use: control-plane URL first, then the default) and passes them to the plugins at create time via a shared `PluginDatabaseConfig`, along with separate connection limits for writes (default 2) and reads (default 5, tunable via `RBAC_DATABASE_*_CONNECTION_LIMIT` and `SSO_DATABASE_*_CONNECTION_LIMIT`). A plugin can then route hot-path reads (per-request auth checks, login routing) to the read replica and keep only rare mutations on the primary. With no replica configured, or no plugin installed, nothing changes: the OSS fallback ignores the new option and keeps reading through the Prisma clients it is already given. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Chris Arderne
|
7faa52597d |
chore: format prisma schemas (#4224)
Creating a Prisma migration now formats its schema first, keeping migration-related schema edits consistently formatted without adding work to the repository-wide format command. Run `pnpm run format:prisma` to format either schema on demand. |
||
|
Eric Allam
|
02cf9c81ad |
fix(tsql): make JSON functions work on the output and error columns (#4221)
## Summary A Query page (TRQL) query that pulls fields out of a run's `output` with JSON functions (`JSONExtractString`, `JSONExtractInt`, `JSONHas`, and the rest of the family) failed with "The first argument of function ... should be a string containing JSON, illegal type: JSON". Those queries now work. ## Root cause and fix `output` is a native ClickHouse `JSON` column, but `JSONExtract*`, `JSONHas`, `JSONLength`, and `JSONType` all expect a String containing JSON text. The compiler already swaps in the column's String companion (`output_text`) when a JSON column is selected or compared, but not inside function-call arguments, so it emitted `JSONExtractInt(output, 'x')` against the native column. The fix prints the companion column for the first argument of these functions when it resolves to a bare JSON field, keeping the table alias when qualified (so it works in JOINs): JSONExtractInt(output, 'x') -> JSONExtractInt(output_text, 'x') JSONExtractArrayRaw(assumeNotNull(output), 'y') -> JSONExtractArrayRaw(assumeNotNull(output_text), 'y') It also reaches through value-preserving passthrough wrappers like `assumeNotNull(...)`, while leaving value-changing wrappers like `toJSONString(output)` on the native column (that argument is already a String). The swap is also semantically correct, not just a type fix: `output_text` is the unwrapped data JSON that the TRQL `output` model already represents, so field paths line up. Covered by printer unit tests and a ClickHouse integration test that runs the whole family (plus the wrapped and `toJSONString` cases) against a real native-JSON column. Both new cases fail with the exact "illegal type: JSON" error without the fix. |
||
|
Eric Allam
|
1a0198cc5e |
perf(webapp,clickhouse): move runs empty-state check to ClickHouse (#4202)
## Summary
The runs page's empty-state check (whether an environment has ever had a
run, which decides between the "getting started" and "no runs match your
filters" states) ran a `findFirst` against the Postgres `TaskRun` table.
This moves it to ClickHouse, the same store the runs list itself reads
from, so the check no longer queries `TaskRun`.
## Design
Only the runs list triggers the check now (via an `includeHasAnyRuns`
flag); the other presenters that reuse `NextRunListPresenter` (API,
schedule detail, waitpoint detail, error group) no longer issue it. When
the list is empty it runs `SELECT 1 FROM task_runs_v2 ... LIMIT 1`
filtered on the full `(organization_id, project_id, environment_id)`
sort-key prefix with a configurable `created_at` lower bound
(`RUN_LIST_HAS_RUNS_LOOKBACK_DAYS`, default 30), so it hits the primary
index and reads minimal granules.
Results are cached in a tiered memory + Redis SWR cache. Only positive
("has runs") results are cached, so an environment with no runs is
always re-checked and its first run shows up immediately.
|
||
|
Chris Arderne
|
34b1a181c2 | fix: security release 2026-07-06 (#4199) | ||
|
Chris Arderne
|
aa74e68c71 |
feat(sdk): add bulk replay to api and sdk (#4105)
## Summary
Adds SDK and API support for run bulk actions. You can now create bulk
cancel or replay actions from `@trigger.dev/sdk` using run IDs or the
same filters as `runs.list()`, then retrieve, list, poll, or abort the
action by its `bulk_` handle.
Tests, docs, changesets added.
## Design
The dashboard bulk action service now accepts structured filters instead
of reading directly from a dashboard request, so the dashboard and API
share the same creation path. Replay actions created through the API are
attributed with the existing `api` trigger source, while
dashboard-created actions keep `dashboard`.
The SDK exposes the new surface under `runs.bulk.*`, including
`targetRegion` for replay region overrides and cursor pagination for
listing bulk actions.
## Filters and runIds
Nuance on filters. If `filter` is provided, it MUST have at least one
key. This is to remove the footgun of passing no filter and selecting
all runs.
```typescript
{ action: "cancel", runIds: ["run_1"] } // valid
{ action: "cancel", runIds: [] } // invalid, min(1)
{ action: "cancel", filter: { status: "FAILED" } } // valid
{ action: "cancel", filter: {} } // invalid
{ action: "cancel", filter: {}, runIds: ["run_1"] } // invalid
```
|
||
|
Daniel Sutton
|
d59743bd35 |
fix(webapp,run-ops-database): keep run-ops batch items co-resident with their batch (#4178)
## Summary Three fixes to the run-ops database split (the Cloud-only mode where run-lifecycle rows live on a dedicated Postgres). All are inert in the default single-database deployment. The main fix: on the batch trigger paths, a parentless batch's item runs chose their physical store from a fresh per-org mint-flag read at processing time, so flipping an org's flag mid-batch could land an item in a different store than its batch, breaking the `TaskRun.batchId` foreign key (or silently orphaning the item). The other two harden the split's safety nets: the schema-parity test now actually compares columns, and the read fan-out gate now signals when it has been silently disabled. ## Batch item residency `RunEngineBatchTriggerService` (api.v2) and the BatchQueue item callback (api.v3) now anchor each item's id mint on the batch's own friendlyId, mirroring the already-safe `BatchTriggerV3Service`. Residency is a pure id-shape check, so an item can no longer diverge from its batch across a mid-batch flag flip. The pre-failed-run fallback is anchored the same way (it also sets `batchId`), and the shared mint branch is consolidated into one helper so every mint path stays in lockstep. No new database queries; single-database mode is unchanged (a cuid-shaped batch friendlyId yields a cuid item). ## Schema parity test The parity test previously read only the dedicated schema and matched model headers with regexes, so it never compared columns and could not catch a run-subgraph column that diverged between the two physical schemas. It now parses both schemas and asserts bidirectional scalar-column parity (type, nullability, array-ness, default) across the run-subgraph models, and fails on any field line it can't parse. Scoped to the run-subgraph models so unrelated control-plane edits don't break it. ## Read fan-out signal The split read fan-out gate is decided by the object identity of the NEW vs control-plane clients. It now warns when both run-ops URLs are set but the NEW client isn't a distinct instance (fan-out silently off), and a new test exercises the real topology-into-gate wiring so a future refactor that aliases the clients can't disable fan-out unnoticed. ## Verification New unit and glue tests cover all three changes; the DB-backed residency, store-routing, and topology suites pass against real Postgres; `typecheck` is clean for both packages. |
||
|
Wes Mason
|
ce123c13e1 |
fix(llm-model-catalog): stop generated files showing as modified (#4170)
📚 Publish docs / publish (push) Has been cancelled
## Summary Running `pnpm run db:migrate` locally left `defaultPrices.ts` and `modelCatalog.ts` in `llm-model-catalog` showing as modified every time, a ~10k-line diff that only ever changed formatting. This stops the churn. ## Root cause The root `db:migrate` script ends in `&& turbo run generate`, which runs the `generate` script in every package that has one, including this one. The generator writes its output with `JSON.stringify` (quoted keys, no trailing commas), but the checked-in copies had been reformatted by oxfmt (unquoted keys, trailing commas). So the generator output never matched what was committed, even though the parsed data was identical. ## Fix Add the two generated files to `.oxfmtrc.json`'s ignore list and commit the raw generator output, matching how other codegen files in the repo are already handled (e.g. the tsql grammar). Generation is deterministic, so `generate` and `format` are both no-ops on a clean tree now. No changeset: internal package, dev tooling only, no runtime or public API change. |
||
|
Daniel Sutton
|
e4ae8cbcd4 |
fix(run-engine,run-store,webapp): stop split-mode waits hanging on resume (#4164)
## Summary On the run-ops database split, a run that waits (`triggerAndWait`, `batchTriggerAndWait`, `wait.forToken`) could hang forever after its wait had already completed. The runner reads a resume from `/snapshots/since` exactly once: if that read returned the resume snapshot without its completed-waitpoints, the runner logged "executing without completed waitpoints", advanced its cursor, and never re-read it, so the awaiting run never continued. ## Root cause The resume snapshot and its completed-waitpoint rows were written as two separate commits. This regressed when the split replaced Prisma's atomic nested `connect` with an FK-free insert (in [#4163](https://github.com/triggerdotdev/trigger.dev/pull/4163)), and `/snapshots/since` is served from a read replica. A fetch landing in the sub-millisecond gap between the two commits, or a multi-reader replica serving the snapshot from a different point in time than its join rows, delivered an empty resume. Because the runner consumes each snapshot once and treats an empty resume as terminal, a single stale read was fatal and produced a permanent, nondeterministic hang. ## Fixes - Commit a snapshot and its completed-waitpoint links in one transaction, restoring the atomicity the split removed. - Repair the completed-waitpoints from the owning primary when a multi-reader replica serves the snapshot without its join rows. This covers single-waitpoint resumes, which carry no `completedWaitpointOrder` and so were missed by the count-based repair. - Read the primary in the checkpoint `WAIT_FOR_BATCH` pre-check, so a batch that already resumed is not re-suspended into a stall. - Fall back to the primary when a waitpoint token misses both read replicas, so a token completed immediately after it was minted no longer returns a spurious 404. - Route batch-item creation by `batchTaskRunId`, consistent with the batch-completion count and the row's foreign key. - Reject control-plane-only relation selects on the dedicated schema with a clear error instead of an opaque Prisma failure, and stop `createDateTimeWaitpoint` bypassing residency routing through a caller transaction. Verified against the deployed split topology: a resume snapshot and its completed-waitpoints are now always delivered together, so the runner can no longer drop a resume. |
||
|
Oskar Otwinowski
|
de65370fb9 |
feat(webapp): Directory Sync (SCIM) for Identity & Access (#4148)
Extend the SSO plugin contract for directory sync and apply membership effects from the accounts webhook worker: provision users in mapped groups (role from group mapping, else the org default role), deprovision on removal, and keep a sticky-removal tombstone so JIT never silently re-adds a removed user. JIT and Directory Sync coexist; roles default to Developer (the JIT default-role picker has no 'None'). Changing a group's role in the dashboard re-applies it to that group's current members immediately. The Directory Sync settings section (group→role mapping, external-domain + manual-membership policy, deferred Save) appears once a domain is verified — independent of SSO — gated by the hasSso flag. The settings page polls the whole page while entitled with override-aware drafts so in-progress edits are never clobbered. |
||
|
Daniel Sutton
|
f101983a70 |
fix(run-store,run-engine): fix run-ops split hangs from wrong-store reads on the resume path (#4163)
## Summary On the run-ops split, NEW-residency runs could hang. Time-based waits (`wait.for`, `wait.until`, `delay`, waitpoint tokens), `batchTriggerAndWait`, and attempt starts stalled and never resumed. Each was a run-ops read or update that hit the wrong database: either the owning store's read replica when it needed read-your-writes, or the wrong store entirely because it routed by an id that does not encode residency. ## Fixes **Waitpoint resume (the main hang).** The managed resume path reads a run's completed waitpoints by snapshot id (`findSnapshotCompletedWaitpointIds`). Snapshot ids are cuids, which always classify to the legacy store, so a NEW run's join rows (which live on the new store) were never found. The resumed run saw zero completed waitpoints and hung. It now fans out across both stores and merges, like its sibling readers. **Batch completion.** Batch item completion (`updateManyBatchTaskRunItems`) routed by the item id, which is also a cuid, so a NEW batch's items were updated on the wrong store, matched zero rows, and the batch was treated as already complete (its parent's `batchTriggerAndWait` then hung). It now routes by the batch id, which does encode residency, matching the sibling `countBatchTaskRunItems`. **Read-your-writes on the resume path.** The block-time pending-waitpoint check (`countPendingWaitpoints`) and the attempt-start lock check (`findRun` in `startRunAttempt`) both read the owning store's replica with no read-your-writes guarantee, so a just-committed waitpoint completion or dequeue lock could be missed under replica lag and strand the run. Both now read the owning primary. Each fix ships with a two-database store or engine test that reproduces the hang and passes with the fix. |
||
|
Daniel Sutton
|
092b9ef07a |
fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154)
## Problem
The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.
## Fix: change the encoding, not the structure
Mint a **26-char lowercase base32hex** run id:
```
run_<24-char base32hex core><region char><version char>
[ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```
- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.
DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.
## Residency discriminator: length → version char
`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.
## Scope / verification
- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.
## Open decisions (flagged, not silently chosen)
1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.
---
## ⚠️ Required migration — deploy in lockstep
This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):
1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
- key `runOpsMintKsuid` → `runOpsMintKind`
- value `"ksuid"` → `"runOpsId"`
Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
d977691219 | fix(run-store): route caller-passed read clients to the owning store's primary (#4153) | ||
|
Daniel Sutton
|
962bc48738 |
feat(run-ops): automatically migrate the dedicated run-ops database (#4150)
## What
Adds the ability to **automatically migrate the dedicated run-ops
database** (the NEW DB in the run-ops split), matching how every other
database in the system is migrated. Follow-up to the run-ops split
activation.
## Changes
- **Migrate runner** — new
`internal-packages/run-ops-database/scripts/migrate.mjs`, exposed as
`db:migrate:deploy` / `db:migrate:status`. Connects via
`RUN_OPS_DATABASE_URL` (the same var the app uses) and expands `${VAR}`
refs like Prisma's dotenv.
- **Self-host** — `docker/scripts/entrypoint.sh` runs the run-ops
migration on boot when the DB is configured, gated by
`SKIP_RUN_OPS_MIGRATIONS`. Single-DB installs never set the URL, so it's
a clean no-op.
- **Single env-var family** — the run-ops DB is now addressed by one
canonical `RUN_OPS_*` family, connect path and migrations resolving the
identical URL:
- `RUN_OPS_DATABASE_URL` (writer) — replaces `TASK_RUN_DATABASE_URL`
- `RUN_OPS_LEGACY_DATABASE_URL` — replaces
`TASK_RUN_LEGACY_DATABASE_URL`
- `RUN_OPS_DATABASE_READ_REPLICA_URL` — replaces
`TASK_RUN_DATABASE_READ_REPLICA_URL`
- the old `TASK_RUN_*` aliases, the `??` coalesce, the
`runOpsNewDatabaseUrl` indirection, and the migrate-only `directUrl` are
all removed (consumers read `env.RUN_OPS_DATABASE_URL` directly).
`directUrl` was dropped because it was only ever used by `prisma
migrate` (never the app runtime) to bypass a pooler for advisory locks —
premature here since the run-ops connection isn't wired to the app yet.
If a pooler is later introduced for the app, a direct URL can be
reintroduced then.
## Safety
- **Pure rename** — nothing deployed sets any `TASK_RUN_*` var (the
split isn't activated anywhere yet; `.env.example`, docker-compose, and
cloud already use `RUN_OPS_*`), so there is no config migration.
- **Single-DB / self-host** — no new required env var; entrypoint and
migrate are no-ops when `RUN_OPS_DATABASE_URL` is unset.
- **Cloud** — runs migrations as pre-deploy ECS tasks (companion cloud
PR), calling these same `db:migrate:deploy` / `db:migrate:status`
commands.
## Verification
- Live migration against a fresh scratch DB with only
`RUN_OPS_DATABASE_URL` set: both migrations applied, no `P1012`/`P1013`;
`${VAR}` expansion, idempotent re-run, `status`, and no-op skip all
pass.
- Schema parity 4/4; `typecheck --filter webapp` 18/18; affected
split/replication tests 34/34.
## Scope
This delivers automatic migrations only. Enabling the app to *use* the
new DB (setting `RUN_OPS_DATABASE_URL` + `RUN_OPS_SPLIT_ENABLED` on the
service) is a separate activation step.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
c9f427e21f |
fix(replication): key logical replication leader lock on slot name (#4151)
## Problem
`LogicalReplicationClient` uses a Redlock leader lock to guarantee a
single active consumer per Postgres logical replication slot. The lock
resource was keyed on the client `name`:
```
logical-replication-client:${this.options.name}
```
A slot permits exactly one consumer, so the lock's job is to serialize
consumers **of a given slot**. Keying it on `name` breaks that whenever
two clients target the same slot with different names — most notably
across a rolling deploy where the client `name` changes but `slotName`
does not. Both acquire *distinct* locks, both consider themselves
leader, and the second to reach `START_REPLICATION` hits `replication
slot "<slot>" is active for PID <n>`. Because that query was
fire-and-forget and its failure was only logged (no retry), the consumer
stopped and replication stalled until the process was restarted.
## Fix
**1. Key the leader lock on `slotName`** — the actual single-consumer
resource:
```
logical-replication-client:${this.options.slotName}
```
Consumers of the same slot now contend on the same lock and hand off
cleanly across restarts/deploys; different slots stay independent.
`name` is kept for logging and the pg `application_name`.
**2. Self-healing resubscribe** (`resubscribeOnFailure`, opt-in) —
instead of logging-and-dying, a client re-subscribes with exponential
backoff after a lost election or a failed `START_REPLICATION`, so a
rolling deploy self-heals: the incoming pod retries until the draining
pod releases the slot, then takes over. Safety:
- `#cleanupAttempt()` unconditionally ends the pg client (freeing the
walsender) and releases the leader lock before rescheduling — retries
never leak connections/locks.
- `shutdown()` sets an intentional-stop latch re-checked after every
`await` in `subscribe()` (and aborts the lock-acquire spin), so a
resubscribe can never race or outlive an intentional shutdown.
- Backoff resets only on genuine stream start, so a permanently stuck
slot backs off to the ceiling and logs loudly rather than tight-looping;
an epoch guard neutralises stale `START_REPLICATION` catches.
Runs- and sessions-replication opt in and use `shutdown()` for all
intentional stops.
**3. Observability** — the admin runs-replication status route probed
the old name-keyed Redis key (would report `leader:false` for every
source after fix #1); now probes the slot-keyed key.
## Tests
`internal-packages/replication/src/client.test.ts` (real Postgres +
Redis containers):
- same-slot/different-name → second client must not double-lead or race
into "slot is active" (the regression)
- a failing `START_REPLICATION` retry loop must not leak connections or
locks
- `shutdown()` during an in-flight `subscribe()` must not leave a zombie
leader
- `subscribe()` after `shutdown()` re-arms `resubscribeOnFailure`
- self-heals once the leader releases the slot
Plus the multi-source wiring test updated to the slot-keyed lock keys.
## Rollout
With the self-healing resubscribe, this ships as a **plain rolling
deploy** — the incoming pods retry across the one-time lock-key
transition and take over once the old pods drain (a brief replication
stall that the durable slot replays on reconnect — no data loss). No
stop-before-start required.
|
||
|
Daniel Sutton
|
70bca82d84 | feat(run-ops): activation — drop cross-DB FKs, provision run-ops DB, enable split (#4124) | ||
|
Daniel Sutton
|
c266e96c87 |
feat(run-ops): run-store routing seam + run-engine read seams (#4116)
## What Introduces the run-store routing seam and the run-engine read seams that let run lifecycle operations be dispatched to either the control-plane database or a separately-generated run-ops database, depending on where a run/batch resides. - **run-store** (`internal-packages/run-store`): adds `runOpsStore.ts` and substantially expands `PostgresRunStore.ts` so the store can resolve residency and route reads/writes to the correct backing client. `types.ts` grows the routing/residency types; `NoopRunStore.ts` is removed. - **run-engine** (`internal-packages/run-engine`): adds `engine/controlPlaneResolver.ts` and routes the per-system read paths (dequeue, enqueue, waitpoint, checkpoint, run-attempt, ttl, delayed-run, execution-snapshot, pending-version, debounce, batch) through the resolver/store instead of talking to a single Prisma client directly. `engine/errors.ts`, `engine/types.ts`, and `engine/index.ts` are extended to support injecting the store/resolver. Three fixes are included on top of the seam work: - `c6cadd85f` — routes read-your-writes to the owning store's **writer**, not its lagging replica, so an operation immediately reading back what it just wrote sees a consistent result. - `05c912e05` — normalizes run-ops-generation Prisma errors to the control-plane error class at the store **write boundary**, so `instanceof` checks and the `P2002` → 422 handling continue to work across the separately-generated run-ops Prisma client. - `88d12907f` — resolves NEW-resident batches in `ApiBatchResultsPresenter` by routing the batch read through the store, so a dedicated-DB batch resolves instead of returning 404. The change is heavily test-first: the bulk of the diff is new unit/integration coverage for the store routing, residency, and each run-engine system's control-plane resolver path. ## Why PR4 of the run-ops split stack (PR1–PR3 land the ClickHouse test-container and earlier plumbing). This PR is the read-path foundation: it adds the seam and read-routing but leaves the write path to route through the same seam in a later PR. Behavior-changing where the three fixes above touch existing read-your-writes / error-normalization / batch-resolution paths; otherwise additive (new store module, new resolver, injectable dependencies with existing single-client behavior preserved when no dedicated store is configured). ## Tests Extensive new vitest coverage under `run-store/src/*.test.ts` (routing, residency, dual-schema select, cross-generation error normalization, read-after-write, idempotency dedup, mixed residency, waitpoint co-location) and `run-engine/src/engine/**/*.test.ts` (per-system `controlPlaneResolver` tests, injectability, block-edge residency, waitpoint read residency, trigger-create routing, lifecycle router). Testcontainers-backed; no mocks. ## Notes Draft, **stacked on #4114** (`runops/pr03-clickhouse-tc`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
a93b101d44 |
feat(run-ops): cross-producer ClickHouse version helper + cross-Postgres-version compat tests (#4114)
## What - Adds `composeTaskRunVersion` to `@internal/clickhouse` (exported from the package index). It packs a small `originGeneration` epoch into the top 8 bits of the ReplacingMergeTree version and keeps the producer's own LSN in the low 56 bits, so `task_runs_v2` rows replicated from more than one Postgres producer become globally comparable while preserving in-producer ordering. Single-producer setups never call it and keep using the raw LSN version. - Adds unit coverage for the helper in `taskRuns.test.ts` (bit layout, ordering, epoch precedence, range validation). - Adds two run-engine tests that exercise the cross-Postgres-version testcontainer fixture: - `heteroPostgresFixture.test.ts` — a smoke test asserting byte-identity and identical `ORDER BY` (under a pinned ICU collation) across two different Postgres major versions. - `crossVersionCompat.test.ts` — mirrors the run-engine's real raw-SQL surfaces and asserts byte-identical, ordering-identical results across the two versions. Ships with an env-gated block (skipped by default) that can be pointed at a real dedicated database in CI. ## Why Third PR in the run-ops split stack. It is purely additive: it introduces one new exported helper plus tests and changes no runtime call sites, so on its own it has no runtime behavior change. It lays down the version-composition primitive and the cross-version compatibility proof that later PRs in the stack rely on. ## Tests - New unit tests for `composeTaskRunVersion` in `internal-packages/clickhouse/src/taskRuns.test.ts` (run against a real ClickHouse testcontainer). - New cross-version tests in `internal-packages/run-engine/src/engine/tests/` running against real Postgres containers of two different major versions (no mocks). The env-gated dedicated-database block is skipped unless its URL is set. ## Notes Draft, **stacked on #4113** (`runops/pr02-db-foundation`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
84588bd236 |
feat(run-ops): dedicated run-ops database package + docker service + migration runner (#4113)
## What The **dedicated run-ops database** foundation for the split: a standalone Prisma package plus the infra to run and migrate it. - **`internal-packages/run-ops-database`** — a new Prisma package (`@internal/run-ops-database`) whose schema mirrors the run-execution tables that will live on the dedicated DB, with its own generated client, migrations, and migration runner. - **`prisma/schema.parity.test.ts`** — a parity test that guards the run-ops schema against drift from the control-plane schema for the mirrored tables. - **Docker** — a Postgres 17 service (`docker/Dockerfile.postgres17`, `docker/docker-compose.yml`) so the dedicated DB is available locally under the run-ops compose profile. - **Testcontainers** — hetero fixtures (PG14 legacy + PG17 dedicated) so later PRs can exercise cross-database behaviour with real containers rather than mocks. ## Why This is the **second PR in the run-ops split stack**, stacked on the core primitives. It stands up the dedicated database and its tooling. There is **no runtime wiring** into the webapp here — the app does not read or write this DB yet; that arrives in later PRs. On its own this PR only adds a package, a docker service, and test fixtures. ## Tests Schema-parity test for the run-ops schema; hetero testcontainer fixture smoke test. ## Notes - Draft, **stacked on #4112** (`runops/pr01-core-residency`). Review that one first; this diff is against it. - Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris Arderne
|
448443a024 |
chore: bump internal node to 22 and standardise (#4084)
Bumps the internal/toolchain Node version to the latest 22.x LTS (`22.23.1`) and standardises it across the repo. Scope is the **platform toolchain + the repo's own runtime images** (all `20 → 22` *upgrades*, off the now-EOL node 20). ### Main changes - Node `20.20.2 → 22.23.1` across all CI workflows, `.nvmrc`, `CONTRIBUTING.md`, and the OSS `docker/Dockerfile` (digest-pinned). - `@types/node → 22.20.0` (root dep + pnpm `overrides`, so the whole workspace resolves to it); lockfile regenerated. - `sdk-compat` matrix: adds Node 24 + 26 (keeps 20, still in `engines`). - **App runtime images → node 22** (were on EOL node 20): `apps/coordinator` → `node:22.23.1-bookworm-slim`; `apps/docker-provider` + `apps/kubernetes-provider` → `node:22-alpine` (reusing the exact digest `apps/supervisor` already runs, so all four worker images are now identical). Stage aliases renamed off `node-20`. ### Possible issues / test notes - `@types/node` 22.x can surface new TS errors — typecheck (now on 22) is the gate. - **Smoke-test the v3 worker path** — `coordinator` (`crictl`/CRI calls) and the docker/kubernetes providers (talking to their daemons) now run on node 22 (alpine/musl for the providers). Upgrade off EOL so low-risk, but it's deployed runtime code with its own `publish-worker.yml` pipeline. |
||
|
Matt Aitken
|
fd4f02b2f8 |
fix(webapp): onboard new cloud orgs via plan selection; allow Free plan without GitHub verification (#4109)
## What & why Two related fixes to how new cloud organizations get onboarded onto the Free plan. ### 1. Route new cloud orgs through plan selection New cloud organizations were created already activated, so they skipped the plan-selection step and went straight to creating projects — which meant their plan and usage limits were never set up. They're now created deactivated and routed through plan selection, which activates them once a plan is chosen. Self-hosted installs have no plan-selection step, so they're activated immediately on creation and are unaffected. The `Organization.v3Enabled` field is renamed to `isActivated` to better describe what it now gates. It's mapped to the existing `v3Enabled` column, so there's no data migration — only a schema/code rename. ### 2. Allow selecting the Free plan without GitHub verification Choosing the Free plan no longer requires connecting and verifying a GitHub account. The plan is applied immediately when selected. This removes: - the "Connect to GitHub" dialog and the GitHub-verified badge from the plan picker - the account-rejected state - the now-unreachable GitHub-connect return routes ## Notes - These changes pair with the corresponding change in the billing service that applies the Free plan directly; they should be released together. ## Testing Verified locally end to end: a new cloud org is routed to plan selection, the Free plan applies in one click with no GitHub step, the org is activated, its usage allowance is provisioned, and it lands on the new-project page. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Chris Arderne
|
c7861be520 |
chore: activate no-unused-vars and import linters (#4096)
Once this is merged, oxlint is at a pretty sensible baseline. **Enable `no-unused-vars`, `typescript/consistent-type-imports`, and `import/no-duplicates` lint rules** Turns on three previously-disabled oxlint rules across the monorepo and fixes all violations: - **`no-unused-vars`** – enabled as an error with standard ignore patterns: unused function arguments are ignored by default (`args: "none"`), variables/caught errors/destructured array elements prefixed with `_` are allowed, and rest siblings are permitted. - **`typescript/consistent-type-imports`** – enforced as an error; all type-only imports now use the `import type` syntax. - **`import/no-duplicates`** – enforced as an error; duplicate import statements from the same module have been merged. The remaining commits clean up the violations found across the codebase: removing unused variables/imports/type aliases, adding `_` prefixes to intentionally unused bindings, fixing duplicate imports, and converting value imports to `import type` where appropriate. |
||
|
Matt Aitken
|
536731a5d6 |
feat(clickhouse): infer mixed-type JSON arrays as Array(Dynamic) on insert (#4095)
## ✅ Checklist - [ ] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [ ] I ran and tested the code works --- ## Testing `pnpm run typecheck --filter @internal/clickhouse` passes. This only adds a ClickHouse input-format setting to existing insert calls; the setting affects type inference for newly-inserted/merged data and is non-destructive to existing rows. --- ## Changelog Sets `input_format_json_infer_array_of_dynamic_from_array_of_different_types = 1` on every native-JSON insert path: - `task_runs_v2` (`output`, `error`) — `insertTaskRuns`, `insertTaskRunsCompactArrays`, and the async-insert variants - `task_events_v1` / `task_events_v2` (`attributes`) - `metrics_v1` - `sessions_v1` ### Why Our JSON columns contain arrays with mixed element types (e.g. `[{"key":"value"}, "string", "string"]`). With this setting off — which is the effective default under `24.12` compatibility — ClickHouse infers those as deeply nested unnamed `Tuple(JSON, Nullable(String), …)` types. ClickHouse 26.2 introduced `input_format_binary_max_type_complexity` (default 1000), and those tuple type trees exceeded the limit, causing background merges to fail with **Code 117**. With the setting on (the default since 25.8), mixed-type arrays are inferred as a single `Array(Dynamic)` — a simpler, flatter type representation that never approaches the complexity limit, even once the upstream default limit is restored. Setting this explicitly at insert time keeps behavior deterministic and version-controlled, so it does not depend on the server profile or a future compatibility bump. This is a forward-only change: it only affects newly inserted/merged data and does not rewrite existing parts. Our read path re-serializes these columns to strings (`toJSONString` via the materialized `*_text` columns), so the internal Tuple → Array(Dynamic) representation change is transparent to the application. ### Companion server-side setting To also apply this on the ClickHouse side (covers merges and any writes not going through these code paths), set it on the default user: ```sql ALTER USER default SETTINGS input_format_json_infer_array_of_dynamic_from_array_of_different_types = 1; ``` --- ## Screenshots _N/A_ 💯 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01AaChyhestFMBYBWh6bgcCF --- _Generated by [Claude Code](https://claude.ai/code/session_01AaChyhestFMBYBWh6bgcCF)_ --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Chris Arderne
|
bfa902bd18 |
chore: enable more linters (#4080)
Re-enables ~15 oxlint rules that were blanket-disabled before. |
||
|
Eric Allam
|
aae1c6a512 |
fix(schedule-engine,webapp): log out-of-entitlements scheduled triggers as warnings (#4067)
## Summary When a scheduled task fires for an organization that is out of entitlements, the trigger can't proceed. That's an expected outcome, but it was being logged at error level and surfaced as a failure. ## Fix The trigger callback now classifies an out-of-entitlements result as its own error type (`OUT_OF_ENTITLEMENTS`), and the schedule engine logs both that and the existing queue-limit result as warnings rather than errors. The run still doesn't fire and the `schedule_execution_failure` metric still records the outcome (now tagged `out_of_entitlements`), so nothing about observability or behavior changes beyond the log level. |
||
|
Chris Arderne
|
0119cf8f9f | fix(dashboard-agent-db): load .env for local migrations (#4069) |