fix/changeset-release-server-changes
185 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Eric Allam
|
1a0198cc5e |
perf(webapp,clickhouse): move runs empty-state check to ClickHouse (#4202)
## Summary
The runs page's empty-state check (whether an environment has ever had a
run, which decides between the "getting started" and "no runs match your
filters" states) ran a `findFirst` against the Postgres `TaskRun` table.
This moves it to ClickHouse, the same store the runs list itself reads
from, so the check no longer queries `TaskRun`.
## Design
Only the runs list triggers the check now (via an `includeHasAnyRuns`
flag); the other presenters that reuse `NextRunListPresenter` (API,
schedule detail, waitpoint detail, error group) no longer issue it. When
the list is empty it runs `SELECT 1 FROM task_runs_v2 ... LIMIT 1`
filtered on the full `(organization_id, project_id, environment_id)`
sort-key prefix with a configurable `created_at` lower bound
(`RUN_LIST_HAS_RUNS_LOOKBACK_DAYS`, default 30), so it hits the primary
index and reads minimal granules.
Results are cached in a tiered memory + Redis SWR cache. Only positive
("has runs") results are cached, so an environment with no runs is
always re-checked and its first run shows up immediately.
|
||
|
Chris Arderne
|
34b1a181c2 | fix: security release 2026-07-06 (#4199) | ||
|
Daniel Sutton
|
80d4819a03 |
fix(webapp): stop slow database cleanup on project deletion (#4191)
## Summary Deleting a project triggered an unbounded database cleanup that scanned the project's entire run history, so deleting a project with many runs could be very slow. Project deletion is a soft delete again: run data is retained and the deletion completes quickly. ## Fix Project deletion ran a cascade hard-delete whose `BulkActionItem` step filtered through a relation to `TaskRun` scoped by `projectId`. Prisma compiles that to an `EXISTS`-join over the project's entire `TaskRun` set (a large, hot table with no `projectId` index), and it ran on every project deletion unconditionally. Removing the cascade-cleanup call restores the prior soft-delete behaviour: queues are removed, the project is marked deleted, and run data is retained. The cascade-cleanup service (added in [#4117](https://github.com/triggerdotdev/trigger.dev/pull/4117)) had no other callers, so it and its test are deleted. |
||
|
nicktrn
|
94b30fc1a6 |
fix(webapp): reject deploy images with runtime-incompatible zstd layers (#4184)
Container runtimes (cri-o / containerd / podman) can't pull zstd-compressed layers carried in a Docker v2s2 manifest (`application/vnd.docker.image.rootfs.diff.tar.zstd`). A deploy built with an outdated CLI can produce exactly that combination - and today it's promoted to current and then fails every run at image-pull time. This extends the pre-promotion image check (#4049) to also inspect the manifest's layer media types. If any layer uses the unpullable zstd/v2s2 media type, the deploy is rejected at finalize with a clear message to upgrade the CLI and re-deploy, instead of silently shipping a version that can't start. The manifest is already returned by the existing ECR `BatchGetImage` call, so there's no extra registry request for single-arch images. Parsing is a lenient Zod schema and **fails open** - a manifest we can't read never blocks a deploy. Manifest lists / OCI indexes (no top-level `layers[]`) and OCI zstd (`...tar+zstd`, which runtimes support) pass unaffected. Also clarifies in the contributor docs that changesets and `.server-changes/` notes are user-facing and should be written for users, not maintainers. refs TRI-11702 |
||
|
claude[bot]
|
8bf5879b60 |
test(webapp): poll for replicated rows instead of fixed sleeps in runs replication tests (#4181)
<!-- ccr-slack-attribution --> _Requested by **Matt Aitken** · [Slack thread](https://triggerdotdev.slack.com/archives/C032WA2S43F/p1783430373189849?thread_ts=1783430373.189849&cid=C032WA2S43F)_ ## ✅ Checklist - [x] The PR title follows the convention. - [x] I ran and tested the code works (typecheck of the edited files is clean; see Testing) --- ## Testing **Before:** the webapp run-replication test shard failed on nearly every PR because assertions waited a fixed 1s for rows to replicate from Postgres → ClickHouse and intermittently checked before the row arrived under CI load. **After:** those assertions poll (up to 30s, 250ms interval) until the rows land, so they pass as soon as replication completes and stop flaking, without slowing the happy path. These tests are testcontainers-backed (need Docker + Postgres + ClickHouse), so the full suite is exercised in CI. Locally I confirmed the edited `runsReplicationService.part1..part8.test.ts` files type-check with no new errors. --- ## Changelog **How:** wrapped the ~21 present-row assertions across `runsReplicationService.part1..part8.test.ts` in `vi.waitFor`, matching the existing poll pattern in `part9.test.ts`. Left absence assertions (expecting 0 rows / no spans) on a fixed settle delay since there is nothing to poll for. Tests only — no production code changed. Note: this does NOT touch the `subscribe()` startup race in `internal-packages/replication/src/client.ts` (a riskier, separate follow-up). 💯 --- _Generated by [Claude Code](https://claude.ai/code/session_01KtUdSLKrK17eFVuRYXT6uj)_ --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Chris Arderne
|
aa74e68c71 |
feat(sdk): add bulk replay to api and sdk (#4105)
## Summary
Adds SDK and API support for run bulk actions. You can now create bulk
cancel or replay actions from `@trigger.dev/sdk` using run IDs or the
same filters as `runs.list()`, then retrieve, list, poll, or abort the
action by its `bulk_` handle.
Tests, docs, changesets added.
## Design
The dashboard bulk action service now accepts structured filters instead
of reading directly from a dashboard request, so the dashboard and API
share the same creation path. Replay actions created through the API are
attributed with the existing `api` trigger source, while
dashboard-created actions keep `dashboard`.
The SDK exposes the new surface under `runs.bulk.*`, including
`targetRegion` for replay region overrides and cursor pagination for
listing bulk actions.
## Filters and runIds
Nuance on filters. If `filter` is provided, it MUST have at least one
key. This is to remove the footgun of passing no filter and selecting
all runs.
```typescript
{ action: "cancel", runIds: ["run_1"] } // valid
{ action: "cancel", runIds: [] } // invalid, min(1)
{ action: "cancel", filter: { status: "FAILED" } } // valid
{ action: "cancel", filter: {} } // invalid
{ action: "cancel", filter: {}, runIds: ["run_1"] } // invalid
```
|
||
|
Daniel Sutton
|
d59743bd35 |
fix(webapp,run-ops-database): keep run-ops batch items co-resident with their batch (#4178)
## Summary Three fixes to the run-ops database split (the Cloud-only mode where run-lifecycle rows live on a dedicated Postgres). All are inert in the default single-database deployment. The main fix: on the batch trigger paths, a parentless batch's item runs chose their physical store from a fresh per-org mint-flag read at processing time, so flipping an org's flag mid-batch could land an item in a different store than its batch, breaking the `TaskRun.batchId` foreign key (or silently orphaning the item). The other two harden the split's safety nets: the schema-parity test now actually compares columns, and the read fan-out gate now signals when it has been silently disabled. ## Batch item residency `RunEngineBatchTriggerService` (api.v2) and the BatchQueue item callback (api.v3) now anchor each item's id mint on the batch's own friendlyId, mirroring the already-safe `BatchTriggerV3Service`. Residency is a pure id-shape check, so an item can no longer diverge from its batch across a mid-batch flag flip. The pre-failed-run fallback is anchored the same way (it also sets `batchId`), and the shared mint branch is consolidated into one helper so every mint path stays in lockstep. No new database queries; single-database mode is unchanged (a cuid-shaped batch friendlyId yields a cuid item). ## Schema parity test The parity test previously read only the dedicated schema and matched model headers with regexes, so it never compared columns and could not catch a run-subgraph column that diverged between the two physical schemas. It now parses both schemas and asserts bidirectional scalar-column parity (type, nullability, array-ness, default) across the run-subgraph models, and fails on any field line it can't parse. Scoped to the run-subgraph models so unrelated control-plane edits don't break it. ## Read fan-out signal The split read fan-out gate is decided by the object identity of the NEW vs control-plane clients. It now warns when both run-ops URLs are set but the NEW client isn't a distinct instance (fan-out silently off), and a new test exercises the real topology-into-gate wiring so a future refactor that aliases the clients can't disable fan-out unnoticed. ## Verification New unit and glue tests cover all three changes; the DB-backed residency, store-routing, and topology suites pass against real Postgres; `typecheck` is clean for both packages. |
||
|
Daniel Sutton
|
4c2c25511b |
test(webapp): split triggerTask engine test into per-concern files (#4167)
The engine `triggerTask` suite was a single 2447-line file with 23 `containerTest` cases, each spinning its own Postgres + Redis. vitest shards by whole file, so all 23 container setups landed on one shard and dominated its wall-clock. The recorded entry in `test-timings.json` badly under-counts the real cost (it does not capture the per-`containerTest` container startup that dominates on CI), so the duration-sharding sequencer treated the file as light and stacked it, producing one ~21 minute shard. Splitting does not reduce the number of container setups; it lets those 23 cases distribute across shards instead of stacking on one. The webapp unit-test stage is gated by its slowest shard, so this cuts the stage's wall-clock roughly in half. ## CI timing (before vs after) Real CI wall-clock of the `Unit Tests: Webapp` shards (`--shard=i/10`). "Before" is sampled from recent runs on other branches (unsplit file, from `main`); "after" is this PR. | Shard | Before (s) | After (s) | |------:|-----------:|----------:| | 1 | 250 | 359 | | 2 | 444 | 411 | | 3 | 497 | 659 | | 4 | **1257** | 284 | | 5 | 545 | 641 | | 6 | 284 | 644 | | 7 | 244 | 214 | | 8 | 340 | 445 | | 9 | 188 | 395 | | 10 | 234 | 567 | | **Slowest shard (gates the stage)** | **~1247s (≈21m)** | **659s (≈11m)** | | Sum of all shards | 4283 | 4619 | Before: shard 4 is the long pole at 1237s / 1247s / 1257s across three sampled runs (the `triggerTask` file plus whatever else the packer put with it). After: the six pieces spread across shards, the slowest drops to 659s. The small rise in summed time is the extra per-file container startup, paid in parallel across shards, so the gating number still falls by about 10 minutes. ## Change Split into six per-concern files that share a `triggerTaskTestHelpers` module (the `vi.mock` calls stay per-file, since vitest hoists them): - `triggerTask.test.ts` (3): trigger + concurrencyKey coercion - `triggerTask.idempotency.test.ts` (4): idempotency + queue resolution - `triggerTask.debounce.test.ts` (4): retries + debounce validation - `triggerTask.mollifier.test.ts` (4): mollifier call-site behaviour - `triggerTask.metadataCache.test.ts` (4): DefaultQueueManager task metadata cache - `triggerTask.residency.test.ts` (4): child run residency inheritance All 23 cases are preserved. The file's `test-timings.json` entry is split across the new files so bin-packing stays balanced. While rewriting these files, cleanup was moved to `onTestFinished(() => engine.quit())` so an `engine`/`Redis` leaked on a failing assertion no longer persists on the worker-scoped Redis and cascades into later cases (`hookTimeout` raised to 60s so the after-cleanup gets the full budget). Prisma lookups switched from `findUnique` to `findFirst` to match the repo convention. Verified: all six files run green locally (23/23), oxlint and oxfmt clean. |
||
|
Daniel Sutton
|
e4ae8cbcd4 |
fix(run-engine,run-store,webapp): stop split-mode waits hanging on resume (#4164)
## Summary On the run-ops database split, a run that waits (`triggerAndWait`, `batchTriggerAndWait`, `wait.forToken`) could hang forever after its wait had already completed. The runner reads a resume from `/snapshots/since` exactly once: if that read returned the resume snapshot without its completed-waitpoints, the runner logged "executing without completed waitpoints", advanced its cursor, and never re-read it, so the awaiting run never continued. ## Root cause The resume snapshot and its completed-waitpoint rows were written as two separate commits. This regressed when the split replaced Prisma's atomic nested `connect` with an FK-free insert (in [#4163](https://github.com/triggerdotdev/trigger.dev/pull/4163)), and `/snapshots/since` is served from a read replica. A fetch landing in the sub-millisecond gap between the two commits, or a multi-reader replica serving the snapshot from a different point in time than its join rows, delivered an empty resume. Because the runner consumes each snapshot once and treats an empty resume as terminal, a single stale read was fatal and produced a permanent, nondeterministic hang. ## Fixes - Commit a snapshot and its completed-waitpoint links in one transaction, restoring the atomicity the split removed. - Repair the completed-waitpoints from the owning primary when a multi-reader replica serves the snapshot without its join rows. This covers single-waitpoint resumes, which carry no `completedWaitpointOrder` and so were missed by the count-based repair. - Read the primary in the checkpoint `WAIT_FOR_BATCH` pre-check, so a batch that already resumed is not re-suspended into a stall. - Fall back to the primary when a waitpoint token misses both read replicas, so a token completed immediately after it was minted no longer returns a spurious 404. - Route batch-item creation by `batchTaskRunId`, consistent with the batch-completion count and the row's foreign key. - Reject control-plane-only relation selects on the dedicated schema with a clear error instead of an opaque Prisma failure, and stop `createDateTimeWaitpoint` bypassing residency routing through a caller transaction. Verified against the deployed split topology: a resume snapshot and its completed-waitpoints are now always delivered together, so the runner can no longer drop a resume. |
||
|
Daniel Sutton
|
f101983a70 |
fix(run-store,run-engine): fix run-ops split hangs from wrong-store reads on the resume path (#4163)
## Summary On the run-ops split, NEW-residency runs could hang. Time-based waits (`wait.for`, `wait.until`, `delay`, waitpoint tokens), `batchTriggerAndWait`, and attempt starts stalled and never resumed. Each was a run-ops read or update that hit the wrong database: either the owning store's read replica when it needed read-your-writes, or the wrong store entirely because it routed by an id that does not encode residency. ## Fixes **Waitpoint resume (the main hang).** The managed resume path reads a run's completed waitpoints by snapshot id (`findSnapshotCompletedWaitpointIds`). Snapshot ids are cuids, which always classify to the legacy store, so a NEW run's join rows (which live on the new store) were never found. The resumed run saw zero completed waitpoints and hung. It now fans out across both stores and merges, like its sibling readers. **Batch completion.** Batch item completion (`updateManyBatchTaskRunItems`) routed by the item id, which is also a cuid, so a NEW batch's items were updated on the wrong store, matched zero rows, and the batch was treated as already complete (its parent's `batchTriggerAndWait` then hung). It now routes by the batch id, which does encode residency, matching the sibling `countBatchTaskRunItems`. **Read-your-writes on the resume path.** The block-time pending-waitpoint check (`countPendingWaitpoints`) and the attempt-start lock check (`findRun` in `startRunAttempt`) both read the owning store's replica with no read-your-writes guarantee, so a just-committed waitpoint completion or dequeue lock could be missed under replica lag and strand the run. Both now read the owning primary. Each fix ships with a two-database store or engine test that reproduces the hang and passes with the fix. |
||
|
Daniel Sutton
|
092b9ef07a |
fix(run-ops): DNS-safe, sortable base32hex run id (replace base62 KSUID) (#4154)
## Problem
The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.
## Fix: change the encoding, not the structure
Mint a **26-char lowercase base32hex** run id:
```
run_<24-char base32hex core><region char><version char>
[ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```
- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.
DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.
## Residency discriminator: length → version char
`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.
## Scope / verification
- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.
## Open decisions (flagged, not silently chosen)
1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.
---
## ⚠️ Required migration — deploy in lockstep
This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):
1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
- key `runOpsMintKsuid` → `runOpsMintKind`
- value `"ksuid"` → `"runOpsId"`
Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
962bc48738 |
feat(run-ops): automatically migrate the dedicated run-ops database (#4150)
## What
Adds the ability to **automatically migrate the dedicated run-ops
database** (the NEW DB in the run-ops split), matching how every other
database in the system is migrated. Follow-up to the run-ops split
activation.
## Changes
- **Migrate runner** — new
`internal-packages/run-ops-database/scripts/migrate.mjs`, exposed as
`db:migrate:deploy` / `db:migrate:status`. Connects via
`RUN_OPS_DATABASE_URL` (the same var the app uses) and expands `${VAR}`
refs like Prisma's dotenv.
- **Self-host** — `docker/scripts/entrypoint.sh` runs the run-ops
migration on boot when the DB is configured, gated by
`SKIP_RUN_OPS_MIGRATIONS`. Single-DB installs never set the URL, so it's
a clean no-op.
- **Single env-var family** — the run-ops DB is now addressed by one
canonical `RUN_OPS_*` family, connect path and migrations resolving the
identical URL:
- `RUN_OPS_DATABASE_URL` (writer) — replaces `TASK_RUN_DATABASE_URL`
- `RUN_OPS_LEGACY_DATABASE_URL` — replaces
`TASK_RUN_LEGACY_DATABASE_URL`
- `RUN_OPS_DATABASE_READ_REPLICA_URL` — replaces
`TASK_RUN_DATABASE_READ_REPLICA_URL`
- the old `TASK_RUN_*` aliases, the `??` coalesce, the
`runOpsNewDatabaseUrl` indirection, and the migrate-only `directUrl` are
all removed (consumers read `env.RUN_OPS_DATABASE_URL` directly).
`directUrl` was dropped because it was only ever used by `prisma
migrate` (never the app runtime) to bypass a pooler for advisory locks —
premature here since the run-ops connection isn't wired to the app yet.
If a pooler is later introduced for the app, a direct URL can be
reintroduced then.
## Safety
- **Pure rename** — nothing deployed sets any `TASK_RUN_*` var (the
split isn't activated anywhere yet; `.env.example`, docker-compose, and
cloud already use `RUN_OPS_*`), so there is no config migration.
- **Single-DB / self-host** — no new required env var; entrypoint and
migrate are no-ops when `RUN_OPS_DATABASE_URL` is unset.
- **Cloud** — runs migrations as pre-deploy ECS tasks (companion cloud
PR), calling these same `db:migrate:deploy` / `db:migrate:status`
commands.
## Verification
- Live migration against a fresh scratch DB with only
`RUN_OPS_DATABASE_URL` set: both migrations applied, no `P1012`/`P1013`;
`${VAR}` expansion, idempotent re-run, `status`, and no-op skip all
pass.
- Schema parity 4/4; `typecheck --filter webapp` 18/18; affected
split/replication tests 34/34.
## Scope
This delivers automatic migrations only. Enabling the app to *use* the
new DB (setting `RUN_OPS_DATABASE_URL` + `RUN_OPS_SPLIT_ENABLED` on the
service) is a separate activation step.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
c9f427e21f |
fix(replication): key logical replication leader lock on slot name (#4151)
## Problem
`LogicalReplicationClient` uses a Redlock leader lock to guarantee a
single active consumer per Postgres logical replication slot. The lock
resource was keyed on the client `name`:
```
logical-replication-client:${this.options.name}
```
A slot permits exactly one consumer, so the lock's job is to serialize
consumers **of a given slot**. Keying it on `name` breaks that whenever
two clients target the same slot with different names — most notably
across a rolling deploy where the client `name` changes but `slotName`
does not. Both acquire *distinct* locks, both consider themselves
leader, and the second to reach `START_REPLICATION` hits `replication
slot "<slot>" is active for PID <n>`. Because that query was
fire-and-forget and its failure was only logged (no retry), the consumer
stopped and replication stalled until the process was restarted.
## Fix
**1. Key the leader lock on `slotName`** — the actual single-consumer
resource:
```
logical-replication-client:${this.options.slotName}
```
Consumers of the same slot now contend on the same lock and hand off
cleanly across restarts/deploys; different slots stay independent.
`name` is kept for logging and the pg `application_name`.
**2. Self-healing resubscribe** (`resubscribeOnFailure`, opt-in) —
instead of logging-and-dying, a client re-subscribes with exponential
backoff after a lost election or a failed `START_REPLICATION`, so a
rolling deploy self-heals: the incoming pod retries until the draining
pod releases the slot, then takes over. Safety:
- `#cleanupAttempt()` unconditionally ends the pg client (freeing the
walsender) and releases the leader lock before rescheduling — retries
never leak connections/locks.
- `shutdown()` sets an intentional-stop latch re-checked after every
`await` in `subscribe()` (and aborts the lock-acquire spin), so a
resubscribe can never race or outlive an intentional shutdown.
- Backoff resets only on genuine stream start, so a permanently stuck
slot backs off to the ceiling and logs loudly rather than tight-looping;
an epoch guard neutralises stale `START_REPLICATION` catches.
Runs- and sessions-replication opt in and use `shutdown()` for all
intentional stops.
**3. Observability** — the admin runs-replication status route probed
the old name-keyed Redis key (would report `leader:false` for every
source after fix #1); now probes the slot-keyed key.
## Tests
`internal-packages/replication/src/client.test.ts` (real Postgres +
Redis containers):
- same-slot/different-name → second client must not double-lead or race
into "slot is active" (the regression)
- a failing `START_REPLICATION` retry loop must not leak connections or
locks
- `shutdown()` during an in-flight `subscribe()` must not leave a zombie
leader
- `subscribe()` after `shutdown()` re-arms `resubscribeOnFailure`
- self-heals once the leader releases the slot
Plus the multi-source wiring test updated to the slot-keyed lock keys.
## Rollout
With the self-healing resubscribe, this ships as a **plain rolling
deploy** — the incoming pods retry across the one-time lock-key
transition and take over once the old pods drain (a brief replication
stall that the durable slot replays on reconnect — no data loss). No
stop-before-start required.
|
||
|
Daniel Sutton
|
70bca82d84 | feat(run-ops): activation — drop cross-DB FKs, provision run-ops DB, enable split (#4124) | ||
|
Daniel Sutton
|
618b921a51 | feat(run-ops): webapp routes — friendlyId reads, cross-seam token resolution, co-location writes (#4123) | ||
|
Daniel Sutton
|
0a51341347 | feat(run-ops): read presenters — de-join control-plane relations + read-through hydration (#4122) | ||
|
Daniel Sutton
|
5be6a4fe38 |
feat(run-ops): ClickHouse multi-source replication fan-in + admin ops (#4119)
## What Extends the ClickHouse runs-replication service to fan in from multiple Postgres sources (the control-plane DB and the run-ops DB) instead of a single source, plus the admin operations to run and observe it. - **Multi-source fan-in** (`services/runsReplicationService.server.ts`, new `runsReplicationInstance.server.ts`, `runsReplicationGlobal.server.ts`): factors the replication service into per-source instances and a coordinator so a single ClickHouse target is fed from more than one Postgres source. - **Admin ops** (`routes/admin.api.v1.runs-replication.status.ts`, `admin.api.v1.runs-replication.backfill.ts`, `v3/services/adminWorker.server.ts`): adds a status endpoint reporting per-source replication state and updates the backfill entrypoint for the multi-source shape. ## Why PR7 of the run-ops split stack, and the final piece: once run state can live in a separate run-ops DB (earlier PRs), the analytics replication into ClickHouse has to consume both sources so runs remain queryable regardless of residency. Behavior-changing for the replication service internals; the ClickHouse-facing output is unchanged (still one runs stream), and single-source operation is preserved when the split is not enabled. ## Tests New vitest coverage: `runsReplicationInstance.test.ts` (per-source instance behavior) and `runsReplicationService.part8`/`part9` suites exercising the multi-source coordinator. Testcontainers-backed (ClickHouse + Postgres); no mocks. ## Notes Draft, **stacked on #4118** (`runops/pr06-write-path`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
84f3e1b39c |
feat(run-ops): webapp write path — trigger/batch minting, idempotency routing, run lifecycle (#4118)
## What Routes the webapp write path through the run-ops split seam: trigger/batch minting, idempotency-key resolution, and the run-lifecycle services now determine residency and dispatch writes to the correct store. - **Trigger & batch** (`runEngine/services/triggerTask.server.ts`, `batchTrigger.server.ts`, `createBatch.server.ts`, `streamBatchItems.server.ts`, `v3/services/batchTriggerV3.server.ts`): mint ids with the run-ops-aware minting and route creation/streaming through the store; batch children inherit the parent's residency. - **Idempotency** (`runEngine/concerns/idempotencyKeys.server.ts` + new `idempotencyResidency.server.ts`): idempotency-key lookup/dedup is residency-aware so a keyed retrigger resolves against the store that owns the original run. - **Run lifecycle services** (`createCheckpoint`, `createTaskRunAttempt`, `enqueueDelayedRun`, `expireEnqueuedRun`, `finalizeTaskRun`, `resumeBatchRun`, `cancelDevSessionRuns`, `executeTasksWaitingForDeploy`, `triggerFailedTask`): resolve their target run through the store rather than a fixed client. - **Reads that fan out from writes** (`runsRepository` + `clickhouseRunsRepository`, `BulkActionV2` + batch read-through, realtime `sessions`/`runReader`, alerts `deliverAlert`/`performTaskRunAlerts`): route through the read-through resolver. - `9535ae63d` — resolves the parent run through an injectable run store in `TriggerFailedTaskService`. - `bf8f7c881` — drops the "known-migrated" concept from write-path and read repos; residency is id-shape only. - `515b897ea` — self-defaults `resolveWaitpointThroughReadThrough` to the safe run-ops clients. ## Why PR6 of the run-ops split stack. This is the write-path counterpart to the read foundation in the previous PRs: with it in place, both reads and writes route through the seam. Additive when the split is disabled (id-shape resolution collapses to the control-plane client); behavior-changing on the minting, idempotency, and lifecycle paths when enabled. ## Tests Large new/expanded vitest suite under `apps/webapp/test/` and colocated service tests: trigger-task and batch-trigger store routing, residency inheritance, idempotency dedup residency + legacy-authority, bulk-action read routing, cancel-dev-session routing, alerts store routing, runs-repository read-through, realtime session/run-reader read-through and stream-registration routing, and the waitpoint read-through default. Testcontainers-backed; no mocks. ## Notes Draft, **stacked on #4117** (`runops/pr05-webapp-foundation`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
8465ac5ac3 |
feat(run-ops): webapp db topology, flags, and split-mode resolver wiring (#4117)
## What Wires the run-ops split into the webapp: database topology, environment flags, split-mode gating, and the control-plane resolver/cache layer that the run-store and run-engine seams from the previous PR plug into. - **DB topology & env** (`apps/webapp/app/db.server.ts`, `env.server.ts`, `entry.server.tsx`): adds the run-ops database clients/topology and the environment variables that configure and gate the split. - **runOpsMigration module** (new `apps/webapp/app/v3/runOpsMigration/`): the webapp-side machinery — `splitMode.server.ts`, `controlPlaneResolver.server.ts` + `controlPlaneCache.server.ts`, `readThrough.server.ts`, `crossSeamGuard.server.ts`, `distinctDbSentinel.server.ts`, id-minting helpers (`mintBatchFriendlyId`, `runOpsMintKind`, `resolveInheritedMintKind`), `runOpsCascadeCleanup.server.ts`, the split read gate, and route/unblock catalogs. - **Store/engine wiring** (`app/v3/runStore.server.ts`, `runEngine.server.ts`, `runEngineHandlers.server.ts` + new `runEngineHandlersShared.server.ts`): points the webapp's store/engine construction at the resolver, and factors shared handler logic out so both seams use one path. - **Read-path touch-ups**: `runtimeEnvironment.server.ts`, `eventRepository/index.server.ts`, `taskRunHeartbeatFailed.server.ts`, `engineVersion.server.ts` route their run/environment lookups read-through the resolver. - `413a94511` — interlocks split mode against the native realtime backend so the two aren't enabled in an incompatible combination (see `.server-changes/run-ops-split-realtime-interlock.md`). - `dc74c57fd` — drops the earlier "known-migrated" read layer; residency is determined by id-shape only. ## Why PR5 of the run-ops split stack. This is the webapp foundation layer: it stands up the DB topology, flags, and resolver/cache the rest of the stack depends on, and repoints webapp read paths through the resolver. Additive when the split is not enabled (existing single-DB behavior preserved behind flags); behavior-changing on the read-through paths and the realtime interlock. ## Tests New vitest coverage across `apps/webapp/test/` and colocated `*.server.test.ts` files: db topology, split mode, split read gate, cross-seam guard, mint cutover / flip latency, control-plane cache, control-plane resolver, distinct-db sentinel, read-through loaders (route loaders, run-detail loaders, `findEnvironmentFromRun`), and the run-engine handlers. Testcontainers-backed; no mocks. `pnpm-lock.yaml` synced for the two new webapp deps. ## Notes Draft, **stacked on #4116** (`runops/pr04-store-engine`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
c266e96c87 |
feat(run-ops): run-store routing seam + run-engine read seams (#4116)
## What Introduces the run-store routing seam and the run-engine read seams that let run lifecycle operations be dispatched to either the control-plane database or a separately-generated run-ops database, depending on where a run/batch resides. - **run-store** (`internal-packages/run-store`): adds `runOpsStore.ts` and substantially expands `PostgresRunStore.ts` so the store can resolve residency and route reads/writes to the correct backing client. `types.ts` grows the routing/residency types; `NoopRunStore.ts` is removed. - **run-engine** (`internal-packages/run-engine`): adds `engine/controlPlaneResolver.ts` and routes the per-system read paths (dequeue, enqueue, waitpoint, checkpoint, run-attempt, ttl, delayed-run, execution-snapshot, pending-version, debounce, batch) through the resolver/store instead of talking to a single Prisma client directly. `engine/errors.ts`, `engine/types.ts`, and `engine/index.ts` are extended to support injecting the store/resolver. Three fixes are included on top of the seam work: - `c6cadd85f` — routes read-your-writes to the owning store's **writer**, not its lagging replica, so an operation immediately reading back what it just wrote sees a consistent result. - `05c912e05` — normalizes run-ops-generation Prisma errors to the control-plane error class at the store **write boundary**, so `instanceof` checks and the `P2002` → 422 handling continue to work across the separately-generated run-ops Prisma client. - `88d12907f` — resolves NEW-resident batches in `ApiBatchResultsPresenter` by routing the batch read through the store, so a dedicated-DB batch resolves instead of returning 404. The change is heavily test-first: the bulk of the diff is new unit/integration coverage for the store routing, residency, and each run-engine system's control-plane resolver path. ## Why PR4 of the run-ops split stack (PR1–PR3 land the ClickHouse test-container and earlier plumbing). This PR is the read-path foundation: it adds the seam and read-routing but leaves the write path to route through the same seam in a later PR. Behavior-changing where the three fixes above touch existing read-your-writes / error-normalization / batch-resolution paths; otherwise additive (new store module, new resolver, injectable dependencies with existing single-client behavior preserved when no dedicated store is configured). ## Tests Extensive new vitest coverage under `run-store/src/*.test.ts` (routing, residency, dual-schema select, cross-generation error normalization, read-after-write, idempotency dedup, mixed residency, waitpoint co-location) and `run-engine/src/engine/**/*.test.ts` (per-system `controlPlaneResolver` tests, injectability, block-edge residency, waitpoint read residency, trigger-create routing, lifecycle router). Testcontainers-backed; no mocks. ## Notes Draft, **stacked on #4114** (`runops/pr03-clickhouse-tc`). Review that first; this diff is against it. Server-change / changeset note to be added at stack-assembly time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
nicktrn
|
0f349dd636 |
test(webapp): regression tests for resuming manually paused environments (#4127)
Follow-up to #4120, adding regression coverage for the pause/resume path in `PauseEnvironmentService`. Three tests against the real service with testcontainers Postgres (no mocks), seeded org/project/environment rows, and the same `AuthenticatedEnvironment` coercion production uses: 1. **resumes a manually paused env** - the actual regression: pause leaves `pauseSource` null, resume must succeed. Verified this fails with the exact pre-#4120 symptom (false billing-limit error) when the fix is reverted locally. 2. **rejects resume of a billing-limit paused env** - the guard still blocks manual resume while `pauseSource = BILLING_LIMIT`, and the env stays paused. 3. **manual pause while billing-limit paused is a no-op** - returns success without overwriting `pauseSource`, so billing-limit converge can still find and unpause the environment. Tests-only change, no runtime behavior touched. |
||
|
Matt Aitken
|
6563793132 |
fix(webapp): dashboard timezone display + preference persistence (#4104)
## Summary
Two related timezone bugs in the dashboard.
1. The date/time tooltip could show a UTC offset label that contradicted
the time it displayed. A viewer whose machine clock differs from their
saved timezone (or when a date falls in the other DST phase) would see
something like `Local (UTC +0)` next to a value that isn't at +0.
2. A user's timezone preference silently failed to save whenever their
browser reported a zone like `UTC`, `Etc/UTC`, or `Asia/Kolkata`,
leaving their timestamps stuck in a previously-saved timezone.
## Offset label
The "Local" row formatted its time using the viewer's configured
timezone but computed the `(UTC +n)` label from `new
Date().getTimezoneOffset()`, the browser's offset at the current moment.
Those are two independent sources, so they disagreed when the configured
timezone differed from the machine, and also when the displayed date was
in the opposite DST phase. The label is now derived from the same date
and timezone used to render the row (via `Intl.DateTimeFormat` with
`timeZoneName: "longOffset"`), so it always matches the displayed time.
## Preference persistence
`/resources/timezone` validated the incoming zone against
`Intl.supportedValuesOf("timeZone")`, which lists only canonical zone
ids. Browsers report zones that aren't in that list via
`resolvedOptions().timeZone`, notably `UTC` (and `Etc/UTC`,
`Asia/Kolkata`, `GMT`), so those requests returned 400 and the
preference was never stored. Validation now checks whether the runtime
can resolve the zone at all, which accepts every real zone and still
rejects invalid input.
Added unit tests for both.
|
||
|
Matt Aitken
|
fd4f02b2f8 |
fix(webapp): onboard new cloud orgs via plan selection; allow Free plan without GitHub verification (#4109)
## What & why Two related fixes to how new cloud organizations get onboarded onto the Free plan. ### 1. Route new cloud orgs through plan selection New cloud organizations were created already activated, so they skipped the plan-selection step and went straight to creating projects — which meant their plan and usage limits were never set up. They're now created deactivated and routed through plan selection, which activates them once a plan is chosen. Self-hosted installs have no plan-selection step, so they're activated immediately on creation and are unaffected. The `Organization.v3Enabled` field is renamed to `isActivated` to better describe what it now gates. It's mapped to the existing `v3Enabled` column, so there's no data migration — only a schema/code rename. ### 2. Allow selecting the Free plan without GitHub verification Choosing the Free plan no longer requires connecting and verifying a GitHub account. The plan is applied immediately when selected. This removes: - the "Connect to GitHub" dialog and the GitHub-verified badge from the plan picker - the account-rejected state - the now-unreachable GitHub-connect return routes ## Notes - These changes pair with the corresponding change in the billing service that applies the Free plan directly; they should be released together. ## Testing Verified locally end to end: a new cloud org is routed to plan selection, the Free plan applies in one click with no GitHub step, the org is activated, its usage allowance is provisioned, and it lands on the new-project page. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Chris Arderne
|
c7861be520 |
chore: activate no-unused-vars and import linters (#4096)
Once this is merged, oxlint is at a pretty sensible baseline. **Enable `no-unused-vars`, `typescript/consistent-type-imports`, and `import/no-duplicates` lint rules** Turns on three previously-disabled oxlint rules across the monorepo and fixes all violations: - **`no-unused-vars`** – enabled as an error with standard ignore patterns: unused function arguments are ignored by default (`args: "none"`), variables/caught errors/destructured array elements prefixed with `_` are allowed, and rest siblings are permitted. - **`typescript/consistent-type-imports`** – enforced as an error; all type-only imports now use the `import type` syntax. - **`import/no-duplicates`** – enforced as an error; duplicate import statements from the same module have been merged. The remaining commits clean up the violations found across the codebase: removing unused variables/imports/type aliases, adding `_` prefixes to intentionally unused bindings, fixing duplicate imports, and converting value imports to `import type` where appropriate. |
||
|
Chris Arderne
|
bfa902bd18 |
chore: enable more linters (#4080)
Re-enables ~15 oxlint rules that were blanket-disabled before. |
||
|
Chris Arderne
|
1e6cc19088 |
chore(webapp): upgrade @conform-to to v1 (#4044)
## Summary Upgrades the dashboard form layer from `@conform-to` 0.9 to 1.x. No behaviour change is intended; this is the conform API migration only. conform 1.x peer-depends on `zod` `^3.21 || ^4`, so it runs on the current zod 3 and is a prerequisite for upgrading the repo to zod 4: conform 0.9 imports `ZodNativeEnum`, `ZodEffects`, and `ZodPipeline`, all removed in zod 4, so the webapp cannot build against zod 4 until conform is on 1.x. Landing this first (on zod 3) keeps the zod 4 PR focused on zod alone. ## Testing Did a bunch of local smoke tests and E2E playwright tests, several rounds of different reviewers, all clean. |
||
|
Katia Bulatova
|
cd9d20cc99 |
fix(webapp): accept invites for orgs with many projects (#4043)
Invite acceptance could fail for cloud organizations with many projects because the whole flow ran inside a single transaction and did too much work before it completed. In larger orgs, that pushed the transaction past its timeout and blocked the invite from being accepted. This PR moves the expensive parts of invite acceptance out of the transaction, excludes deleted projects from environment setup, fixes error handling on /invites, and adds regression coverage for the failure cases. |
||
|
Katia Bulatova
|
cc752cc375 |
fix(webapp): fetch run-scoped trace subtrees for large traces (#4024)
Fixed trace rendering for child and nested runs in large traces. Dashboard and trace API responses now load the requested run's trace subtree instead of depending on the run span appearing in the initial trace slice. |
||
|
James Ritchie
|
b2b4c510e2 |
feat(webapp): improve task and dashboard activity charts (#4064)
## Summary Improves and unifies the run-activity charts by extracting a shared set of chart primitives and adopting them on the three task landing pages (agent, standard, scheduled), with the density and label fixes also carried over to the dashboard and custom query charts. Main changes a reviewer should know about: - **Shared primitives (DRY).** New `ChartCard` (title + maximize/fullscreen), `ChartSyncContext` (cross-chart hover + zoom state), `useXAxisTicks` (width-aware tick selection), `activityTimeAxis`, and `statusColors`, plus a server-side `activitySeries.server.ts` holding `chooseBucketSeconds`, status grouping, and the zero-fill helpers. The three task routes and both presenters were refactored onto these, removing roughly 3x duplicated tick logic, status-color tables, and bucket-ladder code. - **Denser bars on short ranges.** Server-side bucketing now uses `chooseBucketSeconds` (nice-interval ladder, ~72 target, capped at 120 buckets) instead of the hardcoded 1h/6h/1d ladder, so a 5m or 1h range no longer collapses into a single bar. - **Width-aware x-axis labels.** Labels are selected to fit the measured plot width (always first + last, evenly spaced, de-duplicated by rendered text), stay horizontal, and reflow on panel/window resize. Y-axis values default to compact form (8K, 1.2M) in `ChartBar`/`ChartLine`. - **Synced hover line.** Hovering one of the agent page's three charts draws a dashed vertical line at the same bucket on the *other* two, and suppresses it on the hovered chart. It is opt-in via `ChartSyncProvider`, so single-chart pages are unaffected. - **Maximize button.** Each chart gets a fullscreen dialog toggle (reuses the existing dashboard-widget pattern, `v` shortcut while hovered). - **Drag-to-zoom on task pages.** Dragging across a task chart sets the Time/Date filter (`from`/`to` URL params, clearing `period`/`cursor`), with a From/To tooltip shown during the drag. - **Custom query charts.** Long categorical x labels (run IDs, task names) middle-truncate and auto-rotate only when needed, and label thinning is now width-aware for both bar and line variants. Dashboard line-chart label density is also width-aware, tuned by a `TIME_AXIS_LABEL_SPACING_PX` constant. - **Tests.** 46 new unit tests for the pure logic (bucket selection, tick spacing, time-axis formatting, zoom range, truncation). ## Intentionally unchanged - **No click/drag zoom on the dashboard or custom query charts.** Drag-to-zoom is wired up on the task landing pages only; zooming the dashboard and custom charts is deliberately deferred to a separate follow-up PR. A plain click (without a drag) on a task chart is a no-op. - **The 25 mini activity charts on the Task list (`_index`) page are untouched.** They are hand-rolled raw-Recharts sparklines kept deliberately lightweight and do not use these primitives. - **Other raw-Recharts sparklines are untouched** (the usage sparkline, errors and prompts pages). - **No ClickHouse query semantics changed** beyond the bucket-interval parameter (same filters, same FINAL / `_is_deleted` handling). - **Webapp-only.** No public package (`packages/*`) changes, so there is no changeset; the `.server-changes/` entries cover it. --- ## Testing Added 46 unit tests covering server-side bucket selection, width-aware tick spacing, time-axis formatting, zoom-range math, and categorical label truncation (`pnpm --filter webapp run test`), and `pnpm run typecheck --filter webapp` passes. Manually exercised each task landing page (agent, standard, scheduled) plus the dashboard and custom query charts, stepping the Date/Time filter through 5m, 1h, 24h, 7d, and 30d to confirm dense bars on short ranges, non-overlapping labels that reflow on resize, the synced hover line across the agent charts, the maximize button, and drag-to-zoom updating the filter. --- ## Changelog The activity charts on the task landing pages and the dashboard and custom query charts now share one set of reusable primitives. X-axis labels are width-aware so they never overlap and reflow when a panel resizes, y-axis values are abbreviated (8K, 1.2M), and short time ranges render dense bars instead of collapsing into a single bar. Hovering any agent chart mirrors a vertical line on the others, every chart gains a maximize button, and dragging across a task chart zooms the Time/Date filter. Long categorical labels such as run IDs and task names middle-truncate and auto-rotate only when needed. --- https://github.com/user-attachments/assets/6be09e38-3a0e-4947-b6e3-4839daa2fbe0 |
||
|
Katia Bulatova
|
b1987dc090 |
feat(webapp): billing limits — pause, reject, recovery, and settings UI (#3996)
## Summary Adds Billing Limits to the webapp. Customers can set a monthly spend cap. When usage crosses the limit, billable environments enter a grace period. If the limit is not resolved before grace expires, new triggers are rejected until the organization increases or removes the limit. |
||
|
Chris Arderne
|
4fde283e76 |
chore: format and lint webapp also (#4056)
#3977 added formatting and linting everywhere else. This extends it to the webapp. |
||
|
Chris Arderne
|
df78ef96d9 |
feat: multi dev branches (#4023)
Closes this feature request: [https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances) ### Feature notes: - CLI `trigger dev` works as before - `trigger dev --branch my-branch` to create a new branch and run against it. - `trigger dev archive --branch my-branch` to archive (or in webapp). - New webapp page to manage and archive dev branches, currently feature flagged. ### Implementation details: - No changes to data model, no backfill. `isBranchableEnvironment` column is ignored for dev branches, we use `parentEnvironmentId IS NULL` instead. - `x-trigger-branch` overloaded for preview and dev branches - New `TRIGGER_DEV_BRANCH` env var available locally. `TRIGGER_PREVIEW_BRANCH` overloaded for child runs. - Lots of new glue code to sanitise the branch checks. ### Rollout - Deploy webapp/API changes (all backwards compatible) - Manual tests on some orgs - Deploy docs, release CLI, flip feature flag for webapp feature ### NB - `api.v1.projects.$projectRef.environments.ts` will return `isBranchableEnvironment: true` for all dev environments. ### Prerequisites - [x] Typecheck will not pass until we make a new release of `@trigger.dev/platform` and bump it here |
||
|
nicktrn
|
bc605eedaf |
fix(webapp): verify deployment image exists before finalizing (#4049)
A deployment could be marked deployed and promoted to current without its image ever landing in the registry. Finalize trusted the CLI: the v1 path never pushed or checked, and the v2/v3 path skips its own push when the CLI sends `skipPushToRegistry` - which the local-build path always does. In the happy path the CLI pushes the image itself, so this stayed latent. But any deviation - `--no-push`/`--load`, a push that lands in a different registry, or an old CLI - promoted a version whose image can't be pulled, so every run failed at pull time while the deploy itself reported success. This adds a registry existence check after push and before finalize. If the image isn't there, the deploy fails loudly instead of promoting a version that can't start. The check is ECR-only (a no-op for other registries, so self-hosted setups are unaffected) and uses `BatchGetImage`, which the deploy role already allows. It fails open on an ambiguous registry error so the check can't itself turn into a deploy outage. The image reference is the platform-generated value and the lookup is bound to the configured registry host; the CLI-supplied digest is validated before use. Can be turned off with `DEPLOY_IMAGE_VERIFICATION_ENABLED=0` for setups that push images out of band (e.g. an air-gapped registry the platform can't reach). refs TRI-11243 |
||
|
Eric Allam
|
2fa84ea124 |
feat(webapp): gate worker dequeues by worker queue via env var (#4030)
## Summary Adds a `RUN_ENGINE_DEQUEUE_DISABLED_WORKER_QUEUES` setting that refuses worker dequeue requests for the listed worker queues (or base regions), so their runs stay queued instead of being handed to workers that can't run them. Blocked dequeues are counted via a `run_engine.dequeue.blocked` OTel counter (labeled by `worker_queue` and `region`). |
||
|
James Ritchie
|
a90a495542 |
feat(webapp,database): show a Test column for agent sessions (#4011)
## Summary Sessions started from the agent Test playground were tagged with a `"playground"` tag that rendered in the Sessions table's Tags column. They are now flagged with a real `Session.isTest` boolean (mirroring `TaskRun.isTest`) and surfaced as a dedicated **Test** column with a check icon, to the left of Tags, on both the Sessions page and the Agent landing page, plus a matching **Test** property on the session detail page. This mirrors how Standard and Scheduled task runs already indicate test runs. ## Design `isTest` is a new `Session` column (Postgres) replicated into ClickHouse `sessions_v1` alongside the existing fields. The Sessions list reads `isTest` from Postgres for display (ClickHouse only supplies the ordered session IDs), so the column renders correctly without a ClickHouse backfill. The playground action now sets `isTest: true` on session create instead of writing the `"playground"` tag. The triggered run still carries `playground:true` in its own tags (unchanged). A migration backfills existing sessions, setting `isTest = true` and stripping the now-redundant `"playground"` tag where it is present, so the list and detail views render consistently without read-time tag filtering. |
||
|
Daniel Sutton
|
65c545da4e |
refactor(run-store,webapp,run-engine): route Postgres TaskRun reads through the run store (#3990)
## Summary Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`, `findRuns`) and routes every Postgres read of `TaskRun` through them, mirroring how writes already go through the store. Behavior-preserving: each relocated read keeps its exact query, field selection, and database client (writer, replica, or transaction). This lets `TaskRun` reads be retargeted to a different backing store later without touching call sites. Stacked on #3981 (the write adapter); that PR is the base of this one. ## Scope In scope: the run engine, webapp services, presenters, and route loaders. Three reads that pulled `TaskRun` in through a parent model's relation `include` (alert delivery, batch results, attempt-dependency cancellation) are decomposed to fetch the run(s) through the store and stitch them back, since a relation include would not follow `TaskRun` to a new table. Left reading the existing table (out of scope): the legacy MarQS paths, the legacy trigger idempotency read, and one raw-SQL recovery script (commented for revisiting at cutover). ## Notes Reads default to the read replica; callers pass the writer or a transaction client wherever the original read did, so writer-vs-replica behavior is unchanged. |
||
|
Daniel Sutton
|
315baf2e54 | refactor(run-engine,webapp): route TaskRun writes through a new RunStore adapter (#3981) | ||
|
Matt Aitken
|
5740955357 |
feat(webapp): enforce RBAC permissions on run, prompt, member, and billing routes (#3948)
## Summary Several dashboard routes performed actions a restricted role should not be able to do (cancel or replay runs, manage prompt versions, invite and manage members, manage billing) without any permission check. This adds role-based permission enforcement to those routes, and disables the matching UI controls (with a tooltip) when the current role lacks permission. Covered actions: - Runs: cancel and replay (single, bulk create, bulk abort) - Prompts: create or edit override versions, and promote a version to current - Members: invite, resend invite, revoke invite - Billing: change plan, billing alerts, and the customer portal ## How Each affected route now goes through the `dashboardLoader` / `dashboardAction` route builders with an `authorization` block declaring the required permission (or a per-intent check where one route handles several intents). Existing tenancy and data-scoping queries are untouched; this only layers permission checks on top. The UI follows disable-don't-hide: controls stay visible but disabled with a "You don't have permission to ..." tooltip. Two reusable pieces support this: `checkPermissions(ability, checks)` turns a set of checks into a boolean map a loader returns to the client, and `PermissionButton` / `PermissionLink` disable the underlying control and show a tooltip when a permission flag is false. ## Behaviour No change in the default configuration: permissions are permissive, so every control stays enabled and every route behaves as before. The checks only take effect when an RBAC plugin is installed. This also makes role assignment on invite-accept non-fatal, so a failure there cannot block joining an org. Verified with `pnpm run typecheck --filter webapp`; `checkPermissions` has unit tests. |
||
|
Daniel Sutton
|
d34b699950 |
fix(webapp): capture Prisma infra errors and obfuscate leaked messages (#3960)
## Summary Prisma infrastructure failures (P1xxx-class: database unreachable, timed out, connection dropped, engine init/panic) carry the database hostname in their `.message`. This captures them centrally for observability and ensures they never reach API clients verbatim. ## Design A `$allOperations` client extension on the writer and replica clients logs infrastructure errors with the originating model and operation, then rethrows the **original** error unchanged — call sites that branch on `error.code` (unique-violation idempotency, not-found handling) and transaction retries keep working. Only infrastructure errors are logged; routine query/validation errors (P2xxx) are left alone. `$allOperations` can't see the transaction boundary (`$transaction` is a client method, not an operation), so infrastructure errors surfacing from `$transaction()` without a Prisma code — e.g. `PrismaClientInitializationError` — are logged separately at the transaction wrapper, where the existing coded-error path would otherwise miss them. `clientSafeErrorMessage()` swaps an infrastructure error's message for `"Internal Server Error"` at the API routes that previously returned `error.message` raw. Status codes, headers, and every non-infrastructure message are unchanged. ## Test plan - [x] P2002 / P2025 rethrow with code intact and are not logged - [x] Statement errors inside `$transaction` keep their code (retry logic intact) - [x] Raw queries wrapped without crashing on the undefined model - [x] A genuine connectivity failure is logged with model/operation/code - [x] `clientSafeErrorMessage` obfuscates infra messages, preserves all others - [x] `pnpm run typecheck --filter webapp` (12/12) ## Note Overlaps with #3391 (Prisma 7 migration) on `apps/webapp/app/db.server.ts` — coordinate rebasing. |
||
|
nicktrn
|
6bdf800a11 |
feat(clickhouse): replicate run plan type to task_runs_v2 (#3978)
Replicates `TaskRun.planType` into the `task_runs_v2` ClickHouse table so run analytics can group by plan type. Adds a `plan_type` column (goose migration `033`, `LowCardinality(String)`), the replication insert mapping, and the matching schema/column/type entries - same shape as the recent `region` addition. Write-once at trigger, so it just rides along on existing replicated rows. Internal analytics only; not exposed in the Query API. |
||
|
nicktrn
|
7aa871f37b |
feat(webapp): plan-aware compute migration (#3957)
Adds an opt-in mechanism to route a configurable percentage of organizations onto the compute (MicroVM) backing of their region at trigger time, without changing their stored region settings. Routing is gated by three global feature flags - `computeMigrationEnabled`, `computeMigrationFreePercentage`, `computeMigrationPaidPercentage` - plus a per-org `computeMigrationEnabled` override that wins in both directions. A region's compute backing is resolved from a new `WorkerInstanceGroup.region` column: a container group and its MicroVM group share one geo `region`, so the migration swaps the resolved worker queue to the backing group's queue. Orgs are bucketed deterministically by id, so ramping a percentage down keeps a strict subset rather than reshuffling, and a region with no compute backing is never touched. Everything is off by default - behaviour is unchanged unless the flags are set. The flags and the worker-region groups are read on the trigger hot path from in-memory snapshots rather than the database: a small `createReloadingRegistry` helper loads each at startup and refreshes them on an interval, so no per-trigger query is added and a percentage or kill-switch change propagates within the reload interval. A cold replica whose snapshot hasn't loaded yet reads as not-migrated (the container path) and self-corrects on the next load - the same cold-start contract as the datastore / LLM-pricing registries, with a `reloading_registry_loaded` metric so a never-loaded registry is alertable. The same migration decision is consulted at deploy-time template creation so a migrated org gets a compute template built ahead of its first run. This runs in shadow mode (best-effort, never fails the deploy) by default, or - when the `computeMigrationRequireTemplate` flag is on - in required mode, built synchronously at deploy so the first run never builds on-demand and template errors surface at deploy time. So operators keep "which runs ran where" while customers only see geography: the run's actual worker queue is stored raw, and the geo region is stamped separately on `TaskRun.region` (and a new ClickHouse `region` column) at trigger time. Read surfaces - the dashboard, the API, and the Query/Logs page - show the geo region, falling back to the worker queue for runs written before the column existed. Minor follow-ups left out of scope: the percentage flags render as text inputs on the admin flags page (the catalog UI has no numeric control type yet), and `createReloadingRegistry` could later gain pub/sub for sub-second cross-replica propagation if the reload interval proves too slow. |
||
|
Oskar Otwinowski
|
cf4aa7e918 |
fix(webapp): Vercel env var sync rejecting batches containing only reserved keys (#3966)
Fix Vercel onboarding wizard to properly filter out reserved TRIGGER_ env vars |
||
|
Katia Bulatova
|
85d93ffe0e |
perf(webapp): skip queue search count (#3925)
### Summary Queue searches previously executed both a count query and a page query with identical filters. This PR switches filtered searches to `hasMore` pagination, removing the extra count query while preserving existing search behavior. ### Testing cd apps/webapp && pnpm run test ./test/queueListSearch.test.ts --run passes ### Changelog Improve filtered queue search performance. |
||
|
Eric Allam
|
52320679ab |
fix(webapp): stop locked-version triggers failing on stale replica reads (#3930)
## Summary `triggerAndWait` (and other locked-version triggers) could intermittently fail with `Task '<id>' not found on locked version '<version>'` for a task that was registered on that version. The failures came in bursts and recovered on their own, so a retry minutes later would succeed. ## Root cause For a locked-version trigger, the queue resolver looks up the task's `BackgroundWorkerTask` metadata from the read replica (behind a Redis cache). On a cache miss it queried the replica, and a `null` result was treated as "task not registered" and turned into a non-retryable 422. A read replica can return an empty result for a row that already exists on the primary, so a momentarily-behind replica produced a false negative even though the locked worker (resolved on the primary in the same request) clearly had the task. ## Fix On a cache miss, when the replica returns no row the resolver now re-checks the primary before concluding the task is missing. If the primary has the row it is used (and the cache is back-filled); the error fires only when the primary genuinely lacks it, which is the only case where the 422 is correct. The extra read happens on the cache-miss-and-replica-empty path only, so the hot path is unchanged. Verified with a unit test (replica stub vs. real primary) and end-to-end against a local streaming replica with replication paused to reproduce the stale read. TRI-10868 |
||
|
Matt Aitken
|
f48c89752c |
perf(webapp): parallelize streaming batch-item ingest (#3777)
## Problem
The item-streaming endpoint of the two-phase batch API (`POST
/api/v3/batches/:batchId/items`) processed streamed items strictly
sequentially. For a batch of many large payloads, each offloaded to
object storage inline, this serialized N object-store round-trips inside
a single request and could exceed Node's default `server.requestTimeout`
(300s). The webapp then returned `408`, which the SDK reads as `408
terminated` and retries up to 5 times, turning a slow ingest into a
failure that takes tens of minutes to surface.
## Fix
Ingest now runs through `p-map` over the NDJSON async iterable with
bounded concurrency (`STREAMING_BATCH_INGEST_CONCURRENCY`, default 10):
- `p-map` pulls lazily from the stream, so at most `concurrency` items
are read and in-flight at once. Peak memory stays bounded to roughly
`concurrency × STREAMING_BATCH_ITEM_MAXIMUM_SIZE` and request-body
backpressure is preserved.
- Set the env to `1` for fully sequential ingestion (escape hatch).
## Why this is safe (ordering and idempotency unchanged)
- Ordering derives from each item's index (enqueue `timestamp =
batch.createdAt + index`), not enqueue order.
- Dedup is atomic per index in `enqueueBatchItem`.
- The NDJSON parser now stamps oversized-item markers with their emit
position, removing the consumer's sequential `lastIndex` assumption (the
only order-dependent bit).
- The count-check and conditional-seal path is untouched.
## Scope
This speeds up every batch ingested through the streaming endpoint, not
just large-payload batches. Each item does a per-item Redis enqueue
regardless of size, and those now overlap. Large payloads benefit most
because they add an object-store offload round-trip on top of the
enqueue.
## Verification
Added an integration test (`streamBatchItems.test.ts`) that drives the
real service against Postgres + Redis + RunEngine and times a 150-item
batch at increasing concurrency. Object-store offload is modelled as a
fixed per-item latency (local round-trips are too small to compare
meaningfully):
```
runCount=150
large payloads (10ms/item offload):
concurrency=1 1739ms
concurrency=10 192ms (9.1x faster)
concurrency=50 57ms (30.7x faster)
small payloads (Redis enqueue only, no offload):
concurrency=1 90ms
concurrency=10 24ms (3.7x faster)
```
The test asserts correctness at every concurrency (all items accepted,
sealed, enqueued exactly once), that parallel ingest beats the
sequential floor, and that the small-payload case is strictly faster
than sequential, so the win is not specific to large payloads.
Also exercised end-to-end over real HTTP against a local server: a
20-item batch (12MB body) ingests and seals, a re-stream of the sealed
batch returns `sealed: true` with zero re-accepted items (idempotent
retry), and an oversized item still seals at its correct index.
Existing coverage stays green: concurrent ingest of a 100-item batch,
in-flight processing never exceeding the configured concurrency,
concurrent dedup on streaming retry, and emit-position marker indexing.
## Follow-ups (not in this PR)
- SDK pre-offload of large item payloads (send `application/store` refs
instead of raw blobs) to remove object-store work from the request hot
path and shrink the request body.
- Optional `server.requestTimeout` bump as a safety net.
## CI fix
Added `.github/workflows/codeql.yml` to replace GitHub's automatic
("dynamic") CodeQL scanning. The dynamic setup was failing to upload
SARIF results because the auto-generated `GITHUB_TOKEN` lacked the
`security-events: write` permission. The explicit workflow grants that
permission at the job level and pins all actions to commit SHAs,
consistent with the repo's security conventions.
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
- Integration test (`streamBatchItems.test.ts`) validates correctness
and performance at concurrency 1, 10, and 50 for both large and small
payloads.
- End-to-end verified over real HTTP: 20-item/12MB batch ingests and
seals, idempotent retry returns `sealed: true`, oversized item seals at
correct index.
---
## Changelog
Streaming batch ingest now processes items with bounded concurrency
instead of one at a time, so batches of many large payloads ingest far
faster and no longer time out. Concurrency is configurable via
`STREAMING_BATCH_INGEST_CONCURRENCY` (default 10); set it to 1 for fully
sequential ingestion.
---
## Screenshots
_[Screenshots]_
💯
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Eric Allam
|
954ee5c572 |
fix(webapp): deliver realtime changes with current content when the read replica lags (#3910)
## Summary When the realtime runs feed (the backend behind the `realtimeBackend` feature flag) hydrates a change from a Postgres read replica, the read can race the replica's apply of the very write that triggered it. The delivered row then carries the previous change's content, and an isolated final change (for example a last `metadata.set` before a run goes quiet) is not corrected until the roughly 20 second backstop poll. Measured against a replica with deliberate apply delay, every delivery trailed exactly one change behind and a final change stranded for the full backstop interval. ## Fix Publishers stamp each change record with the committed row's `updatedAt`, taken from writes they already perform, so the stamp costs no extra queries. The router delays its wake hydrate until the replica's measured lag has passed, anchored to that timestamp: a record that has already spent longer than the lag in transit is hydrated immediately, so only the racing leading edge ever waits. After hydrating, a tripwire compares each row against its record's watermark. Still-stale rows are withheld and retried briefly, and each detection feeds the lag estimate. If retries run out, the rows are delivered anyway (liveness over freshness) and follow-up re-hydrates emit the fresh version through the normal working-set diff once the replica catches up, with the backstop as the terminal net. Replica lag is sampled reader-side only, and only while feeds are active. Aurora reports live lag via `aurora_replica_status()`; vanilla Postgres can only report "caught up or not" (mid-apply lag is not honestly measurable from a replica), so tripwire observations floor the estimate there. Deployments without a replica resolve to zero lag and skip the gate entirely. Tunables live under `REALTIME_BACKEND_NATIVE_REPLICA_LAG_*`, and `realtime_native.stale_hydrates` plus `realtime_native.replica_lag_estimate_ms` make replica health observable. Two adjacent fixes: a metadata update that writes nothing no longer publishes a change record, and buffered parent and root metadata operations now publish when the flusher writes them, so those changes wake live feeds instead of waiting for the backstop. For local testing, `docker-compose` gains an opt-in `database-replica` service (compose profile `replica`) with a configurable `recovery_min_apply_delay`, which reproduces replica-lag behavior deterministically. With the gate disabled this rig reproduces the one-change-behind delivery exactly; with it enabled, deliveries arrive with current content at roughly the true replica lag, across write rates faster and slower than the lag itself. |
||
|
Eric Allam
|
8dc77c0ccd |
fix(webapp): only load env var values for displayed environments (#3903)
## Summary The environment variables page loaded every variable value in the project, unfiltered by environment. Archiving a preview branch does not delete its environment variable value rows, so projects that churn preview branches accumulate values forever, and every page view loaded all of them. On large projects this made the page loader take many seconds and stalled the server while deserializing the oversized result. ## Fix The presenter now loads the displayed environments first and filters the `values` relation to those environment IDs. That matches the display semantics exactly (per-user dev environments and active branch environments included), and the lookup is covered by the existing unique index on `(variableId, environmentId)`. Values in archived branch environments are no longer fetched at all. Covered by a new testcontainers test asserting that values from active environments (including branch environments) are returned while archived branch environments are excluded. |
||
|
Eric Allam
|
cc9eabd14d |
test(webapp): use relative fixture dates in runs cursor pagination tests (#3912)
## Summary `test/runsRepositoryCursor.test.ts` pinned its fixture runs to `createdAt = 2026-06-04T16:55:07Z`. `listRuns` applies the default 7 day window when no time filter is given, so the fixtures aged out of the window at 16:55 UTC on 2026-06-11 and all five tests started failing for every branch, regardless of what the branch changed. The tests were green on their own CI two days earlier because the fixtures were only five days old at the time. This switches the fixture base to a relative timestamp (one hour ago), so the fixtures stay inside the default window permanently. Verified the suite goes 5/5 green with this change on the same environment where the pinned dates fail 5/5. |
||
|
Eric Allam
|
187c0476c3 |
perf(webapp): shrink run trace loader payload and add trace span cap controls (#3906)
## Summary The run trace page loader serialized every span's raw OTel events (with full properties) into the response, even though the tree UI only renders the derived `timelineEvents` and the span detail panel refetches what it needs. On event-heavy traces that inflated both the loader payload and the server-side heap copies built per request. This PR keeps raw span events server-side and pairs that with a few related trace-view improvements: - A new optional `TRACE_VIEW_EMERGENCY_SPAN_CAP` env var (unset by default) clamps the trace summary and detailed trace summary span limits on both event store paths, including the public run trace endpoint, so operators can bound trace query sizes in one place without retuning the per-store limits. - The TreeView virtualizer resolved every rendered row with a linear scan over the whole tree (and `getNodeProps` did the same via `findIndex`); rows now resolve through memoized id lookup maps, which matters once traces reach tens of thousands of spans. - The run stream SSE lookup now applies the same organization membership scoping as the rest of the run page presenters, for consistency. Behavior is unchanged by default: the trace tree renders from the same `timelineEvents` it always has, and the new cap only takes effect when set. |
||
|
Daniel Sutton
|
7b4443a437 |
test(webapp): stop streamBatchItems container tests timing out on cold start (#3900)
Fixes an intermittent `Test timed out in 30000ms` in the `streamBatchItems` suite. Not a logic hang — the 30s budget covers container setup, and each case boots its own per-test Redis container + a full `RunEngine`, so under CI Docker contention a cold boot can cross 30s (which is why the failure moved between tests). - New `containerTestWithIsolatedRedisNoClickhouse` fixture (Postgres clone + per-test Redis, no ClickHouse) — this suite never uses ClickHouse, but the old fixture's auto `resetClickhouse` forced a ClickHouse boot + migration onto the cold-start test. - Raised `testTimeout` 30s → 120s, matching the run-engine convention for this footprint. |