triggerdotdev--trigger.dev

github-featured/triggerdotdev--trigger.dev

Author	SHA1	Message	Date
Matt Aitken	6bcd369ea1	feat(webapp,rbac): REQUIRE_PLUGINS=1 fail-fast for required plugin loads [TRI-9852] (#3734 ) ## Summary - `internal-packages/rbac/src/index.ts` — in `LazyController.load()`'s catch block, throw an Error when `process.env.REQUIRE_PLUGINS === "1"` instead of silently falling back. The throw is captured into the lazy controller's init promise, so it surfaces on the first method call. - `apps/webapp/app/routes/healthcheck.tsx` — `await rbac.isUsingPlugin()` after the DB ping. With `REQUIRE_PLUGINS=1` and a failed plugin load, the throw surfaces here and the healthcheck returns 500 → readiness probe fails → rollout is rolled back. Noop for self-hosters. - `.server-changes/require-plugins-fail-fast.md` — server-changes entry. - `internal-packages/rbac/src/require-plugins.test.ts` — 4 unit tests covering loader branching: unset → fallback, `=1` → throw, `forceFallback: true` wins, only exactly `"1"` enforces. - `internal-packages/testcontainers/src/webapp.ts` — adds `requirePlugins?: boolean` to `StartWebappOptions`. Implies `forceRbacFallback: false`. - `apps/webapp/test/healthcheck-require-plugins.e2e.test.ts` — e2e closes the loop: spawns a real webapp, hits `/healthcheck` via HTTP, asserts 500 with `REQUIRE_PLUGINS=1` and 200 without. ## Motivation Today the RBAC plugin loader catches any plugin-load failure (missing module, broken transitive dep, init throw) and silently returns the default fallback implementation. This is the correct behaviour for self-hosters who don't ship the plugin — but it's dangerous in deployments where the plugin is expected to load: an accidentally-missing or broken plugin would silently disable enforcement. `REQUIRE_PLUGINS=1` makes the loader fail loudly in those deployments. The variable name is intentionally plural and generic — future plugin contracts (audit logs, SSO) can read the same flag without renaming. Closes [TRI-9852](https://linear.app/triggerdotdev/issue/TRI-9852/require-plugins1-fail-fast-for-required-plugin-loads). ## Test plan - [x] `pnpm run test --filter @trigger.dev/rbac` — 38/38 tests pass, including the 4 new loader tests - [x] `pnpm run typecheck --filter webapp` — passes - [x] `pnpm run typecheck --filter @trigger.dev/rbac --filter @internal/testcontainers` — passes - [x] e2e test added (`healthcheck-require-plugins.e2e.test.ts`) — CI runs it via `e2e-webapp.yml`. Couldn't run locally (no Docker daemon up); CI has Docker provisioned. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-09 12:08:20 +01:00
Oskar Otwinowski	93532cdb99	feat(supervisor): forward per-run labels to the compute provider (#3821 ) Add an optional network_labels field to the internal compute client's create and restore request schemas and forward per-VM endpoint labels on both paths, so a restored VM keeps the same labels as a freshly-booted one. Mirrors the label the Kubernetes workload manager already sets on the run pod. --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>	2026-06-08 18:45:01 +02:00
Matt Aitken	ef04cc39ef	fix(webapp): use composite keyset cursor for run pagination (#3852 ) ## Problem `ClickHouseRunsRepository.listRunIds` / `listRuns` order results by the composite key `(created_at, run_id)`, but the cursor predicate cut on `run_id` alone: ```ts .where("run_id < {runId: String}", { runId: cursor }) .orderBy("created_at DESC, run_id DESC") ``` This is only sound when `run_id` lexicographic order matches `created_at` order. `run_id`s are cuids — only coarsely time-sortable — so when a burst of runs is created within a sub-second window, the two orders can diverge. When they do, the next-page predicate (`run_id < cursor`, where `cursor` is the last page element = the smallest `created_at`, not necessarily the smallest `run_id`): - re-includes rows already returned on a previous page (duplicates), and - skips rows it should have returned (silent data loss). For bulk replay this caused runs to be replayed more than once (replay has no idempotency guard). For the dashboard and the `runs.list` API it could silently repeat or skip runs at page boundaries. ## Fix Make the cursor predicate match the composite ordering: - Cursors now encode the full `(created_at, run_id)` key as an opaque URL-safe base64 token (`base64url({"c":<createdAtMs>,"r":"<runId>"})`), and the query cuts on the matching tuple — `(created_at, run_id) < (…)` forward / `> (…)` backward. - The `ORDER BY` is unchanged, so the query stays aligned with the table's primary key — no performance regression (the tuple range predicate is actually more index-friendly than `run_id <` alone). - Cursors are server-issued opaque tokens (the SDK only echoes `pagination.next` / `pagination.previous` back), so this needs no client/SDK update. Legacy cursors were the bare internal `run_id`; they're detected by decode failure (a cuid isn't a valid base64-wrapped JSON payload) and fall back to the old `run_id`-only predicate, so in-flight cursors keep working and drain naturally. New cursors also no longer expose a bare internal run id. - `listRunIds` is now the single cursor-aware list primitive: it returns `{ runIds, pagination: { nextCursor, previousCursor } }`, and `listRuns` builds on it (one place constructs cursors). Bulk actions consume the same method and advance by `pagination.nextCursor`, finishing when it's `null`. - `getTaskRunsQueryBuilder` now also selects `toUnixTimestamp64Milli(created_at) AS created_at_ms`, using a dedicated `TaskRunListQueryResult` schema. The shared `TaskRunV2QueryResult` stays `run_id`-only so the run-engine pending-version lookup (`getPendingVersionIdsQueryBuilder`, which selects only `run_id`) doesn't fail validation on a column it doesn't query. ## Tests New `runsRepositoryCursor.test.ts` (testcontainer-backed, real Postgres→ClickHouse replication): - forward pagination returns every run exactly once when `run_id` order is the reverse of `created_at` order (reproduces the duplicate/skip bug — fails on `main`; this walk-until-`nextCursor`-null-and-assert-complete is exactly the bulk action's iteration), - backward pagination round-trips to the previous page across a boundary, - legacy bare-`run_id` cursor still uses the old predicate (backwards compatibility). The existing `runsRepository` suites (part1–4) still pass; `part4`'s `count new runs with listRunIds` test was updated for the new `{ runIds, pagination }` return shape, and the `clickhouse` `taskRuns` query-builder snapshots were regenerated for the added `created_at_ms` column. ## Notes - Separate, pre-existing issue (out of scope, not introduced here): `listRuns`' backward display-slicing (`rows.slice(1, size+1)` when `hasMore`) has an off-by-one that can return a straddled page. Tracked separately. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-08 12:57:47 +01:00
nicktrn	f261ff2b85	chore(docker): tidy dev postgres + clickhouse images (#3859 ) Two small hygiene tweaks to dev-only images: - `docker/Dockerfile.postgres`: add `--no-install-recommends` to the partman install (leaner image, skips unneeded recommended packages). - `internal-packages/clickhouse/Dockerfile`: run the migration helper as a non-root user. Both are local-dev images (the `pnpm run docker` stack) - no impact on the published webapp image, prod, or self-hosting.	2026-06-07 12:22:56 +01:00
nicktrn	fa15438e42	perf(ci): speed up unit tests with LPT sharding + container scoping (#3855 ) Speeds up and de-flakes the unit-test suite: testcontainers booted once per vitest worker (per-test isolation kept only where a test runs background redis work that outlives it), a duration-weighted shard sequencer so each shard does roughly equal work, the slowest suites split, two genuine flakes fixed (`streamBatchItems` shared-redis leak; run-engine waits that relied on fixed sleeps), and transient DockerHub pulls retried. Timings (CI, per-shard wall): worst unit-test shard ~771s → ~294s; packages/webapp shards ~250-270s, most internal ~190-240s. All 25 shards green. A shard breaks down as ~70s fixed setup (install / image-pull / generate) + ~70s cold `^build` + the actual container tests. So the remaining cost is mostly the tests themselves plus that fixed setup. Next (separate, timings): - typecheck (~6m24s) — the slowest check overall; bound by full-graph `tsc`, not the TS version (a TS6 branch is still ~6m17s). The real lever is tsgo (the Go compiler). - Possible later: turbo CI caching could trim the ~70s cold build on warm runs, but it's conditional (cold runs rebuild anyway) and doesn't touch setup or test time — secondary. `cli-v3` e2e and `sdk-compat` are path-gated (don't run on test-infra changes) and already comfortably fast.	2026-06-07 12:00:32 +01:00
Eric Allam	97036fb741	feat(webapp,clickhouse): export run traces as log, markdown, or jsonl (#3851 ) ## Summary Adds a trace export to the run page. From the new Export trace menu you can copy a run's full trace to the clipboard as Markdown (for pasting into an AI assistant) or download it as a flat Log, a Markdown table, or JSON Lines. Internal engine-debug events are filtered out by default, and errors are surfaced inline with their message. ## Design The export streams events from the store to the gzipped response one at a time and never materialises the span tree, so a trace of any size exports with bounded memory and without stalling the server. Output is flat and chronological: each line carries its own `spanId ← parentSpanId`, so the hierarchy is reconstructable without nesting. Formats share a single streaming pipeline and are pluggable via `?format=log\|jsonl\|markdown`, so adding a format is an isolated change. ## Screenshots Export menu <img width="370" height="252" alt="trace-export-menu" src="https://github.com/user-attachments/assets/3d10304a-8c49-4606-b15d-2859b137419f" /> In context <img width="2400" height="1802" alt="trace-export-run-page" src="https://github.com/user-attachments/assets/46c80b30-303b-47c6-9ace-a2fb06f6cb61" />	2026-06-06 21:24:37 +01:00
nicktrn	707bf1adb4	ci: reduce unit test flakiness and shard re-run cost (#3844 ) A unit-test shard recently failed on a timing race rather than a real regression - a run-engine waitpoint test sleeps 1250ms waiting on a 1000ms timeout that's processed by a ~1000ms worker poll, so on a CPU-starved shard the margin evaporates and the whole matrix goes red. Because `fail-fast` defaults on, that one flake cancels the sibling shards, and the only recovery is re-running the entire matrix "just to be sure" - which is itself slow. This is the low-risk first pass at that pain: - `fail-fast: false` on the webapp and internal shard matrices, so one flaky shard no longer cancels its siblings. "Re-run failed jobs" now re-runs just the failed shard instead of the whole matrix. - CI-scoped `retry: process.env.CI ? 2 : 0` on the timing-sensitive packages (`run-engine`, `redis-worker`, `schedule-engine`). Flakes self-heal in CI; local runs stay at `retry: 0` so they still surface in dev. A stopgap until the timing tests are made deterministic. - `fetch-depth: 1` on the unit-test checkouts - they don't use git history, so the full clone was wasted setup time across ~20 jobs. - Reconcile the pre-pull image tags with what testcontainers actually pulls (`redis:7-alpine` -> `redis:7.2`, `ryuk:0.11.0` -> `ryuk:0.14.0`) and add `minio/minio:latest` to the webapp pre-pull. Otherwise those images pull unauthenticated at test time and risk Docker Hub rate-limit flakes (worst on fork PRs, where the authenticated pre-pull is skipped entirely). Deeper follow-ups - bigger runners, turbo remote cache, runtime-weighted sharding, and the real root-cause fix (container reuse / template-DB isolation + deterministic timing tests) - are tracked under TRI-10484.	2026-06-05 17:59:11 +01:00
nicktrn	16d59aa9e7	chore: harden webapp docker image (#3845 ) Hardens the webapp Docker image and adds a CVE scan of each published image. - Base image `bullseye-slim` → `bookworm-slim` (Debian 12), pinned by digest. Adds `apt-get upgrade` + `--no-install-recommends` + apt-cache cleanup across the build stages so OS packages are patched at build time. - Moves the `react-email` CLI to `devDependencies` in `internal-packages/emails` — only the `email dev` preview script uses it; the runtime render path is `@react-email/render` + `@react-email/components`. This also drops the bundled `esbuild` binary from the production image. - Bumps `goose` v3.26.0 → v3.27.1 and its Go builder image 1.23 → 1.26. - Adds a reusable Trivy image-scan workflow wired into `publish.yml`, so every published image (main builds and releases) is scanned for OS-package CVEs right after it's pushed to GHCR. Report-only (writes to the run summary), runs alongside the worker publishes so it never blocks a deploy. Verified locally: the image builds clean on the new base, and `@react-email/render` carries no `esbuild` dependency so email rendering is unaffected.	2026-06-05 17:52:42 +01:00
Eric Allam	aa9f1112ea	fix(database): include the Prisma CLI in production builds (#3843 ) ## Summary The Prisma CLI was missing from production builds of the webapp image, so anything that shells out to `prisma` at startup failed. The container entrypoint and the standalone migration step both run `prisma migrate deploy` / `prisma migrate status`, and those broke with `Command "prisma" not found`. ## Fix `prisma` was a `devDependency` of `@trigger.dev/database`. It had only been landing in the pruned `--prod` install as a side effect of pnpm auto-installing it as a peer of `@prisma/client`. A recent dependency change shifted peer resolution so prisma stopped being materialized into the production tree, and the CLI disappeared from the image. Moving `prisma` into `dependencies` of `@trigger.dev/database` makes the CLI an explicit part of production installs. It lands in the webapp image only: the separately deployed supervisor, coordinator, and provider images don't reach the database package in their production trees (`core` only `devDepends` on it, so it isn't transitive), so they're unaffected. Verified against a locally built production image: `pnpm --filter @trigger.dev/database exec prisma --version` now resolves the CLI and the schema engine instead of failing.	2026-06-05 13:08:49 +00:00
Eric Allam	359e2503c9	feat(database,webapp): add LlmModel pricing_unit column and admin selector (#3820 ) ## Summary Adds a nullable `pricing_unit` column to the LLM model registry's `llm_models` table, recording how each model is billed ("tokens", "characters", "images", "minutes", "requests", "free", "not_findable"). It lets pricing-coverage reporting exclude models that aren't priced per-token (image/video/audio models currently drag the "% priced" number down even though they can never carry a per-token price), and lays the groundwork for non-token pricing. The default model catalog is entirely per-token, so `seed` and `syncLlmCatalog` set `pricing_unit="tokens"` on those rows. The admin LLM model form (create + edit) and the admin API get a pricing-unit selector so admin-curated models can set it; existing rows can stay unset. Auto-discovered models get their unit from the model-registry pipeline, which lands separately. --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-06-03 15:08:36 +00:00
nicktrn	55d85d0b23	chore(emails): upgrade react-email to latest (#3819 ) ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Upgraded the email packages in `internal-packages/emails`: \| Package \| Before \| After \| \| --- \| --- \| --- \| \| `@react-email/components` \| `0.0.16` \| `1.0.12` \| \| `@react-email/render` \| `^0.0.12` \| `^2.0.8` \| \| `react-email` (CLI) \| `^2.1.1` \| `^6.5.0` \| \| `react-dom` \| _(missing)_ \| `^18.2.0` (now a required peer of render) \| Breaking change handled: `render()` is now async (`Promise<string>`) in `@react-email/render` v1+. Added `await` in the `aws-ses`, `smtp` and `null` transports. `EmailClient` and the webapp callers were already async and needed no changes. Verification: - `pnpm run typecheck --filter emails` ✅ - `pnpm run typecheck --filter webapp` (consumer of the `emails` package) ✅ - Before/after render comparison: rendered all 11 templates (magic-link, invite, welcome, alert-attempt/run/error-group, deployment-failure/success, mfa-enabled/disabled, bulk-action-complete) to HTML with both the old and new packages and compared them visually + via HTML diff. Output is visually identical. The only HTML changes come from upstream improvements: `<Body>` now wraps content in a `<table>`/`<td>` for better email-client compatibility, an `x-apple-disable-message-reformatting` meta tag was added, and CSS shorthand (e.g. `margin`) is now also emitted as longhand. No visual regressions; the `CodeBlock`/dracula theme, buttons, and row/column layouts all render correctly. No changeset or `.server-changes/` entry is added: `emails` is a private internal package (not under `packages/`), and there is no user-facing behavior change. --- ## Changelog Upgrade `react-email` and `@react-email/{components,render}` in `internal-packages/emails` to their latest versions and adapt the mail transports to the now-async `render()` API. --- ## Screenshots Rendered email templates before vs after the upgrade (visually identical): Before (`@react-email/components@0.0.16`, `render@0.0.12`)	2026-06-03 13:24:55 +01:00
nicktrn	a4d8c9f65f	chore(deps): update OpenTelemetry suite to 0.218.0 / 2.7.1 (#3810 ) Brings the OpenTelemetry packages up to the latest coherent release across the webapp and the published packages (`@trigger.dev/core`, the CLI, `@trigger.dev/sdk`) plus `internal-packages/{tracing,testcontainers}`: - `@opentelemetry/sdk-node` 0.218.0 - `@opentelemetry/core` 2.7.1 - `@opentelemetry/host-metrics` 0.38.3 We were already on the otel 2.x line, so this is a same-major minor move - the versions are pinned to `@opentelemetry/sdk-node@0.218.0`'s own declared dependency set so the experimental (0.2xx) and stable (2.x) packages stay coherent (mixing them is the usual otel breakage). One code change: otel 0.215 made `forceFlush()` a required method on `LogRecordExporter`, so `ExternalLogRecordExporterWrapper` (core's tracing SDK) gains a `forceFlush()` that delegates to the underlying exporter. Notable upgrades along the way: OTLP exporters can take a custom HTTP agent (connection pooling/keepAlive on the export path), HTTP request headers are captured at span creation, and core hot-path perf improvements in 2.6.1/2.7. `host-metrics` 0.37→0.38 is a clean upgrade. Patch changeset added for the three published packages. References projects are intentionally untouched. Verified: `@trigger.dev/core` / CLI / `@trigger.dev/sdk` build, webapp + `@internal/tracing` typecheck - all green.	2026-06-02 21:25:38 +01:00
nicktrn	d541caeb5e	feat(supervisor): wide events + warm-start trace propagation (#3669 ) Adds wide-event observability for the supervisor: one flat-keyed JSON line per dequeue iteration, workload-server route, and run socket lifecycle event. Events carry `trace_id` sourced from the inbound W3C traceparent plus `meta.run_id` and related identifiers, so they join across services by run. The outbound warm-start POST also forwards the inbound traceparent so the upstream receiver continues the same trace instead of minting a new one. Off by default behind `TRIGGER_WIDE_EVENTS_ENABLED`. With the flag off, no events are emitted, no ALS state is allocated, and the outbound warm-start request is unchanged — every call site was audited to confirm the off path is byte-identical to current behavior. Dequeue-path phase timings recorded under `phase.<name>.duration_ms`: `restore`, `warm_start`, `workload_create`. A `path_taken` extra distinguishes `restore` / `warm_start` / `cold_create` / `skipped_no_image`. Refs TRI-9480.	2026-06-02 21:11:01 +01:00
Katia Bulatova	4f8cf4cc63	feat(webapp): runs live updating ## Summary The Runs list now updates live without requiring a page refresh. Status changes and other run fields are updated in place while runs are executing. When new runs matching the current filters are created, a "New runs created" refresh button appears above the list. Root runs now show a live child-run status breakdown directly in the status tooltip. ### List live update - Visible runs update in place while they are still running. - A "New runs created" refresh button appears when new matching runs are detected. - Polling stops when all visible runs have finished and a refresh button is already shown. - Polling pauses when the browser tab is not visible. - Runs list status updates and new-run detection share a single runs/live polling path. ### Child-status tooltip - Root run tooltips now display a breakdown of child run statuses. - Child statuses are loaded when the tooltip opens (after a 400ms hover delay). - The tooltip stays up to date while child runs are still changing state. - Handles cases where child runs continue running after their parent run has completed, or have not yet been created. ### Supporting changes - Added hidden-tab awareness to polling. - Added safeguards around polling inputs (`runIds` deduping and limits). ## Test plan - [x] pnpm run typecheck --filter webapp passes - [x] cd apps/webapp && pnpm run test ./test/presenters/mapRunToLiveFields.test.ts --run passes - [x] cd apps/webapp && pnpm run test ./test/runsRepository.part2.test.ts --run -t "hasNewRuns" passes ### Manual smoke: - [x] Active runs update without a page refresh. - [x] A new matching run shows the refresh banner and the banner actions work as expected. - [x] Root run tooltips show live child-status updates and stop polling once child runs settle. --------- Co-authored-by: Ekaterina Bulatova <kathiekiwi@Ekaterinas-MacBook-Pro.local>	2026-06-02 19:26:46 +02:00
Dan	cd252801eb	feat: dashboard agent - package upgrades (#3793 ) 1. in webapp folder update ai-sdk to 6.x.x 2. update vitest to 4.xx	2026-06-02 10:46:34 +01:00
Daniel Sutton	4745754a7a	feat(webapp,run-engine): mollifier drainer replay + stale sweep + cancelled-run engine API (#3754 ) ## Summary The replay side of the mollifier: - `DrainerHandler`: reads buffered snapshots and replays them through `engine.trigger` to materialise PG rows. - `RunEngine.createCancelledRun`: new public method the handler uses to write CANCELED rows directly from snapshots (bypass queue + waitpoint, emit `runCancelled`). Tolerates the cjson empty-table tags edge case found during validation. - Drainer fairness: org → env rotation so a heavy env doesn't starve light ones in the same org. - Stale-entry sweep + telemetry + alertable gauge so a stuck/offline drainer surfaces in alerts. Both the drainer and sweep default-off; nothing fires unless flagged on (`TRIGGER_MOLLIFIER_DRAINER_ENABLED`, `TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`). Stacked on the trigger-time decisions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierDrainerHandler.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\` passes - [x] \`pnpm run test --filter @internal/run-engine src/engine/tests/createCancelledRun.test.ts\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/drainer.test.ts\` passes --- ## Ship-gate follow-up fix Drainer writes SYSTEM_FAILURE on max-attempts exhaustion. Adds an `onTerminalFailure` callback on `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `MAX_ATTEMPTS` on a retryable PG error (previously `buffer.fail()` was called with no row written → silent data loss). The callback runs before `buffer.fail()` on every terminal path (non-retryable AND max-attempts-exhausted), and re-throwing a retryable error from the callback causes the drainer to requeue rather than fail. Bumps `@trigger.dev/redis-worker` to a minor changeset (additive option + new exported types). Includes 5 unit tests covering both terminal causes plus the requeue-on-retryable-callback-failure path and no-callback back-compat. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-01 13:20:05 +01:00
Eric Allam	9211032733	chore(database): drop unused TaskRun status composite index (#3743 ) ## Summary Drops the `TaskRun_status_runtimeEnvironmentId_createdAt_id_idx` index from the `TaskRun` table. After #3742 gated the legacy `WAITING_FOR_DEPLOY` drain to V1-engine workers only, this index sees zero scans on both writer and reader replicas. Removing it cuts index maintenance on every `TaskRun` INSERT/UPDATE. ## Why The index existed to support `WHERE status = X AND runtimeEnvironmentId = Y` queries from `ExecuteTasksWaitingForDeployService`, which is V1-only and no longer triggered on V2 deployments. A code grep across `apps/webapp` and `internal-packages/run-engine` confirmed no V2 production query uses this access pattern — every other `status:` filter on `TaskRun` is paired with `id`/`friendlyId`/`parentSpanId` and uses a different index. Dropping it also unlocks HOT updates on the dequeue path. The dequeue `UPDATE` modifies `status` (`QUEUED` -> `DEQUEUED`), and `status` is the leading column of this index — its presence blocked HOT eligibility for every `TaskRun` UPDATE. With the index gone, dequeue UPDATEs can become HOT, reducing WAL bytes and removing the B-tree page contention on this index's right-edge leaves. Uses `DROP INDEX CONCURRENTLY` to avoid blocking writes during the drop. ## Sequencing Should only ship once #3742 has soaked long enough to confirm the index is genuinely cold (24h+ of zero scans on `pg_stat_user_indexes`).	2026-06-01 09:37:37 +01:00
Eric Allam	61ca40b4b1	perf(run-engine,webapp): look up PENDING_VERSION runs via ClickHouse (#3707 ) ## Summary When a background worker registers, the engine resolves runs that were queued before the worker was ready (status `PENDING_VERSION`). That lookup used to scan a Postgres status index on `TaskRun`. Move it to ClickHouse: query candidate run ids from `task_runs_v2`, then refetch the actual rows from Postgres by primary key with a `status = 'PENDING_VERSION'` guard for idempotency. ## Design The lookup is a pluggable interface on the run engine (`PendingVersionRunIdLookup`). The webapp wires a ClickHouse-backed implementation through the org-scoped `clickhouseFactory` using a new `"engine"` client type, configured by `RUN_ENGINE_CLICKHOUSE_*` env vars. The URL falls back to `CLICKHOUSE_URL` when unset, so self-hosted deployments don't need new config to keep working. When the lookup returns no candidates, one bounded retry is scheduled ~5s later to cover ClickHouse replication lag against `task_runs_v2`. The Postgres status guard on both the candidate refetch and the inner `updateMany` prevents double-promotion when a retry races with a concurrent deploy. Tests cover three existing PENDING_VERSION cases via a small Postgres-backed test adapter; new ClickHouse-backed integration tests will follow.	2026-05-22 17:18:41 +01:00
Eric Allam	0d4891a5f2	perf(database): drop unused TaskRun(scheduleId, createdAt) index (#3706 ) ## Summary Drops the unused composite Postgres index `TaskRun_scheduleId_createdAt_idx`. The schedule list view reads from ClickHouse, so this index served no Prisma query while still being maintained on every `TaskRun` INSERT/UPDATE. Removing it reduces write amplification on the primary database. Sibling to the prior drop of `TaskRun_scheduleId_idx` and the earlier removal of the `TaskRun.scheduleId` foreign key — all stemming from migrating schedule-aware reads to ClickHouse. ## Verification - Sampled `pg_stat_user_indexes` for `TaskRun` over multiple hours — zero scans against this index. - Grepped the codebase for any Prisma query filtering `TaskRun.scheduleId` — none found. All schedule-aware listing routes through `clickhouseRunsRepository`.	2026-05-22 16:34:00 +01:00
Matt Aitken	71d98b4e6b	Support for org-scoped ClickHouse (#3333 ) Added `OrganizationDataStore` which allows orgs to have data stored in specific separate services. For now this is just used for ClickHouse. When using ClickHouse we get a client for the factory and pass in the org id. Particular care has to be made with two hot-insert paths: 1. RunReplicationService 2. OTLPExporter --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-05-22 14:43:03 +01:00
Eric Allam	d343727021	fix(webapp,sdk): keep chat.agent snapshots on one object store (#3679 ) (`OBJECT_STORE_BASE_URL`) and a named protocol provider (`OBJECT_STORE_DEFAULT_PROTOCOL=s3`), chat.agent session snapshot writes landed in the named provider but reads fell through to the default — so the recovery boot couldn't find the snapshot it had just written. After a mid-stream cancel, the missing snapshot triggered a fallback replay path that dropped the user's follow-up message, leaving the chat stuck in `submitted` indefinitely. Fix: - New `/api/v1/sessions/:id/snapshot-url` route handles PUT + GET symmetrically — both prefix unprefixed keys with `OBJECT_STORE_DEFAULT_PROTOCOL` so they always round-trip through the same store. - `Session.chatSnapshotStoragePath` persists the resolved URI on first write so future protocol changes don't strand existing snapshots. Reads prefer the stored URI and fall back to the computed default for pre-column sessions. - SDK calls `createChatSnapshotUploadUrl` / `getChatSnapshotUrl`; the generic v1/v2 packets endpoints are unchanged. ## Test plan - [x] Configure local with two providers (R2 default + MinIO `s3` named) and `OBJECT_STORE_DEFAULT_PROTOCOL=s3`. - [x] Reproduce hang: send a message, cancel mid-stream, send another — without the fix it hangs in `submitted`; with the fix it streams. - [x] Snapshot lands in the `s3`-protocol bucket and `Session.chatSnapshotStoragePath` is set after first write. - [x] SDK unit tests pass; webapp typecheck passes.	2026-05-20 20:22:59 +01:00
Eric Allam	aec7e0a93d	perf(webapp): index EnvironmentVariableValue.environmentId (#3675 ) Env-var lookups via `GET /api/v1/projects/:projectRef/envvars/:slug/:name` run a Prisma `findMany` on `EnvironmentVariableValue` filtered by `environmentId` + `isSecret`. The only existing indexes are the primary key and a unique on `(variableId, environmentId)`, so `environmentId` is never the leading column — the planner falls back to a Parallel Seq Scan over the whole table to find what is, in practice, a handful of rows per environment. Two changes: - Add a btree index on `EnvironmentVariableValue(environmentId)` so the planner switches to an index scan. The composite `(variableId, environmentId)` unique stays in place; the new index is purely additive. - Route the `findMany` inside `getEnvironmentWithRedactedSecrets` through the read replica via a new `replicaClient` constructor param on the repository (defaulting to `$replica`, mirroring how `prismaClient` defaults to `prisma`). Writes and read-after-write methods stay on the primary. ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] Confirm `EXPLAIN` plan flips from Parallel Seq Scan to an index scan - [ ] Existing env-var route tests still pass	2026-05-20 13:43:44 +01:00
Eric Allam	6c9f1f197e	chore: parameterize docker host ports and wire s2-lite by default (#3642 ) ## Summary Two papercuts new contributors hit running this repo locally: 1. Fresh clones default to v1 (Redis-only) realtime streams, so Sessions and `chat.agent` error with `"S2 configuration is missing"`, even though the `s2` service is already in `docker/docker-compose.yml` and pre-seeds a `trigger-local` basin. Wire `REALTIME_STREAMS_S2_` to it in `.env.example` so the new-contributor flow just works. (Also drop the s2 healthcheck: the image is distroless, so the `wget` check always reports unhealthy.) 2. Two clones can't both run `pnpm run docker` because ports, project name, and container names are all hardcoded. Parameterize every host port as `${VAR:-default}`, drive the project name via `COMPOSE_PROJECT_NAME` (with a top-level `name:` field as the default), prefix container names with `${CONTAINER_PREFIX:-}`, and pass `--env-file .env` so compose reads the same root `.env` the webapp does. The "Running multiple instances side by side" block in `.env.example` lists every overridable knob. Also split the optional services (`electric-shard-1`, `ch-ui`, `toxiproxy`, `nginx-h2`, `otel-collector`, `prometheus`, `grafana`) into `docker-compose.extras.yml` behind a new `pnpm run docker:full` script. The core stack keeps everything the webapp actually needs to boot: postgres, redis, electric, minio, clickhouse + migrator, s2-lite. Defaults match every previous hardcoded value, so existing setups keep working without touching `.env`. ## Test plan - [x] `pnpm run docker` on a clean clone brings up the core services on the standard ports under the `triggerdotdev-docker` project name. - [x] Setting `COMPOSE_PROJECT_NAME=triggerdotdev-docker-alt` + the `_HOST_PORT` overrides in `.env` brings up a second stack alongside the default one with no port or container-name clashes. - [x] Webapp boots cleanly against the default `.env.example` values; `/healthcheck` returns 200, no S2 errors. - [x] s2-lite basin `trigger-local` accepts an append + read via the same REST endpoints the webapp uses. - [x] `pnpm run docker:full` brings up the optional services alongside the core ones in the same project.	2026-05-18 09:28:58 +00:00
Eric Allam	05d3ab1059	docs(clickhouse): require max+1 numbering and idempotent DDL (#3633 ) ## Summary Codify two rules for ClickHouse migration authors that came out of the 029/030 ordering incident on the TRI-9367 test cloud deploy: 1. Number files to `max(existing) + 1`, never slot in below the latest. Goose runs in strict mode in the cloud deploy pipeline and refuses to apply a missing version below the current version — slotting a file in below an already-applied number blocks the next deploy. 2. DDL must be idempotent (`ADD COLUMN IF NOT EXISTS`, `DROP COLUMN IF EXISTS`, `CREATE TABLE IF NOT EXISTS`, etc.) so a retry or out-of-order apply (`goose up --allow-missing` for local recovery, manual fixups) is a no-op rather than an error. ## Where the rules live - `internal-packages/clickhouse/CLAUDE.md` — full rules + example for migration authors (and AI agents writing migrations). - `.claude/REVIEW.md` — added a 🔴 finding under "What makes a 🔴 Important finding" so PR reviewers flag either fault as blocking. The existing migration files are left untouched; the idempotency requirement applies going forward. ## Test plan - [ ] Next ClickHouse migration PR uses `IF NOT EXISTS` / `IF EXISTS` forms - [ ] No new migration files numbered below an already-applied version on test/prod	2026-05-15 17:25:29 +00:00
Eric Allam	032b5a117a	fix(clickhouse): renumber task_kind migration 029 → 031 (#3631 ) ## Summary Renumber `029_add_task_kind_to_task_runs_v2.sql` → `031_add_task_kind_to_task_runs_v2.sql` to fix a deploy-blocking out-of-order migration, and make the DDL idempotent with `ADD COLUMN IF NOT EXISTS` / `DROP COLUMN IF EXISTS`. ## Root cause - Migration `030_create_sessions_v1.sql` landed on main on 2026-04-28 (PR #3417) and was applied to test cloud ClickHouse on a subsequent deploy. Current goose version on test ClickHouse: 30. - Migration `029_add_task_kind_to_task_runs_v2.sql` was authored later on 2026-05-10 as part of the Sessions primitive PR series (`be1a6cf8`). - The next test cloud deploy failed because goose strict-mode refused to apply a missing version before the current version: ``` goose run: error: found 1 missing migrations before current version 30: version 29: 029_add_task_kind_to_task_runs_v2.sql ``` ## Fix 1. *Rename to `031_` (next available number after 030). Goose now treats it as a new migration after 030 and applies it cleanly on test/prod where the column does not yet exist. 2. Make the DDL idempotent** (`ADD COLUMN IF NOT EXISTS`). The original 029 may have been applied in environments that ran goose with `--allow-missing` (e.g. some local dev databases) — those would have the column already, and the rename causes goose to see 031 as new and re-attempt the ADD. Idempotent DDL keeps that path safe. The `Down` mirrors with `DROP COLUMN IF EXISTS`. ## Test plan - [ ] Test cloud deploy (after this lands) successfully runs the ClickHouse migration step - [ ] `task_kind` column shows up on `trigger_dev.task_runs_v2` post-migration - [ ] Local environments that had previously applied 029 do not error on the next `goose up`	2026-05-15 17:17:17 +00:00
Eric Allam	be1a6cf8de	feat: Sessions primitive — durable run-aware streams + dashboard Adds Sessions, a durable, run-aware stream primitive that scopes session.in / session.out records to a session (not a single run). Records survive run boundaries; reconnect-from-last-event-id is built in. Server foundation: - New /realtime/v1/sessions/:session/:io/append + /records routes - sessionRunManager + sessionsRepository + clickhouseSessionsRepository - mintRunToken for short-lived per-session tokens - s2Append retry-with-backoff + undici cause diagnostics - /api/v[12]/packets/* exempt from customer rate limits - BackgroundWorker schema gains taskKind enum (TASK, AGENT, SCHEDULED) - TaskRun.taskKind column + clickhouse 029_add_task_kind_to_task_runs_v2 Core types: - new sessionStreams, inputStreams, realtimeStreams packages in @trigger.dev/core - session-streams-api / realtime-streams-api surface Sessions dashboard UI (the primitive's own viewer): - /sessions index + detail routes - SessionsTable, SessionFilters, SessionStatus, CloseSessionDialog - AGENT/SCHEDULED filter in RunFilters + TaskTriggerSource Includes the sessions-primitive changeset.	2026-05-14 13:12:36 +01:00
Eric Allam	e8ef374fe0	fix(webapp,run-engine): honor per-queue length cap on concurrency-key queues (#3558 ) ## Summary Queues that use concurrency keys can no longer bypass the per-queue length cap, and the "Queued \| Running" columns in the dashboard now show the true total across all CK variants instead of 0. The cap and the dashboard both relied on `ZCARD` of the base queue key, but CK-keyed runs live under `<base>:ck:<variant>` keys. Any queue that used concurrency keys read 0 — letting a single CK variant grow unbounded past the user's configured cap. ## Fix Two per-base-queue counters are maintained inside the CK Lua scripts: `<base>:lengthCounter` and `<base>:runningCounter`. Non-CK enqueue/dequeue paths are untouched. Counters are lazy-initialized the first time a CK enqueue (or nack) lands on a queue: the Lua script sums `ZCARD` across the variants tracked by `ckIndex`, sets the counter, then `INCR`s. Pre-existing CK backlog on already-populated queues is captured automatically — no batch migration required. `INCR`/`DECR` is gated on `ZADD`/`SADD` returning 1 (a new entry vs an idempotent no-op), so duplicate enqueues or re-dequeues don't inflate the counter. The counter is `SET` with a 24-hour TTL on init. `INCR`/`DECR` do not extend the TTL, so the counter expires daily and the next CK operation re-seeds it from `ckIndex`. This bounds any drift that accumulates during the rolling-deploy overlap window — where old (un-Tracked) and new (Tracked) webapp instances briefly coexist — to ≤24 hours, with no admin sweep or background reconciler needed. Read paths pipeline `ZCARD`/`SCARD` on the base key + `GET` on the counter and sum. A missing counter is treated as 0, so pure non-CK queues see the same answer as before. The counter-aware scripts ship alongside the originals with a `Tracked` suffix for rolling-deploy safety; a follow-up PR will drop the originals once this has rolled out. ## Test plan - [ ] `pnpm run test --filter @internal/run-engine` — 116 tests pass, including a new `ckCounters.test.ts` covering lazy init from pre-existing backlog, churn, floor-at-zero, the non-CK regression case, mixed CK + non-CK on the same base queue, idempotent re-enqueue (ZADD-already-exists), 24h TTL on the counter, and nack re-seeding after counter expiry. - [ ] Verified end-to-end against a live local environment: - Triggered 24 CK enqueues across 4 variants → `lengthCounter=16`, `runningCounter=8`, dashboard showed Queued=16 / Running=8 for the CK queue. - Set the env queue cap to 16, triggered 12 more enqueues → 8 succeeded, 4 rejected with `QueueSizeLimitExceededError`. - Deleted the counter on a queue with 31 messages already sitting in CK variants, triggered one more enqueue → counter materialized to 31 from the `ckIndex` sum, then INCR'd.	2026-05-12 18:37:19 +01:00
Matt Aitken	e4981d1b11	feat(webapp): consolidate auth path + add comprehensive auth tests (#3499 ) ## Summary Consolidates the webapp's authentication and authorization into a small set of route helpers, replacing the ad-hoc `requireUser` / `requireUserId` / `authenticatedEnvironmentForAuthentication` calls scattered across routes. Same security model, but the per-request flow (authenticate → authorize → load) now lives in one place per route family. Introduces a plugin seam (`@trigger.dev/plugins`) that lets the cloud build install a richer RBAC implementation without touching webapp code. The OSS fallback keeps the pre-RBAC permissive behaviour intact, so self-hosted deployments work unchanged. Adds a comprehensive end-to-end auth test suite that didn't exist before — 193 `it()` blocks (vitest reports ~199 after `it.each` expansion) covering API key, PAT and JWT auth across the public API surface, plus dashboard session auth for admin pages. ## Changes ### Plugin contract — `@trigger.dev/plugins` `RoleBaseAccessController` interface authoritative for both OSS (fallback) and cloud (enterprise plugin): - `authenticateBearer(request, { allowJWT? })` — API-key / public-JWT auth, returns env + ability - `authenticateSession(request, { userId, organizationId?, projectId? })` — dashboard auth, caller resolves `userId` from the session cookie and passes it in (no `helpers.getSessionUserId` callback — decouples the plugin host from session-cookie code) - `authenticatePat(request, { organizationId?, projectId? })` — PAT auth, returns identity + `lastAccessedAt` so the host can throttle the per-request update - `authenticateAuthorize` variants for the auth-and-check-in-one-call cases - `isUsingPlugin(): Promise<boolean>` — capability flag for UI / branching where plugin-present-ness matters; replaces the sentinel-string coupling that had `personalAccessToken.server` matching `"RBAC plugin not installed"` literally ### Dashboard auth (started, partial rollout) Admin and settings pages migrated to a unified `dashboardLoader` / `dashboardAction` helper that authenticates the session, runs an authorization check, and exposes the result to the route. Other dashboard routes still on the old pattern; remaining migration tracked in TRI-8730. Migrated routes: - `admin.` (14 admin / back-office / feature-flags / LLM-models / notifications / orgs / concurrency pages) - `_app.orgs.$organizationSlug.settings.team` - `_app.orgs.$organizationSlug.settings.roles` ### API / realtime / engine auth (complete for the migrated families) 71 routes migrated to a unified `apiBuilder` that centralizes Bearer / PAT / Public-JWT authentication and applies the per-route authorization check before the handler runs. Includes: - `api.v1.` and `api.v2.` and `api.v3.` — tasks, runs, batches, queues, prompts, deployments, query, sessions, waitpoints, packets, workers, idempotency keys - `realtime.v1.` — runs, batches, sessions, streams - `engine.v1.` — dev / worker-action protocols 29 routes still on the legacy `authenticateApiRequest` helpers — tracked as a post-deploy follow-up in TRI-9228. Multi-resource auth direction is now explicit at the call site via `anyResource(...)` (OR) and `everyResource(...)` (AND). Bare arrays no longer typecheck — fixes a class of bug where a JWT scoped to one resource could implicitly access others under OR semantics. PAT auth path consolidated: was three DB queries per request (legacy `authenticateApiRequestWithPersonalAccessToken` findFirst + `rbac.authenticatePat` join + `lastAccessedAt` update). Now one query in the steady state — plugin returns `lastAccessedAt`, host smart-skips the update via JS-side throttle when fresh. Side effect: action aliases preserved historic JWT scope semantics where the new model is stricter (e.g. a `write:tasks` JWT now also satisfies `trigger` / `batchTrigger` / `update` actions on the same resource — matched at the auth boundary, not in the route handler). ### Backwards-compat fixes The strict-match model regressed several real-world JWT shapes. Each preserved via explicit `anyResource(...)` entries in the route's authz block: - Batch retrieve routes (`api.v1.batches.$batchId`, `api.v2.`, `realtime.v1.batches.`) accept `read:runs` JWTs again (pre-RBAC literal-match superScope behaviour) - Runs list routes (`api.v1.runs`, `realtime.v1.runs`) accept type-level `read:tasks` / `read:tags` on unfiltered queries (matched the legacy `Object.keys` iteration semantic) - PAT/OAT auth shape normalized through `toAuthenticated` so all auth methods return the same slim `AuthenticatedEnvironment` (was: API-key returned the slim shape but PAT/OAT returned raw Prisma `Decimal` / no `orgMember`) - Scope `:` preservation in resource ids — `read:tags:env:staging` now correctly identifies the tag id as `env:staging`, not `env` ### Slim `AuthenticatedEnvironment` Extracted to `@trigger.dev/core/v3/auth/environment` — a structural shape independent of `@trigger.dev/database`. The plugin contract returns this; webapp consumers import from there; the cloud plugin (Drizzle) returns the same shape without Prisma's `Decimal` class leaking into the public surface. Lets internal-packages (run-engine, etc.) refer to `AuthenticatedEnvironment` without pulling Prisma in. ### Auth test suite (new — `.e2e.full.test.ts`) 193 e2e tests run against a real spawned webapp + Postgres (no mocks). Coverage matrix: - API key auth* — read / write / trigger / batchTrigger / deploy actions across runs, batches, deployments, prompts, queues, query, sessions, input-streams, waitpoints, tasks, idempotency keys; multi-key resources (a run carries batch / tag / task identifiers — auth must accept any matching scope) - Personal Access Token auth — comprehensive matrix: scope match, scope mismatch, missing scope, expired token, malformed token - Public JWT auth — sub-vs-URL environment resolution, expired JWTs, signature verification, scope checking, otu (one-time-use) token semantics, branch-environment signing-key fallback - Dashboard session auth — admin-only pages reject non-admins; per-action gating - Cross-cutting edge cases — revoked API key grace window, JWT cross-environment isolation, MissingResource branch behaviour ### Hygiene cleanups - Deleted dead `app/services/authorization.server.ts` (legacy `checkAuthorization` + types — no live consumers post-migration) and its orphaned test - Dropped the never-populated `scopes` field from `ApiAuthenticationResultSuccess` - `scheduleEmail` moved out of `email.server.ts` into its own module — breaks a `commonWorker → marqs/V1` import chain that was poisoning the auth test graph - OSS Roles page shows a deployment-aware empty state ("Roles aren't available in this self-hosted deployment" vs the plan-upsell copy) via `rbac.isUsingPlugin()` - Team action handler: explicit per-intent ability gates (`manage:billing` for purchase-seats, `manage:members` for set-role + remove-member with self-leave carve-out) ### Cross-repo coordination All public-package contract changes paired in `triggerdotdev/cloud#763` (rbac-packages branch) — the enterprise plugin implements the same `RoleBaseAccessController` interface against Drizzle. ## Test plan - [x] `pnpm run typecheck --filter webapp` clean - [x] `pnpm --filter webapp exec vitest run --config vitest.e2e.full.config.ts` — 193/193 pass (requires Docker for testcontainers) - [x] Spot-check an authed API endpoint with a valid + invalid API key against a local stack - [x] Spot-check the migrated admin pages render and gate non-admins --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-12 17:16:20 +01:00
Eric Allam	2301ed608c	refactor(run-engine): make taskIdentifier optional on run-queue messages (#3559 ) ## Summary Make `taskIdentifier` optional on the run-queue message schema. No behavior change in this PR; readers continue to accept payloads that include the field. A separate change will stop writing it on the wire to shrink the per-run payload that lives in Redis while runs wait to be dequeued. ## Design The field is written into every payload at enqueue time but no consumer reads it back on the dequeue path. Both the run-engine and supervisor derive `taskIdentifier` from the loaded `TaskRun` row instead. Relaxing the schema first means readers tolerate payloads that omit it, so the writer-side change can ship without producing schema-parse errors during a rolling deploy. `projectId` is left required: `WorkerQueueResolver.#getOverride` reads it for project-scoped runtime worker-queue overrides. ## Test plan - [x] `pnpm run typecheck --filter @internal/run-engine` - [x] `pnpm run typecheck --filter webapp` - [x] `pnpm run test ./src/run-queue/tests/enqueueMessage.test.ts ./src/run-queue/tests/workerQueueResolver.test.ts --run` (28/28 passing)	2026-05-12 11:03:34 +01:00
Eric Allam	5f1a3e1653	refactor(run-engine): route TTL expiration through the batch path only (#3554 ) ## Summary TTL expiration on queued runs was being scheduled twice: once via a per-run `expireRun` worker job (the original implementation) and once via the batch TTL system (added more recently). Both paths attempt to flip the same run to `EXPIRED`. The per-run job almost always won the race, leaving the batch consumer to observe runs already expired by the older path. This collapses TTL expiration onto the batch path so every queued TTLed run goes through a single Redis-backed sorted set + batch consumer instead of also getting its own scheduled redis-worker job. ## Design `engine.trigger` and `delayedRunSystem.enqueueDelayedRun` no longer call `ttlSystem.scheduleExpireRun`. The remaining `enqueueSystem.enqueueRun({ includeTtl: true })` already adds the run to the TTL sorted set; `TtlSystem.expireRunsBatch` flips it to `EXPIRED` when the TTL fires. Delayed runs get the same coverage by passing `includeTtl: true` on their post-delay enqueue, so the TTL is armed from the moment the run enters the queue (matching how the old job behaved — `parseNaturalLanguageDuration` is evaluated at enqueue time). The new path explicitly does not re-expire runs once they have been allocated a concurrency slot. That is intentional: TTL is for runs that are queued and have never started. Once a run has a slot it is on its way to executing. ## Test plan - [x] `pnpm run test --filter @internal/run-engine ./src/engine/tests/ttl.test.ts` — 15 tests, including a new "Re-enqueued runs are not expired by TTL once they have started" that locks in the queued-and-never-started contract. - [x] `pnpm run test --filter @internal/run-engine ./src/engine/tests/delays.test.ts` — 5 tests, including "Delayed run with a ttl" which now also asserts the TTL is armed from queue-enter time, not `createdAt`. - [x] `pnpm run test --filter @internal/run-engine ./src/engine/tests/lazyWaitpoint.test.ts` — 12 tests. - [x] `pnpm run typecheck --filter @internal/run-engine`.	2026-05-12 07:58:42 +01:00
Eric Allam	a5ba406530	feat(webapp,redis): handle UNBLOCKED during ElastiCache role change (#3549 ) ## Summary When ElastiCache demotes a primary to replica — during a Multi-AZ failover or a vertical node-type change — the demoting primary issues an `UNBLOCKED` reply to any in-flight blocking commands (`BLPOP`, `BRPOP`, `BLMOVE`, `XREADGROUP ... BLOCK`, etc.) to clear them before the role flips. ioredis surfaces these as `ReplyError` to caller code. The shared `defaultReconnectOnError` added in #3548 only matches `READONLY` and `LOADING`. This extends it to `UNBLOCKED` so the disconnect-reconnect-retry cycle handles BLPOP-shaped errors the same way the existing two cases handle non-blocking-command errors. ## Fix ```ts export function defaultReconnectOnError(err: Error): boolean \| 1 \| 2 { const msg = err.message ?? ""; if ( msg.startsWith("READONLY") \|\| msg.startsWith("LOADING") \|\| msg.startsWith("UNBLOCKED") ) { return 2; } return false; } ``` Returning `2` tells ioredis to disconnect, reconnect, and re-issue the command. For a BLPOP that means a fresh BLPOP against the new primary instead of the `UNBLOCKED` error escaping to the caller. ## Test plan - [ ] CI green - [ ] Trigger a Multi-AZ failover or a vertical scale event on an ElastiCache replication group whose clients are running blocking commands and confirm no `UNBLOCKED` errors surface to caller code during the cutover.	2026-05-11 11:02:40 +01:00
Eric Allam	567e2a2c32	feat(webapp,redis): handle READONLY / LOADING during ElastiCache failover (#3548 ) ## Summary During an ElastiCache role swap (failover) or node-type change (vertical scale), the ioredis TCP/TLS connection stays open but the server starts answering with `READONLY` (the client is talking to a node that became a replica) or `LOADING` (node still loading data from disk). Without an explicit hook, those errors surface to caller code as `ReplyError` instances — every write op on the affected connection fails until the cluster fully cuts over. This PR adds `reconnectOnError` to every prod ioredis client so the disconnect + reconnect + retry cycle absorbs these errors and caller code never sees them. ## Fix ```ts export function defaultReconnectOnError(err: Error): boolean \| 1 \| 2 { const msg = err.message ?? ""; if (msg.startsWith("READONLY") \|\| msg.startsWith("LOADING")) return 2; return false; } ``` Returning `2` tells ioredis to disconnect, reconnect, and re-issue the failed command. After reconnect, DNS / SG state routes the new socket to a writable node. The helper lives in `@internal/redis` and is wired into both the shared `createRedisClient` (which covers RunQueue, schedule-engine, redis-worker, and every other internal-package consumer) and the direct `new Redis(...)` call sites in the webapp. V1-only marqs files are intentionally not migrated. ## Test plan - [x] `pnpm run typecheck --filter webapp` - [x] `pnpm run typecheck --filter @internal/run-engine` - [x] Verified end-to-end against a live ElastiCache vertical-scale event — caller-surfaced errors went from tens of thousands during the cutover window down to a handful per ioredis client - [ ] Confirm steady-state behavior unchanged after deploy	2026-05-11 07:17:07 +01:00
Matt Aitken	62e006617e	fix(cli): fail attempt on uncaught exception instead of hanging to maxDuration (TRI-9117) (#3529 ) ⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled Details When a Node EventEmitter (e.g. node-redis) emits an "error" event with no listener attached, Node escalates it to process.on("uncaughtException") in the task worker. The worker reported the error via the UNCAUGHT_EXCEPTION IPC event but did not exit, and the supervisor-side handler in taskRunProcess only logged the message at debug level — leaving the run() promise orphaned until maxDuration fired and producing empty attempts (durationMs=0, costInCents=0). The supervisor now rejects the in-flight attempt with an UncaughtExceptionError and gracefully terminates the worker (preserving the OTEL flush window) on UNCAUGHT_EXCEPTION. The attempt fails fast with TASK_EXECUTION_FAILED, surfacing the original error name, message, and stack trace, and falls under the normal retry policy. This mirrors the existing indexing-side behavior in indexWorkerManifest. Apply the same handling to unhandled promise rejections, which Node already routes through uncaughtException by default.	2026-05-06 19:35:43 +01:00
Eric Allam	386b4f65ff	feat(webapp): per-org S2 basin migration (#3516 ) ## Summary Move from a single shared S2 basin to per-org basins with retention tied to the org's billing plan. Stops S2 from deleting streams out from under live chat sessions when basin retention fires before the chat ends, and unlocks per-org cost attribution. OSS / s2-lite installs are unaffected: provisioning is gated by `REALTIME_STREAMS_PER_ORG_BASINS_ENABLED` (default `false`), and the read precedence falls back to the global basin env var when an entity has no stamped basin. ``` basin = run.streamBasinName ?? session.streamBasinName ?? env.REALTIME_STREAMS_S2_BASIN ``` ## Design Three nullable `streamBasinName` columns (`Organization`, `TaskRun`, `Session`) plus a provisioner that idempotently creates the basin and reconfigures retention on plan changes. The trigger and session-create paths stamp the org's basin onto new rows; the realtime read path picks the basin from the entity context. Admin routes back-fill existing orgs and force-reconfigure a single org. ## Test plan - [x] `pnpm run typecheck --filter webapp --filter @internal/run-engine` - [x] Backfill admin route end-to-end (provision + DB stamp + S2 basin config). - [x] Reconfigure on plan change (all retention tiers). - [x] chat.agent multi-turn drives streams into the per-org basin. - [x] Legacy fallback when entity has no stamped basin. - [x] Provisioner is a no-op when the flag is off.	2026-05-05 10:06:58 +01:00
James Ritchie	45ec23cc73	feat(webapp): app auto session logout (#3473 ) <img width="2284" height="2028" alt="CleanShot 2026-05-01 at 18 53 50@2x" src="https://github.com/user-attachments/assets/4f58cbb1-0168-40fb-a523-017f2ba625a1" /> ## Performance - Per-request DB hit: `getUserId` runs `getEffectiveSessionDuration` (User lookup + Org `aggregate`) on every authenticated request, including each fetcher poll. Consider caching the effective duration in the session cookie with a short TTL (e.g. 60s) and revalidating in the background. - Double session commit in `root.tsx`: `getUser` already runs the expiry check; then `commitAuthenticatedSessionLazy` commits the cookie again. Fine, but doubles `Set-Cookie` headers on every page load — worth a quick perf check. ## Correctness / Edge cases - Lazy backfill assumes a root.tsx hit first: users whose first post-deploy request is a fetcher/API route (`/resources/`) skip the backfill until they navigate to a page. Not a security hole, but `getUserId` could backfill itself for completeness. - No upper bound on `Organization.maxSessionDuration`: admin API accepts `1` second, which would instant-logout every member on next request. Add a `min(60)` (or `min(300)` to match the lowest user option) to the Zod schema. - No clock-skew tolerance: `isSessionExpired` is exact-millisecond. Multi-instance deploys with skewed clocks could log users out a few seconds early/late. Probably fine for the 5-min minimum, but worth noting. ## Security - Auto-logout audit log lacks IP/orgId: HIPAA forensics typically wants source IP and which org context. Currently logs only `userId` + path. IP isn't PII for audit purposes; orgIds help correlate. Add both. - Cookie `Max-Age` is 1 year regardless of user's setting: intentional (server-side `issuedAt` is the source of truth), but reviewers will ask. Add a one-line comment on the cookie config explaining why. ## API surface - `maxSessionDuration` is admin-PAT only: no in-app UI for org owners to set/change their own cap. If this is "Trigger staff sets it during HIPAA onboarding", say so in the PR description; otherwise add an org-settings UI. - Auto-submit dropdown has no confirmation: misclicking "5 minutes" immediately shortens the user's session window with no undo. Consider a save button or 3-sec undo toast. ## Schema / migration - `User.sessionDuration NOT NULL DEFAULT 31556952`: instant on PG 11+ (metadata-only), but call out in the PR description so reviewers don't worry about a table rewrite on the User table. - No DB-level constraint matching `SESSION_DURATION_OPTIONS`: if the option list changes, existing users keep orphaned values. The dropdown's tag-along behaviour hides this — fine for now, but if you ever drop an option you'll need a backfill. ## UX - Session expiry only fires on next request: an idle authenticated tab keeps showing UI past the cap (until SSE/polling catches it, ~60s). Add a client-side timer based on the user's effective duration that triggers a fetcher to `/account` or `/logout` at expiry. - No "you were signed out" message on logout*: users hitting their cap are bounced to `/` with no explanation. Was intentionally reverted in this PR — call that out so reviewers don't request it. ## Tests - Unit coverage on `sessionDuration.server.ts` is solid (215 lines). Missing: integration test for `getUserId` → expired session → redirect to `/logout`, and one for the loader's clamping fix (the most recent bug). Add at least the second one to lock in the regression. --------- Co-authored-by: Matt Aitken <matt@mattaitken.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-04 15:02:26 +01:00
nicktrn	ee3887a321	feat(webapp): configurable deploy template machine presets (#3492 ) The webapp's compute template creation hardcoded a single machine preset (`small-1x`) at deploy time, regardless of which presets a project actually uses. Tasks running on any other preset paid full cold-snapshot creation cost on first run. Two new env vars: - `COMPUTE_TEMPLATE_MACHINE_PRESETS` - CSV of preset names to build boot snapshots for during deploy. Defaults to `small-1x` so existing deploys don't change behavior. - `COMPUTE_TEMPLATE_MACHINE_PRESETS_REQUIRED` - CSV of presets whose failure fails a required-mode deploy. Defaults to the full `PRESETS` list. Optional preset failures are logged but don't block the deploy. The compute client now sends the multi-config request shape; the service evaluates per-preset outcomes against the required set and surfaces a combined failure message when a required preset fails. Both env vars are validated at boot via the env schema - unknown preset names or `_REQUIRED` entries that aren't a subset of `_PRESETS` fail loudly at startup rather than silently per-deploy.	2026-05-01 15:10:26 +01:00
Eric Allam	ac7177d61f	feat(schedule-engine): stop persisting per-tick schedule state (#3476 ) ## Summary Each scheduled-task tick previously issued 3 Prisma `UPDATE`s against `TaskSchedule.lastRunTriggeredAt`, `TaskScheduleInstance.lastScheduledTimestamp`, and `TaskScheduleInstance.nextScheduledTimestamp`. All three were pure denormalization — every value can be derived without persisting. After this PR `TaskSchedule` and `TaskScheduleInstance` become near read-only: writes happen only on schedule create / update / delete (rare admin actions), so the per-tick autovacuum churn on these hot tables disappears. ## Design The previous fire time travels forward through the schedule worker payload, not through the database. Concretely: - The `schedule.triggerScheduledTask` worker payload gains an optional `lastScheduleTime: z.coerce.date().optional()` field. - When the engine fires a schedule, it re-enqueues the next tick with `lastScheduleTime = scheduleTimestamp` (the just-fired time). - When the next tick dequeues, `payload.lastTimestamp` is sourced from `params.lastScheduleTime` directly. No DB round-trip, no cron-derivation drift across DST boundaries, no caveats around recently-edited cron expressions. `payload.lastTimestamp` keeps its `Date \| undefined` SDK shape. First-ever fires still report `undefined`, so customer `if (!payload.lastTimestamp)` first-run patterns keep working. For Redis jobs that were enqueued before this change (which lack `lastScheduleTime` in their payload), the engine falls back to `instance.lastScheduledTimestamp` once. Once those drain, the column is never read again. Revert is code-only; the columns stay in place and can be dropped in a follow-up once the rollout is stable. ## Files - `internal-packages/schedule-engine/` — engine refactor, `workerCatalog` schema field, `TriggerScheduleParams` extension, tests updated to assert on the worker-payload flow rather than DB readbacks. - `internal-packages/database/prisma/schema.prisma` — `/// @deprecated` triple-slash docstrings on the three columns. No migration. - `apps/webapp/app/presenters/v3/ScheduleListPresenter.server.ts` — drops the `lastRunTriggeredAt` Prisma select; "Last run" cell is approximated from the cron expression's previous slot, gated on `schedule.createdAt` so brand-new schedules show "–". UI is best-effort; the runs page is the source of truth. - `apps/webapp/app/v3/utils/calculateNextSchedule.server.ts` — adds a `previousScheduledTimestamp` helper for the UI cell above. Public API responses (`api.v1.schedules.`) already compute `nextRun` from cron and don't expose `lastTimestamp` — no public API change. - `references/scheduled-tasks/` — new reference project with declarative schedules at multiple cadences and three throw-on-fail validators (`first-fire-detector`, `interval-validator`, `upcoming-validator`) for E2E-verifying the worker-payload flow. Refs TRI-8891 ## Test plan - [x] `pnpm run typecheck --filter @internal/schedule-engine --filter webapp` - [x] `pnpm run build --filter @trigger.dev/core` - [x] `pnpm run test --filter @internal/schedule-engine` — integration test asserts first-fire `lastTimestamp === undefined`, second fire carries the previous fire's timestamp exactly. - [x] E2E against local webapp via `references/scheduled-tasks`: - Fresh schedules attached → all three deprecated columns stay `NULL` after multiple fires. - Redis payload at second fire contains `"lastScheduleTime":"<previous fire timestamp>"`. - `TaskRun.payload` and the every-minute task's returned output both confirm `lastTimestamp = null` on first fire and `lastTimestamp = <prev fire>` on second fire, exactly 60s apart. - All three throw-on-FAIL validators completed successfully on every non-first fire. - [x] Schedules REST API end-to-end (`POST` / `GET` / `PUT` / `activate` / `deactivate` / `DELETE`) — `nextRun` recomputed live from cron + tz on every response, no reads of deprecated columns.	2026-05-01 08:22:39 +01:00
Eric Allam	dac9c83bdc	chore(webapp,run-engine): downgrade boundary log noise to warn (#3462 ) ## Summary Several boundary catches and customer-input validation paths were logging at `error` level for failures the system already handles gracefully — disconnect on auth failure, return undefined, skip retries, etc. This batch routes them to `warn` (which stays in stdout) or counts them as OTel metrics, so visibility is preserved without surfacing them as alerts. ## Changes New helper / pattern: - `apiBuilder.server.ts` — `logBoundaryError(message, error, url)` inspects the inner error type at loader/action boundary catches; downgrades to `warn` for `AbortError`, `ServiceValidationError`, and `EngineServiceValidationError`. - `platform.v3.server.ts` — `platform_client.failures_total` OTel counter with `{function, kind}` labels; helper `recordPlatformFailure(fn, kind)` replaces the previous error-level logging across all `BillingClient` wrappers. Log-level downgrades: - `handleSocketIo.server.ts` — `Worker authentication failed` → warn (system disconnects on failure; refs TRI-8863) - `waitpointSystem.ts` — when `runStatus === "CANCELED"` in the suspended-without-checkpoint branch, skip the throw and warn instead (benign cancel-vs-resume race, nothing to resume) - `runAttemptSystem.ts` — `flushedMetadata` parse/validate failures → warn (customer-side data shape, system returns gracefully) - `batch-queue/index.ts` — final-attempt failures with `result.skipRetries` → warn (callbacks already opted out of retry, e.g. queue size limit hit) - `queryPerformanceMonitor.server.ts` — slow queries → warn (observability signal, not an application error) - `timeoutDeployment.server.ts` — deployment-state mismatch in the timeout job → warn (timeout-vs-completion race) Inner error preservation: - `waitpointCompletionPacket.server.ts` — `logger.error(uploadError)` before throwing the `ServiceValidationError` wrapper, so the underlying upload error stays visible. ## Why The pattern across all of these is the same: a boundary log treated any thrown/returned error as `error` regardless of cause, even when the cause was an expected, system-handled condition (client disconnect, customer quota, race condition, schema validation of customer data). That made the logs noisy and made it harder to spot real bugs. Where the underlying signal is still useful operationally (slow queries, billing call failures), we route it to OTel metrics with low-cardinality labels so dashboards and alerts can be tuned independently of error logs. ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run build --filter @internal/run-engine` - [ ] Trigger a run on hello-world and verify task lifecycle is unaffected - [ ] Cancel a suspended run and verify the cancel-while-suspended branch in `waitpointSystem.ts` returns `{status: "skipped"}` instead of throwing - [ ] Confirm `platform_client.failures_total` counter shows up in metrics with `{function, kind}` labels when the billing client errors	2026-04-29 10:00:22 +01:00
Eric Allam	c69e939c34	feat: Sessions - bidirectional durable agent streams (#3417 ) > ⚠️ Not released yet. This PR is the server-side foundation only. The SDK changes that customers will actually use (`chat.agent` migration, `chat.createStartSessionAction`, `useTriggerChatTransport` updates) live on a separate branch and ship together in an upcoming `@trigger.dev/sdk` prerelease. Until that prerelease is published, this surface is reachable only via direct HTTP. ## What this gives Trigger.dev users A new first-class primitive, Session, for durable, task-bound, bidirectional I/O that outlives any single run. Sessions are the run manager for `chat.agent` going forward, and they unblock anything else that needs "one identifier, many runs over time" with a stable channel pair the client can write to and subscribe to. ### Use cases unblocked - Chat agents that persist across many runs. One session per chat (keyed on your own `chatId` via `externalId`), turns 1..N attach to the same Session, the UI subscribes once and keeps receiving output as new runs take over. - Approval loops and long-running tasks with user feedback. The task waits on `.in`, the client writes to `.in`, the server enforces no-writes-after-close. - Workflow progress streams that live past the run. Subscribe to `.out` after the task finishes to replay history. - Resume-next-day flows. A session is a durable row, not a transient stream. Send a message a day later and the server triggers a fresh run on the same session. ### How it works (Session-as-run-manager) A Session row is task-bound (`taskIdentifier` + `triggerConfig` are required) and owns its current run via `currentRunId` + `currentRunVersion` for optimistic claim. Three trigger paths: 1. Session create — `POST /api/v1/sessions` creates the row and triggers the first run synchronously. 2. Append-time probe — `POST /realtime/v1/sessions/:session/in/append` checks if the current run is alive; if it has terminated (idle exit, crash, etc.), the server triggers a new run before processing the append. 3. End-and-continue handoff — `POST /api/v1/sessions/:session/end-and-continue`, called by the running agent, triggers a fresh run and atomically swaps `currentRunId`. Used by `chat.requestUpgrade()` for version handoffs. Every triggered run is recorded in the `SessionRun` audit table with a reason (`initial`, `continuation`, `upgrade`, `manual`). ## Public API surface ### Control plane - `POST /api/v1/sessions` — create. Idempotent on `(env, externalId)`. Triggers the first run, returns the session and a session-scoped public access token. Returns 409 if the upserted row is already closed. - `GET /api/v1/sessions/:session` — retrieve by friendlyId (`session_abc...`) or by your own externalId (server disambiguates by prefix). - `GET /api/v1/sessions` — list with filters (`type`, `tag`, `taskIdentifier`, `externalId`, derived `status` ACTIVE/CLOSED/EXPIRED, created-at range) and cursor pagination. Backed by ClickHouse. - `PATCH /api/v1/sessions/:session` — update tags / metadata / externalId. - `POST /api/v1/sessions/:session/close` — terminate. Idempotent, hard-blocks new server-brokered writes. - `POST /api/v1/sessions/:session/end-and-continue` — agent-only handoff to a fresh run. ### Realtime - `PUT /realtime/v1/sessions/:session/:io` — initialize a channel. Returns S2 credentials in headers so high-throughput clients can write direct to S2. - `GET /realtime/v1/sessions/:session/:io` — SSE subscribe. Supports Last-Event-ID resume and an opt-in `X-Peek-Settled: 1` header that fast-closes the stream when the upstream is already settled (`trigger:turn-complete`), eliminating long-poll wait on reconnect-on-reload paths. - `POST /realtime/v1/sessions/:session/:io/append` — server-side appends. - `POST /api/v1/runs/:runFriendlyId/session-streams/wait` — runs wait on a session stream as a waitpoint, with a race-check to avoid suspending if data already landed. ### Auth scopes `sessions` is a new resource type. `read:sessions:{id}`, `write:sessions:{id}`, `admin:sessions:{id}` flow through the existing JWT validator. Session-scoped public access tokens minted by the server replace browser-held trigger-task tokens for chat-style flows — the browser never sees a run identifier or a run-scoped token in steady state. ## What's coming after this PR - SDK + chat.agent migration: separate branch, separate PR, ships in the next `@trigger.dev/sdk` prerelease alongside this server deploy. Customers using the prerelease `chat.agent` will follow the [upgrade guide](https://github.com/triggerdotdev/trigger.dev/blob/docs/tri-7532-ai-sdk-chat-transport-and-chat-task-system/docs/ai-chat/upgrade-guide.mdx). - Dashboard surfaces: dedicated agent list, agent playground, agent view on the run dashboard. Tracking separately. ## Implementation notes - Postgres `Session` table: scalar scoping columns (`projectId`, `runtimeEnvironmentId`, `environmentType`, `organizationId`) without FKs, matching the January TaskRun FK-removal decision. Point-lookup indexes only — list queries go to ClickHouse. Terminal markers (`closedAt`, `expiresAt`) are write-once. - ClickHouse `sessions_v1`: ReplacingMergeTree, partitioned by month, ordered by `(org_id, project_id, environment_id, created_at, session_id)`. Tags indexed via `tokenbf_v1` skip index. - `SessionsReplicationService`: mirrors `RunsReplicationService` exactly — leader-locked logical replication consumer, `ConcurrentFlushScheduler`, retry with exponential backoff + jitter, identical metric shape. Dedicated slot + publication so the two consume independently. - S2 keys: `sessions/{addressingKey}/{out\|in}`. The existing `runs/{runId}/{streamId}` key format for run-scoped streams is untouched. - Optimistic claim: `ensureRunForSession` triggers a run upfront (cheap to cancel if it loses the race), then attempts an `updateMany` keyed on `currentRunVersion`. Loser cancels its triggered run and reuses the winner's. No DB lock held across the trigger. ### What did NOT change Run-scoped `streams.pipe` / `streams.input` and the existing `/realtime/v1/streams/{runId}/...` routes are unchanged. Sessions are net-new — not a reshaping of the current streams API. ## Deploy notes - Set `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1` to enable the replication consumer. - The `Session` table needs `REPLICA IDENTITY FULL` set on the prod source DB before the publication is created (same one-time DDL we did for `TaskRun`). Required for delete events to carry full column values. - Cross-form authorization on the `GET /api/v1/sessions/:session` loader (a JWT minted for either form authorizes both URL forms). Action routes are URL-form-specific, matching how the SDK mints PATs. ## Verification - Webapp typecheck clean (10/10). - `apps/webapp/test/sessionsReplicationService.test.ts` — round-trip tests for insert/update/delete through Postgres logical replication into ClickHouse via testcontainers. - Live end-to-end against local dev: create + retrieve (both forms) + update + close, `.out.initialize` + `.out.append` x2 + `.in.send` + `.out.subscribe` over SSE, list with all filter combinations + pagination, `end-and-continue` swap, `X-Peek-Settled` fast-close (verified in browser via reconnect-on-reload and via curl). Replicated row lands in ClickHouse within ~1s. - Multi-round Devin + CodeRabbit review feedback addressed (read-after-write paths use `prisma` writer, info-leak on auth-routes masked as 403, peek-settled discriminator parsing fix, etc.). ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run test --filter webapp ./test/sessionsReplicationService.test.ts --run` - [ ] Start the webapp with `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1`. Confirm the slot and publication auto-create on boot. - [ ] `POST /api/v1/sessions` and verify the row replicates to `trigger_dev.sessions_v1` within a couple of seconds. - [ ] `POST /api/v1/sessions/:id/close`, then confirm `POST /realtime/v1/sessions/:id/out/append` returns 400. - [ ] Reuse a closed session's `externalId` on `POST /api/v1/sessions` and confirm 409. - [ ] `GET /realtime/v1/sessions/:id/out` with `X-Peek-Settled: 1` after a turn completes and confirm `X-Session-Settled: true` response header + immediate close.	2026-04-28 12:35:55 +01:00
Eric Allam	e134da7306	fix(run-engine): debounce hot-key lock contention and 5xx feedback loop (#3453 ) ## Changes Three changes in `internal-packages/run-engine/src/engine/systems/debounceSystem.ts`, in order of impact: 1. Fast-path skip before the lock. In `handleExistingRun`, do an unlocked read of `delayUntil` (and `createdAt` for the max-duration check) from the run row before entering `runLock.lock("handleDebounce", ...)`. If `newDelayUntil <= currentDelayUntil` and the run is still within its max-duration window, return the existing run immediately without taking the lock. Safe because debounce is monotonic-forward only — a stale read either matches reality or undershoots, both of which decay correctly (re-checked properly inside the lock by whichever caller is actually pushing forward). Trailing-mode triggers carrying `updateData` still take the lock so the data update is applied. 2. Quantize `newDelayUntil`. Round the computed `newDelayUntil` to 1-second buckets (configurable via `quantizeNewDelayUntilMs`, set to 0 to disable). Without quantization, every call has a slightly larger `newDelayUntil` than the last and they all pass the fast-path check. With it, concurrent callers on the same key share a target time and ~95% short-circuit. User-visible effect: a debounced run might fire up to 1s earlier than the strict spec — non-issue for typical debounce use cases (chat summarization, batched notifications, etc.). 3. Graceful lock-contention fallback. Wrap the `runLock.lock(...)` call so `LockAcquisitionTimeoutError` and Redlock `ExecutionError` / `ResourceLockedError` return the existing run id with success instead of propagating a 5xx. Debounce is best-effort: if we can't take the lock, the herd is already updating it for us; fall in line. This kills the 5xx → SDK-retry feedback loop. With (1)+(2) this rarely fires; without them it's the difference between 5xx and 200. Defaults preserve current behaviour aside from quantization (1s) and fast-path (on). Both are configurable via `RunEngineOptions.debounce`. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Changelog Reduce 5xx feedback loops on hot debounce keys by quantizing `delayUntil`, adding an unlocked fast-path skip before the redlock, and gracefully handling redlock contention in `handleDebounce` so the SDK no longer retries into a herd. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-04-28 11:22:00 +01:00
devin-ai-integration[bot]	4b28080ed4	feat: add `isReplay` to run context (#3454 ) ## Summary Adds `isReplay` boolean to the run context (`ctx.run.isReplay`), following the same pattern as the existing `isTest`. The value is derived from the existing `replayedFromTaskRunFriendlyId` database field, so no schema migration is needed. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing - Verified `@trigger.dev/core` builds successfully - Verified `webapp` typechecks successfully - All new fields use `default(false)` for backwards compatibility --- ## Changelog - Added `isReplay` to `TaskRun` and `V3TaskRun` schemas in `common.ts` - Added `RUN_IS_REPLAY` semantic attribute and wired it in `taskContext` - Propagated `isReplay` through the dequeue system, run attempt system, and all execution context construction paths (V1 + V2) - Added `isReplay` to `DequeuedMessage` and `TaskRunExecutionLazyAttemptPayload` schemas - Added patch changeset for `@trigger.dev/core` - Updated docs: added `isReplay` to context reference, added "Detecting replays" section to replaying page --- 💯 Link to Devin session: https://app.devin.ai/sessions/1d6f1b3cc39a4623b72d05bf00f2d70c --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: nick <55853254+nicktrn@users.noreply.github.com>	2026-04-28 11:57:44 +02:00
nicktrn	91fd8a8a03	chore(security): close dependabot alerts q2 (#3456 ) Closes ~80 dependabot alerts (3 critical, ~25 high, ~31 medium) by bumping direct deps where possible and narrowly overriding the rest. Cloud uses `resend` email transport and Node 20 - all bumps are safe for both cloud and self-hosters. ## Direct upgrades \| Package \| Where \| From \| To \| Why \| \|---\|---\|---\|---\|---\| \| `vite` \| root devDeps \| ^5.4.21 \| (removed) \| dead pin; vitest pulls vite transitively \| \| `dompurify` \| apps/webapp \| ^3.2.6 \| ^3.4.1 \| XSS CVEs \| \| `effect` \| apps/webapp \| ^3.11.7 \| ^3.21.2 \| AsyncLocalStorage CVE in Effect fibers \| \| `nodemailer` \| internal-packages/emails \| ^7.0.11 \| ^8.0.6 \| SMTP CRLF injection (only affects self-hosters w/ smtp/aws-ses transport) \| \| `uuid` \| apps/webapp \| ^9.0.0 \| ^14.0.0 \| buffer bounds check; ESM-only but bundled by Remix \| \| `uuid` + `@types/uuid` \| packages/trigger-sdk \| ^9.0.0 \| (removed) \| dead deps, no usage \| \| `@types/uuid` \| apps/webapp \| ^9.0.0 \| (removed) \| uuid 14 ships its own types \| \| `tar` \| packages/cli-v3 \| ^7.5.4 \| ^7.5.13 \| path traversal CVEs \| \| `testcontainers` + `@testcontainers/postgresql` + `@testcontainers/redis` \| internal-packages/testcontainers \| ^10.28.0 \| ^11.14.0 \| dev/test cleanup; one-line API fix for `RedisContainer(image)` \| \| `rimraf` \| webapp + 6 packages \| ^3.0.2 / ^5.0.7 \| ^6.0.1 \| dev/build tool consolidation \| ## Scoped overrides All bound by both `>=` and `<` to avoid major-version yanks. \| Override \| Closes \| \|---\|---\| \| `tar@>=7 <7.5.11` → `^7.5.11` \| supervisor's `@kubernetes/client-node 1.0.0` chain \| \| `axios@>=1.0.0 <1.15.0` → `^1.15.0` \| replaces older 1.9.0 pin \| \| `systeminformation@>=5.0.0 <5.31.0` → `^5.31.0` \| bumps existing 5.27.14 pin \| \| `lodash@>=4.0.0 <4.18.0` → `^4.18.0` \| bumps existing 4.17.23 pin \| \| `lodash-es@>=4.0.0 <4.18.0` → `^4.18.0` \| new (mirrors lodash) \| \| `dompurify@>=3 <3.4.0` → `^3.4.1` \| catches transitive dompurify via mermaid \| \| `vite@>=5.0.0 <6.4.2` → `^6.4.2` \| path traversal; vite 5 has no patch \| \| `rollup@>=4 <4.59.0` → `^4.59.0` \| path traversal in vite/vitest chain \| \| `flatted@>=3 <3.4.2` → `^3.4.2` \| prototype pollution in eslint flat-cache \| \| `picomatch@>=2 <2.3.2` → `^2.3.2` \| ReDoS in 2.x branch (transitive) \| \| `picomatch@>=4 <4.0.4` → `^4.0.4` \| ReDoS in 4.x branch (vitest/tinyglobby) \| \| `minimatch@>=3 <3.1.3` → `^3.1.3` \| ReDoS in eslint 8 chain \| \| `protobufjs@>=7 <7.5.5` → `^7.5.5` \| critical RCE via @opentelemetry/otlp-transformer \| \| `fast-xml-parser@>=4 <4.5.5` → `^4.5.5` \| DOCTYPE bypass + others (4.x branch via aws-sdk in supervisor) \| \| `fast-xml-parser@>=5 <5.7.0` → `^5.7.0` \| critical + others (5.x branch via aws-sdk in webapp) \| \| `path-to-regexp@>=0.1 <0.1.13` → `^0.1.13` \| ReDoS in express 4 / @remix-run/express \| \| `ajv@>=8 <8.18.0` → `^8.18.0` \| DoS \| \| `socket.io-parser@>=4 <4.2.6` → `^4.2.6` \| DoS in @trigger.dev/core's socket.io \| \| `postcss@>=8 <8.5.10` → `^8.5.10` \| XSS via stringify \| \| `yaml@>=2 <2.8.3` → `^2.8.3` \| DoS \| \| `semver@>=5 <5.7.2` → `^5.7.2` \| ReDoS in 5.x \| \| `defu@>=6 <6.1.5` → `^6.1.5` \| prototype pollution via __proto__ in @prisma/config c12 chain \| ## Dismissed (~47) \| Reason \| Cluster \| Count \| \|---\|---\|---\| \| `not_used` \| langsmith + next 15.x in references/* \| 10 \| \| `not_used` \| minimatch 8.x via prisma-generator-ts-enums (references/prisma-6) \| 3 \| \| `not_used` \| basic-ftp via puppeteer in references/hello-world + references/seed \| 2 \| \| `not_used` \| hono / @hono/node-server / express-rate-limit / path-to-regexp 8.x / @modelcontextprotocol/sdk - all via mcp-sdk chain (dormant in webapp; dev-only localhost in cli-v3) \| 22 \| \| `not_used` \| fastify / @fastify/static / file-type via evalite devDep \| 5 \| \| `tolerable_risk` \| rollup 3 + minimatch 5/8/9/10 dev/build tooling \| 13 \| ## Notes - mcp-sdk chain: `@vercel/sdk` in webapp imports `Vercel` API client only; `mcp-server/` subpath isn't loaded at runtime. cli-v3's MCP server runs only via `trigger mcp` on developer machines. Bumping `@modelcontextprotocol/sdk` to latest (1.29.0) wouldn't close these alerts anyway - it ships hono ^4.11.4 which is still vulnerable - so dismissal is the cleaner call. - References ignore list: confirmed with current dependabot ignore config; added `references/seed/package.json` (only gap). - undici* alerts (CVE-2026-1527, 4 alerts) will auto-close: lockfile already at 6.25.0 > patched 6.24.0; just needs Dependabot rescan. - Effect 3.20 fix is a runtime-only scheduler fix, no public API changes - verified with research agent against our four `effect/` imports. - uuid 14* is ESM-only; we only call `validate`/`version` (no crypto needed) so Node 20 requirement isn't load-bearing for us. ## Public packages (`packages/*`) Minimal surface, deliberately. None of these change published runtime behaviour - all changesets-worthy public package changes are deferred to a regular release pass. \| Package \| Change \| Runtime impact \| \|---\|---\|---\| \| `packages/trigger-sdk` \| Removed dead `uuid` dep (no source imports) \| None - dep was unused \| \| `packages/cli-v3` \| `tar` ^7.5.4 → ^7.5.13 \| Patch bump within already-allowed 7.x range; nothing CLI consumers see \| \| `packages/core` / `packages/build` / `packages/python` / `packages/rsc` / `packages/react-hooks` / `packages/schema-to-json` \| `rimraf` ^3.0.2 → ^6.0.1 in devDeps \| Build-time only, no runtime change \| No changeset added because nothing in these packages affects what published consumers run. ## Validation - Webapp typecheck (forced, no cache) passes after every commit - Smoke-tested testcontainers v11 changes via real `postgresTest` + `redisTest` (sync.test.ts, releaseConcurrency.test.ts) - both pass - Webapp built + verified `require("uuid")` no longer in CJS server output (now bundled inline) - Test env webapp deployed at `dependabot-q2.rc0` (cloud#740) - no issues observed - Test suite run with package prerelease passed	2026-04-28 10:22:44 +01:00
Matt Aitken	8aa1e55588	test: e2e auth baseline tests + webapp testcontainer infrastructure (#3438 ) Adds a minimal end-to-end test harness that spawns the compiled webapp as a child process against a throwaway Postgres container, plus a baseline of 8 auth-behaviour tests. These tests will be used as a regression check before and after the upcoming apiBuilder RBAC migration to confirm auth behaviour is unchanged. ## What's included `internal-packages/testcontainers/src/webapp.ts` (new) Spawns `build/server.js` with a dynamically allocated port, polls `/healthcheck`, and exposes `WebappInstance` and `startTestServer()` (postgres container + webapp + PrismaClient in one call). Key details: - Uses `process.execPath` so the correct Node binary is found in forked test processes - Sets `NODE_PATH` to `node_modules/.pnpm/node_modules` so pnpm-hoisted transitive deps (e.g. `eventsource-parser`) resolve correctly inside the subprocess - Overrides both `PORT` and `REMIX_APP_PORT` so Vite's automatic `.env` loading doesn't override the dynamically allocated port `internal-packages/testcontainers/package.json` Adds `./webapp` sub-path export so tests can `import from "@internal/testcontainers/webapp"`. `internal-packages/testcontainers/src/index.ts` Exports `createPostgresContainer` (used internally by `webapp.ts`). `apps/webapp/test/helpers/seedTestEnvironment.ts` (new) Creates a minimal org → project → environment row set with random suffixes. `apps/webapp/test/api-auth.e2e.test.ts` (new) 8 tests across two suites: - API-key bearer: valid key (auth passes, 404), missing header (401), invalid key (401), error body shape - JWT bearer: valid JWT on JWT-enabled route (passes), valid JWT on non-JWT route (401), empty-scope JWT (403), wrong signing key (401) ## How to run ```bash # Build required first (one-time) pnpm run build --filter webapp cd apps/webapp && pnpm exec vitest run test/api-auth.e2e.test.ts ``` ## Test plan - [x] All 8 tests pass against the current webapp build - [x] Webapp healthcheck returns 200 on startup - [ ] CI passes --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-04-24 12:06:35 +01:00
Matt Aitken	f7aefb705a	fix: disable RunQueue Worker in priority tests to prevent partial-batch race (#3440 ) ## Summary - The `processMasterQueueForEnvironment` call in the priority test was racing against background `processQueueForWorkerQueue` jobs scheduled 50ms after each trigger - With a 50ms debounce (`processWorkerQueueDebounceMs: 50`) and runs triggered sequentially, the RunQueue Worker could process those jobs mid-sequence, pushing partial batches to the worker queue in the wrong overall priority order - `masterQueueConsumersDisabled: true` only blocks the shard-level polling loops — it does not prevent the RunQueue's own Worker from processing these debounced jobs - Fix: add `worker.disabled: true` to the test 1 engine config, which propagates to `workerOptions.disabled` in the RunQueue constructor and prevents the Worker from starting ## Test plan - [x] Both priority tests pass: `pnpm run test ./src/engine/tests/priority.test.ts --run` - [x] Test 1 log confirms no `✅ Starting run engine worker` or worker loop messages — workers fully disabled - [x] Test 2 unaffected (uses master queue consumers for automatic promotion, no `disabled` flag added) 🤖 Generated with [Claude Code](https://claude.com/claude-code)	2026-04-24 12:03:47 +01:00
Eric Allam	2d3b2e82e6	feat(run-engine): flag to route getSnapshotsSince through read replica (#3423 ) ## Summary Adds `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` (default `"0"`). When enabled, the Prisma reads inside `RunEngine.getSnapshotsSince` run against the read-only replica client instead of the primary. Offloads the snapshot-polling queries fired by every running task runner off the writer. ## Why `getSnapshotsSince` is called from the managed runner's fetch-and-process loop (once per poll interval, plus on every snapshot-change notification). It runs four sequential reads per call — one `findFirst` by snapshot id, one `findMany` on snapshots with `createdAt > X`, one raw SQL against `_completedWaitpoints`, and chunked `findMany` on `waitpoint`. Per concurrent run, every few seconds. It's read-only, tolerates a small amount of staleness, and is an obvious candidate for the replica. ## Replica-lag considerations - Step 1 "since snapshot not found": if the runner just received a snapshot id from the primary and asks the replica before it replicates, the function throws and the caller treats the response as an error (runner falls back to a metadata refresh). Self-correcting, not silent. - Step 2 missing newly-created snapshots: the next poll's `createdAt > sinceSnapshot.createdAt` filter still picks them up once the replica catches up. - Waitpoint junction race: the riskiest path — if a latest snapshot is replicated but its `_completedWaitpoints` join rows aren't yet, the runner could advance past that snapshot with `completedWaitpoints: []`. WAL/storage-level replication replays commits in order, so in practice both should appear atomically on the reader, but the race window is why the flag ships disabled. Aurora reader shrinks all three windows to single-digit ms in typical conditions, and its storage-level replication gives atomic visibility of committed transactions on the reader. ## Test plan - [ ] Flip the flag on in a non-prod environment, confirm snapshot polling behaves normally and `getSnapshotsSince` errors in Sentry stay flat. - [ ] Verify writer query volume drops and reader query volume rises on the snapshot-polling queries. - [ ] Keep an eye on `AuroraReplicaLag` (or equivalent) during rollout.	2026-04-22 11:48:04 +01:00
nicktrn	b570586899	fix(webapp): allow cancelling runs in DEQUEUED status from the runs list (#3421 ) The cancel button was missing from the runs list for runs in `DEQUEUED` status. The runs list gates the button on `run.isCancellable`, which goes through `isCancellableRunStatus` -> `CANCELLABLE_RUN_STATUSES` = `NON_FINAL_RUN_STATUSES`. `DEQUEUED` was never added to that list when it was introduced in the run engine. The single run page uses a separate check (`!run.isFinished`, i.e. the inverse of `FINAL_RUN_STATUSES`), so cancellation already worked there - only the list was affected. Adding `DEQUEUED` to `NON_FINAL_RUN_STATUSES` also flips `isCrashableRunStatus` and `isFailableRunStatus`, but: - The crash path is the right behaviour - a `DEQUEUED` run (worker has claimed but not yet executing) can legitimately crash before `EXECUTING`, same as `PENDING`/`DELAYED` already do. - The fail path (`failedTaskRun.server.ts`) is only reached from V1 code paths (marqs consumers, v1 heartbeat handler). `DEQUEUED` is a V2-engine-only status, so V1 consumers never see it. When cancelling a `DEQUEUED` run the execution snapshot goes to `PENDING_CANCEL` (worker must ack) but `TaskRun.status` flips to `CANCELED` immediately - the UI reflects cancellation without waiting for the worker. Added an integration test in `run-engine/src/engine/tests/cancelling.test.ts` covering the full trigger -> dequeue -> cancel -> worker-ack flow. ## Stall safety The stall recovery path (PENDING_EXECUTING heartbeat miss -> nack-and-requeue -> back to QUEUED) lives entirely inside `@internal/run-engine` and never touches the webapp's `taskStatus.ts` helpers - the engine has zero imports from `~/v3/taskStatus` and doesn't know `CrashTaskRunService` / `FailedTaskRunService` exist. A stalled DEQUEUED run still goes back to the queue for retry; this change cannot cause stalls to crash or fail. The only realistic impact is the intended UI fix - the theoretical V1 crash/fail branches for DEQUEUED are unreachable in practice because V1 runs never have DEQUEUED status.	2026-04-21 11:33:17 +01:00
Eric Allam	03e4d5fe31	feat(webapp,database): API key rotation grace period (#3420 ) ## Summary Regenerating a RuntimeEnvironment API key no longer immediately invalidates the previous one. Rotation is now overlap-based: the old key keeps working for 24 hours so customers can roll it out in their env vars without downtime, then stops working. ## Design - New `RevokedApiKey` table (one row per revocation). Holds the archived `apiKey`, a FK to the env, an `expiresAt`, and a `createdAt`. Indexed on `apiKey` (high-cardinality equality — single-row hits) and on `runtimeEnvironmentId`. - `regenerateApiKey` wraps both writes in a single `$transaction`: insert a `RevokedApiKey` with `expiresAt = now + 24h`, update the env with the new `apiKey`/`pkApiKey`. - `findEnvironmentByApiKey` does a two-step lookup: primary unique-index hit on `RuntimeEnvironment.apiKey` first; on miss, `RevokedApiKey.findFirst({ apiKey, expiresAt: { gt: now } })` with an `include: { runtimeEnvironment }`. Two-step (not `OR`-join) keeps the hot path identical to today and puts the fallback cost only on invalid keys. Both lookups use `$replica`. - Admin endpoint `POST /admin/api/v1/revoked-api-keys/:id` accepts `{ expiresAt }` and updates the row. Setting to `now` ends the grace window immediately; setting to the future extends it. - Modal copy on the regenerate dialog updated — previously warned of downtime, now explains the 24h overlap. ## Why a separate table instead of columns on `RuntimeEnvironment` - Keeps the hot auth path's primary lookup unchanged — no OR/nullable-apiKey semantics to reason about. - Naturally supports multiple in-flight grace windows (regenerate twice in a day → two old keys valid until their independent expiries). - FK + cascade cleans up correctly when an env is deleted; nothing to backfill. ## Test plan Verified locally against hello-world with dev and prod env keys: - [x] baseline — current key authenticates (`GET /api/v1/runs`) → `200` - [x] regenerate via UI — DB shows old key in `RevokedApiKey` with `expiresAt ≈ now+24h`, env has new key - [x] grace window — both old and new keys → `200`; bogus key → `401` - [x] admin endpoint: `expiresAt = now` → old key `401` - [x] admin endpoint: `expiresAt = +1h` (after early-expire) → old key `200` again - [x] admin endpoint: `expiresAt = past` → old key `401` - [x] admin 400 (invalid body), 404 (unknown id), 401 (missing/non-admin PAT) - [x] same flow exercised end-to-end on a PROD-typed env — behavior identical - [x] `pnpm run typecheck --filter webapp` passes	2026-04-20 18:28:16 +01:00
Iss	7d7ebdde52	feat: Increase default project limit per org from 10 to 25 (#3409 )	2026-04-17 11:05:57 -04:00
devin-ai-integration[bot]	ff290dfe2f	perf(run-engine): merge dequeue snapshot creation into taskRun.update transaction [TRI-8450] (#3395 ) ## Summary Nests the `TaskRunExecutionSnapshot` creation inside the `taskRun.update()` Prisma call in the dequeue flow, reducing 2 DB commits → 1 per dequeue operation. This is the highest-volume of the five unmerged flows identified in TRI-8450 (~9,200 commits/sec on the engine service). Pattern: Follows the same nested-write approach already used in the completion path (`runAttemptSystem.ts:735`) and trigger path (`engine/index.ts:674`). Changes: - `dequeueSystem.ts`: Moved snapshot creation into `executionSnapshots: { create: {...} }` within the existing `taskRun.update()`. Pre-generates the snapshot ID via `generateInternalId()` (plain cuid, matching what Prisma's `@default(cuid())` produces) so the event emission, heartbeat enqueue, and return value can all be constructed from data already in scope — no extra DB read needed after the merged write. `SnapshotId.toFriendlyId()` is used only for the return value's `friendlyId` field, matching the original `createExecutionSnapshot` behavior. - `executionSnapshotSystem.ts`: Added public `enqueueHeartbeatIfNeeded()` method that exposes the heartbeat scheduling logic (previously only available internally via `createExecutionSnapshot`). This is needed because `PENDING_EXECUTING` requires a heartbeat, unlike the `FINISHED` status in the completion reference pattern. This method is reusable by future merge targets (retry-immediate, checkpoint, cancel, requeue). Net DB change per dequeue: eliminates 1 write transaction (the separate `TaskRunExecutionSnapshot.create`). No extra reads added — the snapshot ID is pre-generated and the `executionSnapshotCreated` event payload is constructed inline from values already available in the closure. ## Review & Testing Checklist for Human - [ ] Verify manually-constructed event payload matches DB state: The `executionSnapshotCreated` event is now built inline (not read back from DB). Confirm the field values (`runStatus: "PENDING"`, `attemptNumber`, `checkpointId`, `workerId`, `runnerId`, `completedWaitpointIds`) match what Prisma actually writes. A mismatch here would be silent — event consumers would get stale/wrong data. - [ ] Verify `attemptNumber` source is equivalent: Old code used `lockedTaskRun.attemptNumber` (post-update result). New code uses `result.run.attemptNumber` (pre-update). The `taskRun.update()` data payload does NOT include `attemptNumber`, so they should be identical — but confirm this assumption holds for all dequeue scenarios (e.g. retried runs). - [ ] Verify `isValid` defaults to `true` in schema: The old `createExecutionSnapshot` explicitly set `isValid: error ? false : true`. The nested create omits `isValid` (no error in the dequeue happy path). Confirm the Prisma schema default for `TaskRunExecutionSnapshot.isValid` is `true`. - [ ] Verify `runStatus: "PENDING"` hardcoding matches the mapping: The old code passed `lockedTaskRun.status` ("DEQUEUED") to `createExecutionSnapshot`, which mapped it to "PENDING" via `run.status === "DEQUEUED" ? "PENDING" : run.status`. The new code hardcodes `"PENDING"` directly. This is correct but brittle if `status` ever changes from "DEQUEUED" to something else upstream. - [ ] Spot-check `completedWaitpoints` connect + order logic: The nested create replicates the connect/order logic from `createExecutionSnapshot` (lines 387-393). Verify the `snapshot.completedWaitpoints` type provides `id` and `index` fields compatible with this usage. - [ ] Verify `checkpoint` in return value: The return now uses `snapshot.checkpoint` (from the previous snapshot) instead of reading the newly-created snapshot's checkpoint relation. Since `checkpointId` is passed through unchanged, they should be identical — but worth a sanity check. Recommended test plan: deploy to staging, run the `sample_pg_activity.py` sampler for a 5-minute window, and verify the COMMIT count drop on the engine service + proportional `IO:XactSync` reduction. ### Notes - This only covers the dequeue flow (flow #1 from TRI-8450). The remaining four flows (retry-immediate, checkpoint, requeue, cancel) are separate follow-ups. - The new `enqueueHeartbeatIfNeeded` method is deliberately designed for reuse by those follow-up PRs. - CI note: the `priority.test.ts` failure in shard 7 is a flaky ordering assertion unrelated to this change (it compares `friendlyId` values in dequeue order). The `audit` check is also pre-existing/unrelated. Link to Devin session: https://app.devin.ai/sessions/034fe0e7224f49278a2de260203e1377 Requested by: @ericallam --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Eric Allam <eallam@icloud.com>	2026-04-16 15:43:16 +01:00
Eric Allam	79b6053e13	feat(server): add TaskIdentifier registry to replace expensive distinct query (#3368 ) Replace the expensive DISTINCT query for task filter dropdowns with a dedicated TaskIdentifier registry table backed by Redis. Environments migrate automatically on their next deploy, with a transparent fallback to the legacy query for unmigrated environments. Also fixes duplicate dropdown entries when a task changes trigger source, and adds active/archived grouping for removed tasks. Moves BackgroundWorkerTask reads in the trigger hot path to the read replica.	2026-04-16 15:22:19 +01:00

1 2 3 4 5 ...

377 Commits