webapp-dashboard-agent-api
585 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dan
|
cd252801eb |
feat: dashboard agent - package upgrades (#3793)
1. in webapp folder update ai-sdk to 6.x.x 2. update vitest to 4.xx |
||
|
Daniel Sutton
|
e21b68cc5f |
feat(webapp): dashboard parity for mollifier-buffered runs (#3757)
## Summary Dashboard surfaces handle buffered runs by falling back to the mollifier snapshot: - Run detail, span detail, streams view (`_app.../runs.\$runParam`, `resources.../spans.\$spanParam`, `resources.../streams.\$streamKey`). - Redirect routes (`@.runs.\$runParam`, `runs.\$runParam`, `projects.v3.\$projectRef.runs.\$runParam`). - Action routes — cancel / replay / idempotency-reset / debug — under `resources.taskruns/...` and `resources.../idempotencyKey.reset`. - Logs download. - Realtime subscription route + per-run resource (`realtime.v1.runs.\$runId`, `resources.../realtime.v1.*`). - `CancelRunDialog` gains an `onCancelSubmitted` callback so submit isn't raced by the Radix `DialogClose` wrapper. Stacked on the mutations PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeRunResource.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeRunResourceBuffer.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeSubscription.test.ts\` passes - [x] Manual smoke: trigger a buffered run, open it in the dashboard, replay/cancel from the UI --- ## Ship-gate follow-up fixes - **Auto-redirect to root span on direct nav** — loader sets `?span=` from root span (PG) or buffered snapshot spanId before 302'ing, so bookmark/share-link/direct-nav doesn't leave the panel collapsed. - **RunPresenter switches from `findFirstOrThrow` to `findFirst` + typed `RunNotInPgError`** — kills the per-poll `PrismaClient error` log spam for buffered runs without changing the route-loader's fallback flow. - **Span detail panel renders for buffered runs** — `SpanPresenter.call` now falls back to `findRunByIdWithMollifierFallback` + `buildSyntheticSpanRun` instead of returning undefined and triggering the "Event not found" toast loop. - **Logs download for buffered runs returns a gzipped placeholder line** — replaces the 404 with a content-encoded line explaining the run is queued. Same org-membership gate as the PG path. - **Admin Debug-Run button hidden for buffered runs + SpanRun circular type alias broken** (squashed) — buttons gate on a new `isBuffered` flag on the synthetic SpanRun. Required grounding SpanRun in `SpanPresenter.getRun` to break a circular type alias TS no longer tolerates once `isBuffered` is a literal field on the shape. - **Replay action requires user auth + org-membership** (🚩 Devin finding) — `action` was unauthenticated and the PG `findFirst` had no org filter, so any caller with a valid `runParam` could replay any run. Buffered fallback inherited the same gap. Fixed to mirror the cancel route. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
e1950778e2 |
feat(webapp): mollifier API mutations on buffered runs (#3756)
## Summary Cancel, replay, reschedule, metadata, tags, and idempotency-key-reset now succeed against a run that's still in the mollifier buffer. Mutations are applied to the buffered snapshot via Lua CAS; the drainer carries the mutation forward when it replays. Primitives added: - `mutateWithFallback` — PG-first / buffer-fallback resolver with bounded-wait safety net for entries that transition mid-mutation. - `applyMetadataMutation` — buffered metadata PUT mirroring the PG-side retry loop with CAS atomicity. - `resolveRunForMutation` — discriminated-union resolver used by route `findResource` so the route builder's pre-action 404 check sees buffered runs. Routes wired (whole files, no GET/POST splits): - `api.v2.runs.\$runParam.cancel.ts` - `api.v1.runs.\$runParam.replay.ts` - `api.v1.runs.\$runParam.reschedule.ts` - `api.v1.runs.\$runId.metadata.ts` - `api.v1.runs.\$runId.tags.ts` - `resetIdempotencyKey.server.ts` Stacked on the reads PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierMutateWithFallback.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierApplyMetadataMutation.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierResolveRunForMutation.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
a1dc3c5ae1 |
feat(webapp): mollifier API GET read-fallback — synthetic primitives + route wiring (#3755)
## Summary Synthesise QUEUED/FAILED responses from the mollifier buffer when a TaskRun row hasn't landed in Postgres yet. Wires the synthesis into: - `ApiRetrieveRunPresenter` - v1 trace GET route - v1 spans GET route - attempts route gains a GET loader (fixes pre-existing Remix "no loader" 400) The `readFallback` infra itself lives on the trigger PR (consumed by `IdempotencyKeyConcern`); this PR adds the route-level synthetic-rendering primitives. Stacked on the replay PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierSyntheticRedirectInfo.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierSyntheticSpanRun.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
4745754a7a |
feat(webapp,run-engine): mollifier drainer replay + stale sweep + cancelled-run engine API (#3754)
## Summary The replay side of the mollifier: - `DrainerHandler`: reads buffered snapshots and replays them through `engine.trigger` to materialise PG rows. - `RunEngine.createCancelledRun`: new public method the handler uses to write CANCELED rows directly from snapshots (bypass queue + waitpoint, emit `runCancelled`). Tolerates the cjson empty-table tags edge case found during validation. - Drainer fairness: org → env rotation so a heavy env doesn't starve light ones in the same org. - Stale-entry sweep + telemetry + alertable gauge so a stuck/offline drainer surfaces in alerts. Both the drainer and sweep default-off; nothing fires unless flagged on (`TRIGGER_MOLLIFIER_DRAINER_ENABLED`, `TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`). Stacked on the trigger-time decisions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierDrainerHandler.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\` passes - [x] \`pnpm run test --filter @internal/run-engine src/engine/tests/createCancelledRun.test.ts\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/drainer.test.ts\` passes --- ## Ship-gate follow-up fix **Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an `onTerminalFailure` callback on `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `MAX_ATTEMPTS` on a retryable PG error (previously `buffer.fail()` was called with no row written → silent data loss). The callback runs before `buffer.fail()` on every terminal path (non-retryable AND max-attempts-exhausted), and re-throwing a retryable error from the callback causes the drainer to requeue rather than fail. Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive option + new exported types). Includes 5 unit tests covering both terminal causes plus the requeue-on-retryable-callback-failure path and no-callback back-compat. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
577f35eebe |
feat(webapp): mollifier trigger-time decisions — mollify, claim, read fallback (#3753)
## Summary The trigger hot path's mollifier integration: - `mollifyTrigger`: when the gate trips, write the engine.trigger snapshot to the buffer and return a synthesised QUEUED response. Postgres write is deferred to drainer-replay (next PR in the stack). - Pre-gate idempotency-key claim: same-key triggers serialise through Redis so a burst lands in PG / buffer exactly once. - Read-fallback extensions: `findRunByIdWithMollifierFallback` for the trigger-time idempotency lookup that must see buffered runs. - Gate bypasses: `debounce`, `oneTimeUseToken`, `parentTaskRunId`/`triggerAndWait` skip the mollify path entirely. - `triggerTask` + `IdempotencyKeyConcern` wired to the above. All behaviour gated by the master `TRIGGER_MOLLIFIER_ENABLED` switch; off-state hot path is unchanged (the gate is not even consulted). Stacked on the buffer extensions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierMollify.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierIdempotencyClaim.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierReadFallback.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierGate.test.ts\` passes - [x] \`pnpm run test --filter webapp test/engine/triggerTask.test.ts\` passes --- ## Ship-gate follow-up fixes - **Batch items bypass the mollifier gate** — fixes `BatchTaskRunItem_taskRunId_fkey` FK violation on batch triggers when the gate trips. End-state is a drainer-side `BatchTaskRunItem` create-on-materialise; batch traffic passes through the gate until that lands. - **IdempotencyKeyConcern honours buffered-run TTL on expiry** — buffered path now clears expired idempotency claims (read-side) and resets the buffer's `mollifier:idempotency:*` SETNX binding (write-side) so a re-trigger past the customer's TTL lands as a fresh run instead of echoing the stale buffered runId. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
3924aa4adb |
feat(redis-worker,webapp): mollifier buffer extensions + snapshot type (#3752)
## Summary Buffer-side data layer used by the rest of the mollifier phase-3 stack. - `buffer.ts` gains entry inspection (`getEntry`), idempotency lookup (`lookupIdempotency`), in-place snapshot mutation (`mutateSnapshot`), and dwell tracking. All atomic via Lua. - `mollifierSnapshot.server.ts`: shared `MollifierSnapshot` type plus (de)serialise helpers. - Drops the entry-TTL config and its env var. The drainer is the recovery mechanism; an entry that survives the drainer should surface as a stale-sweep alert, not silently TTL away. Adds methods to the buffer interface; nothing consumes them yet. Subsequent PRs in the stack wire trigger-time mollify, read-fallback, and mutation paths against this surface. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/buffer.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Saadi Myftija
|
9cb6fd17c3 |
fix(webapp): idempotent DeploymentBackgroundWorker creation (#3772)
Make `POST /api/v1/deployments/:deploymentId/background-workers` idempotent so client-side retries no longer collide on the `BackgroundWorker` `(project, env, version)` unique index. Helps make deployments more resilient against the class of indexing failures that surfaces in the dashboard as "Indexing timed out", e.g. during transient database issues. |
||
|
Matt Aitken
|
90bbbd1c44 |
fix(webapp): recover from ClickHouse JSON parse failures on out-of-range integers (#3759)
## Summary Second class of poisoned-row failure in the runs replication path. PR #3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer literals outside ClickHouse's `Int64`..`UInt64` range. Recovery stays purely reactive — the existing `sanitizeRows` walker just gains an extra branch, so the hot replication path pays nothing on healthy rows. Fixes the still-firing customer-facing symptom from [TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): `scan-social-profiles` runs continued to be stranded in `EXECUTING` on the Tasks page after #3708 deployed. CloudWatch showed `Dropped batch — ClickHouse JSON parse error but sanitizer found nothing to fix` firing **8/8 times** since the previous deploy (zero successful sanitizations). Root cause: upstream JS Number precision loss on a 21-digit Google Plus ID (`117039831458782873093` → `117039831458782870000`) — the precision-lossy value still serialises as a bare integer that exceeds `UInt64.MAX`, which ClickHouse rejects with `INCORRECT_DATA`. ## How the bug ships The customer task emits an output containing a Poshmark profile's `spec_format`: ```json {"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"} ``` That value is `1.17e20` — comfortably above `UInt64.MAX` (`1.84e19`) but comfortably below `1e21`. `Number.prototype.toString` only switches to exponential form at `|value| >= 1e21`, so `JSON.stringify` emits the bare token `117039831458782870000` and the ClickHouse `JSON(max_dynamic_paths)` column fails with: ``` Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1) : While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x) ``` Same error verbatim as prod. The same number quoted (`"117039831458782870000"`) inserts fine — ClickHouse's dynamic JSON column accepts a `String` subtype on the same path. ## What changed `apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts`: - New private `isUnsafeJsonInteger(value)` helper — true iff `value` is a finite integer-valued JS Number where `|value| < 1e21` (so `JSON.stringify` emits integer form, not exponent) **and** `value` falls outside `[Int64.MIN, UInt64.MAX]`. - `sanitizeUnknownInPlace` gains a number-branch: when the predicate holds, replace the Number with `String(value)`. The downstream JSON column dynamic-types the path as String for that row — fine, since the value was already precision-lossy upstream (no JS Number above 2^53 is numerically meaningful anyway). - Float-valued numbers, large floats (>= 1e21), NaN and Infinity are left alone — `JSON.stringify` emits them with exponents or as `null`, both of which ClickHouse accepts. `apps/webapp/test/sanitizeRowsOnParseError.test.ts`: four new unit tests + an extension to `sanitizeRows` covering surrogate + integer fixes counted together across rows. The unit suite now covers: - Positive value above `UInt64.MAX` (`117039831458782870000` — the actual prod value) - Negative value below `Int64.MIN` - Boundary values pass through (`42`, `Number.MAX_SAFE_INTEGER`, `2^63`) - Non-integer numbers untouched (floats, `1e25`, NaN, Infinity) - The actual `scan-social-profiles` nested shape — finds the offending `gp_id` deep inside `output.data.profiles[].spec_format[].platform_variables[].value` `.server-changes/runs-replication-bigint-recovery.md` — release notes entry. ## Why reactive, not pre-flight `#prepareJson` runs millions of times per day on the replication hot path. Walking every JSON tree to look for oversized integers would add bounded-but-real CPU on every healthy row. `sanitizeRows` only fires after a ClickHouse parse-error rejection, which is a few times a day platform-wide. Extending it costs effectively zero on healthy traffic and gains us recovery on the rare poisoned row. ## Verification - Reproduced 1:1 in a throwaway Docker `clickhouse/clickhouse-server:25.12.11.4` (closest available to the prod `25.12.1.1579` build). Pre-sanitize JSON fails with the exact prod error; post-sanitize JSON inserts cleanly and the row is readable with `gp_id` stored as a String subtype. - `pnpm --filter webapp exec vitest run test/sanitizeRowsOnParseError.test.ts` — 22/22 passing (18 existing + 4 new). - `pnpm run typecheck --filter webapp` — clean. ## Test plan - [x] `pnpm run typecheck --filter webapp` - [x] Unit tests pass against new + existing cases - [x] End-to-end Docker ClickHouse repro confirms recovery - [ ] Post-deploy: confirm `Sanitizing batch after ClickHouse JSON parse error` warns fire instead of `Dropped batch …` errors when `scan-social-profiles` outputs trip CH again - [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stops climbing in `/stp/trigger-app-prod/ecs/replication/service-container/process-logs` ## What this does NOT do - Doesn't backfill the ~120k+ existing stranded `EXECUTING` rows in production. Same as #3708 — that needs a reconciliation/backfill sweep (separate ticket — TRI-9755 fix #3). - Doesn't address the upstream root cause (the customer task emitting a JS-Number-precision-lossy big int). That's a customer-task concern; our replication path needs to be robust to whatever shape arrives. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Eric Allam
|
37eeaa3690 |
fix(webapp): skip V1 WAITING_FOR_DEPLOY drain on V2 promotes (#3742)
## Summary Stops the legacy V1 `WAITING_FOR_DEPLOY` drain from running on every V2 deployment promotion. The drain queries `TaskRun` by `status='WAITING_FOR_DEPLOY'`, which only V1-engine runs ever have — V2 runs use `PENDING_VERSION` and are handled out of band. Calling the drain on V2 promotes produced empty queries against the status index and unnecessary reader-DB load. ## Fix Two layers: 1. Gate the enqueue at the call site in `ChangeCurrentDeploymentService` so it only fires when the deployment's worker is on engine V1. 2. Add a `LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED` env var (default `0`). When set to `1`, the service returns immediately from `call()` — neuters any jobs already sitting in the worker queue from before the deploy lands. V1 customers see no change; V2 promotes no longer trigger the drain. |
||
|
Matt Aitken
|
596a9bb90c |
fix(webapp): retain sessions-replication singleton import via globalThis assignment (#3738)
Linear: [TRI-9864](https://linear.app/triggerdotdev/issue/TRI-9864) (Urgent) Production incident: [TRI-9863](https://linear.app/triggerdotdev/issue/TRI-9863) (mitigated by image revert in cloud#910) ## Bug `apps/webapp/package.json` declares `"sideEffects": false`. PR #3333 (`71d98b4e`) replaced the previous real method-call retention idiom at the two `sessionsReplicationInstance` import sites with: ```ts import { sessionsReplicationInstance } from "..."; void sessionsReplicationInstance; ``` esbuild treats `void <identifier>;` as a pure expression statement under `sideEffects: false` and **tree-shakes the entire import** — including the `singleton(...)` call inside `sessionsReplicationInstance.server.ts` which is the only thing that fires `initializeSessionsReplicationInstance()`. The sessions→ClickHouse logical replication worker never starts, the slot is unconsumed, lag grows. ### How it manifested in production cloud#907's image bump rolled the `SessionReplicationService` ECS task on prod at 14:32 UTC. The new container's startup log emitted `🗃️ Runs replication service enabled` but **not** `🗃️ Sessions replication service enabled` or `🗃️ Sessions replication service started`. CloudWatch `OldestReplicationSlotLag` grew at ~220 MB/min and the `High replication lag` alarm fired at 14:37 UTC. Prod was reverted to the previous image (cloud#910) to stop the bleed. ### Verification `grep` of the built bundle `apps/webapp/build/index.js` (built from `c0365d36`): - **3** occurrences of `Runs replication` / `runsReplicationInstance` strings ✅ - **0** occurrences of `Sessions replication` / `sessionsReplicationInstance` / `SessionsReplicationService` ❌ The runs path survives tree-shaking because `adminWorker.server.ts` and `admin.api.v1.runs-replication.*` routes have real method calls (`.start()`, `.teardown()`, `.backfill()`) — observable uses the tree-shaker must preserve. The sessions singleton has no real callers, only the `void` no-ops, hence its complete elimination from the bundle. ## Fix Replace `void sessionsReplicationInstance;` with an assignment to `globalThis`, an unambiguous observable side effect the bundler cannot eliminate: ```ts (globalThis as Record<string, unknown>).__sessionsReplicationInstance = sessionsReplicationInstance; ``` Applied at both call sites: `apps/webapp/app/entry.server.tsx` and `apps/webapp/app/v3/services/adminWorker.server.ts`. Surrounding comments updated to document the bundler interaction so the next maintainer doesn't reintroduce `void`. ## Out of scope (follow-ups) - **Robustness improvement**: change `apps/webapp/package.json` from `"sideEffects": false` to an allowlist that includes `*Instance.server.ts` files. Prevents the same regression shape via any future `*Instance` singleton. - **Build-time check**: add a `grep` post-build step in `publish.yml` requiring `"Sessions replication"` to appear in `apps/webapp/build/index.js`. Catches this exact regression at CI time. ## Test plan - [x] `pnpm run typecheck --filter webapp` clean - [ ] After merge + publish: confirm new image's `SessionReplicationService` container logs `🗃️ Sessions replication service enabled` and `🗃️ Sessions replication service started` at startup - [ ] After re-deploying to prod: confirm `OldestReplicationSlotLag` stops growing and drains |
||
|
Eric Allam
|
61ca40b4b1 |
perf(run-engine,webapp): look up PENDING_VERSION runs via ClickHouse (#3707)
## Summary When a background worker registers, the engine resolves runs that were queued before the worker was ready (status `PENDING_VERSION`). That lookup used to scan a Postgres status index on `TaskRun`. Move it to ClickHouse: query candidate run ids from `task_runs_v2`, then refetch the actual rows from Postgres by primary key with a `status = 'PENDING_VERSION'` guard for idempotency. ## Design The lookup is a pluggable interface on the run engine (`PendingVersionRunIdLookup`). The webapp wires a ClickHouse-backed implementation through the org-scoped `clickhouseFactory` using a new `"engine"` client type, configured by `RUN_ENGINE_CLICKHOUSE_*` env vars. The URL falls back to `CLICKHOUSE_URL` when unset, so self-hosted deployments don't need new config to keep working. When the lookup returns no candidates, one bounded retry is scheduled ~5s later to cover ClickHouse replication lag against `task_runs_v2`. The Postgres status guard on both the candidate refetch and the inner `updateMany` prevents double-promotion when a retry races with a concurrent deploy. Tests cover three existing PENDING_VERSION cases via a small Postgres-backed test adapter; new ClickHouse-backed integration tests will follow. |
||
|
Matt Aitken
|
71d98b4e6b |
Support for org-scoped ClickHouse (#3333)
Added `OrganizationDataStore` which allows orgs to have data stored in specific separate services. For now this is just used for ClickHouse. When using ClickHouse we get a client for the factory and pass in the org id. Particular care has to be made with two hot-insert paths: 1. RunReplicationService 2. OTLPExporter --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Eric Allam
|
aec7e0a93d |
perf(webapp): index EnvironmentVariableValue.environmentId (#3675)
Env-var lookups via `GET /api/v1/projects/:projectRef/envvars/:slug/:name` run a Prisma `findMany` on `EnvironmentVariableValue` filtered by `environmentId` + `isSecret`. The only existing indexes are the primary key and a unique on `(variableId, environmentId)`, so `environmentId` is never the leading column — the planner falls back to a Parallel Seq Scan over the whole table to find what is, in practice, a handful of rows per environment. Two changes: - Add a btree index on `EnvironmentVariableValue(environmentId)` so the planner switches to an index scan. The composite `(variableId, environmentId)` unique stays in place; the new index is purely additive. - Route the `findMany` inside `getEnvironmentWithRedactedSecrets` through the read replica via a new `replicaClient` constructor param on the repository (defaulting to `$replica`, mirroring how `prismaClient` defaults to `prisma`). Writes and read-after-write methods stay on the primary. ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] Confirm `EXPLAIN` plan flips from Parallel Seq Scan to an index scan - [ ] Existing env-var route tests still pass |
||
|
Iss
|
204a766bb1 |
feat(webapp): expose is_warm_start in TRQL runs schema (#3667)
Add is_warm_start to TRQL runs schema so warm vs cold start data is queryable |
||
|
Oskar Otwinowski
|
02d61afc1c |
fix(webapp): sanitize OTel attributes on ClickHouse JSON parse rejection (#3659)
Before fix: <img width="1264" height="987" alt="image" src="https://github.com/user-attachments/assets/24b8b85c-b89f-4109-9004-8d6af61d2849" /> After fix: <img width="1264" height="987" alt="image" src="https://github.com/user-attachments/assets/89bbc587-c50a-45ab-b203-dbe91028e918" /> |
||
|
Daniel Sutton
|
906d5fafb6 |
feat(mollifier): trigger burst smoothing — Phase 1 (monitoring) (#3614)
## Summary - Introduce the Mollifier: a Redis-backed buffer for `trigger()` API calls during traffic spikes, with a per-env trip evaluator and a drainer ack-loop. - Phase 1 is dual-write monitoring — every mollified trigger is buffered to Redis AND continues to `engine.trigger`. No customer-facing behaviour change. - Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`, `mollifier.drained`, plus the `mollifier.decisions` counter. - Gated behind a feature flag (default off). ## Test plan - [x] `pnpm run test --filter @trigger.dev/redis-worker` - [x] `pnpm run test --filter webapp -- mollifier` - [x] Manual: with flag off, no behaviour change vs main - [x] Manual: with flag on + threshold lowered, observe `mollifier.buffered` + `mollifier.drained` log pairs with matching `runId` --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Eric Allam
|
454f0c949a |
perf(webapp): cache task metadata in Redis for the trigger hotpath (#3625)
## Summary The trigger-task hotpath used to early-return without a DB query when a caller passed both a queue override and a per-trigger TTL — the hottest configuration on the trigger API. Adding `triggerSource` to the resolver so the runs-list "Source" filter could distinguish STANDARD / SCHEDULED / AGENT runs removed those early-returns, costing +2 DB queries per trigger on non-locked calls and +1 on locked calls. This change caches `BackgroundWorkerTask` metadata (`ttl`, `triggerSource`, `queueId`, `queueName`) in Redis so the resolver can satisfy every caller configuration with a single `HGET` on the warm path. PG fallback on miss back-fills the cache. Follow-up to #3542. ## Design Two key spaces: - `task-meta:env:{envId}` — the "current worker" view, refreshed at every deploy promotion. 24h safety TTL. - `task-meta:by-worker:{workerId}` — used for `lockToVersion` triggers. Immutable post-create. 30d sliding TTL so historical workers age out. Cache writes use Lua scripts via `defineCommand` so `DEL` + `HSET` + `EXPIRE` land atomically — concurrent readers never see the empty intermediate state of a naive pipeline. Read-path back-fill uses single-field upserts so concurrent back-fills don't wipe each other's siblings. The cache lives behind its own `TASK_META_CACHE_REDIS_*` env-var prefix that falls back to the default `REDIS_*` set, so operators can route the cache to a dedicated Redis instance if they want. The service/instance file split (`taskMetadataCache.server.ts` for the pure class, `taskMetadataCacheInstance.server.ts` for the env-wired singleton) mirrors the existing `runsReplicationService` / `runsReplicationInstance` pattern. ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run test ./test/engine/triggerTask.test.ts --run` — 8 existing tests untouched + 5 new tests covering warm cache, cold miss with back-fill, queue + ttl path, by-worker vs env keyspace, and the promotion cache write - [ ] End-to-end against a dev worker: registering writes both keyspaces with the expected TTLs, and `redis-cli HGETALL "tr:task-meta:env:<envId>"` returns the cached entries ## Benchmark Measured `DefaultQueueManager.resolveQueueProperties` against a real Postgres + Redis (vitest `containerTest`, single-host docker). 500 sequential calls and 2,000 parallel calls (concurrency=50) per scenario, request shaped as `{ taskId, queue: "bench-queue", ttl: "5m" }` — the hot path this PR restores. ``` sequential (one in flight at a time): [noop cache (baseline)] n=500 mean=1.423ms p50=1.394ms p95=1.735ms p99=2.629ms max=11.100ms [redis cache, cold ] n=500 mean=1.346ms p50=1.283ms p95=1.688ms p99=2.463ms max=5.058ms [redis cache, warm ] n=500 mean=0.084ms p50=0.078ms p95=0.105ms p99=0.156ms max=1.129ms speedup (warm vs baseline, sequential): 16.95x parallel (concurrency=50): [noop cache (baseline)] n=2000 mean=10.069ms p50=8.850ms p95=14.718ms p99=31.887ms total=405ms ops/s=4,940 [redis cache, warm ] n=2000 mean=0.614ms p50=0.568ms p95=1.189ms p99=1.432ms total=25ms ops/s=80,389 throughput speedup (warm vs baseline, parallel): 16.27x ``` Read: - **Warm cache cuts resolver latency 17×** at p50 — from ~1.4 ms to ~78 µs per call. - **Cold cache is on par with baseline** — the extra `HGET` miss adds <50 µs against the two Postgres queries that follow, so the worst case is not worse than today. - **Under burst load (50 concurrent triggers)**, the baseline's p99 jumps to ~32 ms as Postgres connections queue up; warm stays at ~1.4 ms. The cache moves the saturation point from ~5k ops/s (PG pool) to ~80k ops/s (single-client Redis pipelining). Caveats: single-host docker, local Postgres + Redis, resolver-only measurement (excludes the rest of the trigger transaction). Prod adds region-local Redis RTT (~0.3–0.8 ms) which shifts warm absolute numbers up but keeps the ratio intact. |
||
|
Daniel Sutton
|
5022769f70 |
fix(webapp): retry on version collision when initializing a deployment (#3610)
Concurrent `POST /api/v1/deployments` requests for the same environment race on the `WorkerDeployment(environmentId, version)` unique constraint. Both requests read the same latest deployment via `findFirst`, compute the same next version via `calculateNextBuildVersion`, and both attempt `prisma.workerDeployment.create()` — one wins, the other crashes with Prisma `P2002`. The bug is a classic TOCTOU between the version read and the version write; it's been latent since the version-assignment logic was first added but only fires when two deploys land within milliseconds of each other (CI matrices, retried CLI calls, webhook-triggered redeploys). ## Approach Extracts the version assignment + create into a small helper `createDeploymentWithNextVersion` (`apps/webapp/app/v3/services/initializeDeployment/createDeploymentWithNextVersion.server.ts`). The helper retries on `P2002 (environmentId, version)` up to 5 times with randomised 5–50ms jitter so N concurrent racers don't loop in lockstep. Each attempt re-reads the latest version, recomputes via `calculateNextBuildVersion`, and re-runs the caller's `buildData` callback so version-dependent fields (image ref tag, friendlyId) are always consistent with the version actually persisted. A `logger.warn` fires per collision so the retry rate is observable in production logs. When retries are exhausted, the helper throws a dedicated `DeploymentVersionCollisionError` carrying `environmentId`, `attempts`, and `lastAttemptedVersion`, with the original `PrismaClientKnownRequestError` attached as `cause`. Sentry walks the `cause` chain natively, so contention exhaustion shows up as a distinguishable wrapper exception linked to the underlying `P2002` rather than a generic unique-constraint violation that looks identical to every other duplicate-key bug. The behavioural change is limited to "catch P2002 and retry instead of crashing." The image ref computation stays inside the builder callback (same call site as before the refactor), so ECR / non-ECR behaviour, S2 stream creation order, and all downstream side effects are unchanged. ## Non-goals - No new database migrations, no schema changes, no isolation-level / locking changes. A serialisable transaction or advisory lock would also fix this; retry-on-conflict is the smaller change that keeps the existing version-allocation logic intact. - Does not touch the analogous `calculateNextBuildVersion` call in `createBackgroundWorker.server.ts`, which likely has the same race shape against `BackgroundWorker`'s unique constraint — flagged as a follow-up. ## Test plan - [x] `pnpm run typecheck --filter webapp` passes (no new errors in the modified files). - [x] Three real-Postgres tests in `apps/webapp/test/createDeploymentWithNextVersion.test.ts` via `containerTest`: - 5 concurrent calls all produce distinct, persistable versions (`Set(versions).size === concurrency`). The naive read-then-create version of the helper fails this test with the exact same `P2002` seen in production; the retry version passes. - Non-`P2002` errors raised from the `buildData` callback propagate immediately without retry, builder invoked exactly once. - With `maxRetries: 0`, concurrent racers surface the wrapped `DeploymentVersionCollisionError` (not a raw `P2002`); `environmentId`, `attempts`, `lastAttemptedVersion` are populated and `error.cause.code === "P2002"`. - [x] Existing `apps/webapp/test/getDeploymentImageRef.test.ts` still green (the file was untouched in the final diff). ## Follow-ups (not in this PR) - `createBackgroundWorker.server.ts` likely has the same TOCTOU shape against its background-worker version unique constraint — should use the same helper. - Sentry visibility check: confirm `error.cause` chain renders as a linked exception in the Sentry UI when the wrapped error fires (requires a sandboxed triggering of the exhaustion path). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Eric Allam
|
be1a6cf8de |
feat: Sessions primitive — durable run-aware streams + dashboard
Adds Sessions, a durable, run-aware stream primitive that scopes session.in / session.out records to a session (not a single run). Records survive run boundaries; reconnect-from-last-event-id is built in. Server foundation: - New /realtime/v1/sessions/:session/:io/append + /records routes - sessionRunManager + sessionsRepository + clickhouseSessionsRepository - mintRunToken for short-lived per-session tokens - s2Append retry-with-backoff + undici cause diagnostics - /api/v[12]/packets/* exempt from customer rate limits - BackgroundWorker schema gains taskKind enum (TASK, AGENT, SCHEDULED) - TaskRun.taskKind column + clickhouse 029_add_task_kind_to_task_runs_v2 Core types: - new sessionStreams, inputStreams, realtimeStreams packages in @trigger.dev/core - session-streams-api / realtime-streams-api surface Sessions dashboard UI (the primitive's own viewer): - /sessions index + detail routes - SessionsTable, SessionFilters, SessionStatus, CloseSessionDialog - AGENT/SCHEDULED filter in RunFilters + TaskTriggerSource Includes the sessions-primitive changeset. |
||
|
Matt Aitken
|
e4981d1b11 |
feat(webapp): consolidate auth path + add comprehensive auth tests (#3499)
## Summary
Consolidates the webapp's authentication and authorization into a small
set of route helpers, replacing the ad-hoc `requireUser` /
`requireUserId` / `authenticatedEnvironmentForAuthentication` calls
scattered across routes. Same security model, but the per-request flow
(authenticate → authorize → load) now lives in one place per route
family.
Introduces a plugin seam (`@trigger.dev/plugins`) that lets the cloud
build install a richer RBAC implementation without touching webapp code.
The OSS fallback keeps the pre-RBAC permissive behaviour intact, so
self-hosted deployments work unchanged.
Adds a comprehensive end-to-end auth test suite that didn't exist before
— 193 `it()` blocks (vitest reports ~199 after `it.each` expansion)
covering API key, PAT and JWT auth across the public API surface, plus
dashboard session auth for admin pages.
## Changes
### Plugin contract — `@trigger.dev/plugins`
`RoleBaseAccessController` interface authoritative for both OSS
(fallback) and cloud (enterprise plugin):
- `authenticateBearer(request, { allowJWT? })` — API-key / public-JWT
auth, returns env + ability
- `authenticateSession(request, { userId, organizationId?, projectId?
})` — dashboard auth, caller resolves `userId` from the session cookie
and passes it in (no `helpers.getSessionUserId` callback — decouples the
plugin host from session-cookie code)
- `authenticatePat(request, { organizationId?, projectId? })` — PAT
auth, returns identity + `lastAccessedAt` so the host can throttle the
per-request update
- `authenticateAuthorize*` variants for the auth-and-check-in-one-call
cases
- `isUsingPlugin(): Promise<boolean>` — capability flag for UI /
branching where plugin-present-ness matters; replaces the
sentinel-string coupling that had `personalAccessToken.server` matching
`"RBAC plugin not installed"` literally
### Dashboard auth (started, partial rollout)
Admin and settings pages migrated to a unified `dashboardLoader` /
`dashboardAction` helper that authenticates the session, runs an
authorization check, and exposes the result to the route. Other
dashboard routes still on the old pattern; remaining migration tracked
in TRI-8730.
Migrated routes:
- `admin.*` (14 admin / back-office / feature-flags / LLM-models /
notifications / orgs / concurrency pages)
- `_app.orgs.$organizationSlug.settings.team`
- `_app.orgs.$organizationSlug.settings.roles`
### API / realtime / engine auth (complete for the migrated families)
71 routes migrated to a unified `apiBuilder` that centralizes Bearer /
PAT / Public-JWT authentication and applies the per-route authorization
check before the handler runs. Includes:
- `api.v1.*` and `api.v2.*` and `api.v3.*` — tasks, runs, batches,
queues, prompts, deployments, query, sessions, waitpoints, packets,
workers, idempotency keys
- `realtime.v1.*` — runs, batches, sessions, streams
- `engine.v1.*` — dev / worker-action protocols
29 routes still on the legacy `authenticateApiRequest*` helpers —
tracked as a post-deploy follow-up in TRI-9228.
Multi-resource auth direction is now explicit at the call site via
`anyResource(...)` (OR) and `everyResource(...)` (AND). Bare arrays no
longer typecheck — fixes a class of bug where a JWT scoped to one
resource could implicitly access others under OR semantics.
PAT auth path consolidated: was three DB queries per request (legacy
`authenticateApiRequestWithPersonalAccessToken` findFirst +
`rbac.authenticatePat` join + `lastAccessedAt` update). Now one query in
the steady state — plugin returns `lastAccessedAt`, host smart-skips the
update via JS-side throttle when fresh.
Side effect: action aliases preserved historic JWT scope semantics where
the new model is stricter (e.g. a `write:tasks` JWT now also satisfies
`trigger` / `batchTrigger` / `update` actions on the same resource —
matched at the auth boundary, not in the route handler).
### Backwards-compat fixes
The strict-match model regressed several real-world JWT shapes. Each
preserved via explicit `anyResource(...)` entries in the route's authz
block:
- **Batch retrieve routes** (`api.v1.batches.$batchId`, `api.v2.*`,
`realtime.v1.batches.*`) accept `read:runs` JWTs again (pre-RBAC
literal-match superScope behaviour)
- **Runs list routes** (`api.v1.runs`, `realtime.v1.runs`) accept
type-level `read:tasks` / `read:tags` on unfiltered queries (matched the
legacy `Object.keys` iteration semantic)
- **PAT/OAT auth shape** normalized through `toAuthenticated` so all
auth methods return the same slim `AuthenticatedEnvironment` (was:
API-key returned the slim shape but PAT/OAT returned raw Prisma
`Decimal` / no `orgMember`)
- **Scope `:` preservation** in resource ids — `read:tags:env:staging`
now correctly identifies the tag id as `env:staging`, not `env`
### Slim `AuthenticatedEnvironment`
Extracted to `@trigger.dev/core/v3/auth/environment` — a structural
shape independent of `@trigger.dev/database`. The plugin contract
returns this; webapp consumers import from there; the cloud plugin
(Drizzle) returns the same shape without Prisma's `Decimal` class
leaking into the public surface. Lets internal-packages (run-engine,
etc.) refer to `AuthenticatedEnvironment` without pulling Prisma in.
### Auth test suite (new — `*.e2e.full.test.ts`)
193 e2e tests run against a real spawned webapp + Postgres (no mocks).
Coverage matrix:
- **API key auth** — read / write / trigger / batchTrigger / deploy
actions across runs, batches, deployments, prompts, queues, query,
sessions, input-streams, waitpoints, tasks, idempotency keys; multi-key
resources (a run carries batch / tag / task identifiers — auth must
accept any matching scope)
- **Personal Access Token auth** — comprehensive matrix: scope match,
scope mismatch, missing scope, expired token, malformed token
- **Public JWT auth** — sub-vs-URL environment resolution, expired JWTs,
signature verification, scope checking, otu (one-time-use) token
semantics, branch-environment signing-key fallback
- **Dashboard session auth** — admin-only pages reject non-admins;
per-action gating
- **Cross-cutting edge cases** — revoked API key grace window, JWT
cross-environment isolation, MissingResource branch behaviour
### Hygiene cleanups
- Deleted dead `app/services/authorization.server.ts` (legacy
`checkAuthorization` + types — no live consumers post-migration) and its
orphaned test
- Dropped the never-populated `scopes` field from
`ApiAuthenticationResultSuccess`
- `scheduleEmail` moved out of `email.server.ts` into its own module —
breaks a `commonWorker → marqs/V1` import chain that was poisoning the
auth test graph
- OSS Roles page shows a deployment-aware empty state ("Roles aren't
available in this self-hosted deployment" vs the plan-upsell copy) via
`rbac.isUsingPlugin()`
- Team action handler: explicit per-intent ability gates
(`manage:billing` for purchase-seats, `manage:members` for set-role +
remove-member with self-leave carve-out)
### Cross-repo coordination
All public-package contract changes paired in `triggerdotdev/cloud#763`
(rbac-packages branch) — the enterprise plugin implements the same
`RoleBaseAccessController` interface against Drizzle.
## Test plan
- [x] `pnpm run typecheck --filter webapp` clean
- [x] `pnpm --filter webapp exec vitest run --config
vitest.e2e.full.config.ts` — 193/193 pass (requires Docker for
testcontainers)
- [x] Spot-check an authed API endpoint with a valid + invalid API key
against a local stack
- [x] Spot-check the migrated admin pages render and gate non-admins
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Eric Allam
|
567e2a2c32 |
feat(webapp,redis): handle READONLY / LOADING during ElastiCache failover (#3548)
## Summary
During an ElastiCache role swap (failover) or node-type change (vertical
scale), the ioredis TCP/TLS connection stays open but the server starts
answering with `READONLY` (the client is talking to a node that became a
replica) or `LOADING` (node still loading data from disk). Without an
explicit hook, those errors surface to caller code as `ReplyError`
instances — every write op on the affected connection fails until the
cluster fully cuts over.
This PR adds `reconnectOnError` to every prod ioredis client so the
disconnect + reconnect + retry cycle absorbs these errors and caller
code never sees them.
## Fix
```ts
export function defaultReconnectOnError(err: Error): boolean | 1 | 2 {
const msg = err.message ?? "";
if (msg.startsWith("READONLY") || msg.startsWith("LOADING")) return 2;
return false;
}
```
Returning `2` tells ioredis to disconnect, reconnect, and re-issue the
failed command. After reconnect, DNS / SG state routes the new socket to
a writable node.
The helper lives in `@internal/redis` and is wired into both the shared
`createRedisClient` (which covers RunQueue, schedule-engine,
redis-worker, and every other internal-package consumer) and the direct
`new Redis(...)` call sites in the webapp.
V1-only marqs files are intentionally not migrated.
## Test plan
- [x] `pnpm run typecheck --filter webapp`
- [x] `pnpm run typecheck --filter @internal/run-engine`
- [x] Verified end-to-end against a live ElastiCache vertical-scale
event — caller-surfaced errors went from tens of thousands during the
cutover window down to a handful per ioredis client
- [ ] Confirm steady-state behavior unchanged after deploy
|
||
|
Eric Allam
|
ead1e5a53d |
feat(webapp): reload LLM pricing registry on Redis pub/sub (#3534)
## Summary Adds a Redis pub/sub reload path to the webapp's in-memory LLM pricing registry. When enabled on a process, the registry reloads from the database whenever a publish lands on the configured channel — instead of waiting for the existing 5-minute interval. Lets pricing/model changes propagate to cost enrichment within seconds. Subscription is **off by default** and opt-in per process. Only OTel-ingesting services need real-time freshness; dashboard and worker services run fine on the periodic interval and shouldn't pile onto each publish with a full-table reload. ## Design When `LLM_PRICING_RELOAD_PUBSUB_ENABLED=true`, subscribes via `createRedisClient` against `COMMON_WORKER_REDIS_*` and listens on `LLM_PRICING_RELOAD_CHANNEL` (default `llm-registry:reload`). The 5-minute periodic reload stays as a backstop, and a SIGTERM/SIGINT handler closes the subscription cleanly. The publisher side lives outside this PR — any process running in the same Redis namespace can trigger a reload by `PUBLISH llm-registry:reload <anything>`. Includes a `.server-changes/` note for the changelog. ### Debounced reload Bursts of publishes are coalesced. The first publish schedules a reload at T+`LLM_PRICING_RELOAD_DEBOUNCE_MS` (default 1s); subsequent publishes during that window are no-ops because the trailing reload picks up everything when it queries the DB. Bounds reload rate to at most 1 per debounce window regardless of publisher chattiness, so a runaway upstream publisher can't fan out into a flood of full-table-scan reloads. ## Test plan - [ ] With `LLM_PRICING_RELOAD_PUBSUB_ENABLED=false` (default): `redis-cli PUBSUB NUMSUB llm-registry:reload` returns `0` while the webapp is up - [ ] With it set to `true`: returns `>= 1` - [ ] `redis-cli PUBLISH llm-registry:reload test` returns `1` (one subscriber received) on a subscribed process - [ ] Mutate an `LlmModel` row externally, publish on the channel, observe the registry's match() picks up the change without waiting for the 5-min tick - [ ] Publish 100x in rapid succession; confirm only one reload fires within the debounce window |
||
|
Eric Allam
|
31999afcaa |
perf(webapp): trim BackgroundWorker.metadata to the schedule slice on create (#3525)
Large deploys (projects with many tasks or source files) blocked the
webapp event loop for several seconds inside Prisma's client-side
serializer on `BackgroundWorker.create`, tail-latencying every other
in-flight request on the same Node process. The `metadata` JSON column
was being written with the full deploy manifest — every task's config,
every queue and prompt, and the full source of every file — all of which
already live on dedicated columns or in dedicated tables.
Fix: project the manifest to `{ packageVersion, contentHash, tasks: [{
id, filePath, schedule }] }` on insert. The only post-write read site is
`changeCurrentDeployment`, which feeds `tasks[].schedule` into
`syncDeclarativeSchedules` at deploy promotion. The retained top-level
keys and per-task `filePath` are kept solely so
`BackgroundWorkerMetadata.safeParse` still succeeds on read.
## Test plan
- [ ] Deploy a project with declarative schedules; verify schedules are
created on first deploy
- [ ] Modify / remove schedules across subsequent deploys; verify sync
- [ ] Roll back to a previous deploy; verify `changeCurrentDeployment`
re-syncs schedules
- [ ] Inspect `BackgroundWorker.metadata` on a fresh deploy — should be
a small object, not the full manifest
|
||
|
Daniel Sutton
|
14920ce2c4 |
fix(webapp): downgrade expected user-input error logs to warn (#3523)
`dac9c83bd` added `ignoreErrors: /^ServiceValidationError(?::|$)/` in
`apps/webapp/sentry.server.ts` to drop SVEs before they reach Sentry.
The
filter only matches when the captured event's *type* is
`ServiceValidationError`, but nine call sites in the webapp catch SVE
(and
analogous user-input error types — `OutOfEntitlementError`,
`CreateDeclarativeScheduleError`, `QueryError`) and call
`logger.error("wrapper message", { error: e })` *before* the type check.
The captured event is then titled with the wrapper message, with the
inner
error buried in `extra.error` — invisible to the SDK filter. Result: a
steady stream of expected user-input failures escalating as
`error`-level
events when they should be `warn`.
Each catch block now type-discriminates first, logs expected types at
`warn`,
and keeps unknown-error fall-throughs at `error`. For service sites that
wrap into SVE (`createBackgroundWorker`,
`createDeploymentBackgroundWorkerV4`),
the inner error is logged at `error` before wrapping — mirrors the
`waitpointCompletionPacket.server.ts` pattern from `dac9c83bd`.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
devin-ai-integration[bot]
|
30bd567d48 |
fix: sync declarative schedules on deployment rollback (#3468)
## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing - Reviewed the code flow for deployment rollback (`ChangeCurrentDeploymentService`) and confirmed it was missing schedule sync - Verified all 4 callers of `ChangeCurrentDeploymentService` (UI rollback, UI promote, API promote, finalize deployment) are now covered - Ran `pnpm run typecheck --filter webapp` — passes cleanly --- ## Changelog When rolling back (or manually promoting) a deployment, declarative schedules were not being synced to match the target deployment's worker metadata. Schedules remained as configured by the most recent deployment rather than reflecting the target version's schedule configuration. This fix adds a call to `syncDeclarativeSchedules` in `ChangeCurrentDeploymentService` after the deployment promotion is updated. It parses the target deployment's stored `BackgroundWorkerMetadata` to restore the correct schedule state. This covers both rollback and promote paths (UI and API). Errors are handled gracefully so they don't block the deployment change itself. --- ## Screenshots N/A — backend-only change. 💯 Link to Devin session: https://app.devin.ai/sessions/0debf012b58c4132be778f8ea88cd2b6 --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: nick <55853254+nicktrn@users.noreply.github.com> |
||
|
nicktrn
|
ee3887a321 |
feat(webapp): configurable deploy template machine presets (#3492)
The webapp's compute template creation hardcoded a single machine preset (`small-1x`) at deploy time, regardless of which presets a project actually uses. Tasks running on any other preset paid full cold-snapshot creation cost on first run. Two new env vars: - `COMPUTE_TEMPLATE_MACHINE_PRESETS` - CSV of preset names to build boot snapshots for during deploy. Defaults to `small-1x` so existing deploys don't change behavior. - `COMPUTE_TEMPLATE_MACHINE_PRESETS_REQUIRED` - CSV of presets whose failure fails a required-mode deploy. Defaults to the full `PRESETS` list. Optional preset failures are logged but don't block the deploy. The compute client now sends the multi-config request shape; the service evaluates per-preset outcomes against the required set and surfaces a combined failure message when a required preset fails. Both env vars are validated at boot via the env schema - unknown preset names or `_REQUIRED` entries that aren't a subset of `_PRESETS` fail loudly at startup rather than silently per-deploy. |
||
|
Eric Allam
|
ac7177d61f |
feat(schedule-engine): stop persisting per-tick schedule state (#3476)
## Summary
Each scheduled-task tick previously issued **3 Prisma `UPDATE`s**
against
`TaskSchedule.lastRunTriggeredAt`,
`TaskScheduleInstance.lastScheduledTimestamp`,
and `TaskScheduleInstance.nextScheduledTimestamp`. All three were pure
denormalization — every value can be derived without persisting.
After this PR `TaskSchedule` and `TaskScheduleInstance` become **near
read-only**:
writes happen only on schedule create / update / delete (rare admin
actions),
so the per-tick autovacuum churn on these hot tables disappears.
## Design
The previous fire time travels forward through the **schedule worker
payload**,
not through the database. Concretely:
- The `schedule.triggerScheduledTask` worker payload gains an optional
`lastScheduleTime: z.coerce.date().optional()` field.
- When the engine fires a schedule, it re-enqueues the next tick with
`lastScheduleTime = scheduleTimestamp` (the just-fired time).
- When the next tick dequeues, `payload.lastTimestamp` is sourced from
`params.lastScheduleTime` directly. No DB round-trip, no cron-derivation
drift across DST boundaries, no caveats around recently-edited cron
expressions.
`payload.lastTimestamp` keeps its `Date | undefined` SDK shape.
First-ever
fires still report `undefined`, so customer `if
(!payload.lastTimestamp)`
first-run patterns keep working.
For Redis jobs that were enqueued **before** this change (which lack
`lastScheduleTime` in their payload), the engine falls back to
`instance.lastScheduledTimestamp` once. Once those drain, the column is
never read again. Revert is code-only; the columns stay in place and can
be dropped in a follow-up once the rollout is stable.
## Files
- `internal-packages/schedule-engine/*` — engine refactor,
`workerCatalog`
schema field, `TriggerScheduleParams` extension, tests updated to assert
on the worker-payload flow rather than DB readbacks.
- `internal-packages/database/prisma/schema.prisma` — `/// @deprecated`
triple-slash docstrings on the three columns. No migration.
- `apps/webapp/app/presenters/v3/ScheduleListPresenter.server.ts` —
drops
the `lastRunTriggeredAt` Prisma select; "Last run" cell is approximated
from the cron expression's previous slot, gated on `schedule.createdAt`
so brand-new schedules show "–". UI is best-effort; the runs page is the
source of truth.
- `apps/webapp/app/v3/utils/calculateNextSchedule.server.ts` — adds a
`previousScheduledTimestamp` helper for the UI cell above. Public API
responses (`api.v1.schedules.*`) already compute `nextRun` from cron and
don't expose `lastTimestamp` — no public API change.
- `references/scheduled-tasks/` — new reference project with declarative
schedules at multiple cadences and three throw-on-fail validators
(`first-fire-detector`, `interval-validator`, `upcoming-validator`) for
E2E-verifying the worker-payload flow.
Refs TRI-8891
## Test plan
- [x] `pnpm run typecheck --filter @internal/schedule-engine --filter
webapp`
- [x] `pnpm run build --filter @trigger.dev/core`
- [x] `pnpm run test --filter @internal/schedule-engine` — integration
test
asserts first-fire `lastTimestamp === undefined`, second fire carries
the previous fire's timestamp exactly.
- [x] E2E against local webapp via `references/scheduled-tasks`:
- Fresh schedules attached → all three deprecated columns stay `NULL`
after
multiple fires.
- Redis payload at second fire contains
`"lastScheduleTime":"<previous fire timestamp>"`.
- `TaskRun.payload` and the every-minute task's returned output both
confirm
`lastTimestamp = null` on first fire and `lastTimestamp = <prev fire>`
on
second fire, exactly 60s apart.
- All three throw-on-FAIL validators completed successfully on every
non-first fire.
- [x] Schedules REST API end-to-end (`POST` / `GET` / `PUT` / `activate`
/
`deactivate` / `DELETE`) — `nextRun` recomputed live from cron + tz on
every response, no reads of deprecated columns.
|
||
|
DKP
|
19c16759f6 |
feat(webapp): errors page polish and GA rollout (#3477)
## What this does Polish + bug-fix pass on the Errors page so it can ship to everyone. Touches the Slack alert config UX, errors list, error detail page, and unhides the SideMenu entry for non-admins. ## Decisions **"No channel" item over standalone Remove button** Chose pinning a `<XMarkIcon /> No channel` `SelectItem` above the channel list. Rejected the standalone "Remove channel" link in a `<Hint>` — color/hover behaviour clashed with the sibling `<TextLink>`, and "channel selection" is the right context for clearing. Server action already deletes the channel when `slackChannel=""` is submitted. **Slack `<!date^>` token over per-user TZ field for alerts** Chose Slack's native `<!date^TS^…>` token so each viewer sees timestamps in their own timezone (UTC fallback). Rejected per-user/per-org TZ schema work — works for multi-region channels for free. Email/dashboard TZ source-of-truth filed as TRI-8885 / TRI-8886. **Make errors GA** |
||
|
ThullyoCunha
|
f1736595cd |
feat(webapp): apply default repository policy on ECR repo creation (#3467)
🚀 Publish Trigger.dev Docker / units (push) Failing after 13m3s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 13m3s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
## Summary
Self-hosters that operate the webapp's ECR account separately from the
account running the EKS workers (e.g., a shared platform account that
hosts the registry plus per-team accounts that host clusters) currently
hit a 403 Forbidden the first time **any** project is deployed:
```
Failed to pull image "<acct-A>.dkr.ecr.<region>.amazonaws.com/<namespace>/proj_…:…":
unexpected status from HEAD request to .../v2/.../manifests/sha256:…: 403 Forbidden
```
`ensureEcrRepositoryExists` in
`apps/webapp/app/v3/getDeploymentImageRef.server.ts` calls
`CreateRepository` and `PutLifecyclePolicy`, but never
`SetRepositoryPolicy` — so the new repo inherits the AWS default (only
the registry-owner account can read/pull). Workers in the cluster
account get 403 every single deploy. The only workarounds today are
running a one-off post-create script or pre-creating every repo by hand.
## Proposed change
Add an optional env var:
```
DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY (V4 mirror: V4_DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY)
```
Raw IAM policy JSON. When set, the webapp calls `SetRepositoryPolicy`
immediately after `CreateRepository` so every new repo carries that
policy from creation. Operators control the principal/actions; we don't
bake in any opinions about cross-account boundaries.
Example value (for the typical self-host case — grant pull to the
cluster account):
```json
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "AllowClusterAccountPull",
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::<cluster-account-id>:root"},
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
]
}]
}
```
## Why env var (not a chart-level field)
- Mirrors the shape of the sibling vars (`DEPLOY_REGISTRY_ECR_TAGS`,
`DEPLOY_REGISTRY_ECR_ASSUME_ROLE_ARN`, etc.) which are already
operator-supplied via `webapp.extraEnvVars` in self-host setups.
- Cloud is unaffected — the env var is optional, unset by default;
existing behavior unchanged.
- Existing repos are unaffected — only newly-created repos get the
policy.
- `RepositoryCreationTemplate` from the AWS provider isn't an
alternative here: it only applies to repos created via
pull-through-cache or replication, not to `ecr:CreateRepository` API
calls.
## Implementation
- `apps/webapp/app/env.server.ts` — declare
`DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY` and the V4 fallback.
- `apps/webapp/app/v3/registryConfig.server.ts` — propagate
`ecrDefaultRepositoryPolicy` to `RegistryConfig`.
- `apps/webapp/app/v3/getDeploymentImageRef.server.ts` —
`createEcrRepository` accepts the policy; if set, calls
`SetRepositoryPolicy` after `PutLifecyclePolicy`.
- `docs/self-hosting/env/webapp.mdx` — documentation row added under
**Deploy & Registry**.
## Verification
Verified end-to-end against a self-hosted Trigger.dev on EKS where the
ECR account is separate from the cluster account:
- **Without the env var** (current `main`): the new project's first run
pod stays in `ImagePullBackOff` with `403 Forbidden`.
- **With the env var set** to a JSON granting
`ecr:BatchGetImage`/`GetDownloadUrlForLayer`/`BatchCheckLayerAvailability`
to the cluster account: a fresh `trigger.dev deploy --env prod` followed
by a `hello-world` run completes in ~5s end-to-end on the first try.
Manually also confirmed that existing repos are untouched (the call only
fires inside `createEcrRepository`, which only runs when
`DescribeRepositories` returned `RepositoryNotFoundException`).
## Out of scope
- Chart values surface for this — operators already pass the existing
ECR vars via `webapp.extraEnvVars`, so this follows the same pattern.
Happy to add a first-class chart field in a follow-up if that's the
preferred direction.
- IAM-policy validation in the webapp — we forward the JSON verbatim to
AWS and surface AWS's error messages on misuse, matching how
`DEPLOY_REGISTRY_ECR_TAGS` is handled today.
This is a draft pending CI / CodeRabbit pass — happy to iterate on
direction (e.g., split into per-action env vars, or extend the chart
values schema) if any of the above choices feels off.
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
|
||
|
Eric Allam
|
dac9c83bdc |
chore(webapp,run-engine): downgrade boundary log noise to warn (#3462)
## Summary
Several boundary catches and customer-input validation paths were
logging at `error` level for failures the system already handles
gracefully — disconnect on auth failure, return undefined, skip retries,
etc. This batch routes them to `warn` (which stays in stdout) or counts
them as OTel metrics, so visibility is preserved without surfacing them
as alerts.
## Changes
**New helper / pattern:**
- `apiBuilder.server.ts` — `logBoundaryError(message, error, url)`
inspects the inner error type at loader/action boundary catches;
downgrades to `warn` for `AbortError`, `ServiceValidationError`, and
`EngineServiceValidationError`.
- `platform.v3.server.ts` — `platform_client.failures_total` OTel
counter with `{function, kind}` labels; helper
`recordPlatformFailure(fn, kind)` replaces the previous error-level
logging across all `BillingClient` wrappers.
**Log-level downgrades:**
- `handleSocketIo.server.ts` — `Worker authentication failed` → warn
(system disconnects on failure; refs TRI-8863)
- `waitpointSystem.ts` — when `runStatus === "CANCELED"` in the
suspended-without-checkpoint branch, skip the throw and warn instead
(benign cancel-vs-resume race, nothing to resume)
- `runAttemptSystem.ts` — `flushedMetadata` parse/validate failures →
warn (customer-side data shape, system returns gracefully)
- `batch-queue/index.ts` — final-attempt failures with
`result.skipRetries` → warn (callbacks already opted out of retry, e.g.
queue size limit hit)
- `queryPerformanceMonitor.server.ts` — slow queries → warn
(observability signal, not an application error)
- `timeoutDeployment.server.ts` — deployment-state mismatch in the
timeout job → warn (timeout-vs-completion race)
**Inner error preservation:**
- `waitpointCompletionPacket.server.ts` — `logger.error(uploadError)`
before throwing the `ServiceValidationError` wrapper, so the underlying
upload error stays visible.
## Why
The pattern across all of these is the same: a boundary log treated any
thrown/returned error as `error` regardless of cause, even when the
cause was an expected, system-handled condition (client disconnect,
customer quota, race condition, schema validation of customer data).
That made the logs noisy and made it harder to spot real bugs.
Where the underlying signal is still useful operationally (slow queries,
billing call failures), we route it to OTel metrics with low-cardinality
labels so dashboards and alerts can be tuned independently of error
logs.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run build --filter @internal/run-engine`
- [ ] Trigger a run on hello-world and verify task lifecycle is
unaffected
- [ ] Cancel a suspended run and verify the cancel-while-suspended
branch in `waitpointSystem.ts` returns `{status: "skipped"}` instead of
throwing
- [ ] Confirm `platform_client.failures_total` counter shows up in
metrics with `{function, kind}` labels when the billing client errors
|
||
|
Eric Allam
|
c69e939c34 |
feat: Sessions - bidirectional durable agent streams (#3417)
> ⚠️ **Not released yet.** This PR is the server-side foundation only. The SDK changes that customers will actually use (`chat.agent` migration, `chat.createStartSessionAction`, `useTriggerChatTransport` updates) live on a separate branch and ship together in an upcoming `@trigger.dev/sdk` prerelease. Until that prerelease is published, this surface is reachable only via direct HTTP. ## What this gives Trigger.dev users A new first-class primitive, **Session**, for durable, task-bound, bidirectional I/O that outlives any single run. Sessions are the run manager for `chat.agent` going forward, and they unblock anything else that needs "one identifier, many runs over time" with a stable channel pair the client can write to and subscribe to. ### Use cases unblocked - **Chat agents that persist across many runs.** One session per chat (keyed on your own `chatId` via `externalId`), turns 1..N attach to the same Session, the UI subscribes once and keeps receiving output as new runs take over. - **Approval loops and long-running tasks with user feedback.** The task waits on `.in`, the client writes to `.in`, the server enforces no-writes-after-close. - **Workflow progress streams that live past the run.** Subscribe to `.out` after the task finishes to replay history. - **Resume-next-day flows.** A session is a durable row, not a transient stream. Send a message a day later and the server triggers a fresh run on the same session. ### How it works (Session-as-run-manager) A Session row is task-bound (`taskIdentifier` + `triggerConfig` are required) and owns its current run via `currentRunId` + `currentRunVersion` for optimistic claim. Three trigger paths: 1. **Session create** — `POST /api/v1/sessions` creates the row and triggers the first run synchronously. 2. **Append-time probe** — `POST /realtime/v1/sessions/:session/in/append` checks if the current run is alive; if it has terminated (idle exit, crash, etc.), the server triggers a new run before processing the append. 3. **End-and-continue handoff** — `POST /api/v1/sessions/:session/end-and-continue`, called by the running agent, triggers a fresh run and atomically swaps `currentRunId`. Used by `chat.requestUpgrade()` for version handoffs. Every triggered run is recorded in the `SessionRun` audit table with a reason (`initial`, `continuation`, `upgrade`, `manual`). ## Public API surface ### Control plane - `POST /api/v1/sessions` — create. Idempotent on `(env, externalId)`. Triggers the first run, returns the session and a session-scoped public access token. Returns 409 if the upserted row is already closed. - `GET /api/v1/sessions/:session` — retrieve by friendlyId (`session_abc...`) or by your own externalId (server disambiguates by prefix). - `GET /api/v1/sessions` — list with filters (`type`, `tag`, `taskIdentifier`, `externalId`, derived `status` ACTIVE/CLOSED/EXPIRED, created-at range) and cursor pagination. Backed by ClickHouse. - `PATCH /api/v1/sessions/:session` — update tags / metadata / externalId. - `POST /api/v1/sessions/:session/close` — terminate. Idempotent, hard-blocks new server-brokered writes. - `POST /api/v1/sessions/:session/end-and-continue` — agent-only handoff to a fresh run. ### Realtime - `PUT /realtime/v1/sessions/:session/:io` — initialize a channel. Returns S2 credentials in headers so high-throughput clients can write direct to S2. - `GET /realtime/v1/sessions/:session/:io` — SSE subscribe. Supports Last-Event-ID resume and an opt-in `X-Peek-Settled: 1` header that fast-closes the stream when the upstream is already settled (`trigger:turn-complete`), eliminating long-poll wait on reconnect-on-reload paths. - `POST /realtime/v1/sessions/:session/:io/append` — server-side appends. - `POST /api/v1/runs/:runFriendlyId/session-streams/wait` — runs wait on a session stream as a waitpoint, with a race-check to avoid suspending if data already landed. ### Auth scopes `sessions` is a new resource type. `read:sessions:{id}`, `write:sessions:{id}`, `admin:sessions:{id}` flow through the existing JWT validator. Session-scoped public access tokens minted by the server replace browser-held trigger-task tokens for chat-style flows — the browser never sees a run identifier or a run-scoped token in steady state. ## What's coming after this PR - **SDK + chat.agent migration**: separate branch, separate PR, ships in the next `@trigger.dev/sdk` prerelease alongside this server deploy. Customers using the prerelease `chat.agent` will follow the [upgrade guide](https://github.com/triggerdotdev/trigger.dev/blob/docs/tri-7532-ai-sdk-chat-transport-and-chat-task-system/docs/ai-chat/upgrade-guide.mdx). - **Dashboard surfaces**: dedicated agent list, agent playground, agent view on the run dashboard. Tracking separately. ## Implementation notes - **Postgres `Session` table**: scalar scoping columns (`projectId`, `runtimeEnvironmentId`, `environmentType`, `organizationId`) without FKs, matching the January TaskRun FK-removal decision. Point-lookup indexes only — list queries go to ClickHouse. Terminal markers (`closedAt`, `expiresAt`) are write-once. - **ClickHouse `sessions_v1`**: ReplacingMergeTree, partitioned by month, ordered by `(org_id, project_id, environment_id, created_at, session_id)`. Tags indexed via `tokenbf_v1` skip index. - **`SessionsReplicationService`**: mirrors `RunsReplicationService` exactly — leader-locked logical replication consumer, `ConcurrentFlushScheduler`, retry with exponential backoff + jitter, identical metric shape. Dedicated slot + publication so the two consume independently. - **S2 keys**: `sessions/{addressingKey}/{out|in}`. The existing `runs/{runId}/{streamId}` key format for run-scoped streams is untouched. - **Optimistic claim**: `ensureRunForSession` triggers a run upfront (cheap to cancel if it loses the race), then attempts an `updateMany` keyed on `currentRunVersion`. Loser cancels its triggered run and reuses the winner's. No DB lock held across the trigger. ### What did NOT change Run-scoped `streams.pipe` / `streams.input` and the existing `/realtime/v1/streams/{runId}/...` routes are unchanged. Sessions are net-new — not a reshaping of the current streams API. ## Deploy notes - Set `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1` to enable the replication consumer. - The `Session` table needs `REPLICA IDENTITY FULL` set on the prod source DB before the publication is created (same one-time DDL we did for `TaskRun`). Required for delete events to carry full column values. - Cross-form authorization on the `GET /api/v1/sessions/:session` loader (a JWT minted for either form authorizes both URL forms). Action routes are URL-form-specific, matching how the SDK mints PATs. ## Verification - Webapp typecheck clean (10/10). - `apps/webapp/test/sessionsReplicationService.test.ts` — round-trip tests for insert/update/delete through Postgres logical replication into ClickHouse via testcontainers. - Live end-to-end against local dev: create + retrieve (both forms) + update + close, `.out.initialize` + `.out.append` x2 + `.in.send` + `.out.subscribe` over SSE, list with all filter combinations + pagination, `end-and-continue` swap, `X-Peek-Settled` fast-close (verified in browser via reconnect-on-reload and via curl). Replicated row lands in ClickHouse within ~1s. - Multi-round Devin + CodeRabbit review feedback addressed (read-after-write paths use `prisma` writer, info-leak on auth-routes masked as 403, peek-settled discriminator parsing fix, etc.). ## Test plan - [ ] `pnpm run typecheck --filter webapp` - [ ] `pnpm run test --filter webapp ./test/sessionsReplicationService.test.ts --run` - [ ] Start the webapp with `SESSION_REPLICATION_CLICKHOUSE_URL` and `SESSION_REPLICATION_ENABLED=1`. Confirm the slot and publication auto-create on boot. - [ ] `POST /api/v1/sessions` and verify the row replicates to `trigger_dev.sessions_v1` within a couple of seconds. - [ ] `POST /api/v1/sessions/:id/close`, then confirm `POST /realtime/v1/sessions/:id/out/append` returns 400. - [ ] Reuse a closed session's `externalId` on `POST /api/v1/sessions` and confirm 409. - [ ] `GET /realtime/v1/sessions/:id/out` with `X-Peek-Settled: 1` after a turn completes and confirm `X-Session-Settled: true` response header + immediate close. |
||
|
Eric Allam
|
e134da7306 |
fix(run-engine): debounce hot-key lock contention and 5xx feedback loop (#3453)
## Changes
Three changes in
`internal-packages/run-engine/src/engine/systems/debounceSystem.ts`, in
order of impact:
1. **Fast-path skip before the lock.** In `handleExistingRun`, do an
unlocked read of `delayUntil` (and `createdAt` for the max-duration
check) from the run row before entering `runLock.lock("handleDebounce",
...)`. If `newDelayUntil <= currentDelayUntil` and the run is still
within its max-duration window, return the existing run immediately
without taking the lock. Safe because debounce is monotonic-forward only
— a stale read either matches reality or undershoots, both of which
decay correctly (re-checked properly inside the lock by whichever caller
is actually pushing forward). Trailing-mode triggers carrying
`updateData` still take the lock so the data update is applied.
2. **Quantize `newDelayUntil`.** Round the computed `newDelayUntil` to
1-second buckets (configurable via `quantizeNewDelayUntilMs`, set to 0
to disable). Without quantization, every call has a slightly larger
`newDelayUntil` than the last and they all pass the fast-path check.
With it, concurrent callers on the same key share a target time and ~95%
short-circuit. User-visible effect: a debounced run might fire up to 1s
earlier than the strict spec — non-issue for typical debounce use cases
(chat summarization, batched notifications, etc.).
3. **Graceful lock-contention fallback.** Wrap the `runLock.lock(...)`
call so `LockAcquisitionTimeoutError` and Redlock `ExecutionError` /
`ResourceLockedError` return the existing run id with success instead of
propagating a 5xx. Debounce is best-effort: if we can't take the lock,
the herd is already updating it for us; fall in line. This kills the 5xx
→ SDK-retry feedback loop. With (1)+(2) this rarely fires; without them
it's the difference between 5xx and 200.
Defaults preserve current behaviour aside from quantization (1s) and
fast-path (on). Both are configurable via `RunEngineOptions.debounce`.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Changelog
Reduce 5xx feedback loops on hot debounce keys by quantizing
`delayUntil`, adding an unlocked fast-path skip before the redlock, and
gracefully handling redlock contention in `handleDebounce` so the SDK no
longer retries into a herd.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
4b28080ed4 |
feat: add isReplay to run context (#3454)
## Summary Adds `isReplay` boolean to the run context (`ctx.run.isReplay`), following the same pattern as the existing `isTest`. The value is derived from the existing `replayedFromTaskRunFriendlyId` database field, so no schema migration is needed. ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing - Verified `@trigger.dev/core` builds successfully - Verified `webapp` typechecks successfully - All new fields use `default(false)` for backwards compatibility --- ## Changelog - Added `isReplay` to `TaskRun` and `V3TaskRun` schemas in `common.ts` - Added `RUN_IS_REPLAY` semantic attribute and wired it in `taskContext` - Propagated `isReplay` through the dequeue system, run attempt system, and all execution context construction paths (V1 + V2) - Added `isReplay` to `DequeuedMessage` and `TaskRunExecutionLazyAttemptPayload` schemas - Added patch changeset for `@trigger.dev/core` - Updated docs: added `isReplay` to context reference, added "Detecting replays" section to replaying page --- 💯 Link to Devin session: https://app.devin.ai/sessions/1d6f1b3cc39a4623b72d05bf00f2d70c --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: nick <55853254+nicktrn@users.noreply.github.com> |
||
|
Eric Allam
|
ca399565ba |
feat(webapp): add per-worker Node.js heap metrics (#3437)
## Summary Adds direct V8 heap and process-memory gauges to the webapp's OpenTelemetry meter. The webapp already exports per-cluster-worker Node.js runtime metrics (event-loop lag / utilization, active handles, active requests, libuv threadpool size) via a custom meter under the `trigger.dev` scope. Heap and memory were missing; this PR adds them alongside, in the same observable-batch pattern. ## New gauges | Metric | Source | Unit | | --- | --- | --- | | `nodejs.memory.heap.used` | `process.memoryUsage().heapUsed` | bytes | | `nodejs.memory.heap.total` | `process.memoryUsage().heapTotal` | bytes | | `nodejs.memory.heap.limit` | `v8.getHeapStatistics().heap_size_limit` | bytes | | `nodejs.memory.external` | `process.memoryUsage().external` | bytes | | `nodejs.memory.array_buffers` | `process.memoryUsage().arrayBuffers` | bytes | | `nodejs.memory.rss` | `process.memoryUsage().rss` | bytes | Gated by the existing `INTERNAL_OTEL_NODEJS_METRICS_ENABLED` flag, same as the adjacent event-loop / handle gauges. Zero overhead when disabled. ## Why `@opentelemetry/host-metrics` publishes `process.memory.usage`, which is RSS only. RSS is the sum of V8 heap, external memory (Buffers, etc.), native code, and thread stacks. Without a direct heap metric it is not possible to size the V8 heap cap (`--max-old-space-size`) from metrics alone, because RSS overstates heap by the external + native footprint. A worker can have a 4 GB RSS with a 2.5 GB heap and 1.5 GB of buffers; the former constrains `--max-old-space-size`, the latter does not. `nodejs.memory.heap.limit` also surfaces the configured `--max-old-space-size` (read from `v8.getHeapStatistics().heap_size_limit`), so operators can see the current limit in the same dashboard as actual usage rather than cross-referencing container environment variables. ## Risk Minimal. Observable gauges are sampled at the configured metric-export interval. `v8.getHeapStatistics()` and `process.memoryUsage()` are each microsecond-level calls, and six gauges are added to the same batch callback that already reads ~20 other Node.js runtime values per sample. Same registration pattern as the existing event-loop metrics in the file. ## Test plan - [ ] Deploy and confirm the six new gauges appear at the configured exporter - [ ] In cluster mode, confirm per-worker granularity (one series per cluster worker, tagged by `process.executable.name` / `service.instance.id`) - [ ] Confirm `nodejs.memory.heap.limit` reports the configured `--max-old-space-size` value in bytes |
||
|
Eric Allam
|
486f49791d |
fix(webapp): eliminate SSE abort-signal memory leak (#3430)
## Summary Fixes a server-side memory leak in the webapp's SSE helper. Every aborted SSE connection (client tab close, navigation, timeout) was pinning its full request/response graph indefinitely on Node 20, so any long-running webapp process accumulated retained memory proportional to streaming-request churn. ## Root cause `apps/webapp/app/utils/sse.ts` combined four abort signals via `AbortSignal.any([requestAbortSignal, timeoutSignal, internalController.signal])`. The composite signal tracks its source signals in an internal `Set<WeakRef>` registered against a `FinalizationRegistry`; under sustained traffic those entries accumulate faster than they're cleaned up, pinning every source signal (and its listeners, and anything those listeners close over) until the parent signal itself is GC'd or aborts. This is a long-standing Node issue with multiple open reports: - [nodejs/node#54614](https://github.com/nodejs/node/issues/54614) — original report, still open. A [follow-up from ChainSafe](https://github.com/nodejs/node/issues/54614#issuecomment-4055656572) describes the exact same shape in a Lodestar production workload (req + timeout signals composed per request accumulating in long-running worker) and the same mitigation: drop `AbortSignal.any`, compose manually. - [nodejs/node#55351](https://github.com/nodejs/node/issues/55351) — mechanism confirmed by Node member @jasnell: *"the set of dependent signals known to the AbortSignal are kept in an internal Set using WeakRefs. The AbortSignals are being properly gc'd but the Set is never cleaned out of the WeakRefs making those leak."* Partially fixed by [PR #55354](https://github.com/nodejs/node/pull/55354), shipped in Node 22.12.0 — but only covers the tight-loop case, not long-lived parent signals. - [nodejs/node#57584](https://github.com/nodejs/node/issues/57584) — circular-dependency variant, still open. - [nodejs/node#62363](https://github.com/nodejs/node/issues/62363) — regression in Node 24/25 from an unrelated V8 change ("Don't pretenure WeakCells"). Different root cause, same symptom. A separate issue in `apps/webapp/app/entry.server.tsx` — `setTimeout(abort, ABORT_DELAY)` with no `clearTimeout` on success paths — kept the React render tree + `remixContext` alive for 30s per successful HTML request. Same pattern fixed upstream in React Router templates ([react-router#14200](https://github.com/remix-run/react-router/pull/14200)), never backported to Remix v2. ## What changed - **`apps/webapp/app/utils/sse.ts`** — single-signal abort chain. `AbortSignal.any` removed; `AbortSignal.timeout` replaced by a plain `setTimeout` cleared when the controller aborts; named sentinel constants used as stackless abort reasons; request-abort handler explicitly removed on cleanup. - **`apps/webapp/app/entry.server.tsx`** — clears the `setTimeout(abort, ABORT_DELAY)` timer in `onShellReady` / `onAllReady` / `onShellError`. - **`apps/webapp/app/v3/tracer.server.ts` + `env.server.ts`** — gates OpenTelemetry `HttpInstrumentation` and `ExpressInstrumentation` behind `DISABLE_HTTP_INSTRUMENTATION=true` as an escape hatch for future OTel-listener retention patterns. Defaults to enabled. - **`apps/webapp/app/presenters/v3/RunStreamPresenter.server.ts`** — uses the shared `ABORT_REASON_SEND_ERROR` sentinel. ## Verification ### Full-app reproduction (memlab) Isolated local harness, 500 abrupt SSE disconnects against a dev-presence route, GC between passes, heap snapshot diff with [memlab](https://facebook.github.io/memlab/): | Run | Heap delta after 500 conns + GC | memlab retained leaks | | --- | --- | --- | | Before | +16.0 MB (linear with request count) | 158 clusters; 250 `ServerResponse`, 1000 `AbortController`, 250 `SpanImpl` retained | | After | **+3.3 MB (noise)** | **0 app-code leaks** | ### Standalone mechanism isolation To confirm *which* axis of the change is load-bearing, a separate standalone Node script (`/tmp/abort-leak-test.mjs`) ran 2000 requests × 200 KB payload per variant: | Variant | Heap delta after GC | | --- | --- | | baseline (no signal machinery) | 0 MB | | V1: `AbortSignal.any` + string abort reason | **+9.1 MB** | | V2: `AbortSignal.any` only (no reason) | **+10.8 MB** | | V3: string reason only (no `AbortSignal.any`) | 0 MB | | V4: neither (the fix) | 0 MB | | V5: `AbortSignal.any` with no listener on the composite | **+10.2 MB** | This proves `AbortSignal.any` is the sole mechanism. The reason type (`.abort()` vs `.abort("string")`) is irrelevant for retention — V3 is clean, V5 leaks even without a listener on the composite. ## Risk - `sse.ts` is used by the dev-presence routes. Behaviour is equivalent — timeouts and client disconnects still abort the stream. `signal.reason` is now a named string sentinel (`"timeout"`, `"request_aborted"`, etc.) instead of the previous string arg or default `AbortError`. No in-tree reader of `signal.reason` exists. - `entry.server.tsx` change is a standard cleanup of an abort timer, matches upstream React Router guidance. - `tracer.server.ts` change is env-gated and defaults to current behaviour. - Three other webapp `AbortSignal.timeout()` callsites (alert delivery, remote-build status) are fire-and-forget passed directly to `fetch` — not composed with anything long-lived, no retention risk, untouched. ## Test plan - [ ] Existing SSE integration tests pass - [ ] Dev-presence SSE behaves normally across tab open/close cycles - [ ] No heap growth under sustained aborted-connection traffic (heap snapshot diff) ## Follow-up The same `AbortSignal.any([userSignal, internalSignal])` pattern exists in several SDK/core callsites that ship to customers (`packages/core/src/v3/realtimeStreams/manager.ts`, `packages/trigger-sdk/src/v3/{ai,chat,chat-client,sessions}.ts`, `packages/core/src/v3/workers/warmStartClient.ts`). Whether those leak in practice depends on the user passing a long-lived signal. Tracked separately. |
||
|
Eric Allam
|
2d3b2e82e6 |
feat(run-engine): flag to route getSnapshotsSince through read replica (#3423)
## Summary Adds `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` (default `"0"`). When enabled, the Prisma reads inside `RunEngine.getSnapshotsSince` run against the read-only replica client instead of the primary. Offloads the snapshot-polling queries fired by every running task runner off the writer. ## Why `getSnapshotsSince` is called from the managed runner's fetch-and-process loop (once per poll interval, plus on every snapshot-change notification). It runs four sequential reads per call — one `findFirst` by snapshot id, one `findMany` on snapshots with `createdAt > X`, one raw SQL against `_completedWaitpoints`, and chunked `findMany` on `waitpoint`. Per concurrent run, every few seconds. It's read-only, tolerates a small amount of staleness, and is an obvious candidate for the replica. ## Replica-lag considerations - **Step 1 "since snapshot not found"**: if the runner just received a snapshot id from the primary and asks the replica before it replicates, the function throws and the caller treats the response as an error (runner falls back to a metadata refresh). Self-correcting, not silent. - **Step 2 missing newly-created snapshots**: the next poll's `createdAt > sinceSnapshot.createdAt` filter still picks them up once the replica catches up. - **Waitpoint junction race**: the riskiest path — if a latest snapshot is replicated but its `_completedWaitpoints` join rows aren't yet, the runner could advance past that snapshot with `completedWaitpoints: []`. WAL/storage-level replication replays commits in order, so in practice both should appear atomically on the reader, but the race window is why the flag ships disabled. Aurora reader shrinks all three windows to single-digit ms in typical conditions, and its storage-level replication gives atomic visibility of committed transactions on the reader. ## Test plan - [ ] Flip the flag on in a non-prod environment, confirm snapshot polling behaves normally and `getSnapshotsSince` errors in Sentry stay flat. - [ ] Verify writer query volume drops and reader query volume rises on the snapshot-polling queries. - [ ] Keep an eye on `AuroraReplicaLag` (or equivalent) during rollout. |
||
|
Eric Allam
|
7c95ee498e |
feat(webapp): tag Prisma spans with db.datasource attribute (#3422)
## Summary
Stamp every Prisma span with `db.datasource: "writer" | "replica"` so
traces can distinguish which client the query went through.
Both `PrismaClient` instances share the same global
`@prisma/instrumentation`, so their spans come out with identical names
and attributes today. This makes them trivially filterable.
## How
Two pieces in `apps/webapp/app/`:
1. **`v3/tracer.server.ts`** — a `DatasourceAttributeSpanProcessor`
reads an OTel context key in `onStart` and calls
`span.setAttribute("db.datasource", value)`. Registered as the first
span processor.
2. **`db.server.ts`** — `tagDatasource(datasource, client)` wraps each
`PrismaClient` with `$extends({ query: { $allOperations } })`. The
middleware sets the context key around the query and directly tags the
active span (to catch `prisma:client:operation`, which Prisma creates
before the middleware fires).
### Context-propagation gotcha
`PrismaPromise` is lazy — `query(args)` returns a thenable that only
starts when someone `.then()`s it. The naive `context.with(ctx, () =>
query(args))` restores ALS synchronously, so when Prisma's internal code
awaits the thenable later, the engine spans fire with the original ALS.
Wrapping as `async () => await query(args)` forces the `.then()` inside
the `context.with` callback, so ALS stays on our context for the engine
spans.
### Coverage
- **Tagged**: all `prisma:engine:*` (`connection`, `db_query`,
`serialize`, `query`, etc.), `prisma:client:operation`,
`prisma:client:serialize`, `prisma:client:connect`
- **Not tagged**: `prisma:client:load_engine` — one-time startup, fires
before any query
Concurrent `Promise.all([writer.x, replica.y])` correctly tags each pool
separately (ALS isolates per-Promise chain).
### Performance
One `context.with` (~200ns) and one `setAttribute` per span (effectively
free per OTel JS benchmarks) per Prisma op. Negligible against a query
path measured in milliseconds.
## Test plan
- [ ] Verify `db.datasource` appears on `prisma:engine:connection` spans
after the webapp is restarted
- [ ] Spot-check a handful of real traces carry the attribute
|
||
|
nicktrn
|
b570586899 |
fix(webapp): allow cancelling runs in DEQUEUED status from the runs list (#3421)
The cancel button was missing from the runs list for runs in `DEQUEUED` status. The runs list gates the button on `run.isCancellable`, which goes through `isCancellableRunStatus` -> `CANCELLABLE_RUN_STATUSES` = `NON_FINAL_RUN_STATUSES`. `DEQUEUED` was never added to that list when it was introduced in the run engine. The single run page uses a separate check (`!run.isFinished`, i.e. the inverse of `FINAL_RUN_STATUSES`), so cancellation already worked there - only the list was affected. Adding `DEQUEUED` to `NON_FINAL_RUN_STATUSES` also flips `isCrashableRunStatus` and `isFailableRunStatus`, but: - The crash path is the right behaviour - a `DEQUEUED` run (worker has claimed but not yet executing) can legitimately crash before `EXECUTING`, same as `PENDING`/`DELAYED` already do. - The fail path (`failedTaskRun.server.ts`) is only reached from V1 code paths (marqs consumers, v1 heartbeat handler). `DEQUEUED` is a V2-engine-only status, so V1 consumers never see it. When cancelling a `DEQUEUED` run the execution snapshot goes to `PENDING_CANCEL` (worker must ack) but `TaskRun.status` flips to `CANCELED` immediately - the UI reflects cancellation without waiting for the worker. Added an integration test in `run-engine/src/engine/tests/cancelling.test.ts` covering the full trigger -> dequeue -> cancel -> worker-ack flow. ## Stall safety The stall recovery path (PENDING_EXECUTING heartbeat miss -> nack-and-requeue -> back to QUEUED) lives entirely inside `@internal/run-engine` and never touches the webapp's `taskStatus.ts` helpers - the engine has zero imports from `~/v3/taskStatus` and doesn't know `CrashTaskRunService` / `FailedTaskRunService` exist. A stalled DEQUEUED run still goes back to the queue for retry; this change cannot cause stalls to crash or fail. The only realistic impact is the intended UI fix - the theoretical V1 crash/fail branches for DEQUEUED are unreachable in practice because V1 runs never have DEQUEUED status. |
||
|
Eric Allam
|
881288c615 |
feat(webapp): deprecate v3 CLI deploys server-side (#3415)
## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Summary Adds a server-side gate that detects deploy attempts from v3 CLI versions (i.e. `trigger.dev@3.x`) at the `POST /api/v1/deployments` entry point and, when enabled, rejects them with a clear upgrade message. v4 CLI deploys are completely unaffected. The last 3.x CLI release was `3.3.7`, which we can't update. This approach short-circuits the deploy before any DB writes, image-ref generation, S2 stream creation, or queue enqueue — no side effects in either mode. ## How v3 vs v4 are distinguished I pulled the published CLI tarballs for `trigger.dev@3.3.7`, `4.0.0`, `4.0.1`, `4.0.5`, `4.1.0`, `4.2.0`, and the current `4.4.4` in the repo. The cleanest, most reliable signal is the request body to `POST /api/v1/deployments`: | Field on initialize | v3.3.7 CLI | v4.x CLI | |---|---|---| | `type` | **never sent** | always sent — `"MANAGED"` (run_engine_v2) or `"V1"` | | `isNativeBuild` / `gitMeta` / `triggeredVia` / `runtime` | not sent | sent | | `registryHost` / `namespace` | sent (v3-only; stripped by current Zod schema) | not sent | Every v4 call site I inspected sets `type: features.run_engine_v2 ? "MANAGED" : "V1"` unconditionally. `payload.type` is `undefined` if and only if the client is a 3.x CLI. ## Behavior - Detection always runs and emits `logger.warn("Detected deploy from deprecated v3 CLI", { environmentId, projectId, organizationId, enforced })`, which lets us watch how many v3 deploys are still happening before enforcement is flipped. - Enforcement is gated behind `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` (default `"0"`, off). When `"1"`, the server returns `400` with: > The trigger.dev CLI v3 is no longer supported for deployments. Please upgrade your project to v4: https://trigger.dev/docs/migrating-from-v3 The v3 CLI surfaces this verbatim as `Failed to start deployment: <message>` because `zodfetch` throws `ApiError` for non-retryable 4xx (400/422) and `deploy.js` in 3.3.7 prints `error.message`. ## Out of scope (intentionally) - `api.v1.deployments.$deploymentId.finalize.ts` / `FinalizeDeploymentService` / `createDeploymentBackgroundWorkerV3.server.ts` are V1-engine paths, not the v3 CLI gate. Leaving them alone per review. - Container-side `createDeploymentBackgroundWorker` call in `managed-index-controller.ts` is still used by v4's in-image indexer. Not touched. - v3 `trigger dev` flow (different code path) — separate deprecation if/when needed. ## Testing - Ran `pnpm run typecheck --filter webapp` locally — passes. - Verified v4 tarballs (4.0.0, 4.0.1, 4.0.5, 4.1.0, 4.2.0, 4.4.4) all include `type:` in the `initializeDeployment` call site, so none will be accidentally blocked. - Verified v3.3.7 tarball's `initializeDeployment` payload has no `type` field. Rollout plan after merge: 1. Deploy with `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` unset → watch `Detected deploy from deprecated v3 CLI` log volume. 2. When comfortable, set `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1` to enforce. --- ## Changelog Detect v3 CLI deploys on `/api/v1/deployments` and, when `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1`, reject them with an upgrade message pointing at https://trigger.dev/docs/migrating-from-v3. v4 CLI deploys are unaffected. Link to Devin session: https://app.devin.ai/sessions/b242c11bd86e4099aeec8b59bab62143 Requested by: @ericallam |
||
|
Eric Allam
|
9636e43567 |
fix(webapp): reduce error-level log noise for handled/benign cases (#3403)
Two changes to cut error volume from logs that represent handled
conditions, not real errors (combined ~1600/hr in prod):
1. api.v1.waitpoints.tokens.$waitpointFriendlyId.complete.ts
The route throws `json(..., { status: 404 })` when a waitpoint
isn't found, but the generic catch block caught that Response,
logged it as an error (with an empty {} body because Error fields
are non-enumerable), and rethrew as a 500 — so clients saw a 500
instead of the intended 404, and every stale-waitpoint request
produced a Sentry event.
Fix: re-throw Response objects unchanged so the correct status
propagates and we don't log user 404s as errors. Also serialize
remaining Error instances explicitly (name/message/stack) so the
logs are actionable when we do hit a real error.
2. v3/marqs/sharedQueueConsumer.server.ts:603
"Task run has invalid status for execution. Going to ack" — the
message itself says we're handling it gracefully. Benign race
between dequeue and completion/cancellation. Demote to warn.
|
||
|
Eric Allam
|
79b6053e13 |
feat(server): add TaskIdentifier registry to replace expensive distinct query (#3368)
Replace the expensive DISTINCT query for task filter dropdowns with a dedicated TaskIdentifier registry table backed by Redis. Environments migrate automatically on their next deploy, with a transparent fallback to the legacy query for unmigrated environments. Also fixes duplicate dropdown entries when a task changes trigger source, and adds active/archived grouping for removed tasks. Moves BackgroundWorkerTask reads in the trigger hot path to the read replica. |
||
|
nicktrn
|
93f2ca6bf4 |
feat(webapp): extend admin workers endpoint and unify admin api auth (#3390)
Extends the admin worker groups endpoint with a GET loader and more fields on POST (type, hidden, workloadType, cloudProvider, location, staticIPs, enableFastPath), and pulls the PAT + admin check that was inlined or locally duplicated across every admin.api route into a shared helper in personalAccessToken.server.ts. The generic authenticateAdminRequest returns a discriminated result; requireAdminApiRequest is the thin Remix loader/action wrapper that throws. The neverthrow-style route (platform-notifications.ts) now composes the generic helper instead of duplicating the check. Verified locally against GET (listing) and POST (new fields, invalid enum, minimal backwards-compat). |
||
|
Oskar Otwinowski
|
097fab0d8c |
feat(webapp): Vercel integration - disable auto promotions (#3376)
<img width="1044" height="709" alt="image" src="https://github.com/user-attachments/assets/7f2cf25c-3b74-46a9-8794-41be077e04bf" /> |
||
|
Eric Allam
|
ed0c3e4f38 |
fix: stop creating TaskRunTag records and join table entries during triggering (#3369)
The TaskRun.runTags string array already stores tag names, making the TaskRunTag M2M relation redundant write overhead. Remove createTags calls, connect: tags, and join table writes from both V1 and V2 trigger paths. Simplify the add-tags API to just push to runTags directly. |
||
|
Eric Allam
|
417ab876e3 | fix(batch-queue): Batch items that hit the environment queue size limit now fast-fail (#3352) | ||
|
nicktrn
|
e59614a31c |
feat(webapp): gate microvm regions behind compute access feature flag (#3366)
Adds region-level gating so MICROVM regions are only visible and usable by orgs with the `hasComputeAccess` feature flag. Admins and explicit allowlist behavior unchanged. - New shared helper (`regionAccess.server.ts`) with `resolveComputeAccess`, `defaultVisibilityFilter`, and `isComputeRegionAccessible` - `RegionsPresenter` filters out MICROVM regions for non-compute orgs - `SetDefaultRegionService` blocks setting a MICROVM region as default without compute access - `WorkerGroupService` blocks triggering runs in MICROVM regions without compute access - `computeTemplateCreation` refactored to use shared `resolveComputeAccess` - Updated snapshot callback schema |
||
|
Eric Allam
|
f1f1d02f1d | feat(dashboard): a few tweaks to the AI models page (#3315) | ||
|
Matt Aitken
|
0e14b6d750 |
TaskRun optimizations: dropping FKs and some indexes (#3309)
## Summary - Drop all 8 foreign key constraints on TaskRun. The run listing path is now fully ClickHouse-backed so we no longer need Postgres to enforce referential integrity on this table. The FK constraints add write overhead on every insert/update with no remaining benefit. Prisma queries are unaffected. - Remove PostgresRunsRepository and its associated feature flag (runsListRepository), which was the last remaining code path querying TaskRun directly for list/count operations. - Drop three indexes that were only useful for the Postgres run list path and have no remaining query consumers: - TaskRun_runtimeEnvironmentId_id_idx — was the cursor pagination index for PostgresRunsRepository; superseded by the (runtimeEnvironmentId, createdAt DESC) composite index - TaskRun_scheduleId_idx — redundant with the (scheduleId, createdAt DESC) composite index; no direct Postgres queries filter by scheduleId alone - TaskRun_rootTaskRunId_idx — no queries filter TaskRun by rootTaskRunId as a WHERE clause anywhere in the codebase All index drops use CONCURRENTLY IF EXISTS to avoid table locks in production. ## Test plan - pnpm run db:migrate:deploy applies all migrations cleanly - pnpm run typecheck --filter webapp passes - Run list pages load correctly in the dashboard (ClickHouse path) - Scheduled task runs still trigger and appear correctly |
||
|
Matt Aitken
|
68e88d0d71 |
Object Storage seamless migration (#3275)
This allows seamless migration to different object storage. Existing runs that have offloaded payloads/outputs will continue to use the default object store (configured using `OBJECT_STORE_*` env vars). You can add additional stores by setting new env vars: - `OBJECT_STORE_DEFAULT_PROTOCOL` this determines where new run large payloads will get stored. - If you set that you need to set new env vars for that protocol. Example: ``` OBJECT_STORE_DEFAULT_PROTOCOL=“s3" OBJECT_STORE_S3_BASE_URL=https://s3.us-east-1.amazonaws.com OBJECT_STORE_S3_ACCESS_KEY_ID=<val> OBJECT_STORE_S3_SECRET_ACCESS_KEY=<val> OBJECT_STORE_S3_REGION=us-east-1 OBJECT_STORE_S3_SERVICE=s3 ``` --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com> |