cecdfd94be97cadbe041f513aabe9c8d711e5b1e
274 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Arderne
|
cecdfd94be | fix: only count preview branches toward the preview branch limit (#4283) | ||
|
nicktrn
|
0ff0abd776 |
fix(webapp): recover from stale /build assets via a bounded reload (#4282)
## Problem The webapp's HTML references content-hashed `/build` assets, and each running instance contains exactly one build and returns 404 for asset hashes it doesn't have. During a rolling deploy a client can hold HTML from one build while a request for one of its assets is served by an instance on a different build → missing styles or a failed chunk load. ## What this does On a `/build` stylesheet/script/chunk load failure, the client does a **bounded full-document reload** (at most 2 per 5 minutes, tracked in `sessionStorage`) so the page reloads onto a single consistent build. That's the whole mechanism — no polling, no `fetch` interception, no blocking overlay, no form snapshotting. - `apps/webapp/app/components/StaleAssetRecovery.tsx` — authored as a typed, lint-checked function and serialized to an inline script via `.toString()` (so the logic is real, reviewable code, not an opaque string), injected before `<Links />`, production only. - Detection: capture-phase `error` listener for `<link>`/`<script>`/modulepreload failures under `/build/`, plus an `unhandledrejection` guard for dynamic-import failures. - Guards: once-per-page re-entrancy guard, the bounded reload budget, and a `navigator.onLine` check so it never reloads into an offline error page. - Unit tests in `StaleAssetRecovery.test.ts`. ## Relationship to #4260 Replaces the recovery introduced in #4260 (reverted in #4280) with a much smaller, reload-only approach — the previous version intercepted `fetch` and could turn a data request into a navigation, and showed a full-screen overlay on any asset error; this drops both. ## `/build-version` compatibility shim `apps/webapp/server.ts` adds a tiny `GET /build-version` endpoint (build id only, `no-store`). A previously-deployed client build polls it after an asset failure and reloads once it sees a newer build, so those older tabs recover in one reload instead of getting stuck. Temporary — safe to remove once older clients have cycled out. It deliberately does **not** re-add an `X-Build-Id` response header. ## Also Restores the `.server-changes` writing guidance in `.claude/rules/server-apps.md` (reverted alongside #4260). ## Self-hosting note Recovery is most reliable when your load balancer keeps a client on one instance for the duration of a deploy (short session stickiness) — the reload then lands on a consistent build in one hop. |
||
|
Chris Arderne
|
73d966ad22 |
chore(webapp): remove deprecated realtime stream write action (#4250)
Removes the deprecated realtime stream write action kept for retired v3 task clients. Supported clients use the targeted stream write routes, while the existing stream read loader remains unchanged. |
||
|
nicktrn
|
051d7080d6 |
Revert "fix(webapp): survive asset hash rotation across rolling deploys (#4260)" (#4280)
Standard `git revert` of #4260. Its client-side stale-asset recovery is net-negative during normal deploys: - The `fetch` interception treats any `?_data=` request (Remix loader **and** action traffic) as a navigation and, on a build-id mismatch, `location.assign`es the tab to the fetched URL — an open dashboard tab can be hard-navigated to a raw data URL during a rolling deploy, losing unsaved input. - Any transient `/build` asset error (a network blip, an extension, an unrelated failed dynamic import) blanks the page behind a full-screen overlay for ~60s before offering a manual reload. - It serialized form field values to `sessionStorage` to restore them across the reload. This returns the webapp to the pre-#4260 baseline as a fast, low-risk step. Follow-ups (separate PRs): - a minimal reload-only recovery to replace this, - restore the unrelated `.claude/rules/server-apps.md` docs tidy-up from #4260 (via cherry-pick), - a load-balancer stickiness change addressing the root cause. |
||
|
Eric Allam
|
43250522a5 |
fix(run-store): fix batch idempotency lookup on the dedicated run-ops store (#4271)
## Summary `batchTrigger` requests that set a per-item `idempotencyKey` failed with a 500 when the run-store is split across databases: the per-item idempotency lookup errored before any run was created. Batches without per-item keys, single `trigger` idempotency, and batch-level (`idempotency-key` header) idempotency were unaffected. ## Root cause `findRunsByIdempotencyKeys` built its `UNION ALL` of per-key point-lookups with `@trigger.dev/database`'s `Prisma.sql` / `Prisma.join`, then executed it on whichever store client it was handed. On the dedicated run-ops store that client is a *separate* generated Prisma client, and a `Sql` object from a different generated client is not recognized: the bare `$queryRaw(Prisma.join(...))` form dropped the query text entirely (`Argument \`query\` is missing`). The tagged-template form is no better here: joining nested `Prisma.sql` fragments across the two clients mis-numbers the bound parameters (`syntax error at or near "$1"`). ## Fix Build the lookup as a plain parameterized string and run it via `$queryRawUnsafe` with positional placeholders and bound values, so it no longer depends on which generated client executes it. The query text contains only static SQL and integer placeholders; every value (`runtimeEnvironmentId`, `taskIdentifier`, each key) is bound, so it is not a raw-interpolation site. Same per-key point-lookup shape as before, no change on the single-client path. Verified end-to-end against a bundled build with the run-store split enabled: before the fix, `batchTrigger` with a per-item key 500s; after, it returns the runs and dedups correctly across fresh, repeat, and mixed batches. |
||
|
Iss
|
80cbc46bf6 |
fix(webapp): log transient Attio 5xx/429 at warn instead of error (#4270)
The signup → Attio sync (`attio.server.ts` `#assert`) logged every non-2xx response at `error` level and threw the same way regardless of status. Transient upstream failures (5xx/429) are retried by the common worker and self-heal, so treating them as errors created false alerts for something that isn't actually a bug. Now `#assert` splits the two cases: - **5xx / 429** — Logged at `warn` and thrown with `logLevel: "warn"`, so they continue to be retried but don't raise error-level alerts. This reuses the same pattern the worker already honors (`directorySyncEffects`). - **4xx** — Unchanged: logged at `error` and thrown, so genuine integration bugs (schema, permissions, auth, etc.) remain visible. There is no behavior change to retries or the signup flow. This is a server-only change. --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
nicktrn
|
890dd66eb5 |
feat(webapp): route ClickHouse reads to an optional read replica (#4081)
## Summary Adds optional configuration to send ClickHouse read traffic to a separate instance (for example a read replica) while writes stay on the primary `CLICKHOUSE_URL`. This lets operators offload read load (runs list, traces, logs, queries) from the cluster that handles inserts. Fully backwards compatible: with nothing new set, every client resolves to `CLICKHOUSE_URL` exactly as before. ## What it adds - `CLICKHOUSE_READER_URL` (optional): a single reader endpoint that the read-only clients fall back to. Read clients resolve `<own URL> ?? CLICKHOUSE_READER_URL ?? CLICKHOUSE_URL`. The task-events client (which both inserts events and reads traces, spans, and logs) is built as a reader/writer pair so queries use the reader while inserts stay on `CLICKHOUSE_URL`. - `RUNS_LIST_CLICKHOUSE_URL` (optional): a dedicated client for the runs list (dashboard list, runs list API, live reload, child-status counts), so the highest-traffic read path can target its own instance. ## Safety Only read-only clients fall back to the reader: logs, query, admin, runs list, the pending-version lookup, and the realtime run-id resolver. The query page is constrained to read-only (the TSQL parser rejects anything that is not a `SELECT`, and a `readonly` setting is applied). The task-events client routes inserts to the writer and queries to the reader per method, so a write can never reach the reader. Pure-write clients (event inserts, replication) always use `CLICKHOUSE_URL`. Note: this PR targets a baseline branch rather than `main` so the diff stays scoped to the read-replica changes. It will be retargeted to `main` before merge. --------- Co-authored-by: Eric Allam <eallam@icloud.com> |
||
|
Eric Allam
|
1ab5066ed0 |
perf(webapp,run-store): point-lookup batch idempotency keys (#4255)
## Summary Batch triggers that use per-item idempotency keys could take seconds instead of milliseconds when the target task had a large run history. This keeps the idempotency lookup fast regardless of how many runs a task has accumulated. ## Root cause The batch path checks which items already have runs by looking up their idempotency keys with a single `WHERE runtimeEnvironmentId = ? AND taskIdentifier = ? AND idempotencyKey IN (...)` query. On a very large `TaskRun` table Postgres underestimates the row count of a specific `(environment, task)` pair, so once the `IN` list grows past a handful of keys it stops doing per-key index probes and instead scans every run for that `(environment, task)` and filters the keys in memory. The cost is then flat and large regardless of how many keys are being checked, and a routine `ANALYZE` does not correct the estimate at that table size. ## Fix Look each idempotency key up on its own, batched into a `UNION ALL` of point lookups (chunked, run with bounded concurrency). Each branch is an equality on all three columns of the unique index, so the planner can only do a per-key index probe and can never fall back to the range scan. Same results, same columns, confined to the batch trigger path. |
||
|
Katia Bulatova
|
2aa64200f8 |
fix(webapp): survive asset hash rotation across rolling deploys (#4260)
### Problem Webapp HTML references content-hashed /build assets, and each Docker image contains exactly one build with a hard 404 for unknown hashes. During a rolling deploy, a client holding HTML from the old build may request old asset hashes from a replica running the new image, causing missing styles or failed chunk loads. The page should recover automatically once a compatible build becomes available, without reload loops or unnecessary interruptions during normal deployments. ### What changed - Build changes alone do nothing — no polling, no automatic reloads. - If a CSS or JavaScript asset fails to load, a recovery overlay is shown immediately. - While the server still reports the same build, the client polls for a newer build using exponential backoff (up to ~60s). As soon as a newer build is detected, the page reloads automatically. - If no newer build appears within the timeout, recovery falls back to a manual Reload action. - If recovery still fails after the automatic reload, the client stops retrying and displays a final recovery screen instead of entering a reload loop. - Recovery preserves form values and scroll position across the automatic reload. |
||
|
Daniel Sutton
|
a1ca64613b |
fix(webapp): reuse the primary db pool for legacy run-ops when DSNs match (#4253)
## Summary When the run-ops split is enabled, the legacy run-ops database client was always constructed as its own connection pool, even when it points at the same database as the primary (control-plane) client. On setups where those two DSNs resolve to the same physical database, this opened a second, redundant pool and doubled the number of connections used against that database. This change makes the legacy client reuse the primary client's pool whenever their DSNs point at the same database, and only open a separate pool when they genuinely differ. ## Fix A small `sameDatabaseTarget` comparison (host, port, database name, user) decides whether the legacy DSN points at the same database as the primary. When it does, the legacy handle reuses the primary client by reference, so no second pool is opened. When the DSNs diverge, the legacy client is built independently as before, so the split still works once the databases are actually separate. Two smaller changes ride along: - An optional per-pool limit for the run-ops read replica, which connects unpooled and so draws raw backend connections; unset, it falls back to the existing default and behaviour is unchanged. - A startup warning about a missing legacy replica URL is now suppressed when the legacy client shares the primary pool, where it would be misleading. ## Verification Booted the webapp end-to-end in three modes and confirmed the pools opened as expected via the client's own startup logs and live backend connection counts: split off (single pool), split on with a shared database (legacy reuses the primary pool, no doubling), and split on with separate databases (legacy opens its own pool). |
||
|
github-actions[bot]
|
165955781d |
chore: release v4.5.4 (#4228)
📚 Publish docs / publish (push) Has been cancelled
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 1s
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
## Summary 2 new features, 11 improvements, 5 bug fixes. ## Breaking changes - Trigger.dev v3 is no longer supported. For self-hosted deployments, 4.5.0 is the last version we officially support for running v3; stay on 4.5.0 or upgrade to v4. v3 triggers, batch triggers, reschedules, and deploys now return a clear upgrade message instead of running. ([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236)) ## Improvements - You can now mark environment variables synced via the `syncEnvVars` build extension as secrets. Return `{ name, value, isSecret: true }` from your callback and those variables are stored redacted in the dashboard, just like manually created secret env vars. ([#4203](https://github.com/triggerdotdev/trigger.dev/pull/4203)) - Remove the legacy `--mcp` and `--mcp-port` options from the `dev` command. Run the dedicated `trigger mcp` command to start the Trigger.dev MCP server. ([#4246](https://github.com/triggerdotdev/trigger.dev/pull/4246)) - Removed the unused `ResourceMonitor` export from `@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper with no remaining consumers. ([#4244](https://github.com/triggerdotdev/trigger.dev/pull/4244)) - Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the legacy v3 socket message schemas. These were only used by the now-retired v3 engine and have no v4 consumers. ([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236)) ## Bug fixes - Fix a `chat.agent` message-loss race where sending a message right after an action (such as an undo) could drop the follow-up's response from the UI until a refresh. ([#4234](https://github.com/triggerdotdev/trigger.dev/pull/4234)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Added `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` to skip all PostgreSQL task-event writes for deployments that store task events in ClickHouse. Leave it off unless `EVENT_REPOSITORY_DEFAULT_STORE` is `clickhouse_v2`, otherwise task events are lost. ([#4242](https://github.com/triggerdotdev/trigger.dev/pull/4242)) - Promo credits: a /promo signup landing page, redeeming a promo code when a new org selects a plan, and showing remaining credits on the usage page. ([#4138](https://github.com/triggerdotdev/trigger.dev/pull/4138)) - Speed up retrieving a background worker by version. The endpoint no longer runs a slow lookup that scanned the full task table for large deployments; it now reuses data it already loads, so the response is the same but returns much faster. ([#4245](https://github.com/triggerdotdev/trigger.dev/pull/4245)) - Clearer login error when an email address is blocked by the WHITELISTED_EMAILS setting: the message now explains the address isn't allowed on this instance instead of the ambiguous "This email is unauthorized". ([#4220](https://github.com/triggerdotdev/trigger.dev/pull/4220)) - Make the native build server the default in project build settings. It's now opt-out, stored as a new `disableNativeBuildServer` key. Also clarifies in the UI that build settings apply to GitHub-triggered and native build server deployments. ([#3980](https://github.com/triggerdotdev/trigger.dev/pull/3980)) - Optionally process high-volume telemetry ingestion in parallel for higher throughput under heavy load by setting `OTEL_TRANSFORM_WORKER_POOL_ENABLED=1`. Off by default. ([#4232](https://github.com/triggerdotdev/trigger.dev/pull/4232)) - Add a `REALTIME_BACKEND_DEFAULT` env var to choose the default realtime backend (`electric`, `native`, or `shadow`) for environments whose org has no per-org override. Defaults to `electric`, so existing behavior is unchanged. ([#4231](https://github.com/triggerdotdev/trigger.dev/pull/4231)) - Clarified on the Regions page that a region only affects where your runs execute, not where your data is stored. This shows as a tooltip on the Location column and in the confirmation dialog when you change your default region. ([#4226](https://github.com/triggerdotdev/trigger.dev/pull/4226)) - Improved the reliability of how run data is read and written. ([#4237](https://github.com/triggerdotdev/trigger.dev/pull/4237)) - Fixed stale login errors: an error from a previous login attempt (for example a rejected email address) no longer keeps reappearing on the login page and no longer makes later, successful attempts look like they failed. ([#4220](https://github.com/triggerdotdev/trigger.dev/pull/4220)) - The Errors page now shows better details for each error. Errors that don't carry a message — such as errors thrown without a message, or values thrown that aren't `Error` objects — get a meaningful title instead of all reading "Unknown error", and are grouped by their name (or value) rather than collapsed into a single group. The error type now shows the actual error name, and stack traces now appear where previously they were missing. ([#4225](https://github.com/triggerdotdev/trigger.dev/pull/4225)) - Return a clear client error when SSO form submissions use an unsupported content type ([#4238](https://github.com/triggerdotdev/trigger.dev/pull/4238)) - Query page: extracting fields from a run's output with JSON functions (such as JSONExtractString or JSONExtractInt) no longer fails with an "illegal type: JSON" error. ([#4221](https://github.com/triggerdotdev/trigger.dev/pull/4221)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.4 ### Patch Changes - You can now mark environment variables synced via the `syncEnvVars` build extension as secrets. Return `{ name, value, isSecret: true }` from your callback and those variables are stored redacted in the dashboard, just like manually created secret env vars. ([#4203](https://github.com/triggerdotdev/trigger.dev/pull/4203)) - Updated dependencies: - `@trigger.dev/core@4.5.4` ## trigger.dev@4.5.4 ### Patch Changes - Remove the legacy `--mcp` and `--mcp-port` options from the `dev` command. Run the dedicated `trigger mcp` command to start the Trigger.dev MCP server. ([#4246](https://github.com/triggerdotdev/trigger.dev/pull/4246)) - Updated dependencies: - `@trigger.dev/core@4.5.4` - `@trigger.dev/build@4.5.4` - `@trigger.dev/schema-to-json@4.5.4` ## @trigger.dev/core@4.5.4 ### Patch Changes - Removed the unused `ResourceMonitor` export from `@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper with no remaining consumers. ([#4244](https://github.com/triggerdotdev/trigger.dev/pull/4244)) - Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the legacy v3 socket message schemas. These were only used by the now-retired v3 engine and have no v4 consumers. ([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236)) ## @trigger.dev/python@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.4` - `@trigger.dev/core@4.5.4` - `@trigger.dev/build@4.5.4` ## @trigger.dev/react-hooks@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.4` ## @trigger.dev/redis-worker@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.4` ## @trigger.dev/rsc@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.4` ## @trigger.dev/schema-to-json@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.4` ## @trigger.dev/sdk@4.5.4 ### Patch Changes - Fix a `chat.agent` message-loss race where sending a message right after an action (such as an undo) could drop the follow-up's response from the UI until a refresh. ([#4234](https://github.com/triggerdotdev/trigger.dev/pull/4234)) - Updated dependencies: - `@trigger.dev/core@4.5.4` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Eric Allam
|
29598a77b8 |
feat(webapp): add option to disable PostgreSQL task-event writes (#4242)
## Summary Adds `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` (default off), which makes the task-event store skip all PostgreSQL `TaskEvent` writes. It's for deployments that store task events in ClickHouse (`EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2`) and no longer want the PostgreSQL copy. ## How it works The guard sits at the single postgres write boundary, `TaskEventStore.create` / `createMany`, so it covers every write path (OTLP ingestion and run-lifecycle events) with one check. Reads are untouched (`findMany` / trace queries / streaming), so existing PostgreSQL events remain readable. Leave it off unless the default store is `clickhouse_v2`, otherwise task events for any run still routed to PostgreSQL would be dropped. |
||
|
Daniel Sutton
|
e0b42a88d6 |
perf(webapp): avoid unindexed fileId scan in get-background-worker-by-version (#4245)
## What
The `GET
/api/v1/projects/:projectRef/background-workers/:envSlug/:version`
endpoint loaded each file's tasks through the nested `files.tasks`
relation. Prisma resolves that as a separate query:
```sql
SELECT id, slug, "fileId" FROM "BackgroundWorkerTask" WHERE "fileId" IN (...)
```
`BackgroundWorkerTask.fileId` is not indexed — the FK constraint exists,
but Postgres does not auto-create an index for foreign keys — so on a
large table this can only run as a sequential scan, which gets
progressively slower as the table grows and was observed taking minutes
per call in production.
The loader already loads every task for the worker via `tasks: true`,
which uses the indexed `workerId` relation, and those rows already
include `fileId`. This PR groups task slugs by `fileId` in memory from
that already-loaded data and drops the `files.tasks` include entirely.
## Behavior change (latent bug fix)
The response shape is unchanged, but there is a semantic correction for
**source files reused across worker versions** (files are de-duplicated
by `@@unique([projectId, contentHash])`, so one file row can be linked
to many workers).
- **Before:** `file.tasks` came from the `BackgroundWorkerFile.tasks`
relation, i.e. *every* `BackgroundWorkerTask` with that `fileId` —
across all workers sharing the file. So a worker's manifest could list
tasks it doesn't actually have.
- **After:** `file.tasks` is grouped from the queried worker's own
tasks, so it reflects only that worker version's tasks.
Verified on a local DB: 460 files are referenced by tasks from more than
one worker; of 6819 (worker, file) pairs, 6 differ — all one file where
the old union leaked a task slug (`cancellation-test`) into worker
versions that never had it. The new per-worker behavior is the correct
one for a worker-version manifest. (Thanks to the automated review for
flagging this.)
## Analysis
Captured the exact SQL before/after by instrumenting Prisma against real
data (a worker with 62 files):
- **Before:** 5 statements, including the `WHERE "fileId" IN (...)`
scan.
- **After:** 4 statements; the `fileId` query is gone and the other four
are identical.
EXPLAIN of the two access paths:
```
Before WHERE "fileId" IN (...)
Seq Scan on "BackgroundWorkerTask"
Filter: ("fileId" = ANY (...)) -- reads the whole table, scales with table size
After WHERE "workerId" IN (...)
Index Scan using "BackgroundWorkerTask_workerId_slug_key"
Index Cond: ("workerId" = ...) -- bounded by matching rows, scale-independent
```
No new index is required: the `workerId` access path is already covered
by the existing `BackgroundWorkerTask_workerId_slug_key` unique index.
## Testing
- `pnpm run typecheck --filter webapp` passes.
- Query capture + EXPLAIN performed against a local database seeded with
real worker/file/task data.
|
||
|
Daniel Sutton
|
bea7e2be90 |
feat(webapp,run-store): route run-graph reads and writes through the run-store router (#4237)
## Summary Run-graph data (runs, batches, waitpoints, and their related tables) can now live in a database separate from the control plane, with every read and write routed to the correct database by each run's residency. This makes reading and writing run data more reliable once the two are split, and is a no-op for single-database installs. ## Design - Run-graph table access goes through the run-store router, which selects the legacy or the new run-ops store per run instead of assuming one shared client. - The legacy run-ops client is now independently pointable, so legacy run data can be served from its own database (and replica) rather than the control-plane connection. - Run-graph writes go straight to the run-graph database instead of being forwarded through the control plane, and replication targets are split so runs in the new database still replicate to analytics without under-counting. - Read-through slots refuse the control-plane client, so a missing residency fails loudly instead of silently reading the wrong database. - Migration `20260710120000_drop_remaining_run_graph_seam_foreign_keys` drops the foreign keys that still crossed the run-graph / control-plane seam, which is what lets the two live in separate databases. The split stays off unless explicitly enabled and the two databases are confirmed physically distinct; startup fails closed otherwise. Verified by running the full dashboard end-to-end suite against both a single-database configuration and a three-database configuration (control plane, the new database, and a physically separate legacy database), with runs on both residencies. No misrouted reads in either configuration. |
||
|
Eric Allam
|
c23585710c |
docs: note v3 is retired and 4.5.0 is the last version supporting v3 (#4241)
## Summary Refreshes the docs for the v3 sunset: v3 (SDK v3) is end of life, and 4.5.0 is the last version we officially support for running v3. - The self-hosting overview, plus the docker and kubernetes version-locking sections, now tell self-hosters on v3 to stay on 4.5.0 or migrate to v4. 4.5.1 and later reject v3 triggers and deploys with an upgrade message. - The migration guide's deprecation notice was still written in the future tense (with dates that have since passed); it now describes v3 as retired and adds the self-hosted 4.5.0 cutoff. This is the page the server's upgrade message links to. - Fixes a stale "v3 project" reference in the CLI overview. The Mintlify preview will render the callouts for a visual check. |
||
|
Eric Allam
|
5ba8557a51 |
chore(webapp,core): remove the end-of-life v3 (engine V1) execution stack (#4236)
## Summary v3 (the engine that ran the SDK v3 era, internally `RunEngineVersion.V1`) is end-of-life. Following the removal of the v3 execution apps ([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and the legacy dev websocket ([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this removes the remaining v3 execution stack from the server. Clients still on v3 (an old SDK or CLI that has not upgraded) keep getting a clear "upgrade to v4" response. Triggers, batch triggers, reschedules, and deploys that resolve to v3 are rejected with a graceful 4xx pointing at the migration guide, never a 5xx, so a stale client cannot affect server health. Self-hosted instances still running v3 should stay on the 4.5.x release line until they migrate. ## What is removed - The MarQS queue and its shared/dev queue consumers. - The v3 socket.io namespaces (coordinator, provider, shared-queue) and the v3 run lifecycle services (attempt, checkpoint, and batch-resume). - The graphile-worker background job system; all live jobs already run on `@trigger.dev/redis-worker`. - The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally, so the flag is gone. - Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace` subpath and the legacy socket message catalogs) and the now-dead MarQS environment variables. ## What stays The v4 engine is untouched. The graceful v3 rejection boundary stays, `determineEngineVersion` still detects a v3 project so it can reject it, and the batch service plus batch-completion worker stay for current clients. Live queue concurrency limits and metrics now read from the v4 run engine instead of MarQS, and a brand-new dev environment now defaults to v4. ## Dependency cleanup Removes webapp dependencies left unused by this change: `seedrandom` and `semver` (only the removed v3 code used them) plus a set that was already dead, their orphaned `@types` packages, and two dead files. Adds a `knip:deps` script and a `knip.json` config so unused dependencies can be found the same way going forward. |
||
|
Chris Arderne
|
c0f7c803b1 |
fix(webapp): return 415 for invalid SSO form content types (#4238)
## Summary SSO form submissions with an unsupported content type now receive a 415 response instead of failing while parsing the request body. |
||
|
Saadi Myftija
|
5f2541d94f |
feat(webapp): make native build server the default in build settings (#3980)
Switches the native build server from opt-in to opt-out in project build settings. - It's now enabled by default, stored as a new \`disableNativeBuildServer\` opt-out key so previously-saved \`useNativeBuildServer: false\` values aren't treated as deliberate opt-outs. - The "Use native build server" checkbox is checked by default; unchecking it persists the opt-out. - Brief wording: clarifies build settings apply to GitHub-triggered and native build server deployments, and the native build server hint no longer says "in the future". |
||
|
Eric Allam
|
45527e317a |
feat(webapp): opt-in worker pool for OTLP ingest transform (#4232)
## Summary Under high OTLP ingest volume, the whole decode, transform, and enrich pipeline runs on the request event loop, so a single CPU core becomes the ceiling while the rest sit idle. This adds an opt-in worker pool that moves decode, transform, and LLM-cost enrichment onto worker threads, keeping the main thread free for I/O. It is off by default (`OTEL_TRANSFORM_WORKER_POOL_ENABLED`), so behavior is unchanged unless enabled. ## Design Workers do decode, filter, convert, and enrich (including LLM pricing match). The main thread stays the single database reader: it loads the pricing registry and broadcasts the compiled model rows to the workers (re-broadcasting on every reload), so workers never touch the database. The pure transform is extracted into a dependency-light module (no Prisma/Redis/ClickHouse imports) so it can run inside a worker. Importantly, the main thread keeps the existing single consolidated insert path, so ClickHouse insert batching and part count are unchanged. The parallelism buys CPU headroom, not more insert streams (which would add merge pressure). The worker is bundled as a standalone file at build time and ships in the existing image with no Dockerfile change. In local load testing the pool sustained roughly 2.6x the throughput of the single-thread path and kept the main thread responsive under load. |
||
|
Eric Allam
|
5d0e9d9dc5 |
feat(webapp): make the default realtime backend configurable (#4231)
## Summary The default realtime backend was hardcoded to Electric. This adds a `REALTIME_BACKEND_DEFAULT` env var (`electric` | `native` | `shadow`, default `electric`) that chooses the backend for any environment whose org has no `realtimeBackend` override. Behavior is unchanged unless you set it; per-org overrides still win. The default is applied at every point where the per-org flag falls through: the initial value, the flag lookup default, and the error fallback. |
||
|
Matt Aitken
|
2cac63f13a |
fix: improve error labelling, grouping, and stack traces in the Errors feature (#4225)
## Problem Several display/grouping issues in the **Errors** feature, all rooted in how the ClickHouse error materialized views (`errors_mv_v1`, `error_occurrences_mv_v1`) read the stored error JSON produced by `parseError`: 1. **Messageless errors show "Unknown error".** An empty message falls straight through `coalesce(nullIf(message,''), 'Unknown error')` to the literal, even though the error's class `name` is available (e.g. an Effect tagged error `ListMessagesError` with no message). 2. **Unrelated errors collapse into one group.** `calculateErrorFingerprint` keys on `type : message : stack`, where `type` is always the union tag (`BUILT_IN_ERROR`, …), `message` is empty, and the stack isn't read — so every messageless built-in error (and every string/custom error) hashes to the same constant input → one fingerprint. 3. **error_type shows the internal tag.** `coalesce(type, name, …)` always resolves to `type` (always present), so the column shows `BUILT_IN_ERROR` instead of the real class name. 4. **Stack traces never populate.** The MVs read `error.data.stack`, but the serializer stores the trace under `stackTrace` — so the column is always empty. ## Fix All display changes are `ALTER TABLE … MODIFY QUERY` on the two views (migration `035`); the fingerprint change is in the webapp. - **Fingerprint** (`errorFingerprinting.ts`): fall back **message → name → raw**. Messageless errors now group by class name (or raw value for non-Error throws); message-bearing errors are **unchanged** (short-circuits at `message`), so existing groups don't split — only currently-messageless errors get their own group going forward. - **error_message**: same `message → name → raw` fallback before `'Unknown error'`. - **error_type**: coalesce `name → code → 'Error'` (drops the reliance on the union tag). Built-in → class name, internal → `code`, string/custom → `Error`. - **stack trace**: read `error.data.stackTrace`. Bounded as before (serializer caps 50 frames / 1024 chars per line; MV clips to 2000 chars). ## Migration notes - `MODIFY QUERY` swaps the view query in place (no drop/recreate gap); Down restores the previous query. - **Existing rows are left unchanged** — changes apply only to rows inserted after the migration. No backfill. ## Tests `errorFingerprinting.test.ts` — 57 pass, incl. new cases for messageless class names, string/custom raw values, and stability of message-bearing fingerprints. Fixes the display-derivation half of TRI-11938 (error_type + stack trace); relates to TRI-9254 and TRI-9250. |
||
|
DKP
|
25eb0c71a0 |
fix(webapp): clarify that region only affects where runs execute (#4226)
## Summary This adds an always-visible info tooltip on the Location column and a note in the "set default region" confirmation dialog making it explicit. It also removes the obsolete "V4" badge from the Regions page title. |
||
|
Matt Aitken
|
48a0b83ec6 |
feat(webapp): promo credits — /promo signup landing, redeem at plan selection, usage display (#4138)
## What & why Signup promo credits. A new logged-out `/promo?code=<code>` landing page validates the code and carries it through signup via a cookie. When the new organization is activated by selecting a plan, the code is redeemed and its credits are applied; the usage page then shows the remaining promo credits and their expiry. ## Notes - The code is redeemed at **plan selection**, not org creation: the credit grant targets the org's usage allowance, which only exists once a plan is selected — applying at creation would have nothing to grant onto. Redemption is best-effort and never blocks plan selection. - Pairs with the corresponding billing-service change (promo code validate/apply/credits + grant issuance); the two are released together. ## Testing Verified locally end to end: `/promo` shows the offer, a new account carries the code through signup, selecting the Free plan redeems it, and the usage page shows the remaining credits. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Eric Allam
|
02cf9c81ad |
fix(tsql): make JSON functions work on the output and error columns (#4221)
## Summary A Query page (TRQL) query that pulls fields out of a run's `output` with JSON functions (`JSONExtractString`, `JSONExtractInt`, `JSONHas`, and the rest of the family) failed with "The first argument of function ... should be a string containing JSON, illegal type: JSON". Those queries now work. ## Root cause and fix `output` is a native ClickHouse `JSON` column, but `JSONExtract*`, `JSONHas`, `JSONLength`, and `JSONType` all expect a String containing JSON text. The compiler already swaps in the column's String companion (`output_text`) when a JSON column is selected or compared, but not inside function-call arguments, so it emitted `JSONExtractInt(output, 'x')` against the native column. The fix prints the companion column for the first argument of these functions when it resolves to a bare JSON field, keeping the table alias when qualified (so it works in JOINs): JSONExtractInt(output, 'x') -> JSONExtractInt(output_text, 'x') JSONExtractArrayRaw(assumeNotNull(output), 'y') -> JSONExtractArrayRaw(assumeNotNull(output_text), 'y') It also reaches through value-preserving passthrough wrappers like `assumeNotNull(...)`, while leaving value-changing wrappers like `toJSONString(output)` on the native column (that argument is already a String). The swap is also semantically correct, not just a type fix: `output_text` is the unwrapped data JSON that the TRQL `output` model already represents, so field paths line up. Covered by printer unit tests and a ClickHouse integration test that runs the whole family (plus the wrapped and `toJSONString` cases) against a real native-JSON column. Both new cases fail with the exact "illegal type: JSON" error without the fix. |
||
|
James Ritchie
|
32e5edbd04 |
fix(webapp): restore magic link login on the login page (#4220)
## Summary Magic link login could appear completely broken: submitting your email on the login page showed a stale "This email is unauthorized" error instead of the "we've sent you a magic link" confirmation, even when the address was fine. This PR reverts [#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) (whose diagnosis and fix turned out to be wrong) and fixes the actual bug, which was in how login errors are stored and consumed. ## Root cause Two session bugs compounded on the login page: - The `/login` loader read the flashed `auth:error` without committing the session. A Remix flash is only consumed when the session is committed after the read, so once any attempt flashed an error (for example an address rejected on an instance with `WHITELISTED_EMAILS` set), it stayed in the session cookie and reappeared on every later `/login` visit, making successful attempts look like failures. - The `/login/magic` action stored its validation and rate limit errors with `session.set`, which survives every later read and commit, so those errors stuck permanently. [#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) had instead diagnosed a server-only module leaking into the client bundle and crashing navigation. Checking the shipped images' client bundles via their sourcemaps shows `.server` modules were always stubbed out, so that change fixed nothing and is reverted here. ## Fix - `/login` reads the flashed error and commits the session when one was present, so an error renders once and clears. The `redirectTo` branch now surfaces the error too instead of leaving it in the cookie. - `/login/magic` flashes its errors instead of `set`ting them. Verified end-to-end on a live preview environment: a rejected address shows the error once and a reload clears it; a valid address lands on the confirmation screen with the address named; GitHub, Google, and SSO login paths are untouched by this diff. |
||
|
github-actions[bot]
|
9f76c92021 |
chore: release v4.5.3 (#4219)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 3s
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
## Summary 1 improvement, 2 bug fixes. ## Breaking changes - Removed support for the end-of-life v3 `trigger dev` CLI. Starting a dev session with an old v3 CLI now returns an upgrade message instead of connecting - upgrade to the v4 CLI to continue using `trigger dev`. ([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)) ## Bug fixes - Fix TS2742 ("inferred type cannot be named") when exporting a `chat.agent` from a project with declaration emit: `ChatTaskWirePayload` and `ChatInputChunk` are now declared in the public `@trigger.dev/sdk/chat` subpath, so inferred agent types emit portable declarations and the wire types are directly importable. ([#4218](https://github.com/triggerdotdev/trigger.dev/pull/4218)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Reduce primary database load on the runs page by serving its empty-state check from ClickHouse instead of Postgres. ([#4202](https://github.com/triggerdotdev/trigger.dev/pull/4202)) - Fixed submitting your email on the login page reloading back to an empty form instead of showing the magic link confirmation screen. ([#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.3` ## trigger.dev@4.5.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/build@4.5.3` - `@trigger.dev/core@4.5.3` - `@trigger.dev/schema-to-json@4.5.3` ## @trigger.dev/python@4.5.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.3` - `@trigger.dev/build@4.5.3` - `@trigger.dev/core@4.5.3` ## @trigger.dev/react-hooks@4.5.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.3` ## @trigger.dev/redis-worker@4.5.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.3` ## @trigger.dev/rsc@4.5.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.3` ## @trigger.dev/schema-to-json@4.5.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.3` ## @trigger.dev/sdk@4.5.3 ### Patch Changes - Fix TS2742 ("inferred type cannot be named") when exporting a `chat.agent` from a project with declaration emit: `ChatTaskWirePayload` and `ChatInputChunk` are now declared in the public `@trigger.dev/sdk/chat` subpath, so inferred agent types emit portable declarations and the wire types are directly importable. ([#4218](https://github.com/triggerdotdev/trigger.dev/pull/4218)) - Updated dependencies: - `@trigger.dev/core@4.5.3` ## @trigger.dev/core@4.5.3 </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
James Ritchie
|
afc8f9e210 |
fix(webapp): show magic link confirmation instead of reloading login (#4215)
## Summary Submitting your email on the login page could reload back to an empty login form instead of showing the "we've sent you a magic link" confirmation. The magic link email was still sent, so it looked like nothing happened. ## Root cause The `/login/magic` route imported a server-only cookie module (`magicLinkEmailCookie.server.ts`) whose top-level `env.NODE_ENV` read got bundled into the route's client JS. On the client `env` is undefined, so the module threw a `TypeError` at module eval, which aborted Remix's client-side navigation to the confirmation and hard-reloaded back to `/login`. It only surfaced in production builds (local dev auto-logs-in, and local prod builds happen to tree-shake the module out), which is why it slipped through. ## Fix The email-link strategy already stores the submitted address in the session (`auth:email`), so the separate cookie was redundant. Deleted the cookie module and read the address from the session in the loader. With the module gone, nothing server-only can leak into the client bundle regardless of tree-shaking. Verified the confirmation renders with the email address, the SSO domain-policy redirect (with the email prefilled) still works, and a production build no longer bundles the module. |
||
|
Eric Allam
|
1a0198cc5e |
perf(webapp,clickhouse): move runs empty-state check to ClickHouse (#4202)
## Summary
The runs page's empty-state check (whether an environment has ever had a
run, which decides between the "getting started" and "no runs match your
filters" states) ran a `findFirst` against the Postgres `TaskRun` table.
This moves it to ClickHouse, the same store the runs list itself reads
from, so the check no longer queries `TaskRun`.
## Design
Only the runs list triggers the check now (via an `includeHasAnyRuns`
flag); the other presenters that reuse `NextRunListPresenter` (API,
schedule detail, waitpoint detail, error group) no longer issue it. When
the list is empty it runs `SELECT 1 FROM task_runs_v2 ... LIMIT 1`
filtered on the full `(organization_id, project_id, environment_id)`
sort-key prefix with a configurable `created_at` lower bound
(`RUN_LIST_HAS_RUNS_LOOKBACK_DAYS`, default 30), so it hits the primary
index and reads minimal granules.
Results are cached in a tiered memory + Redis SWR cache. Only positive
("has runs") results are cached, so an environment with no runs is
always re-checked and its first run shows up immediately.
|
||
|
nicktrn
|
0631c8373c |
chore: retire legacy v3 dev websocket + delete legacy self-hosting docs (#4198)
Follow-up to #4194 (v3 execution app + core-helper removal). The v3 (engine V1) is end-of-lifed and enforced off in prod, so this removes a self-contained slice of the remaining dead v3 code while **keeping every user-facing deprecation message** - a user still on v3 must still be told to upgrade. ## Legacy dev websocket `app/v3/handleWebsockets.server.ts` backs the `/ws` transport used **only** by the legacy v3 `trigger dev` CLI (v4 dev uses a different transport). It's now authenticate-then-close with `V3_DEV_DEPRECATION_MESSAGE`, so an old CLI is still told what to do - only the legacy `AuthenticatedSocketConnection` / `DevQueueConsumer` execution behind it (which can no longer run) is removed. - Deleted `app/v3/authenticatedSocketConnection.server.ts` (its only consumer). - `engineDeprecation.server.ts` and the deprecation message constants are untouched. ## Docs Deleted the intentionally-legacy "Docker (legacy)" self-hosting page (`open-source-self-hosting.mdx`) and redirected `/open-source-self-hosting` (+ the existing `/v3/open-source-self-hosting` alias) to `/self-hosting/overview`; repointed the two inbound links. The current `self-hosting/*` docs already describe the v4 (single supervisor) setup. ## Deliberately out of scope Despite the branch name, this PR does **not** touch MarQS or the socket.io coordinator/provider namespaces. Investigation found MarQS is entangled with **live v2** queue/metrics/concurrency/project-cleanup code (`runQueue`, `queueSizeLimits`, `taskRunConcurrencyTracker`, `EnvironmentQueuePresenter`, `registerProjectMetrics`, `deleteProject`), so it needs a per-file reviewed pass, not a bulk delete. That remainder stays on TRI-11883. refs TRI-11883 |
||
|
github-actions[bot]
|
188f008715 |
chore: release v4.5.2 (#4180)
## Summary 4 improvements, 5 bug fixes. ## Improvements - Add SDK and API client helpers for run bulk actions. ([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105)) - Large batch payloads now offload to object storage instead of riding inline in the trigger request. `batchTrigger` and `batchTriggerAndWait` (and the by-id and by-task variants) offload any per-item payload over 128KB before sending, the same way single `trigger` and `triggerAndWait` already do, so a big batch no longer blows past the API body limit. ([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165)) - Removed internal helpers that were only used by the end-of-life v3 self-hosted compute providers. ([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) - Add an `onEvent` callback to `TriggerChatTransport` / `useTriggerChatTransport` that emits typed lifecycle events for sends, stream connects, first chunk, and turn completion. Send-success metrics, time-to-first-token, and "sent but never answered" watchdogs become a few lines of client code. ([#4187](https://github.com/triggerdotdev/trigger.dev/pull/4187)) ```ts onEvent: (event) => { if (event.type === "message-sent") metrics.timing("chat.send_ms", event.durationMs); if (event.type === "first-chunk") metrics.timing("chat.ttft_ms", event.sinceSendMs ?? 0); }, ``` ## Bug fixes - fix(cli): honor the MCP server's `--dev-only` flag ([#4199](https://github.com/triggerdotdev/trigger.dev/pull/4199)) - Fix chat turns that throw (for example from an `onTurnStart` hook) leaking their message listener, which lost or duplicated messages sent during later turns. ([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176)) - Fix `chat.agent` and `chat.createSession` permanently dropping user messages when several arrived during a single turn: every buffered message is now dispatched as its own turn instead of only the first. ([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176)) - Fix chat continuation runs replaying already-answered messages: turns delivered while the run was suspended now advance the session.in resume cursor, so a new run picks up exactly where the previous one left off. ([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176)) - Fix `chat.createSession` swallowing a message sent shortly after stopping a turn: the turn's message listener now detaches when the stream settles, so those messages run as the next turn. ([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.2 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.2` ## trigger.dev@4.5.2 ### Patch Changes - fix(cli): honor the MCP server's `--dev-only` flag ([#4199](https://github.com/triggerdotdev/trigger.dev/pull/4199)) - Updated dependencies: - `@trigger.dev/core@4.5.2` - `@trigger.dev/build@4.5.2` - `@trigger.dev/schema-to-json@4.5.2` ## @trigger.dev/core@4.5.2 ### Patch Changes - Add SDK and API client helpers for run bulk actions. ([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105)) - Large batch payloads now offload to object storage instead of riding inline in the trigger request. `batchTrigger` and `batchTriggerAndWait` (and the by-id and by-task variants) offload any per-item payload over 128KB before sending, the same way single `trigger` and `triggerAndWait` already do, so a big batch no longer blows past the API body limit. ([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165)) - Removed internal helpers that were only used by the end-of-life v3 self-hosted compute providers. ([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) ## @trigger.dev/python@4.5.2 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.2` - `@trigger.dev/sdk@4.5.2` - `@trigger.dev/build@4.5.2` ## @trigger.dev/react-hooks@4.5.2 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.2` ## @trigger.dev/redis-worker@4.5.2 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.2` ## @trigger.dev/rsc@4.5.2 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.2` ## @trigger.dev/schema-to-json@4.5.2 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.2` ## @trigger.dev/sdk@4.5.2 ### Patch Changes - Add SDK and API client helpers for run bulk actions. ([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105)) - Fix chat turns that throw (for example from an `onTurnStart` hook) leaking their message listener, which lost or duplicated messages sent during later turns. ([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176)) - Fix `chat.agent` and `chat.createSession` permanently dropping user messages when several arrived during a single turn: every buffered message is now dispatched as its own turn instead of only the first. ([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176)) - Fix chat continuation runs replaying already-answered messages: turns delivered while the run was suspended now advance the session.in resume cursor, so a new run picks up exactly where the previous one left off. ([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176)) - Fix `chat.createSession` swallowing a message sent shortly after stopping a turn: the turn's message listener now detaches when the stream settles, so those messages run as the next turn. ([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176)) - Add an `onEvent` callback to `TriggerChatTransport` / `useTriggerChatTransport` that emits typed lifecycle events for sends, stream connects, first chunk, and turn completion. Send-success metrics, time-to-first-token, and "sent but never answered" watchdogs become a few lines of client code. ([#4187](https://github.com/triggerdotdev/trigger.dev/pull/4187)) ```ts onEvent: (event) => { if (event.type === "message-sent") metrics.timing("chat.send_ms", event.durationMs); if (event.type === "first-chunk") metrics.timing("chat.ttft_ms", event.sinceSendMs ?? 0); }, ``` - Large batch payloads now offload to object storage instead of riding inline in the trigger request. `batchTrigger` and `batchTriggerAndWait` (and the by-id and by-task variants) offload any per-item payload over 128KB before sending, the same way single `trigger` and `triggerAndWait` already do, so a big batch no longer blows past the API body limit. ([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165)) - Updated dependencies: - `@trigger.dev/core@4.5.2` ## @trigger.dev/plugins@4.5.2 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.2` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Chris Arderne
|
34b1a181c2 | fix: security release 2026-07-06 (#4199) | ||
|
James Ritchie
|
bb450e608d |
feat(webapp): SSO & Directory Sync settings UI improvements (#4196)
📚 Publish docs / publish (push) Has been cancelled
## Summary UI/layout/copy pass over the org **SSO & Directory Sync** settings page (formerly "Identity & Access"). No logic, gates, flags, or data flow changed — server-side auth (`manage:sso`), Enterprise entitlement, action validation, and data loading are all untouched. - Renamed the nav item, page title, and meta from "Identity & Access" to "SSO & Directory Sync". - Added a reusable `SettingsLayout` component system (container, section, header, row, block, actions) modeled on `/account/security`, and refactored the SSO page onto it (section titles, dividers, left title/subtitle + right action rows). - Tightened all UI copy: concise, active voice, consistent labels, no em-dashes. - `Select` primitive: additive `wrap`, `popoverClassName`, and `placement` props (all default to prior behavior) so role options show a bright title with a wrapping description, right-aligned popover, and no horizontal overflow. - Removed the external-link arrow icon from buttons that open a modal; kept it only on genuinely external actions (Contact us, Open in new tab). - Polished the admin portal link dialog: smaller description, tighter spacing, `ClipboardField` with a permanent copy button, removed the redundant Copy link button, and a provider-aware Open label (e.g. "Open in WorkOS") derived from the link host with a safe fallback. ### SSO page UI <img width="3568" height="2550" alt="CleanShot 2026-07-08 at 18 52 11@2x" src="https://github.com/user-attachments/assets/009d2437-7552-4ff0-a457-64744a9fcd88" /> ### Login with SSO and normal email test (local) https://github.com/user-attachments/assets/b33a4ce9-c1fa-45c9-bd3c-077cb6fc9473 ## Test plan - [ ] Non-Enterprise org: SSO page shows the upsell state - [ ] Enterprise org, non-Owner without `manage:sso`: 403 - [ ] Enterprise Owner: verify domains, configure SSO, connect directory, JIT/default/group role selects, and enforcement toggle all work - [ ] Role select popovers: bright title + wrapping description, right-aligned, no horizontal scroll - [ ] Admin portal dialog: copy button works, "Open in WorkOS" opens the portal in a new tab --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
nicktrn
|
a6bd370e42 |
chore: remove end-of-life v3 execution components (#4194)
v3 (engine V1) is end-of-lifed and the v3 clusters are gone, so this removes the dead v3 execution code from the monorepo. It's the first pass of TRI-11824 - the webapp v3 code paths are deliberately left untouched and gated for a follow-up. ## Apps Deletes the three v3-only execution apps and their build wiring: - `apps/coordinator`, `apps/kubernetes-provider`, `apps/docker-provider` - `.github/workflows/publish-worker.yml` - it built only those three; the v4 worker publish is a separate workflow - Their references in `.changeset/config.json`, `.cursorignore`, `CHANGESETS.md`, `CONTRIBUTING.md`, `.server-changes/README.md` - `pnpm-lock.yaml` regenerated to prune the apps and their app-only dependencies (`socket.io`, `@kubernetes/client-node`, `p-queue`, `execa`, `prom-client`, `tinyexec`) ## Core Removes the helpers in `@trigger.dev/core` that only those apps used - `ProviderShell`, `SimpleLogger`, the `Exec`/process helpers, `isExecaChildProcess`, `getTextBody`, and `testDockerCheckpoint`. Each was verified to have no remaining consumers anywhere in the repo. Kept the helpers still used elsewhere: `ExponentialBackoff` (warm-start client), `HttpReply`/`getJsonBody` (serverOnly http server), `SimpleStructuredLogger` (widely used), and `ZodNamespace`/`ZodSocketConnection` (still referenced by legacy v3 webapp code, hence the follow-up pass). The `./v3/apps` and `./v3/serverOnly` export subpaths remain - only dead members were trimmed from their barrels, so no `package.json` exports changed. ## Verification `@trigger.dev/core` builds, and `typecheck` passes for core, supervisor, cli-v3, run-engine, redis-worker, and webapp. refs TRI-11824 |
||
|
Iss
|
e0208f3a27 |
fix(webapp): keep playground chat requests same-origin (#4193)
### Problem
The agent playground chat builds its realtime transport baseURL from
apiOrigin, but points it at a same-origin /resources/... dashboard
route. When API_ORIGIN differs from APP_ORIGIN, the in/append POST goes
cross-origin, fails the CORS preflight, and messages never reach the
agent ("Failed to fetch").
It only reproduces where the two origins differ — not locally, where
both default to localhost:3030.
Fixes #4149.
### Fix
Build the base URL from window.location.origin (falling back to
apiOrigin on SSR), so realtime traffic stays same-origin — the same
approach AgentView.tsx already uses.
### Testing
Typecheck passes. The CORS path only manifests when API_ORIGIN !=
APP_ORIGIN, so verify on test-cloud (can't reproduce locally).
|
||
|
Daniel Sutton
|
80d4819a03 |
fix(webapp): stop slow database cleanup on project deletion (#4191)
## Summary Deleting a project triggered an unbounded database cleanup that scanned the project's entire run history, so deleting a project with many runs could be very slow. Project deletion is a soft delete again: run data is retained and the deletion completes quickly. ## Fix Project deletion ran a cascade hard-delete whose `BulkActionItem` step filtered through a relation to `TaskRun` scoped by `projectId`. Prisma compiles that to an `EXISTS`-join over the project's entire `TaskRun` set (a large, hot table with no `projectId` index), and it ran on every project deletion unconditionally. Removing the cascade-cleanup call restores the prior soft-delete behaviour: queues are removed, the project is marked deleted, and run data is retained. The cascade-cleanup service (added in [#4117](https://github.com/triggerdotdev/trigger.dev/pull/4117)) had no other callers, so it and its test are deleted. |
||
|
Katia Bulatova
|
6e827f1da3 |
chore: Tailwind CSS v4 migration (#4139)
Migrates the webapp from Tailwind CSS 3.4 to 4.x. |
||
|
DKP
|
00ee0751ec |
feat(webapp): proxy PostHog through a same-origin /ph path (#4183)
## Summary posthog-js sent product analytics to PostHog Cloud directly from the browser. This points `api_host` at a same-origin `/ph` path that forwards to PostHog Cloud EU server-side, following PostHog's standard first-party reverse-proxy setup. ## How it works A resource route forwards each request server-side, splitting by path: `/ph/static/*` and `/ph/array/*` go to the asset host, everything else (analytics events, feature flags) goes to the ingest host. It rewrites the `Host` header, strips the `/ph` prefix, and streams the response back. Only PostHog's own cookies are forwarded, so the app session cookie stays first-party. Upstream hosts default to PostHog Cloud EU, overridable via `POSTHOG_INGEST_HOST` / `POSTHOG_ASSETS_HOST`. It also sets `cross_subdomain_cookie` so a single PostHog session is shared across the marketing site and app. Verified locally: static assets return 200 from the EU asset host, and analytics events return 200 through the ingest host. |
||
|
Eric Allam
|
fe07de4a2c |
fix(webapp): use provider-reported cost for AI generations when present (#4186)
## Summary The run page could show an AI generation cost well above what the provider actually charged, most visibly for OpenRouter and Vercel AI Gateway requests where a heavily cache-read prompt was priced at the full input rate. When the provider reports an exact per-request cost, we now use that instead of catalog pricing. ## Fix Gateway and OpenRouter include the exact per-request cost in `ai.response.providerMetadata` (`openrouter.usage.cost` / `gateway.cost`). That figure already reflects the cache-read discount and the real per-provider rate, which the catalog cannot reconstruct: cache-read counts do not arrive in `gen_ai.usage.*`, and per-model catalog prices drift from what the provider billed, in either direction. So provider-reported cost is now preferred, and the catalog is used only when no provider cost is present. Fallback routing is covered by the same change: when OpenRouter routes to a different model, `gen_ai.response.model` already carries the served model, so the cost follows the served model and the provider's own figure makes it exact. `extractProviderCost` now runs on every AI span, so it gets a cheap `"cost"` substring guard to skip the JSON parse on reasoning-model spans whose provider metadata carries large reasoning text and no cost field. Regression tests cover the cache-discount overcharge, fallback served-model pricing, gateway cost, and the catalog fallback path. |
||
|
James Ritchie
|
7c5f089d3d | feat(webapp): rework login page and SSO sign-in UI (#4182) | ||
|
nicktrn
|
94b30fc1a6 |
fix(webapp): reject deploy images with runtime-incompatible zstd layers (#4184)
Container runtimes (cri-o / containerd / podman) can't pull zstd-compressed layers carried in a Docker v2s2 manifest (`application/vnd.docker.image.rootfs.diff.tar.zstd`). A deploy built with an outdated CLI can produce exactly that combination - and today it's promoted to current and then fails every run at image-pull time. This extends the pre-promotion image check (#4049) to also inspect the manifest's layer media types. If any layer uses the unpullable zstd/v2s2 media type, the deploy is rejected at finalize with a clear message to upgrade the CLI and re-deploy, instead of silently shipping a version that can't start. The manifest is already returned by the existing ECR `BatchGetImage` call, so there's no extra registry request for single-arch images. Parsing is a lenient Zod schema and **fails open** - a manifest we can't read never blocks a deploy. Manifest lists / OCI indexes (no top-level `layers[]`) and OCI zstd (`...tar+zstd`, which runtimes support) pass unaffected. Also clarifies in the contributor docs that changesets and `.server-changes/` notes are user-facing and should be written for users, not maintainers. refs TRI-11702 |
||
|
Chris Arderne
|
aa74e68c71 |
feat(sdk): add bulk replay to api and sdk (#4105)
## Summary
Adds SDK and API support for run bulk actions. You can now create bulk
cancel or replay actions from `@trigger.dev/sdk` using run IDs or the
same filters as `runs.list()`, then retrieve, list, poll, or abort the
action by its `bulk_` handle.
Tests, docs, changesets added.
## Design
The dashboard bulk action service now accepts structured filters instead
of reading directly from a dashboard request, so the dashboard and API
share the same creation path. Replay actions created through the API are
attributed with the existing `api` trigger source, while
dashboard-created actions keep `dashboard`.
The SDK exposes the new surface under `runs.bulk.*`, including
`targetRegion` for replay region overrides and cursor pagination for
listing bulk actions.
## Filters and runIds
Nuance on filters. If `filter` is provided, it MUST have at least one
key. This is to remove the footgun of passing no filter and selecting
all runs.
```typescript
{ action: "cancel", runIds: ["run_1"] } // valid
{ action: "cancel", runIds: [] } // invalid, min(1)
{ action: "cancel", filter: { status: "FAILED" } } // valid
{ action: "cancel", filter: {} } // invalid
{ action: "cancel", filter: {}, runIds: ["run_1"] } // invalid
```
|
||
|
Daniel Sutton
|
d59743bd35 |
fix(webapp,run-ops-database): keep run-ops batch items co-resident with their batch (#4178)
## Summary Three fixes to the run-ops database split (the Cloud-only mode where run-lifecycle rows live on a dedicated Postgres). All are inert in the default single-database deployment. The main fix: on the batch trigger paths, a parentless batch's item runs chose their physical store from a fresh per-org mint-flag read at processing time, so flipping an org's flag mid-batch could land an item in a different store than its batch, breaking the `TaskRun.batchId` foreign key (or silently orphaning the item). The other two harden the split's safety nets: the schema-parity test now actually compares columns, and the read fan-out gate now signals when it has been silently disabled. ## Batch item residency `RunEngineBatchTriggerService` (api.v2) and the BatchQueue item callback (api.v3) now anchor each item's id mint on the batch's own friendlyId, mirroring the already-safe `BatchTriggerV3Service`. Residency is a pure id-shape check, so an item can no longer diverge from its batch across a mid-batch flag flip. The pre-failed-run fallback is anchored the same way (it also sets `batchId`), and the shared mint branch is consolidated into one helper so every mint path stays in lockstep. No new database queries; single-database mode is unchanged (a cuid-shaped batch friendlyId yields a cuid item). ## Schema parity test The parity test previously read only the dedicated schema and matched model headers with regexes, so it never compared columns and could not catch a run-subgraph column that diverged between the two physical schemas. It now parses both schemas and asserts bidirectional scalar-column parity (type, nullability, array-ness, default) across the run-subgraph models, and fails on any field line it can't parse. Scoped to the run-subgraph models so unrelated control-plane edits don't break it. ## Read fan-out signal The split read fan-out gate is decided by the object identity of the NEW vs control-plane clients. It now warns when both run-ops URLs are set but the NEW client isn't a distinct instance (fan-out silently off), and a new test exercises the real topology-into-gate wiring so a future refactor that aliases the clients can't disable fan-out unnoticed. ## Verification New unit and glue tests cover all three changes; the DB-backed residency, store-routing, and topology suites pass against real Postgres; `typecheck` is clean for both packages. |
||
|
Matt Aitken
|
1a033b665b |
fix(webapp,core): retry run resume through transient database outages (#4161)
## Summary When the platform database is briefly unreachable while a run is resuming from a wait, the run no longer fails with `TASK_EXECUTION_ABORTED`. The worker now retries the resume through the outage instead of aborting on the first blip. ## Root cause Resuming a run calls the engine's `continue` worker-action endpoint. That route caught every error and returned a `422`, which the worker's HTTP client treats as non-retryable. So a transient Prisma infrastructure error (for example `P1001` "Can't reach database server") was flattened into a permanent failure: the worker gave up, force-killed the run process, and completed it with `TASK_EXECUTION_ABORTED`. ## Fix - The `continue` route now lets infrastructure errors propagate to the generic 500 handler (message scrubbed, and retryable by the worker's HTTP client), the same treatment the trigger path already gives them via `isInfrastructureError`. Genuine validation errors (snapshot mismatch, invalid state) still return `422`, so a stale retry stays non-retryable. Resuming is idempotent server-side (guarded by the snapshot id), so retrying is safe. - The worker's `continueRunExecution` calls (both the runner-to-supervisor and supervisor-to-engine hops) retry with a longer, jittered backoff so they can ride out an outage lasting tens of seconds, and the jitter keeps a fleet of resuming runs from stampeding the database the moment it recovers. Builds on #3960, which scrubbed the leaked message on these routes but left the status non-retryable. No changeset: this is a server-side behaviour fix recorded via `.server-changes`. The `@trigger.dev/core` edits are internal run-engine worker plumbing, not a public API change. |
||
|
Chris Arderne
|
f4af13981a | feat: auto-update dev and preview branch selector (#4171) | ||
|
github-actions[bot]
|
c584236937 |
chore: release v4.5.1 (#4126)
🚀 Publish Trigger.dev Docker / units (push) Failing after 22s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 22s
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
## Summary 1 improvement. ## Improvements - Extend the SSO plugin contract with WorkOS Directory Sync (SCIM) support. ([#4148](https://github.com/triggerdotdev/trigger.dev/pull/4148)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.1` ## trigger.dev@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/build@4.5.1` - `@trigger.dev/core@4.5.1` - `@trigger.dev/schema-to-json@4.5.1` ## @trigger.dev/python@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/build@4.5.1` - `@trigger.dev/core@4.5.1` - `@trigger.dev/sdk@4.5.1` ## @trigger.dev/react-hooks@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.1` ## @trigger.dev/redis-worker@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.1` ## @trigger.dev/rsc@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.1` ## @trigger.dev/schema-to-json@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.1` ## @trigger.dev/sdk@4.5.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.1` ## @trigger.dev/core@4.5.1 ## @trigger.dev/plugins@4.5.1 ### Patch Changes - Extend the SSO plugin contract with WorkOS Directory Sync (SCIM) support. ([#4148](https://github.com/triggerdotdev/trigger.dev/pull/4148)) - Updated dependencies: - `@trigger.dev/core@4.5.1` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Daniel Sutton
|
d977691219 | fix(run-store): route caller-passed read clients to the owning store's primary (#4153) | ||
|
Daniel Sutton
|
119189f31a |
fix(webapp): accept all valid run and batch IDs in dashboard filters (#4152)
## Summary The **Run ID** and **Batch ID** filters on the runs list, batches list, and logs view rejected valid IDs. The input showed an error and the **Apply** button stayed disabled, so filtering by an affected run or batch ID from the dashboard was impossible. The filter validators hard-coded exact friendly-id character lengths. Friendly IDs come in three generations that all still exist in the data (`<prefix>_` plus a 21-char nanoid, a 25-char cuid, or a 27-char ksuid), and the hard-coded lengths never covered all three at once. ## Fix All the ID filter validators (run, batch, waitpoint, schedule) now share one helper, `makeFriendlyIdValidator` (`apps/webapp/app/utils/friendlyId.ts`), which validates by prefix plus a base62 body of any known generator length (21 / 25 / 27). The cuid and ksuid lengths are sourced from core so the helper tracks any future change to those formats. Unit tests assert it accepts the output of the real id generators and rejects malformed input. Downstream was already unaffected: run/batch route params and URL-applied filters use unconstrained validation, so only the manual filter inputs needed the fix. |
||
|
Daniel Sutton
|
962bc48738 |
feat(run-ops): automatically migrate the dedicated run-ops database (#4150)
## What
Adds the ability to **automatically migrate the dedicated run-ops
database** (the NEW DB in the run-ops split), matching how every other
database in the system is migrated. Follow-up to the run-ops split
activation.
## Changes
- **Migrate runner** — new
`internal-packages/run-ops-database/scripts/migrate.mjs`, exposed as
`db:migrate:deploy` / `db:migrate:status`. Connects via
`RUN_OPS_DATABASE_URL` (the same var the app uses) and expands `${VAR}`
refs like Prisma's dotenv.
- **Self-host** — `docker/scripts/entrypoint.sh` runs the run-ops
migration on boot when the DB is configured, gated by
`SKIP_RUN_OPS_MIGRATIONS`. Single-DB installs never set the URL, so it's
a clean no-op.
- **Single env-var family** — the run-ops DB is now addressed by one
canonical `RUN_OPS_*` family, connect path and migrations resolving the
identical URL:
- `RUN_OPS_DATABASE_URL` (writer) — replaces `TASK_RUN_DATABASE_URL`
- `RUN_OPS_LEGACY_DATABASE_URL` — replaces
`TASK_RUN_LEGACY_DATABASE_URL`
- `RUN_OPS_DATABASE_READ_REPLICA_URL` — replaces
`TASK_RUN_DATABASE_READ_REPLICA_URL`
- the old `TASK_RUN_*` aliases, the `??` coalesce, the
`runOpsNewDatabaseUrl` indirection, and the migrate-only `directUrl` are
all removed (consumers read `env.RUN_OPS_DATABASE_URL` directly).
`directUrl` was dropped because it was only ever used by `prisma
migrate` (never the app runtime) to bypass a pooler for advisory locks —
premature here since the run-ops connection isn't wired to the app yet.
If a pooler is later introduced for the app, a direct URL can be
reintroduced then.
## Safety
- **Pure rename** — nothing deployed sets any `TASK_RUN_*` var (the
split isn't activated anywhere yet; `.env.example`, docker-compose, and
cloud already use `RUN_OPS_*`), so there is no config migration.
- **Single-DB / self-host** — no new required env var; entrypoint and
migrate are no-ops when `RUN_OPS_DATABASE_URL` is unset.
- **Cloud** — runs migrations as pre-deploy ECS tasks (companion cloud
PR), calling these same `db:migrate:deploy` / `db:migrate:status`
commands.
## Verification
- Live migration against a fresh scratch DB with only
`RUN_OPS_DATABASE_URL` set: both migrations applied, no `P1012`/`P1013`;
`${VAR}` expansion, idempotent re-run, `status`, and no-op skip all
pass.
- Schema parity 4/4; `typecheck --filter webapp` 18/18; affected
split/replication tests 34/34.
## Scope
This delivers automatic migrations only. Enabling the app to *use* the
new DB (setting `RUN_OPS_DATABASE_URL` + `RUN_OPS_SPLIT_ENABLED` on the
service) is a separate activation step.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
c9f427e21f |
fix(replication): key logical replication leader lock on slot name (#4151)
## Problem
`LogicalReplicationClient` uses a Redlock leader lock to guarantee a
single active consumer per Postgres logical replication slot. The lock
resource was keyed on the client `name`:
```
logical-replication-client:${this.options.name}
```
A slot permits exactly one consumer, so the lock's job is to serialize
consumers **of a given slot**. Keying it on `name` breaks that whenever
two clients target the same slot with different names — most notably
across a rolling deploy where the client `name` changes but `slotName`
does not. Both acquire *distinct* locks, both consider themselves
leader, and the second to reach `START_REPLICATION` hits `replication
slot "<slot>" is active for PID <n>`. Because that query was
fire-and-forget and its failure was only logged (no retry), the consumer
stopped and replication stalled until the process was restarted.
## Fix
**1. Key the leader lock on `slotName`** — the actual single-consumer
resource:
```
logical-replication-client:${this.options.slotName}
```
Consumers of the same slot now contend on the same lock and hand off
cleanly across restarts/deploys; different slots stay independent.
`name` is kept for logging and the pg `application_name`.
**2. Self-healing resubscribe** (`resubscribeOnFailure`, opt-in) —
instead of logging-and-dying, a client re-subscribes with exponential
backoff after a lost election or a failed `START_REPLICATION`, so a
rolling deploy self-heals: the incoming pod retries until the draining
pod releases the slot, then takes over. Safety:
- `#cleanupAttempt()` unconditionally ends the pg client (freeing the
walsender) and releases the leader lock before rescheduling — retries
never leak connections/locks.
- `shutdown()` sets an intentional-stop latch re-checked after every
`await` in `subscribe()` (and aborts the lock-acquire spin), so a
resubscribe can never race or outlive an intentional shutdown.
- Backoff resets only on genuine stream start, so a permanently stuck
slot backs off to the ceiling and logs loudly rather than tight-looping;
an epoch guard neutralises stale `START_REPLICATION` catches.
Runs- and sessions-replication opt in and use `shutdown()` for all
intentional stops.
**3. Observability** — the admin runs-replication status route probed
the old name-keyed Redis key (would report `leader:false` for every
source after fix #1); now probes the slot-keyed key.
## Tests
`internal-packages/replication/src/client.test.ts` (real Postgres +
Redis containers):
- same-slot/different-name → second client must not double-lead or race
into "slot is active" (the regression)
- a failing `START_REPLICATION` retry loop must not leak connections or
locks
- `shutdown()` during an in-flight `subscribe()` must not leave a zombie
leader
- `subscribe()` after `shutdown()` re-arms `resubscribeOnFailure`
- self-heals once the leader releases the slot
Plus the multi-source wiring test updated to the slot-keyed lock keys.
## Rollout
With the self-healing resubscribe, this ships as a **plain rolling
deploy** — the incoming pods retry across the one-time lock-key
transition and take over once the old pods drain (a brief replication
stall that the durable slot replays on reconnect — no data loss). No
stop-before-start required.
|
||
|
Daniel Sutton
|
70bca82d84 | feat(run-ops): activation — drop cross-DB FKs, provision run-ops DB, enable split (#4124) |