## Stacked on #4284 — tests only
This PR contains **only the tests** that guard the production fixes in
#4284 (its base). Review #4284 first; this branch adds no production
code.
## What
Caller-driven replica-lag and idempotency guards for every fixed site:
- Each guard **drives the real exported caller** (route loader/action,
presenter `.call()`, service, or engine method) against a **real
Postgres** with the owning replica frozen via the shared
`laggingReplica` testcontainer primitive — never a store-seam
reimplementation.
- For a **fixed** site the guard goes **RED when the production change
is reverted**; for a **tolerated read-view** site it's a caller-driven
**GREEN** proof the miss self-heals (returns null/empty, no mutation,
row live on primary).
- The **global-scope idempotency** guard drives the real dedup + claim
path through a **real `MollifierBuffer` over a Redis testcontainer**
(real SETNX/poll/publish), and covers the cross-DB **andWait** waitpoint
wiring and the **expired/failed clear-and-recreate** reacquire cases.
Run with `vitest --no-file-parallelism` (testcontainers). Verified
GREEN, and revert→RED verified per fixed site.
## What & why
Two related correctness fixes for the run-ops DB split. Under the split,
run-store reads can route to a **lagging read replica**; a just-written
run/waitpoint/batch can then be missed, causing a wrong decision.
**1. Read-your-writes → owning primary.** Surfaced first as an
intermittent `wait.until({ idempotencyKey })` re-wait on retry. Auditing
the run-store read surface found the same class at sibling sites (some
gating mutations or returning spurious 404s, others
tolerable/self-healing). Reads that must observe their own writes now
route to the owning **primary**
(`findRun`/`findWaitpoint`/`findBatchTaskRunByFriendlyId` →
`*OnPrimary`, a primary re-read on a miss, or a retryable 404 where the
SDK polls). Read-view reads stay on the replica. All additive — the
happy path is unchanged.
**2. Global-scope idempotency across the split.** A `global`-scope key
carries no per-run salt, so the same `(env, task, key)` triggered
concurrently from parents resident on **different** run-ops DBs could
dedup-miss on each DB and create a duplicate (the per-DB unique index
can't enforce cross-DB uniqueness). Such triggers (global scope, or
scope-absent, while split is active) are serialized through the existing
Redis idempotency claim, the loser resolves the winner by id across both
DBs, and the claim is reacquired on the expired/failed
clear-and-recreate path. `run`/`attempt` scope embed the run id and
never contend.
## Stacked for review
This is the **base** of a 2-PR stack, split so review is easier:
- **This PR** — production code only (34 files).
- **Stacked tests PR →
https://github.com/triggerdotdev/trigger.dev/pull/4285** — the
caller-driven guards (55 test files) on top of this branch.
## Validation
Local run-ops split, **both 2-DB and 3-DB**, fresh boot on this branch:
SDK canary 64/71 (only the known concurrency/input-streams/s3 failures),
quarantine sweep **0 unexpected** (340 pass / 16 known / 4 local) in
each topology, dashboard e2e 0 failed. No product regressions.
## What
- Runs `apps/webapp/scripts/runOpsLegacyGuard.ts --check` as its own PR
job (`runops-guard`), so code that reaches a run-graph table through the
control-plane Prisma client instead of the RunStore fails the build.
- Adds a `trigger-runops` oxlint plugin with two fast, in-editor rules
scoped to `apps/webapp/app`: one for direct `prisma.taskRun`-style
access, one for a control-plane client wired into a read-through slot.
These are the cheap fence; the guard is the type-aware gate.
- Fixes `CancelTaskRunService.callV1`: historical V1 runs are
legacy-resident, so its two finalize writes now go through
`runOpsLegacyPrisma` instead of the control-plane client (they'd miss
the row once legacy is a separate database).
- Regenerates the guard baseline, which had drifted stale (it referenced
files deleted in an earlier PR).
## Why
The guard existed but ran nowhere, so its baseline rotted and a real
residency gap (the V1 cancel writes) sat undetected. Wiring it into CI
turns it into a ratchet against new control-plane run-graph access.
## Verification
Local, against a clean regen: `oxfmt --check`, `oxlint .`, `guard
--check`, and `typecheck --filter webapp` all pass. Remaining baseline
entries are 4 batch-results router reads through type-opaque `as
PrismaReplicaClient` casts (correct at runtime, accepted) + 2 sanctioned
legacy annotations.
## Summary
Correctness and performance fixes for deployments that split run data
across more than one database. Single-database / self-hosted deployments
are unaffected (they collapse to a single read/write path).
- **Batches list (dashboard):** for some organizations the Batches list
could hide older batches or show them out of order. It now orders and
paginates by creation time (with the id as a stable tiebreak), so every
batch appears exactly once, newest first. The pagination cursor format
changes; older in-flight cursors simply restart from the first page.
- **Reads:** waitpoint and snapshot lookups that are keyed by a single
run now read only the database that holds that run instead of querying
both, removing redundant queries on hot paths (unblock, snapshot reads).
- **Writes:** environment-scoped writes with no owning run (standalone
wait tokens, waitpoint tags, idempotency-key resets) now land in the
same database as that environment's runs, rather than defaulting to the
other one. An idempotency-key reset also falls back to the other
database when it matches nothing, so a reset still clears the key
wherever the run actually lives.
## Notes
Verified end-to-end against multi-database setups: run-keyed reads and
env-scoped writes land on the correct database with no cross-database
writes, and the batches list surfaces every batch in creation order. New
tests cover the batches ordering/reachability and the write-residency
routing.
## Problem
The webapp's HTML references content-hashed `/build` assets, and each
running
instance contains exactly one build and returns 404 for asset hashes it
doesn't
have. During a rolling deploy a client can hold HTML from one build
while a
request for one of its assets is served by an instance on a different
build →
missing styles or a failed chunk load.
## What this does
On a `/build` stylesheet/script/chunk load failure, the client does a
**bounded
full-document reload** (at most 2 per 5 minutes, tracked in
`sessionStorage`) so
the page reloads onto a single consistent build. That's the whole
mechanism — no
polling, no `fetch` interception, no blocking overlay, no form
snapshotting.
- `apps/webapp/app/components/StaleAssetRecovery.tsx` — authored as a
typed,
lint-checked function and serialized to an inline script via
`.toString()` (so
the logic is real, reviewable code, not an opaque string), injected
before
`<Links />`, production only.
- Detection: capture-phase `error` listener for
`<link>`/`<script>`/modulepreload
failures under `/build/`, plus an `unhandledrejection` guard for
dynamic-import
failures.
- Guards: once-per-page re-entrancy guard, the bounded reload budget,
and a
`navigator.onLine` check so it never reloads into an offline error page.
- Unit tests in `StaleAssetRecovery.test.ts`.
## Relationship to #4260
Replaces the recovery introduced in #4260 (reverted in #4280) with a
much
smaller, reload-only approach — the previous version intercepted `fetch`
and
could turn a data request into a navigation, and showed a full-screen
overlay on
any asset error; this drops both.
## `/build-version` compatibility shim
`apps/webapp/server.ts` adds a tiny `GET /build-version` endpoint (build
id only,
`no-store`). A previously-deployed client build polls it after an asset
failure
and reloads once it sees a newer build, so those older tabs recover in
one reload
instead of getting stuck. Temporary — safe to remove once older clients
have
cycled out. It deliberately does **not** re-add an `X-Build-Id` response
header.
## Also
Restores the `.server-changes` writing guidance in
`.claude/rules/server-apps.md`
(reverted alongside #4260).
## Self-hosting note
Recovery is most reliable when your load balancer keeps a client on one
instance
for the duration of a deploy (short session stickiness) — the reload
then lands
on a consistent build in one hop.
Removes the deprecated realtime stream write action kept for retired v3
task clients. Supported clients use the targeted stream write routes,
while the existing stream read loader remains unchanged.
Standard `git revert` of #4260.
Its client-side stale-asset recovery is net-negative during normal
deploys:
- The `fetch` interception treats any `?_data=` request (Remix loader
**and** action traffic) as a navigation and, on a build-id mismatch,
`location.assign`es the tab to the fetched URL — an open dashboard tab
can be hard-navigated to a raw data URL during a rolling deploy, losing
unsaved input.
- Any transient `/build` asset error (a network blip, an extension, an
unrelated failed dynamic import) blanks the page behind a full-screen
overlay for ~60s before offering a manual reload.
- It serialized form field values to `sessionStorage` to restore them
across the reload.
This returns the webapp to the pre-#4260 baseline as a fast, low-risk
step.
Follow-ups (separate PRs):
- a minimal reload-only recovery to replace this,
- restore the unrelated `.claude/rules/server-apps.md` docs tidy-up from
#4260 (via cherry-pick),
- a load-balancer stickiness change addressing the root cause.
## Summary
Adds experimental Node.js 24 and 26 task runtimes through the
`experimental-node-24` and `experimental-node-26` config values.
Existing runtime defaults and the `node`, `node-22`, and `bun` behavior
remain unchanged. The unprefixed `node-24` and `node-26` config values
remain unavailable until the runtimes are ready for general use.
## Design
Experimental config values normalize to canonical runtime identifiers
before build manifests are created, keeping deployment metadata and
execution behavior consistent. Kubernetes task pods also use the
runtime-default seccomp profile so modern Node.js versions fall back
from io_uring to checkpoint-compatible system calls.
The signup → Attio sync (`attio.server.ts` `#assert`) logged every
non-2xx response at `error` level and threw the same way regardless of
status. Transient upstream failures (5xx/429) are retried by the common
worker and self-heal, so treating them as errors created false alerts
for something that isn't actually a bug.
Now `#assert` splits the two cases:
- **5xx / 429** — Logged at `warn` and thrown with `logLevel: "warn"`,
so they continue to be retried but don't raise error-level alerts. This
reuses the same pattern the worker already honors
(`directorySyncEffects`).
- **4xx** — Unchanged: logged at `error` and thrown, so genuine
integration bugs (schema, permissions, auth, etc.) remain visible.
There is no behavior change to retries or the signup flow. This is a
server-only change.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Summary
Adds optional configuration to send ClickHouse read traffic to a
separate instance (for example a read replica) while writes stay on the
primary `CLICKHOUSE_URL`. This lets operators offload read load (runs
list, traces, logs, queries) from the cluster that handles inserts.
Fully backwards compatible: with nothing new set, every client resolves
to `CLICKHOUSE_URL` exactly as before.
## What it adds
- `CLICKHOUSE_READER_URL` (optional): a single reader endpoint that the
read-only clients fall back to. Read clients resolve `<own URL> ??
CLICKHOUSE_READER_URL ?? CLICKHOUSE_URL`. The task-events client (which
both inserts events and reads traces, spans, and logs) is built as a
reader/writer pair so queries use the reader while inserts stay on
`CLICKHOUSE_URL`.
- `RUNS_LIST_CLICKHOUSE_URL` (optional): a dedicated client for the runs
list (dashboard list, runs list API, live reload, child-status counts),
so the highest-traffic read path can target its own instance.
## Safety
Only read-only clients fall back to the reader: logs, query, admin, runs
list, the pending-version lookup, and the realtime run-id resolver. The
query page is constrained to read-only (the TSQL parser rejects anything
that is not a `SELECT`, and a `readonly` setting is applied). The
task-events client routes inserts to the writer and queries to the
reader per method, so a write can never reach the reader. Pure-write
clients (event inserts, replication) always use `CLICKHOUSE_URL`.
Note: this PR targets a baseline branch rather than `main` so the diff
stays scoped to the read-replica changes. It will be retargeted to
`main` before merge.
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
## Summary
Updates the internal development, CI, and runtime-image Node version to
24.18.0. SDK compatibility coverage continues to include Node 20, 22,
24, and 26.
The Node type definitions and the package-manager lockfiles now resolve
against Node 24 types.
## Summary
Adds a set of PAT-authenticated management API endpoints so orgs,
projects, members/invites, environment variables, and a few
project/environment settings can be managed programmatically (scripting,
automation) rather than only through the dashboard. Each route is a thin
wrapper over the **existing** service the dashboard already uses, with
the same authorization applied at the route layer - no new business
logic.
## Endpoints
**Organizations**
- `POST /api/v1/orgs` - create an org (`createOrganization`)
- `PATCH /api/v1/orgs/:orgParam` - rename (title)
- `DELETE /api/v1/orgs/:orgParam` - soft-delete
(`DeleteOrganizationService`; keeps the active-subscription guard)
**Members & invites**
- `GET /api/v1/orgs/:orgParam/members` - list members + pending invites
- `DELETE /api/v1/orgs/:orgParam/members/:memberId` - remove a member
(last-member guarded)
- `POST /api/v1/orgs/:orgParam/invites` - invite by email
(`inviteMembers`, sends the invite email)
- `DELETE /api/v1/orgs/:orgParam/invites/:inviteId` - revoke an invite
**Projects**
- `PATCH /api/v1/projects/:projectRef` - rename
(`ProjectSettingsService`)
- `DELETE /api/v1/projects/:projectRef` - soft-delete
(`DeleteProjectService`)
- `PUT /api/v1/projects/:projectRef/default-region` - set the default
region by worker-group name (`SetDefaultRegionService`)
- project GET/list now return `defaultRegion` (worker-group name, or
null when unset)
**Environments**
- `POST /api/v1/projects/:projectRef/:env/pause` and `/resume`
(`PauseEnvironmentService`)
- `POST /api/v1/projects/:projectRef/:env/regenerate-api-key` - rotate
the env secret key (`regenerateApiKey`, RBAC `write:apiKeys`)
- env var create now accepts an optional `isSecret` flag
## Auth & authorization
- All routes authenticate with a **Personal Access Token**
(`Authorization: Bearer tr_pat_...`).
- Org/project routes are built on the PAT route builders in
`apiBuilder.server.ts`: `createLoaderPATApiRoute` (already existed) and
**`createActionPATApiRoute`** (added here - the loader builder had no
mutation counterpart). The builder runs auth, resolves the org/project
role-floor via `context`, and enforces a declarative `authorization`
block using the same RBAC actions the dashboard applies
(`manage:organization` / `read:members` / `manage:members` /
`manage:project`). Handlers keep a membership-scoped query as the floor,
so a non-member gets a 404. This also gives these routes `tenantContext`
user attribution (Sentry) and `ServiceValidationError`-to-status mapping
for free.
- **Membership floor (important).** The OSS RBAC fallback grants a
permissive ability, so `ability.can(...)` can't reject a non-member on
self-hosted. Every handler therefore resolves the target scoped to the
caller's membership (`members: { some: { userId } }`) → 404 for
non-members. `authorization` is the *role* gate; this is the *tenant*
gate. `resolveOrganizationForApiUser`
(`organizationApiAccess.server.ts`) is the org-tier version of the
existing `findProjectByRef` - org-addressed PAT routes are new, so no
such helper existed before.
- Env-tier routes reuse the existing `authorizePatEnvironmentAccess`
(`write:apiKeys`).
### What `createActionPATApiRoute` gives you
A route is pure declaration - the builder handles auth, RBAC,
validation, tracing, and error mapping:
```ts
export const action = createActionPATApiRoute(
{
method: "PUT", // one verb, or ["PATCH", "DELETE"] for multi-verb routes
params: ParamsSchema,
body: SetDefaultRegionRequestBody, // zod-validated
context: async ({ projectRef }) => { // resolve the org for the RBAC role-floor
const project = await prisma.project.findFirst({
where: { externalRef: projectRef, deletedAt: null },
select: { organizationId: true },
});
return project ? { organizationId: project.organizationId } : {};
},
authorization: { action: "manage", resource: () => ({ type: "project" }) },
},
async ({ params, body, authentication, ability }) => {
// auth + authz already enforced. Just do the work.
// `throw new ServiceValidationError("Region not found", 400)` → mapped to that status.
return json({ ok: true });
}
);
```
Handled for you, so handlers stay thin:
- **Method allowlist** - `method` accepts a verb or an array; any other
verb → `405` with an `Allow` header, *before* auth runs:
```ts
const allowedMethods = method ? (Array.isArray(method) ? method :
[method]) : undefined;
if (allowedMethods && !(allowedMethods as
string[]).includes(request.method.toUpperCase())) {
return json({ error: "Method not allowed" }, { status: 405, headers: {
Allow: allowedMethods.join(", ") } });
}
```
- **PAT / user-actor auth** in a single roundtrip → `401` on
missing/invalid/revoked token.
- **RBAC** - `context` computes the caller's role-floor for the target
org/project; `authorization` gates it → `403` with a structured error
body.
- **Sentry attribution** - `tenantContext.enrich({ userId })` so events
from the handler carry the acting user.
- **Typed errors** - a thrown `ServiceValidationError` is mapped to its
`.status` (default 400); anything else → `500`, and expected boundary
errors are logged as `warn` (kept out of Sentry).
- **Validation** - params / query / headers / body are all zod-checked →
`400` with details.
## Notes for reviewers
- Everything wraps an existing service; the intent is API parity for
things that are currently dashboard-only, not new behaviour.
- `createActionPATApiRoute` is new shared infra (the PAT + RBAC mutation
builder that didn't exist). It's self-contained - the loader builder and
existing routes are untouched.
- `@trigger.dev/core` gets one additive field (`defaultRegion` on the
project response, optional/nullable for client-server version skew) -
changeset included, patch.
- `removeTeamMember`'s last-member guard is now atomic (Serializable
transaction via the `$transaction` helper, with retry), so the dashboard
and API both get it server-side. Added a `## Transactions` rule to
`apps/webapp/CLAUDE.md` (always use the `$transaction` helper);
migrating the remaining direct usages is tracked in TRI-11698.
## Open questions
- ~~Is PAT the right auth (vs OAT for automation)?~~ **Resolved: PAT.**
Organization Access Tokens are currently internal-only (used by the
image builder) and not user-accessible, so they can't back this yet.
- Should any of these be gated behind a flag or scope?
- Naming/shape of the routes.
## Summary
Batch triggers that use per-item idempotency keys could take seconds
instead of milliseconds when the target task had a large run history.
This keeps the idempotency lookup fast regardless of how many runs a
task has accumulated.
## Root cause
The batch path checks which items already have runs by looking up their
idempotency keys with a single `WHERE runtimeEnvironmentId = ? AND
taskIdentifier = ? AND idempotencyKey IN (...)` query. On a very large
`TaskRun` table Postgres underestimates the row count of a specific
`(environment, task)` pair, so once the `IN` list grows past a handful
of keys it stops doing per-key index probes and instead scans every run
for that `(environment, task)` and filters the keys in memory. The cost
is then flat and large regardless of how many keys are being checked,
and a routine `ANALYZE` does not correct the estimate at that table
size.
## Fix
Look each idempotency key up on its own, batched into a `UNION ALL` of
point lookups (chunked, run with bounded concurrency). Each branch is an
equality on all three columns of the unique index, so the planner can
only do a per-key index probe and can never fall back to the range scan.
Same results, same columns, confined to the batch trigger path.
### Problem
Webapp HTML references content-hashed /build assets, and each Docker
image contains exactly one build with a hard 404 for unknown hashes.
During a rolling deploy, a client holding HTML from the old build may
request old asset hashes from a replica running the new image, causing
missing styles or failed chunk loads.
The page should recover automatically once a compatible build becomes
available, without reload loops or unnecessary interruptions during
normal deployments.
### What changed
- Build changes alone do nothing — no polling, no automatic reloads.
- If a CSS or JavaScript asset fails to load, a recovery overlay is
shown immediately.
- While the server still reports the same build, the client polls for a
newer build using exponential backoff (up to ~60s). As soon as a newer
build is detected, the page reloads automatically.
- If no newer build appears within the timeout, recovery falls back to a
manual Reload action.
- If recovery still fails after the automatic reload, the client stops
retrying and displays a final recovery screen instead of entering a
reload loop.
- Recovery preserves form values and scroll position across the
automatic reload.
## Summary
When the run-ops split is enabled, the legacy run-ops database client
was always constructed as its own connection pool, even when it points
at the same database as the primary (control-plane) client. On setups
where those two DSNs resolve to the same physical database, this opened
a second, redundant pool and doubled the number of connections used
against that database. This change makes the legacy client reuse the
primary client's pool whenever their DSNs point at the same database,
and only open a separate pool when they genuinely differ.
## Fix
A small `sameDatabaseTarget` comparison (host, port, database name,
user) decides whether the legacy DSN points at the same database as the
primary. When it does, the legacy handle reuses the primary client by
reference, so no second pool is opened. When the DSNs diverge, the
legacy client is built independently as before, so the split still works
once the databases are actually separate.
Two smaller changes ride along:
- An optional per-pool limit for the run-ops read replica, which
connects unpooled and so draws raw backend connections; unset, it falls
back to the existing default and behaviour is unchanged.
- A startup warning about a missing legacy replica URL is now suppressed
when the legacy client shares the primary pool, where it would be
misleading.
## Verification
Booted the webapp end-to-end in three modes and confirmed the pools
opened as expected via the client's own startup logs and live backend
connection counts: split off (single pool), split on with a shared
database (legacy reuses the primary pool, no doubling), and split on
with separate databases (legacy opens its own pool).
The `ResourceMonitor` server-side logging helper is no longer used. It
periodically logged the webapp process own memory, disk, and CPU usage
behind the `RESOURCE_MONITOR_ENABLED` flag (off by default), and was
also exported from `@trigger.dev/core/v3/serverOnly` with no other
consumers.
This removes the helper, its `@trigger.dev/core` export, the webapp
wiring, and the `RESOURCE_MONITOR_ENABLED` env var. The supervisor has
its own unrelated `ResourceMonitor` class, which is left untouched.
## Summary
Adds `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` (default off), which
makes the task-event store skip all PostgreSQL `TaskEvent` writes. It's
for deployments that store task events in ClickHouse
(`EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2`) and no longer want the
PostgreSQL copy.
## How it works
The guard sits at the single postgres write boundary,
`TaskEventStore.create` / `createMany`, so it covers every write path
(OTLP ingestion and run-lifecycle events) with one check. Reads are
untouched (`findMany` / trace queries / streaming), so existing
PostgreSQL events remain readable.
Leave it off unless the default store is `clickhouse_v2`, otherwise task
events for any run still routed to PostgreSQL would be dropped.
## What
The `GET
/api/v1/projects/:projectRef/background-workers/:envSlug/:version`
endpoint loaded each file's tasks through the nested `files.tasks`
relation. Prisma resolves that as a separate query:
```sql
SELECT id, slug, "fileId" FROM "BackgroundWorkerTask" WHERE "fileId" IN (...)
```
`BackgroundWorkerTask.fileId` is not indexed — the FK constraint exists,
but Postgres does not auto-create an index for foreign keys — so on a
large table this can only run as a sequential scan, which gets
progressively slower as the table grows and was observed taking minutes
per call in production.
The loader already loads every task for the worker via `tasks: true`,
which uses the indexed `workerId` relation, and those rows already
include `fileId`. This PR groups task slugs by `fileId` in memory from
that already-loaded data and drops the `files.tasks` include entirely.
## Behavior change (latent bug fix)
The response shape is unchanged, but there is a semantic correction for
**source files reused across worker versions** (files are de-duplicated
by `@@unique([projectId, contentHash])`, so one file row can be linked
to many workers).
- **Before:** `file.tasks` came from the `BackgroundWorkerFile.tasks`
relation, i.e. *every* `BackgroundWorkerTask` with that `fileId` —
across all workers sharing the file. So a worker's manifest could list
tasks it doesn't actually have.
- **After:** `file.tasks` is grouped from the queried worker's own
tasks, so it reflects only that worker version's tasks.
Verified on a local DB: 460 files are referenced by tasks from more than
one worker; of 6819 (worker, file) pairs, 6 differ — all one file where
the old union leaked a task slug (`cancellation-test`) into worker
versions that never had it. The new per-worker behavior is the correct
one for a worker-version manifest. (Thanks to the automated review for
flagging this.)
## Analysis
Captured the exact SQL before/after by instrumenting Prisma against real
data (a worker with 62 files):
- **Before:** 5 statements, including the `WHERE "fileId" IN (...)`
scan.
- **After:** 4 statements; the `fileId` query is gone and the other four
are identical.
EXPLAIN of the two access paths:
```
Before WHERE "fileId" IN (...)
Seq Scan on "BackgroundWorkerTask"
Filter: ("fileId" = ANY (...)) -- reads the whole table, scales with table size
After WHERE "workerId" IN (...)
Index Scan using "BackgroundWorkerTask_workerId_slug_key"
Index Cond: ("workerId" = ...) -- bounded by matching rows, scale-independent
```
No new index is required: the `workerId` access path is already covered
by the existing `BackgroundWorkerTask_workerId_slug_key` unique index.
## Testing
- `pnpm run typecheck --filter webapp` passes.
- Query capture + EXPLAIN performed against a local database seeded with
real worker/file/task data.
## Summary
Run-graph data (runs, batches, waitpoints, and their related tables) can
now live in a database separate from the control plane, with every read
and write routed to the correct database by each run's residency. This
makes reading and writing run data more reliable once the two are split,
and is a no-op for single-database installs.
## Design
- Run-graph table access goes through the run-store router, which
selects the legacy or the new run-ops store per run instead of assuming
one shared client.
- The legacy run-ops client is now independently pointable, so legacy
run data can be served from its own database (and replica) rather than
the control-plane connection.
- Run-graph writes go straight to the run-graph database instead of
being forwarded through the control plane, and replication targets are
split so runs in the new database still replicate to analytics without
under-counting.
- Read-through slots refuse the control-plane client, so a missing
residency fails loudly instead of silently reading the wrong database.
- Migration `20260710120000_drop_remaining_run_graph_seam_foreign_keys`
drops the foreign keys that still crossed the run-graph / control-plane
seam, which is what lets the two live in separate databases.
The split stays off unless explicitly enabled and the two databases are
confirmed physically distinct; startup fails closed otherwise.
Verified by running the full dashboard end-to-end suite against both a
single-database configuration and a three-database configuration
(control plane, the new database, and a physically separate legacy
database), with runs on both residencies. No misrouted reads in either
configuration.
## Summary
v3 (the engine that ran the SDK v3 era, internally
`RunEngineVersion.V1`) is end-of-life. Following the removal of the v3
execution apps
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and
the legacy dev websocket
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this
removes the remaining v3 execution stack from the server.
Clients still on v3 (an old SDK or CLI that has not upgraded) keep
getting a clear "upgrade to v4" response. Triggers, batch triggers,
reschedules, and deploys that resolve to v3 are rejected with a graceful
4xx pointing at the migration guide, never a 5xx, so a stale client
cannot affect server health. Self-hosted instances still running v3
should stay on the 4.5.x release line until they migrate.
## What is removed
- The MarQS queue and its shared/dev queue consumers.
- The v3 socket.io namespaces (coordinator, provider, shared-queue) and
the v3 run lifecycle services (attempt, checkpoint, and batch-resume).
- The graphile-worker background job system; all live jobs already run
on `@trigger.dev/redis-worker`.
- The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally,
so the flag is gone.
- Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace`
subpath and the legacy socket message catalogs) and the now-dead MarQS
environment variables.
## What stays
The v4 engine is untouched. The graceful v3 rejection boundary stays,
`determineEngineVersion` still detects a v3 project so it can reject it,
and the batch service plus batch-completion worker stay for current
clients. Live queue concurrency limits and metrics now read from the v4
run engine instead of MarQS, and a brand-new dev environment now
defaults to v4.
## Dependency cleanup
Removes webapp dependencies left unused by this change: `seedrandom` and
`semver` (only the removed v3 code used them) plus a set that was
already dead, their orphaned `@types` packages, and two dead files. Adds
a `knip:deps` script and a `knip.json` config so unused dependencies can
be found the same way going forward.
## Summary
Two threads on the run-ops split path.
Read path: per-item run reads are batched into grouped queries, a
waitpoint's connected-run reads are bounded, and the dedicated-schema
relation hydrators fetch only the requested columns instead of whole
rows. Retrieve also falls back to the other database when a routed read
misses, so a run whose physical residency diverges from its id shape is
still found rather than returning a spurious not-found. Fewer and
lighter queries on the run read path, with no change to results.
Mint-kind flip safety: flipping which database new runs mint to is now a
deterministic wall-clock cutover, for both per-org and global flips. For
a grace window every process resolves the same database, so a flip
cannot route two concurrent triggers that share an idempotency key to
different databases (which would bypass the per-database unique
constraint and create a duplicate run).
Supersedes the earlier #4205 and #4208.
Draft: validation in progress.
Switches the native build server from opt-in to opt-out in project build
settings.
- It's now enabled by default, stored as a new
\`disableNativeBuildServer\` opt-out key so previously-saved
\`useNativeBuildServer: false\` values aren't treated as deliberate
opt-outs.
- The "Use native build server" checkbox is checked by default;
unchecking it persists the opt-out.
- Brief wording: clarifies build settings apply to GitHub-triggered and
native build server deployments, and the native build server hint no
longer says "in the future".
## Summary
Under high OTLP ingest volume, the whole decode, transform, and enrich
pipeline runs on the request event loop, so a single CPU core becomes
the ceiling while the rest sit idle. This adds an opt-in worker pool
that moves decode, transform, and LLM-cost enrichment onto worker
threads, keeping the main thread free for I/O. It is off by default
(`OTEL_TRANSFORM_WORKER_POOL_ENABLED`), so behavior is unchanged unless
enabled.
## Design
Workers do decode, filter, convert, and enrich (including LLM pricing
match). The main thread stays the single database reader: it loads the
pricing registry and broadcasts the compiled model rows to the workers
(re-broadcasting on every reload), so workers never touch the database.
The pure transform is extracted into a dependency-light module (no
Prisma/Redis/ClickHouse imports) so it can run inside a worker.
Importantly, the main thread keeps the existing single consolidated
insert path, so ClickHouse insert batching and part count are unchanged.
The parallelism buys CPU headroom, not more insert streams (which would
add merge pressure).
The worker is bundled as a standalone file at build time and ships in
the existing image with no Dockerfile change. In local load testing the
pool sustained roughly 2.6x the throughput of the single-thread path and
kept the main thread responsive under load.
## Summary
Sending a chat message immediately after an action (for example an undo)
could make the message's response vanish from the UI. The transport
opened a response stream that closed on the *earlier* turn's completion
instead of waiting for the send's own turn. The agent still produced and
persisted the answer, so it reappeared on refresh. Same "disappearing
message" class as
[#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176),
different cause.
## Fix
A send's response stream had no way to tell whether a `turn-complete`
belonged to its turn. `POST /realtime/v1/sessions/:id/in/append` now
returns the appended record's sequence number, and the transport skips
any turn-complete whose `session-in-event-id` (the agent's committed
`.in` cursor) is below that seq, closing only on its own turn. Older
webapps omit the seq, in which case the transport falls back to the
previous behavior, so the SDK and server can ship independently.
Because the fix spans the SDK and the server, both a webapp deploy and
an SDK release are needed for the full effect.
Verified end to end with the ai-chat reference app:
undo-then-immediate-send loses the follow-up's answer before the fix and
streams it inline after, with a revert-the-guard run reproducing the
loss on the same script. Unit tests cover the skip and the no-seq
fallback.
## Summary
The default realtime backend was hardcoded to Electric. This adds a
`REALTIME_BACKEND_DEFAULT` env var (`electric` | `native` | `shadow`,
default `electric`) that chooses the backend for any environment whose
org has no `realtimeBackend` override. Behavior is unchanged unless you
set it; per-org overrides still win.
The default is applied at every point where the per-org flag falls
through: the initial value, the flag lookup default, and the error
fallback.
## Problem
Several display/grouping issues in the **Errors** feature, all rooted in
how the ClickHouse error materialized views (`errors_mv_v1`,
`error_occurrences_mv_v1`) read the stored error JSON produced by
`parseError`:
1. **Messageless errors show "Unknown error".** An empty message falls
straight through `coalesce(nullIf(message,''), 'Unknown error')` to the
literal, even though the error's class `name` is available (e.g. an
Effect tagged error `ListMessagesError` with no message).
2. **Unrelated errors collapse into one group.**
`calculateErrorFingerprint` keys on `type : message : stack`, where
`type` is always the union tag (`BUILT_IN_ERROR`, …), `message` is
empty, and the stack isn't read — so every messageless built-in error
(and every string/custom error) hashes to the same constant input → one
fingerprint.
3. **error_type shows the internal tag.** `coalesce(type, name, …)`
always resolves to `type` (always present), so the column shows
`BUILT_IN_ERROR` instead of the real class name.
4. **Stack traces never populate.** The MVs read `error.data.stack`, but
the serializer stores the trace under `stackTrace` — so the column is
always empty.
## Fix
All display changes are `ALTER TABLE … MODIFY QUERY` on the two views
(migration `035`); the fingerprint change is in the webapp.
- **Fingerprint** (`errorFingerprinting.ts`): fall back **message → name
→ raw**. Messageless errors now group by class name (or raw value for
non-Error throws); message-bearing errors are **unchanged**
(short-circuits at `message`), so existing groups don't split — only
currently-messageless errors get their own group going forward.
- **error_message**: same `message → name → raw` fallback before
`'Unknown error'`.
- **error_type**: coalesce `name → code → 'Error'` (drops the reliance
on the union tag). Built-in → class name, internal → `code`,
string/custom → `Error`.
- **stack trace**: read `error.data.stackTrace`. Bounded as before
(serializer caps 50 frames / 1024 chars per line; MV clips to 2000
chars).
## Migration notes
- `MODIFY QUERY` swaps the view query in place (no drop/recreate gap);
Down restores the previous query.
- **Existing rows are left unchanged** — changes apply only to rows
inserted after the migration. No backfill.
## Tests
`errorFingerprinting.test.ts` — 57 pass, incl. new cases for messageless
class names, string/custom raw values, and stability of message-bearing
fingerprints.
Fixes the display-derivation half of TRI-11938 (error_type + stack
trace); relates to TRI-9254 and TRI-9250.
Applying a directory-sync effect that would demote the org's last Owner
(a
group remap, or a provision) previously threw and 500'd the settings
save. Now
rbac.setUserRole reports code:"last_owner" and applyEffect skips just
that
member (they keep Owner) while the rest of the batch applies.
Adds the machine-readable RoleAssignmentResult.code to the plugin
contract so
callers can tell the last-owner guard apart from a real failure.
## Summary
The RBAC and SSO auth plugins can own their own database client, but
they could only read `DATABASE_URL`, so every connection they opened
landed on the primary. The host webapp now resolves writer and
read-replica URLs from its env (the same fallback chain its own Prisma
clients use: control-plane URL first, then the default) and passes them
to the plugins at create time via a shared `PluginDatabaseConfig`, along
with separate connection limits for writes (default 2) and reads
(default 5, tunable via `RBAC_DATABASE_*_CONNECTION_LIMIT` and
`SSO_DATABASE_*_CONNECTION_LIMIT`).
A plugin can then route hot-path reads (per-request auth checks, login
routing) to the read replica and keep only rare mutations on the
primary. With no replica configured, or no plugin installed, nothing
changes: the OSS fallback ignores the new option and keeps reading
through the Prisma clients it is already given.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
This adds an always-visible info tooltip on the Location column and a
note in the "set default region" confirmation dialog making it explicit.
It also removes the obsolete "V4" badge from the Regions page title.
## What & why
Signup promo credits. A new logged-out `/promo?code=<code>` landing page
validates the code and carries it through signup via a cookie. When the
new organization is activated by selecting a plan, the code is redeemed
and its credits are applied; the usage page then shows the remaining
promo credits and their expiry.
## Notes
- The code is redeemed at **plan selection**, not org creation: the
credit grant targets the org's usage allowance, which only exists once a
plan is selected — applying at creation would have nothing to grant
onto. Redemption is best-effort and never blocks plan selection.
- Pairs with the corresponding billing-service change (promo code
validate/apply/credits + grant issuance); the two are released together.
## Testing
Verified locally end to end: `/promo` shows the offer, a new account
carries the code through signup, selecting the Free plan redeems it, and
the usage page shows the remaining credits.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- Verified the rule emits exactly five errors for un-awaited throws of
the known
async redirect helpers while ignoring awaited throws, returned promises,
and
synchronous `redirect(...)`.
- Verified `--fix` inserts `await` in async functions and produces a
clean
second lint run.
- Verified synchronous functions remain diagnostic-only so autofix
cannot
introduce invalid syntax.
- Ran `pnpm run format`, `pnpm run lint`,
`pnpm run typecheck --filter webapp`, and `git diff --check`.
---
## Changelog
Adds an Oxlint rule that prevents async redirect helpers from being
thrown
without awaiting their `Response`. Existing violations are fixed, the
autofix
is limited to async functions, and the plugin uses an explicit ESM
extension.
---
## Screenshots
See the test-results comment for CLI evidence.
💯
Link to Devin session:
https://app.devin.ai/sessions/e60ad7610773401da3d3040cf1252337
Requested by: @ericallam
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Eric Allam <eric@trigger.dev>
## Summary
Magic link login could appear completely broken: submitting your email
on the login page showed a stale "This email is unauthorized" error
instead of the "we've sent you a magic link" confirmation, even when the
address was fine.
This PR reverts
[#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) (whose
diagnosis and fix turned out to be wrong) and fixes the actual bug,
which was in how login errors are stored and consumed.
## Root cause
Two session bugs compounded on the login page:
- The `/login` loader read the flashed `auth:error` without committing
the session. A Remix flash is only consumed when the session is
committed after the read, so once any attempt flashed an error (for
example an address rejected on an instance with `WHITELISTED_EMAILS`
set), it stayed in the session cookie and reappeared on every later
`/login` visit, making successful attempts look like failures.
- The `/login/magic` action stored its validation and rate limit errors
with `session.set`, which survives every later read and commit, so those
errors stuck permanently.
[#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) had
instead diagnosed a server-only module leaking into the client bundle
and crashing navigation. Checking the shipped images' client bundles via
their sourcemaps shows `.server` modules were always stubbed out, so
that change fixed nothing and is reverted here.
## Fix
- `/login` reads the flashed error and commits the session when one was
present, so an error renders once and clears. The `redirectTo` branch
now surfaces the error too instead of leaving it in the cookie.
- `/login/magic` flashes its errors instead of `set`ting them.
Verified end-to-end on a live preview environment: a rejected address
shows the error once and a reload clears it; a valid address lands on
the confirmation screen with the address named; GitHub, Google, and SSO
login paths are untouched by this diff.
## Summary
Submitting your email on the login page could reload back to an empty
login form instead of showing the "we've sent you a magic link"
confirmation. The magic link email was still sent, so it looked like
nothing happened.
## Root cause
The `/login/magic` route imported a server-only cookie module
(`magicLinkEmailCookie.server.ts`) whose top-level `env.NODE_ENV` read
got bundled into the route's client JS. On the client `env` is
undefined, so the module threw a `TypeError` at module eval, which
aborted Remix's client-side navigation to the confirmation and
hard-reloaded back to `/login`. It only surfaced in production builds
(local dev auto-logs-in, and local prod builds happen to tree-shake the
module out), which is why it slipped through.
## Fix
The email-link strategy already stores the submitted address in the
session (`auth:email`), so the separate cookie was redundant. Deleted
the cookie module and read the address from the session in the loader.
With the module gone, nothing server-only can leak into the client
bundle regardless of tree-shaking.
Verified the confirmation renders with the email address, the SSO
domain-policy redirect (with the email prefilled) still works, and a
production build no longer bundles the module.
Condense the kept rationale comments (logLevel/warn, last-Owner dedup,
role
overwrite) and drop the obvious function-header comments that just
restated
the code. No behavior change.
Directory-sync effects are idempotent and the accounts-webhook worker
retries
the whole event, so a single failed attempt (typically a role assignment
losing a serializable race during a backfill burst) is self-healing
rather
than alert-worthy. Tag those thrown errors with logLevel "warn" so the
worker
logs at warn instead of error, keeping them visible for triage without
paging.
## Summary
The runs page's empty-state check (whether an environment has ever had a
run, which decides between the "getting started" and "no runs match your
filters" states) ran a `findFirst` against the Postgres `TaskRun` table.
This moves it to ClickHouse, the same store the runs list itself reads
from, so the check no longer queries `TaskRun`.
## Design
Only the runs list triggers the check now (via an `includeHasAnyRuns`
flag); the other presenters that reuse `NextRunListPresenter` (API,
schedule detail, waitpoint detail, error group) no longer issue it. When
the list is empty it runs `SELECT 1 FROM task_runs_v2 ... LIMIT 1`
filtered on the full `(organization_id, project_id, environment_id)`
sort-key prefix with a configurable `created_at` lower bound
(`RUN_LIST_HAS_RUNS_LOOKBACK_DAYS`, default 30), so it hits the primary
index and reads minimal granules.
Results are cached in a tiered memory + Redis SWR cache. Only positive
("has runs") results are cached, so an environment with no runs is
always re-checked and its first run shows up immediately.
Follow-up to #4194 (v3 execution app + core-helper removal). The v3
(engine V1) is end-of-lifed and enforced off in prod, so this removes a
self-contained slice of the remaining dead v3 code while **keeping every
user-facing deprecation message** - a user still on v3 must still be
told to upgrade.
## Legacy dev websocket
`app/v3/handleWebsockets.server.ts` backs the `/ws` transport used
**only** by the legacy v3 `trigger dev` CLI (v4 dev uses a different
transport). It's now authenticate-then-close with
`V3_DEV_DEPRECATION_MESSAGE`, so an old CLI is still told what to do -
only the legacy `AuthenticatedSocketConnection` / `DevQueueConsumer`
execution behind it (which can no longer run) is removed.
- Deleted `app/v3/authenticatedSocketConnection.server.ts` (its only
consumer).
- `engineDeprecation.server.ts` and the deprecation message constants
are untouched.
## Docs
Deleted the intentionally-legacy "Docker (legacy)" self-hosting page
(`open-source-self-hosting.mdx`) and redirected
`/open-source-self-hosting` (+ the existing
`/v3/open-source-self-hosting` alias) to `/self-hosting/overview`;
repointed the two inbound links. The current `self-hosting/*` docs
already describe the v4 (single supervisor) setup.
## Deliberately out of scope
Despite the branch name, this PR does **not** touch MarQS or the
socket.io coordinator/provider namespaces. Investigation found MarQS is
entangled with **live v2** queue/metrics/concurrency/project-cleanup
code (`runQueue`, `queueSizeLimits`, `taskRunConcurrencyTracker`,
`EnvironmentQueuePresenter`, `registerProjectMetrics`, `deleteProject`),
so it needs a per-file reviewed pass, not a bulk delete. That remainder
stays on TRI-11883.
refs TRI-11883
## Summary
UI/layout/copy pass over the org **SSO & Directory Sync** settings page
(formerly "Identity & Access"). No logic, gates, flags, or data flow
changed — server-side auth (`manage:sso`), Enterprise entitlement,
action validation, and data loading are all untouched.
- Renamed the nav item, page title, and meta from "Identity & Access" to
"SSO & Directory Sync".
- Added a reusable `SettingsLayout` component system (container,
section, header, row, block, actions) modeled on `/account/security`,
and refactored the SSO page onto it (section titles, dividers, left
title/subtitle + right action rows).
- Tightened all UI copy: concise, active voice, consistent labels, no
em-dashes.
- `Select` primitive: additive `wrap`, `popoverClassName`, and
`placement` props (all default to prior behavior) so role options show a
bright title with a wrapping description, right-aligned popover, and no
horizontal overflow.
- Removed the external-link arrow icon from buttons that open a modal;
kept it only on genuinely external actions (Contact us, Open in new
tab).
- Polished the admin portal link dialog: smaller description, tighter
spacing, `ClipboardField` with a permanent copy button, removed the
redundant Copy link button, and a provider-aware Open label (e.g. "Open
in WorkOS") derived from the link host with a safe fallback.
### SSO page UI
<img width="3568" height="2550" alt="CleanShot 2026-07-08 at 18 52
11@2x"
src="https://github.com/user-attachments/assets/009d2437-7552-4ff0-a457-64744a9fcd88"
/>
### Login with SSO and normal email test (local)
https://github.com/user-attachments/assets/b33a4ce9-c1fa-45c9-bd3c-077cb6fc9473
## Test plan
- [ ] Non-Enterprise org: SSO page shows the upsell state
- [ ] Enterprise org, non-Owner without `manage:sso`: 403
- [ ] Enterprise Owner: verify domains, configure SSO, connect
directory, JIT/default/group role selects, and enforcement toggle all
work
- [ ] Role select popovers: bright title + wrapping description,
right-aligned, no horizontal scroll
- [ ] Admin portal dialog: copy button works, "Open in WorkOS" opens the
portal in a new tab
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
v3 (engine V1) is end-of-lifed and the v3 clusters are gone, so this
removes the dead v3 execution code from the monorepo. It's the first
pass of TRI-11824 - the webapp v3 code paths are deliberately left
untouched and gated for a follow-up.
## Apps
Deletes the three v3-only execution apps and their build wiring:
- `apps/coordinator`, `apps/kubernetes-provider`, `apps/docker-provider`
- `.github/workflows/publish-worker.yml` - it built only those three;
the v4 worker publish is a separate workflow
- Their references in `.changeset/config.json`, `.cursorignore`,
`CHANGESETS.md`, `CONTRIBUTING.md`, `.server-changes/README.md`
- `pnpm-lock.yaml` regenerated to prune the apps and their app-only
dependencies (`socket.io`, `@kubernetes/client-node`, `p-queue`,
`execa`, `prom-client`, `tinyexec`)
## Core
Removes the helpers in `@trigger.dev/core` that only those apps used -
`ProviderShell`, `SimpleLogger`, the `Exec`/process helpers,
`isExecaChildProcess`, `getTextBody`, and `testDockerCheckpoint`. Each
was verified to have no remaining consumers anywhere in the repo.
Kept the helpers still used elsewhere: `ExponentialBackoff` (warm-start
client), `HttpReply`/`getJsonBody` (serverOnly http server),
`SimpleStructuredLogger` (widely used), and
`ZodNamespace`/`ZodSocketConnection` (still referenced by legacy v3
webapp code, hence the follow-up pass).
The `./v3/apps` and `./v3/serverOnly` export subpaths remain - only dead
members were trimmed from their barrels, so no `package.json` exports
changed.
## Verification
`@trigger.dev/core` builds, and `typecheck` passes for core, supervisor,
cli-v3, run-engine, redis-worker, and webapp.
refs TRI-11824
### Problem
The agent playground chat builds its realtime transport baseURL from
apiOrigin, but points it at a same-origin /resources/... dashboard
route. When API_ORIGIN differs from APP_ORIGIN, the in/append POST goes
cross-origin, fails the CORS preflight, and messages never reach the
agent ("Failed to fetch").
It only reproduces where the two origins differ — not locally, where
both default to localhost:3030.
Fixes#4149.
### Fix
Build the base URL from window.location.origin (falling back to
apiOrigin on SSR), so realtime traffic stays same-origin — the same
approach AgentView.tsx already uses.
### Testing
Typecheck passes. The CORS path only manifests when API_ORIGIN !=
APP_ORIGIN, so verify on test-cloud (can't reproduce locally).
## Summary
Deleting a project triggered an unbounded database cleanup that scanned
the project's entire run history, so deleting a project with many runs
could be very slow. Project deletion is a soft delete again: run data is
retained and the deletion completes quickly.
## Fix
Project deletion ran a cascade hard-delete whose `BulkActionItem` step
filtered through a relation to `TaskRun` scoped by `projectId`. Prisma
compiles that to an `EXISTS`-join over the project's entire `TaskRun`
set (a large, hot table with no `projectId` index), and it ran on every
project deletion unconditionally.
Removing the cascade-cleanup call restores the prior soft-delete
behaviour: queues are removed, the project is marked deleted, and run
data is retained. The cascade-cleanup service (added in
[#4117](https://github.com/triggerdotdev/trigger.dev/pull/4117)) had no
other callers, so it and its test are deleted.
## Summary
posthog-js sent product analytics to PostHog Cloud directly from the
browser. This points `api_host` at a same-origin `/ph` path that
forwards to PostHog Cloud EU server-side, following PostHog's standard
first-party reverse-proxy setup.
## How it works
A resource route forwards each request server-side, splitting by path:
`/ph/static/*` and `/ph/array/*` go to the asset host, everything else
(analytics events, feature flags) goes to the ingest host. It rewrites
the `Host` header, strips the `/ph` prefix, and streams the response
back. Only PostHog's own cookies are forwarded, so the app session
cookie stays first-party. Upstream hosts default to PostHog Cloud EU,
overridable via `POSTHOG_INGEST_HOST` / `POSTHOG_ASSETS_HOST`.
It also sets `cross_subdomain_cookie` so a single PostHog session is
shared across the marketing site and app.
Verified locally: static assets return 200 from the EU asset host, and
analytics events return 200 through the ingest host.
## Summary
The run page could show an AI generation cost well above what the
provider actually charged, most visibly for OpenRouter and Vercel AI
Gateway requests where a heavily cache-read prompt was priced at the
full input rate. When the provider reports an exact per-request cost, we
now use that instead of catalog pricing.
## Fix
Gateway and OpenRouter include the exact per-request cost in
`ai.response.providerMetadata` (`openrouter.usage.cost` /
`gateway.cost`). That figure already reflects the cache-read discount
and the real per-provider rate, which the catalog cannot reconstruct:
cache-read counts do not arrive in `gen_ai.usage.*`, and per-model
catalog prices drift from what the provider billed, in either direction.
So provider-reported cost is now preferred, and the catalog is used only
when no provider cost is present.
Fallback routing is covered by the same change: when OpenRouter routes
to a different model, `gen_ai.response.model` already carries the served
model, so the cost follows the served model and the provider's own
figure makes it exact.
`extractProviderCost` now runs on every AI span, so it gets a cheap
`"cost"` substring guard to skip the JSON parse on reasoning-model spans
whose provider metadata carries large reasoning text and no cost field.
Regression tests cover the cache-discount overcharge, fallback
served-model pricing, gateway cost, and the catalog fallback path.
Daniel Sutton