Files
Eric Allam b98dd79fe4 feat(webapp,run-store,database): env-configurable transaction resilience (maxWait + tx-start retry) (#4623)
## What

Makes two transaction-resilience behaviors real and env-var
configurable, defaults set to the good values, so we can tune during and
after the Aug 15 database patch window without a redeploy:

- **maxWait 2s → 10s** (TRI-12982): how long Prisma waits to borrow a
connection before it can `BEGIN`. A restart freeze holds the pool full,
and the only thing that errored was transaction starts giving up at 2s.
- **Retry transaction-start P2028-at-acquisition** (TRI-12984): when
Prisma can't borrow a connection within `maxWait` it raises P2028
(`Unable to start a transaction in the given time`) and **no SQL ran**,
so retrying is safe. Scoped narrowly: only that error (never P2024
pool-exhaustion), 2 attempts, jittered backoff, and a token-bucket
budget so a mass freeze can't amplify into a retry storm.

## Env vars (`DATABASE_*` convention)

Generic defaults:

| var | default |
|---|---|
| `DATABASE_TRANSACTION_MAX_WAIT_MS` | `10000` |
| `DATABASE_TRANSACTION_START_RETRY_ENABLED` | `true` (kill switch) |
| `DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS` | `2` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS` | `250` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST` | `100` |

Per-writer-pool overrides, each falling back to the generic when unset
(same pattern as the per-client pool/connect-timeout work):
`RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all 7 knobs each). Transactions
only open on writer pools, so those are the only pools with their own
knobs. Each pool gets its **own** token bucket, so a storm on one pool
can't drain another's retry budget.

## Design

- The retry primitives live in `internal-packages/database` and never
read `process.env` (IoC): a P2028-at-acquisition classifier, a
`TokenBucketRetryBudget`, and `withTransactionStartRetry`, folded into
the `$transaction` helper via a new `startRetry` option. Config is
resolved at the app boundary and threaded in.
- The `$transaction` helper is the chokepoint (wraps the whole
transaction), not the per-statement `$allOperations` extension.
- The run engine's writes go through `PostgresRunStore`'s own
`.$transaction(...)`, not the webapp helper, so both the helper and the
two `PostgresRunStore` sites apply maxWait + retry (sharing the per-pool
config). Builds on the `options?: { timeout, maxWait }` seam added in
#4514.
- Webapp `$transaction` call sites get the default `maxWait` + retry
injected at one merge point, so no call site needed editing.

## Evidence

- Unit red/green in `internal-packages/database`: reverting the helper
wiring turned the acquisition-retry test red (`Unable to start a
transaction in the given time`), re-applying it green. Full package
suite 25/25. Covers: classifier (P2028-acq yes, P2024 no, in-tx P2028
no), retry (retry-then-succeed, no-retry P2024, stop at maxAttempts,
disabled, budget-exhausted, jitter bounds), token bucket, and
`$transaction` wiring.
- Typecheck clean: webapp, run-store, run-engine.
- Full-stack run: bounded queue-ay pass (15 projects, real dev runs
through the run-engine `PostgresRunStore` transaction path). 13 pass;
the 2 failures are one documented known-failure and one
stale-worker-state flake that passes 2/2 with this change active on a
fresh app.
- Boots cleanly with per-pool overrides set.

## Configuration & rollout

Ship **inert** first (zero behavior change), then flip to the good
values **live via env** — no redeploy needed for either.

### Inert — behaves exactly as today

```
DATABASE_TRANSACTION_MAX_WAIT_MS=2000            # Prisma's built-in default (change defaults to 10000)
DATABASE_TRANSACTION_START_RETRY_ENABLED=false   # disable the new retry entirely
```

`maxWait=2000` is what every path used before (Prisma's default; the
run-store sites and the helper passed no maxWait). `retry=false`
short-circuits `withTransactionStartRetry` to a single run and makes the
serialization-retry exclusion a no-op. Verified on the pooler-freeze
rig: identical fail-fast P2028 at ~2003ms with zero retries —
byte-for-byte current behavior, across all pools.

### Production ("good") — the baked defaults

Rely on defaults (nothing to set) or set explicitly:

```
DATABASE_TRANSACTION_MAX_WAIT_MS=10000
DATABASE_TRANSACTION_START_RETRY_ENABLED=true
DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS=3      # 3 attempts (2 retries); ~30s acquisition tolerance covers a ~20-25s freeze
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS=50
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS=250
DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC=50
DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST=100
```

Per-pool overrides `RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all seven knobs each) are
optional and fall back to the generic set — not needed for v1; the
generic set covers the control-plane, run-ops, and run-ops-legacy writer
pools. Readers open no transactions and take nothing.

**Guardrail:** the retry only engages when a pool's `pool_timeout` >
`maxWait`. Prod is fine (`DATABASE_POOL_TIMEOUT=60` >> 10). Do not set
any writer pool's `pool_timeout` at or under `maxWait`, or saturation
failures flip from retryable P2028 to non-retryable P2024 and the retry
silently stops helping.

### Rollback

Env flip (set inert) or revert. Retry only fires where no SQL ran, and
the per-pool token bucket caps a storm. No migration.

refs TRI-13295, TRI-12982, TRI-12984
2026-08-15 09:03:10 +01:00

140 lines
5.4 KiB
TypeScript

import { PostgresRunStore, RoutingRunStore, type RunStore } from "@internal/run-store";
import { ownerEngine, type Residency } from "@trigger.dev/core/v3/isomorphic";
import type { PrismaClient, PrismaReplicaClient } from "@trigger.dev/database";
import type { RunOpsPrismaClient } from "@internal/run-ops-database";
import {
$replica,
prisma,
runOpsLegacyPrisma,
runOpsLegacyReplica,
runOpsNewPrismaClient,
runOpsNewReplicaClient,
} from "~/db.server";
import { env } from "~/env.server";
import { singleton } from "~/utils/singleton";
import {
resilienceForClient,
type TransactionResilienceConfig,
} from "./transactionResilience.server";
type BuildRunStoreDeps = {
/** Boot constant: true only when both run-ops DBs are configured and the split flag is on. */
splitEnabled: boolean;
/** Split-only handles. Required when splitEnabled is true; omitted entirely when OFF
* so single-DB callers never touch the run-ops clients (keeps mocks/passthrough clean). */
newWriter?: RunOpsPrismaClient;
newReplica?: RunOpsPrismaClient;
legacyWriter?: PrismaClient;
legacyReplica?: PrismaReplicaClient;
/** Single-DB store handles (control-plane pair). Used verbatim when split is OFF. */
singleWriter: PrismaClient;
singleReplica: PrismaReplicaClient;
/** Residency classifier; defaults to ownerEngine inside RoutingRunStore. */
classify?: (id: string) => Residency;
/** Per-pool transaction-resilience configs threaded into the store(s) this builds (IoC). */
singleResilience?: TransactionResilienceConfig;
newResilience?: TransactionResilienceConfig;
legacyResilience?: TransactionResilienceConfig;
};
/**
* Pure run-store builder (no env / no boot side effects — webapp testability rule).
*
* Split OFF (default / self-host): returns the exact passthrough PostgresRunStore we
* have always returned, built from the single control-plane handles. No second store
* is constructed and no marker predicate is consulted, so behavior is byte-identical
* to single-DB today.
*
* Split ON: returns a RoutingRunStore that selects between a NEW store (where new runs
* are born) and a LEGACY store (draining) by run-id residency (id shape). There is no cuid
* migration, so a LEGACY-classified id is always LEGACY-resident.
*/
export function buildRunStore(deps: BuildRunStoreDeps): RunStore {
if (!deps.splitEnabled) {
return new PostgresRunStore({
prisma: deps.singleWriter,
readOnlyPrisma: deps.singleReplica,
maxWait: deps.singleResilience?.maxWait,
transactionStartRetry: deps.singleResilience?.startRetry,
});
}
if (!deps.newWriter || !deps.newReplica || !deps.legacyWriter || !deps.legacyReplica) {
throw new Error("buildRunStore: split is enabled but run-ops store handles are missing");
}
// The NEW store is backed by the dedicated RunOpsPrismaClient (subset schema): relation-shaped
// ops branch onto FK-free scalars + explicit join models. The LEGACY store keeps the default
// "legacy" variant (full @trigger.dev/database schema with implicit M2M + @relations).
const newStore = new PostgresRunStore({
prisma: deps.newWriter,
readOnlyPrisma: deps.newReplica,
schemaVariant: "dedicated",
maxWait: deps.newResilience?.maxWait,
transactionStartRetry: deps.newResilience?.startRetry,
});
const legacyStore = new PostgresRunStore({
prisma: deps.legacyWriter,
readOnlyPrisma: deps.legacyReplica,
maxWait: deps.legacyResilience?.maxWait,
transactionStartRetry: deps.legacyResilience?.startRetry,
});
return new RoutingRunStore({
new: newStore,
legacy: legacyStore,
classify: deps.classify ?? ownerEngine,
});
}
// Build the routing store whenever BOTH run-ops DBs are configured, independent of
// RUN_OPS_SPLIT_ENABLED. Reads must fan out across both DBs so a run that lives on the new
// DB stays visible even with the flag off (matches the db.server topology factory). The flag
// governs write/mint residency + migration via isSplitEnabled(), not read visibility.
const ROUTING_ENABLED = !!env.RUN_OPS_DATABASE_URL && !!env.RUN_OPS_LEGACY_DATABASE_URL;
// Resolve the run-ops handles, tolerating contexts where they are absent — tests that mock
// ~/db.server minimally omit them, and accessing a missing export under vi.mock throws. A
// miss means "no run-ops handles here" and we fall back to single-store.
function tryResolveRunOpsHandles() {
try {
if (
!runOpsNewPrismaClient ||
!runOpsNewReplicaClient ||
!runOpsLegacyPrisma ||
!runOpsLegacyReplica
) {
return null;
}
return {
newWriter: runOpsNewPrismaClient,
newReplica: runOpsNewReplicaClient,
legacyWriter: runOpsLegacyPrisma,
legacyReplica: runOpsLegacyReplica,
};
} catch {
return null;
}
}
export const runStore: RunStore = singleton("RunStore", () => {
const handles = ROUTING_ENABLED ? tryResolveRunOpsHandles() : null;
// Single-store passthrough: self-host (one DB), or a context without run-ops handles.
if (!handles) {
return buildRunStore({
splitEnabled: false,
singleWriter: prisma,
singleReplica: $replica,
singleResilience: resilienceForClient(prisma),
});
}
return buildRunStore({
splitEnabled: true,
...handles,
singleWriter: prisma,
singleReplica: $replica,
singleResilience: resilienceForClient(prisma),
newResilience: resilienceForClient(handles.newWriter),
legacyResilience: resilienceForClient(handles.legacyWriter),
});
});