feat(webapp,database): opt-in per-client Prisma driver adapters (#4539)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
📚 Docs Checks / check-broken-links (push) Has been cancelled
🧭 Helm Chart Prerelease / lint-and-test (push) Has been cancelled
Workflow Checks / Actionlint (push) Has been cancelled
Workflow Checks / Zizmor (push) Has been cancelled
🧭 Helm Chart Prerelease / prerelease (push) Has been cancelled

## What

Adds an opt-in path to run each Prisma client through
**`@prisma/adapter-pg`** (the node-postgres driver) instead of the
built-in engine driver, controlled by a **per-client env var, all off by
default**:

| env var | client |
|---|---|
| `CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER` | control-plane writer
|
| `CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER` | control-plane
replica |
| `RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER` | new run-ops writer |
| `RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER` | new run-ops replica |
| `RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER` | legacy run-ops
writer |
| `RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER` | legacy run-ops
replica |

With every flag unset the construction path is byte-identical to today
(`datasources` URL + Rust engine), so this is inert until a flag is
turned on. Per-client granularity allows enabling the adapter only where
it's wanted.

## How

- Enables the `driverAdapters` preview feature on both schemas
(`@trigger.dev/database` and `@internal/run-ops-database`). This keeps
the **Rust query engine** — it does NOT add `queryCompiler` — so query
behavior, result types, and engine tracing spans are unchanged.
- A shared `buildDriverAdapterPool` builds each client's `pg.Pool` with
an explicit `max`, a bounded `connectionTimeoutMillis` (the
node-postgres pool otherwise waits unbounded on acquire), and an
`onPoolError` handler (an unhandled idle-connection error would
otherwise crash the process). Threaded through all four client builders
via a `useDriverAdapter` flag.
- Adds `@prisma/adapter-pg` + `@types/pg` to the webapp; `pg` is already
pinned at `8.15.6` (adapter-pg 6.x requires `pg < 8.17`).

## Connect-failure handling (the important correctness/security bit)

Under the adapter an unreachable DB no longer surfaces as
`PrismaClientInitializationError` / `P1001`; it becomes a `P2010`
"Database not reachable: <host>" (or a raw
`ECONNREFUSED`/`ENOTFOUND`-class error). Two handlers are updated so a
client on the adapter behaves like today:

- **`isInfrastructureError`** now recognizes those shapes (P2010 with a
connectivity message, and raw connectivity errno codes). Without this,
the DB **hostname would leak into API-client-facing errors** and the
failure would go unlogged. Security-relevant.
- **`isPrismaRetriableError`** treats the adapter's pool-acquire timeout
("timeout exceeded when trying to connect") as retriable, preserving the
`P2024` retry behavior the adapter otherwise drops.

## Evidence

Validated on an isolated stack that mirrors the production DB topology
(chained PgBouncers in front of writer + reader):

- **Behavioral parity:** raw-query results and Prisma error codes/`meta`
are byte-identical between the engine driver and the adapter across the
queried shapes (unique-constraint `meta.target`, record-not-found,
transaction-timeout, serialization-failure, etc.).
- **Feature matrix:** a full 380-project queue-ay pass shows no
adapter-caused regressions — pass/fail parity between adapter-off and
adapter-on, with the residual failures being pre-existing
known-failures/flakes common to both.

## Rollout / rollback

All flags default off; enable per client via env var, roll back by
unsetting and redeploying (no data migration). Recommended first target
is a single writer; enable one client at a time.

## Follow-ups (not in this PR)

- `$metrics`-based pool observability is removed under the adapter (the
Prometheus route + `db.pool.connections.*` instruments); the metrics
replacement (via `pg.Pool` counters) lands in a separate PR.
- Note for operators: on the adapter path, interactive-transaction
`maxWait` does not bound pool acquisition — `connectionTimeoutMillis`
does.

## Note on connection-string parameters

The adapter pool is built from the base DSN, so Prisma-specific DSN
parameters that node-postgres does not understand are not honored when a
client is on the adapter:

- **Prisma TLS spellings** (`sslaccept`, `sslcert`, etc.) —
node-postgres uses `sslmode`/`ssl` instead. Our production DSNs do not
use these Prisma-specific TLS params, but any deployment whose DSN
relies on them must be checked before enabling a flag.
- `pgbouncer=true` and `statement_cache_size` — effectively moot under
the adapter, which uses no persistent named prepared statements.

`connection_limit`, `pool_timeout`, and `schema` are handled explicitly
(passed as `max`/`connectionTimeoutMillis` and PrismaPg's `{schema}`
option).

refs TRI-13039

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Eric Allam
2026-08-08 21:27:20 +01:00
committed by GitHub
parent c526528d8f
commit 90e8bd5c12
12 changed files with 364 additions and 167 deletions
@@ -0,0 +1,6 @@
---
area: webapp
type: improvement
---
Groundwork for an alternative database connection driver, gated behind configuration and disabled by default, so there is no change to default behavior.
+270 -144
View File
@@ -10,6 +10,8 @@ import {
} from "@trigger.dev/database";
import { RunOpsPrismaClient } from "@internal/run-ops-database";
import { markReadReplicaClient } from "@internal/run-store";
import { PrismaPg } from "@prisma/adapter-pg";
import { Pool } from "pg";
import invariant from "tiny-invariant";
import { z } from "zod";
import { env } from "./env.server";
@@ -307,7 +309,14 @@ const runOpsTopology: RunOpsTopology = singleton("runOpsTopology", () => {
controlPlane: { writer: prisma, replica: $replica },
buildNewWriter: (url, clientType) =>
captureInfraErrorsRunOps(
tagDatasourceRunOps("run-ops-writer", buildRunOpsWriterClient({ url, clientType }))
tagDatasourceRunOps(
"run-ops-writer",
buildRunOpsWriterClient({
url,
clientType,
useDriverAdapter: env.RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER === "1",
})
)
),
// Brand the run-ops replica (only built for a real replica URL) so routed replica reads stay
// off the primary. When no replica URL is set, selectRunOpsTopology reuses the writer here —
@@ -315,7 +324,14 @@ const runOpsTopology: RunOpsTopology = singleton("runOpsTopology", () => {
buildNewReplica: (url, clientType) =>
markReadReplicaClient(
captureInfraErrorsRunOps(
tagDatasourceRunOps("run-ops-replica", buildRunOpsReplicaClient({ url, clientType }))
tagDatasourceRunOps(
"run-ops-replica",
buildRunOpsReplicaClient({
url,
clientType,
useDriverAdapter: env.RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER === "1",
})
)
)
),
// Legacy client shares the exact control-plane wrapper stack (the legacy DB carries the full
@@ -329,6 +345,7 @@ const runOpsTopology: RunOpsTopology = singleton("runOpsTopology", () => {
clientType,
poolTimeout: env.RUN_OPS_LEGACY_DATABASE_WRITER_POOL_TIMEOUT,
connectTimeout: env.RUN_OPS_LEGACY_DATABASE_WRITER_CONNECTION_TIMEOUT,
useDriverAdapter: env.RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER === "1",
})
)
),
@@ -342,6 +359,7 @@ const runOpsTopology: RunOpsTopology = singleton("runOpsTopology", () => {
clientType,
poolTimeout: env.RUN_OPS_LEGACY_DATABASE_READ_REPLICA_POOL_TIMEOUT,
connectTimeout: env.RUN_OPS_LEGACY_DATABASE_READ_REPLICA_CONNECTION_TIMEOUT,
useDriverAdapter: env.RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER === "1",
})
)
)
@@ -415,9 +433,40 @@ function getClient() {
clientType: "writer",
poolTimeout: env.DATABASE_WRITER_POOL_TIMEOUT,
connectTimeout: env.DATABASE_WRITER_CONNECTION_TIMEOUT,
useDriverAdapter: env.CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER === "1",
});
}
function buildDriverAdapterPool(
connectionString: string,
clientType: string,
poolTimeoutSeconds: number,
connectionLimit: number
): PrismaPg {
const pool = new Pool({
connectionString,
max: connectionLimit,
connectionTimeoutMillis: poolTimeoutSeconds * 1000,
application_name: env.SERVICE_NAME,
});
pool.on("error", (error) => {
logger.error("prisma driver adapter pool error", {
clientType,
error: error instanceof Error ? error.message : String(error),
ignoreError: true,
});
});
let schema: string | undefined;
try {
schema = new URL(connectionString).searchParams.get("schema") ?? undefined;
} catch {
schema = undefined;
}
return new PrismaPg(pool, { schema, disposeExternalPool: true });
}
// Generalized writer builder shared by the control-plane client and the run-ops
// clients. Returns a RAW, untagged, un-wrapped PrismaClient — the
// caller applies tagDatasource + captureInfrastructureErrors.
@@ -426,11 +475,13 @@ export function buildWriterClient({
clientType,
poolTimeout,
connectTimeout,
useDriverAdapter = false,
}: {
url: string;
clientType: string;
poolTimeout?: number;
connectTimeout?: number;
useDriverAdapter?: boolean;
}): PrismaClient {
const databaseUrl = buildPrismaConnectionUrl(url, {
connectionLimit: env.DATABASE_CONNECTION_LIMIT.toString(),
@@ -439,66 +490,78 @@ export function buildWriterClient({
applicationName: env.SERVICE_NAME,
});
console.log(`🔌 setting up prisma client to ${redactUrlSecrets(databaseUrl)}`);
console.log(
`🔌 setting up prisma client to ${redactUrlSecrets(databaseUrl)}${
useDriverAdapter ? " (pg driver adapter)" : ""
}`
);
const client = new PrismaClient({
datasources: {
db: {
url: databaseUrl.href,
},
const logConfig = [
// events
{
emit: "event",
level: "error",
},
log: [
// events
{
emit: "event",
level: "error",
},
{
emit: "event",
level: "info",
},
{
emit: "event",
level: "warn",
},
// stdout
...((process.env.PRISMA_LOG_TO_STDOUT === "1"
? [
{
emit: "stdout",
level: "error",
},
{
emit: "stdout",
level: "info",
},
{
emit: "stdout",
level: "warn",
},
]
: []) satisfies Prisma.LogDefinition[]),
// Query performance monitoring
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
? [
{
emit: "event",
level: "query",
},
]
: []) satisfies Prisma.LogDefinition[]),
// verbose
...((process.env.VERBOSE_PRISMA_LOGS === "1"
? [
{
emit: "stdout",
level: "query",
},
]
: []) satisfies Prisma.LogDefinition[]),
],
});
{
emit: "event",
level: "info",
},
{
emit: "event",
level: "warn",
},
// stdout
...((process.env.PRISMA_LOG_TO_STDOUT === "1"
? [
{
emit: "stdout",
level: "error",
},
{
emit: "stdout",
level: "info",
},
{
emit: "stdout",
level: "warn",
},
]
: []) satisfies Prisma.LogDefinition[]),
// Query performance monitoring
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
? [
{
emit: "event",
level: "query",
},
]
: []) satisfies Prisma.LogDefinition[]),
// verbose
...((process.env.VERBOSE_PRISMA_LOGS === "1"
? [
{
emit: "stdout",
level: "query",
},
]
: []) satisfies Prisma.LogDefinition[]),
] satisfies Prisma.LogDefinition[];
const client = useDriverAdapter
? new PrismaClient({
adapter: buildDriverAdapterPool(
url,
clientType,
poolTimeout ?? env.DATABASE_POOL_TIMEOUT,
env.DATABASE_CONNECTION_LIMIT
),
log: logConfig,
})
: new PrismaClient({
datasources: { db: { url: databaseUrl.href } },
log: logConfig,
});
// Only use structured logging if we're not already logging to stdout
if (process.env.PRISMA_LOG_TO_STDOUT !== "1") {
@@ -571,6 +634,7 @@ function getReplicaClient() {
clientType: "reader",
poolTimeout: env.DATABASE_READ_REPLICA_POOL_TIMEOUT,
connectTimeout: env.DATABASE_READ_REPLICA_CONNECTION_TIMEOUT,
useDriverAdapter: env.CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER === "1",
});
}
@@ -582,11 +646,13 @@ export function buildReplicaClient({
clientType,
poolTimeout,
connectTimeout,
useDriverAdapter = false,
}: {
url: string;
clientType: string;
poolTimeout?: number;
connectTimeout?: number;
useDriverAdapter?: boolean;
}): PrismaClient {
const replicaUrl = buildPrismaConnectionUrl(url, {
connectionLimit: env.DATABASE_CONNECTION_LIMIT.toString(),
@@ -595,66 +661,78 @@ export function buildReplicaClient({
applicationName: env.SERVICE_NAME,
});
console.log(`🔌 setting up read replica connection to ${redactUrlSecrets(replicaUrl)}`);
console.log(
`🔌 setting up read replica connection to ${redactUrlSecrets(replicaUrl)}${
useDriverAdapter ? " (pg driver adapter)" : ""
}`
);
const replicaClient = new PrismaClient({
datasources: {
db: {
url: replicaUrl.href,
},
const logConfig = [
// events
{
emit: "event",
level: "error",
},
log: [
// events
{
emit: "event",
level: "error",
},
{
emit: "event",
level: "info",
},
{
emit: "event",
level: "warn",
},
// stdout
...((process.env.PRISMA_LOG_TO_STDOUT === "1"
? [
{
emit: "stdout",
level: "error",
},
{
emit: "stdout",
level: "info",
},
{
emit: "stdout",
level: "warn",
},
]
: []) satisfies Prisma.LogDefinition[]),
// Query performance monitoring
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
? [
{
emit: "event",
level: "query",
},
]
: []) satisfies Prisma.LogDefinition[]),
// verbose
...((process.env.VERBOSE_PRISMA_LOGS === "1"
? [
{
emit: "stdout",
level: "query",
},
]
: []) satisfies Prisma.LogDefinition[]),
],
});
{
emit: "event",
level: "info",
},
{
emit: "event",
level: "warn",
},
// stdout
...((process.env.PRISMA_LOG_TO_STDOUT === "1"
? [
{
emit: "stdout",
level: "error",
},
{
emit: "stdout",
level: "info",
},
{
emit: "stdout",
level: "warn",
},
]
: []) satisfies Prisma.LogDefinition[]),
// Query performance monitoring
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
? [
{
emit: "event",
level: "query",
},
]
: []) satisfies Prisma.LogDefinition[]),
// verbose
...((process.env.VERBOSE_PRISMA_LOGS === "1"
? [
{
emit: "stdout",
level: "query",
},
]
: []) satisfies Prisma.LogDefinition[]),
] satisfies Prisma.LogDefinition[];
const replicaClient = useDriverAdapter
? new PrismaClient({
adapter: buildDriverAdapterPool(
url,
clientType,
poolTimeout ?? env.DATABASE_POOL_TIMEOUT,
env.DATABASE_CONNECTION_LIMIT
),
log: logConfig,
})
: new PrismaClient({
datasources: { db: { url: replicaUrl.href } },
log: logConfig,
});
// Only use structured logging if we're not already logging to stdout
if (process.env.PRISMA_LOG_TO_STDOUT !== "1") {
@@ -714,9 +792,11 @@ export function buildReplicaClient({
function buildRunOpsWriterClient({
url,
clientType,
useDriverAdapter = false,
}: {
url: string;
clientType: string;
useDriverAdapter?: boolean;
}): RunOpsPrismaClient {
const databaseUrl = buildPrismaConnectionUrl(url, {
connectionLimit: env.DATABASE_CONNECTION_LIMIT.toString(),
@@ -727,20 +807,42 @@ function buildRunOpsWriterClient({
applicationName: env.SERVICE_NAME,
});
console.log(`🔌 setting up run-ops prisma client to ${redactUrlSecrets(databaseUrl)}`);
console.log(
`🔌 setting up run-ops prisma client to ${redactUrlSecrets(databaseUrl)}${
useDriverAdapter ? " (pg driver adapter)" : ""
}`
);
const client = new RunOpsPrismaClient({
datasources: { db: { url: databaseUrl.href } },
log: [
{ emit: "event", level: "error" },
{ emit: "event", level: "info" },
{ emit: "event", level: "warn" },
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
? [{ emit: "event", level: "query" }]
: []) as { emit: "event"; level: "query" }[]),
],
});
const client = useDriverAdapter
? new RunOpsPrismaClient({
adapter: buildDriverAdapterPool(
url,
clientType,
env.RUN_OPS_DATABASE_WRITER_POOL_TIMEOUT ?? env.DATABASE_POOL_TIMEOUT,
env.DATABASE_CONNECTION_LIMIT
),
log: [
{ emit: "event", level: "error" },
{ emit: "event", level: "info" },
{ emit: "event", level: "warn" },
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
? [{ emit: "event", level: "query" }]
: []) as { emit: "event"; level: "query" }[]),
],
})
: new RunOpsPrismaClient({
datasources: { db: { url: databaseUrl.href } },
log: [
{ emit: "event", level: "error" },
{ emit: "event", level: "info" },
{ emit: "event", level: "warn" },
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
? [{ emit: "event", level: "query" }]
: []) as { emit: "event"; level: "query" }[]),
],
});
if (process.env.PRISMA_LOG_TO_STDOUT !== "1") {
client.$on("info", (log) => logger.info("RunOpsPrismaClient info", { clientType, event: log }));
@@ -767,9 +869,11 @@ function buildRunOpsWriterClient({
function buildRunOpsReplicaClient({
url,
clientType,
useDriverAdapter = false,
}: {
url: string;
clientType: string;
useDriverAdapter?: boolean;
}): RunOpsPrismaClient {
const replicaUrl = buildPrismaConnectionUrl(url, {
connectionLimit: (
@@ -784,20 +888,42 @@ function buildRunOpsReplicaClient({
applicationName: env.SERVICE_NAME,
});
console.log(`🔌 setting up run-ops read replica connection to ${redactUrlSecrets(replicaUrl)}`);
console.log(
`🔌 setting up run-ops read replica connection to ${redactUrlSecrets(replicaUrl)}${
useDriverAdapter ? " (pg driver adapter)" : ""
}`
);
const client = new RunOpsPrismaClient({
datasources: { db: { url: replicaUrl.href } },
log: [
{ emit: "event", level: "error" },
{ emit: "event", level: "info" },
{ emit: "event", level: "warn" },
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
? [{ emit: "event", level: "query" }]
: []) as { emit: "event"; level: "query" }[]),
],
});
const client = useDriverAdapter
? new RunOpsPrismaClient({
adapter: buildDriverAdapterPool(
url,
clientType,
env.RUN_OPS_DATABASE_READ_REPLICA_POOL_TIMEOUT ?? env.DATABASE_POOL_TIMEOUT,
env.RUN_OPS_DATABASE_READ_REPLICA_CONNECTION_LIMIT ?? env.DATABASE_CONNECTION_LIMIT
),
log: [
{ emit: "event", level: "error" },
{ emit: "event", level: "info" },
{ emit: "event", level: "warn" },
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
? [{ emit: "event", level: "query" }]
: []) as { emit: "event"; level: "query" }[]),
],
})
: new RunOpsPrismaClient({
datasources: { db: { url: replicaUrl.href } },
log: [
{ emit: "event", level: "error" },
{ emit: "event", level: "info" },
{ emit: "event", level: "warn" },
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
? [{ emit: "event", level: "query" }]
: []) as { emit: "event"; level: "query" }[]),
],
});
if (process.env.PRISMA_LOG_TO_STDOUT !== "1") {
client.$on("info", (log) => logger.info("RunOpsPrismaClient info", { clientType, event: log }));
+6
View File
@@ -238,6 +238,12 @@ const EnvironmentSchema = z
)
.optional(),
CONTROL_PLANE_DATABASE_READ_REPLICA_URL: z.string().optional(),
CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER: z.string().default("0"),
CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER: z.string().default("0"),
RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER: z.string().default("0"),
RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER: z.string().default("0"),
RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER: z.string().default("0"),
RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER: z.string().default("0"),
// Control-plane cache relax knobs. Unset -> defaults (DEFAULT_CP_CACHE_TTL_MS / _MAX_ENTRIES).
CONTROL_PLANE_CACHE_TTL_MS: z.coerce.number().int().optional(),
CONTROL_PLANE_CACHE_MAX_ENTRIES: z.coerce.number().int().optional(),
+6 -1
View File
@@ -14,7 +14,12 @@ export async function loader({ request }: LoaderFunctionArgs) {
}
// We need to remove empty lines from the prisma metrics, grafana doesn't like them
const prismaMetrics = (await prisma.$metrics.prometheus()).replace(/^\s*[\r\n]/gm, "");
let prismaMetrics = "";
try {
prismaMetrics = (await prisma.$metrics.prometheus()).replace(/^\s*[\r\n]/gm, "");
} catch {
prismaMetrics = "";
}
const coreMetrics = await metricsRegister.metrics();
// Order matters, core metrics end with `# EOF`, prisma metrics don't
+24 -2
View File
@@ -27,6 +27,25 @@ const INFRASTRUCTURE_PRISMA_CODES = new Set([
* (which both scrubs the message and is retryable by the SDK) instead of
* folding `.message` into a client-facing error.
*/
const CONNECTIVITY_ERRNO = new Set([
"ECONNREFUSED",
"ENOTFOUND",
"ETIMEDOUT",
"ECONNRESET",
"EHOSTUNREACH",
"EPIPE",
]);
const CONNECTIVITY_MESSAGE =
/ECONNREFUSED|ENOTFOUND|ETIMEDOUT|ECONNRESET|EHOSTUNREACH|database not reachable|can't reach database|connection terminated|server has closed the connection|timed out fetching a new connection/i;
export function looksLikeConnectivityError(error: unknown): boolean {
const e = error as { code?: unknown; message?: unknown };
if (typeof e?.code === "string" && CONNECTIVITY_ERRNO.has(e.code)) {
return true;
}
return typeof e?.message === "string" && CONNECTIVITY_MESSAGE.test(e.message);
}
export function isInfrastructureError(error: unknown): boolean {
if (
error instanceof Prisma.PrismaClientInitializationError ||
@@ -37,10 +56,13 @@ export function isInfrastructureError(error: unknown): boolean {
}
if (error instanceof Prisma.PrismaClientKnownRequestError) {
return INFRASTRUCTURE_PRISMA_CODES.has(error.code);
if (INFRASTRUCTURE_PRISMA_CODES.has(error.code)) {
return true;
}
return error.code === "P2010" && looksLikeConnectivityError(error);
}
return false;
return looksLikeConnectivityError(error);
}
// One-shot marker so a single infra error is logged exactly once: the client
@@ -7,6 +7,7 @@ import type {
MollifierDrainerTerminalFailureHandler,
} from "@trigger.dev/redis-worker";
import { logger } from "~/services/logger.server";
import { looksLikeConnectivityError } from "~/utils/prismaErrors";
import { recordRunDebugLog } from "~/v3/eventRepository/index.server";
import { PerformTaskRunAlertsService } from "~/v3/services/alerts/performTaskRunAlerts.server";
import { startSpan } from "~/v3/tracing.server";
@@ -29,6 +30,7 @@ export function isRetryablePgError(err: unknown): boolean {
if (msg.includes("Can't reach database server")) return true;
if (msg.includes("Connection lost")) return true;
if (msg.includes("ECONNRESET")) return true;
if (looksLikeConnectivityError(err)) return true;
return false;
}
+7 -1
View File
@@ -551,7 +551,13 @@ function configurePrismaMetrics({ meter }: { meter: Meter }) {
meter.addBatchObservableCallback(
async (res) => {
const { counters, gauges, histograms } = await readPrismaMetrics();
let prismaMetrics: Awaited<ReturnType<typeof readPrismaMetrics>>;
try {
prismaMetrics = await readPrismaMetrics();
} catch {
return;
}
const { counters, gauges, histograms } = prismaMetrics;
// Observe counters
res.observe(queriesTotal, counters.queriesTotal);
+2
View File
@@ -175,6 +175,7 @@
"p-retry": "^4.6.1",
"parse-duration": "^2.1.0",
"pg": "8.15.6",
"@prisma/adapter-pg": "6.14.0",
"posthog-js": "^1.93.3",
"posthog-node": "5.35.6",
"prism-react-renderer": "^2.3.1",
@@ -243,6 +244,7 @@
"@types/marked": "^4.0.3",
"@types/morgan": "^1.9.3",
"@types/node-fetch": "^2.6.2",
"@types/pg": "^8.11.10",
"@types/prismjs": "^1.26.0",
"@types/qs": "^6.9.7",
"@types/react": "18.2.69",
@@ -8,7 +8,7 @@ generator client {
provider = "prisma-client-js"
output = "../generated/prisma"
binaryTargets = ["native", "debian-openssl-1.1.x"]
previewFeatures = ["metrics"]
previewFeatures = ["metrics", "driverAdapters"]
}
model User {
+14 -11
View File
@@ -37,12 +37,15 @@ export function isPrismaKnownError(error: unknown): error is PrismaClientKnownRe
*/
const retryCodes = ["P2024", "P2028", "P2034"];
const ADAPTER_ACQUIRE_TIMEOUT = /timeout exceeded when trying to connect/i;
export function isPrismaRetriableError(error: unknown): boolean {
if (!isPrismaKnownError(error)) {
return false;
if (isPrismaKnownError(error) && retryCodes.includes(error.code)) {
return true;
}
return retryCodes.includes(error.code);
const message = (error as { message?: unknown })?.message;
return typeof message === "string" && ADAPTER_ACQUIRE_TIMEOUT.test(message);
}
/*
@@ -90,15 +93,15 @@ export async function $transaction<R>(
try {
return await (prisma as PrismaClient).$transaction(fn, options);
} catch (error) {
if (isPrismaKnownError(error)) {
if (
retryCodes.includes(error.code) &&
typeof options?.maxRetries === "number" &&
attempt < options.maxRetries
) {
return $transaction(prisma, fn, prismaError, options, attempt + 1);
}
if (
isPrismaRetriableError(error) &&
typeof options?.maxRetries === "number" &&
attempt < options.maxRetries
) {
return $transaction(prisma, fn, prismaError, options, attempt + 1);
}
if (isPrismaKnownError(error)) {
prismaError(error);
if (options?.swallowPrismaErrors) {
@@ -7,7 +7,7 @@ generator client {
provider = "prisma-client-js"
output = "../generated/run-ops"
binaryTargets = ["native", "debian-openssl-1.1.x"]
previewFeatures = ["metrics"]
previewFeatures = ["metrics", "driverAdapters"]
}
// ─────────────────────────────────────────────────────────────────────────────
+25 -6
View File
@@ -391,6 +391,9 @@ importers:
'@popperjs/core':
specifier: ^2.11.8
version: 2.11.8
'@prisma/adapter-pg':
specifier: 6.14.0
version: 6.14.0
'@prisma/instrumentation':
specifier: ^6.14.0
version: 6.14.0(@opentelemetry/api@1.9.1)
@@ -854,6 +857,9 @@ importers:
'@types/node-fetch':
specifier: ^2.6.2
version: 2.6.2
'@types/pg':
specifier: ^8.11.10
version: 8.11.14
'@types/prismjs':
specifier: ^1.26.0
version: 1.26.0
@@ -5679,6 +5685,9 @@ packages:
'@posthog/types@1.376.4':
resolution: {integrity: sha512-EoDEvA925lf6yxPpbP4wozlXgu4b9WEqxZlFBUDd4k2akP5R/RWyHpvQT8aYyfY6BtSLn8TnVwxPQOM4b90isA==}
'@prisma/adapter-pg@6.14.0':
resolution: {integrity: sha512-heUCNPZ3f2Iv/JId280HCoN7NECWkYckC3K1cOKpTRRjiJv0mN0w99V91vBq7aHHTtlOVpfDPQbUMNAjMegIWg==}
'@prisma/client@6.14.0':
resolution: {integrity: sha512-8E/Nk3eL5g7RQIg/LUj1ICyDmhD053STjxrPxUtCRybs2s/2sOEcx9NpITuAOPn07HEpWBfhAVe1T/HYWXUPOw==}
engines: {node: '>=18.18'}
@@ -5700,6 +5709,9 @@ packages:
'@prisma/debug@6.14.0':
resolution: {integrity: sha512-j4Lf+y+5QIJgQD4sJWSbkOD7geKx9CakaLp/TyTy/UDu9Wo0awvWCBH/BAxTHUaCpIl9USA5VS/KJhDqKJSwug==}
'@prisma/driver-adapter-utils@6.14.0':
resolution: {integrity: sha512-On9vTNiJ7J/O1kVqedLtfdhhrfRYprkUOhxjlmmWEv12WNdG6v5x4PsrfZXdBtZqRZaqK1i1TigO6IdtYh8z+A==}
'@prisma/engines-version@6.14.0-25.717184b7b35ea05dfa71a3236b7af656013e1e49':
resolution: {integrity: sha512-EgN9ODJpiX45yvwcngoStp3uQPJ3l+AEVoQ6dMMO2QvmwIlnxfApzKmJQExzdo7/hqQANrz5txHJdGYHzOnGHA==}
@@ -13324,9 +13336,6 @@ packages:
pg-protocol@1.10.3:
resolution: {integrity: sha512-6DIBgBQaTKDJyxnXaLiLR8wBpQQcGWuAESkRBX/t6OwA8YsqP+iVSiond2EDy6Y/dsGk8rh/jtax3js5NeV7JQ==}
pg-protocol@1.9.5:
resolution: {integrity: sha512-DYTWtWpfd5FOro3UnAfwvhD8jh59r2ig8bPtc9H8Ds7MscE/9NYruUQWFAOuraRl29jwcT2kyMFQ3MxeaVjUhg==}
pg-types@2.2.0:
resolution: {integrity: sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==}
engines: {node: '>=4'}
@@ -20315,6 +20324,14 @@ snapshots:
'@posthog/types@1.376.4': {}
'@prisma/adapter-pg@6.14.0':
dependencies:
'@prisma/driver-adapter-utils': 6.14.0
pg: 8.15.6
postgres-array: 3.0.4
transitivePeerDependencies:
- pg-native
'@prisma/client@6.14.0(prisma@6.14.0(magicast@0.3.5)(typescript@7.0.2))(typescript@7.0.2)':
optionalDependencies:
prisma: 6.14.0(magicast@0.3.5)(typescript@7.0.2)
@@ -20340,6 +20357,10 @@ snapshots:
'@prisma/debug@6.14.0': {}
'@prisma/driver-adapter-utils@6.14.0':
dependencies:
'@prisma/debug': 6.14.0
'@prisma/engines-version@6.14.0-25.717184b7b35ea05dfa71a3236b7af656013e1e49': {}
'@prisma/engines@6.14.0':
@@ -23169,7 +23190,7 @@ snapshots:
'@types/pg@8.11.14':
dependencies:
'@types/node': 24.13.3
pg-protocol: 1.9.5
pg-protocol: 1.10.3
pg-types: 4.0.2
'@types/pg@8.6.1':
@@ -29209,8 +29230,6 @@ snapshots:
pg-protocol@1.10.3: {}
pg-protocol@1.9.5: {}
pg-types@2.2.0:
dependencies:
pg-int8: 1.0.1