feat(webapp,database): opt-in per-client Prisma driver adapters (#4539)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
📚 Docs Checks / check-broken-links (push) Has been cancelled
🧭 Helm Chart Prerelease / lint-and-test (push) Has been cancelled
Workflow Checks / Actionlint (push) Has been cancelled
Workflow Checks / Zizmor (push) Has been cancelled
🧭 Helm Chart Prerelease / prerelease (push) Has been cancelled
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
📚 Docs Checks / check-broken-links (push) Has been cancelled
🧭 Helm Chart Prerelease / lint-and-test (push) Has been cancelled
Workflow Checks / Actionlint (push) Has been cancelled
Workflow Checks / Zizmor (push) Has been cancelled
🧭 Helm Chart Prerelease / prerelease (push) Has been cancelled
## What
Adds an opt-in path to run each Prisma client through
**`@prisma/adapter-pg`** (the node-postgres driver) instead of the
built-in engine driver, controlled by a **per-client env var, all off by
default**:
| env var | client |
|---|---|
| `CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER` | control-plane writer
|
| `CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER` | control-plane
replica |
| `RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER` | new run-ops writer |
| `RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER` | new run-ops replica |
| `RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER` | legacy run-ops
writer |
| `RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER` | legacy run-ops
replica |
With every flag unset the construction path is byte-identical to today
(`datasources` URL + Rust engine), so this is inert until a flag is
turned on. Per-client granularity allows enabling the adapter only where
it's wanted.
## How
- Enables the `driverAdapters` preview feature on both schemas
(`@trigger.dev/database` and `@internal/run-ops-database`). This keeps
the **Rust query engine** — it does NOT add `queryCompiler` — so query
behavior, result types, and engine tracing spans are unchanged.
- A shared `buildDriverAdapterPool` builds each client's `pg.Pool` with
an explicit `max`, a bounded `connectionTimeoutMillis` (the
node-postgres pool otherwise waits unbounded on acquire), and an
`onPoolError` handler (an unhandled idle-connection error would
otherwise crash the process). Threaded through all four client builders
via a `useDriverAdapter` flag.
- Adds `@prisma/adapter-pg` + `@types/pg` to the webapp; `pg` is already
pinned at `8.15.6` (adapter-pg 6.x requires `pg < 8.17`).
## Connect-failure handling (the important correctness/security bit)
Under the adapter an unreachable DB no longer surfaces as
`PrismaClientInitializationError` / `P1001`; it becomes a `P2010`
"Database not reachable: <host>" (or a raw
`ECONNREFUSED`/`ENOTFOUND`-class error). Two handlers are updated so a
client on the adapter behaves like today:
- **`isInfrastructureError`** now recognizes those shapes (P2010 with a
connectivity message, and raw connectivity errno codes). Without this,
the DB **hostname would leak into API-client-facing errors** and the
failure would go unlogged. Security-relevant.
- **`isPrismaRetriableError`** treats the adapter's pool-acquire timeout
("timeout exceeded when trying to connect") as retriable, preserving the
`P2024` retry behavior the adapter otherwise drops.
## Evidence
Validated on an isolated stack that mirrors the production DB topology
(chained PgBouncers in front of writer + reader):
- **Behavioral parity:** raw-query results and Prisma error codes/`meta`
are byte-identical between the engine driver and the adapter across the
queried shapes (unique-constraint `meta.target`, record-not-found,
transaction-timeout, serialization-failure, etc.).
- **Feature matrix:** a full 380-project queue-ay pass shows no
adapter-caused regressions — pass/fail parity between adapter-off and
adapter-on, with the residual failures being pre-existing
known-failures/flakes common to both.
## Rollout / rollback
All flags default off; enable per client via env var, roll back by
unsetting and redeploying (no data migration). Recommended first target
is a single writer; enable one client at a time.
## Follow-ups (not in this PR)
- `$metrics`-based pool observability is removed under the adapter (the
Prometheus route + `db.pool.connections.*` instruments); the metrics
replacement (via `pg.Pool` counters) lands in a separate PR.
- Note for operators: on the adapter path, interactive-transaction
`maxWait` does not bound pool acquisition — `connectionTimeoutMillis`
does.
## Note on connection-string parameters
The adapter pool is built from the base DSN, so Prisma-specific DSN
parameters that node-postgres does not understand are not honored when a
client is on the adapter:
- **Prisma TLS spellings** (`sslaccept`, `sslcert`, etc.) —
node-postgres uses `sslmode`/`ssl` instead. Our production DSNs do not
use these Prisma-specific TLS params, but any deployment whose DSN
relies on them must be checked before enabling a flag.
- `pgbouncer=true` and `statement_cache_size` — effectively moot under
the adapter, which uses no persistent named prepared statements.
`connection_limit`, `pool_timeout`, and `schema` are handled explicitly
(passed as `max`/`connectionTimeoutMillis` and PrismaPg's `{schema}`
option).
refs TRI-13039
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
---
|
||||
area: webapp
|
||||
type: improvement
|
||||
---
|
||||
|
||||
Groundwork for an alternative database connection driver, gated behind configuration and disabled by default, so there is no change to default behavior.
|
||||
+270
-144
@@ -10,6 +10,8 @@ import {
|
||||
} from "@trigger.dev/database";
|
||||
import { RunOpsPrismaClient } from "@internal/run-ops-database";
|
||||
import { markReadReplicaClient } from "@internal/run-store";
|
||||
import { PrismaPg } from "@prisma/adapter-pg";
|
||||
import { Pool } from "pg";
|
||||
import invariant from "tiny-invariant";
|
||||
import { z } from "zod";
|
||||
import { env } from "./env.server";
|
||||
@@ -307,7 +309,14 @@ const runOpsTopology: RunOpsTopology = singleton("runOpsTopology", () => {
|
||||
controlPlane: { writer: prisma, replica: $replica },
|
||||
buildNewWriter: (url, clientType) =>
|
||||
captureInfraErrorsRunOps(
|
||||
tagDatasourceRunOps("run-ops-writer", buildRunOpsWriterClient({ url, clientType }))
|
||||
tagDatasourceRunOps(
|
||||
"run-ops-writer",
|
||||
buildRunOpsWriterClient({
|
||||
url,
|
||||
clientType,
|
||||
useDriverAdapter: env.RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER === "1",
|
||||
})
|
||||
)
|
||||
),
|
||||
// Brand the run-ops replica (only built for a real replica URL) so routed replica reads stay
|
||||
// off the primary. When no replica URL is set, selectRunOpsTopology reuses the writer here —
|
||||
@@ -315,7 +324,14 @@ const runOpsTopology: RunOpsTopology = singleton("runOpsTopology", () => {
|
||||
buildNewReplica: (url, clientType) =>
|
||||
markReadReplicaClient(
|
||||
captureInfraErrorsRunOps(
|
||||
tagDatasourceRunOps("run-ops-replica", buildRunOpsReplicaClient({ url, clientType }))
|
||||
tagDatasourceRunOps(
|
||||
"run-ops-replica",
|
||||
buildRunOpsReplicaClient({
|
||||
url,
|
||||
clientType,
|
||||
useDriverAdapter: env.RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER === "1",
|
||||
})
|
||||
)
|
||||
)
|
||||
),
|
||||
// Legacy client shares the exact control-plane wrapper stack (the legacy DB carries the full
|
||||
@@ -329,6 +345,7 @@ const runOpsTopology: RunOpsTopology = singleton("runOpsTopology", () => {
|
||||
clientType,
|
||||
poolTimeout: env.RUN_OPS_LEGACY_DATABASE_WRITER_POOL_TIMEOUT,
|
||||
connectTimeout: env.RUN_OPS_LEGACY_DATABASE_WRITER_CONNECTION_TIMEOUT,
|
||||
useDriverAdapter: env.RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER === "1",
|
||||
})
|
||||
)
|
||||
),
|
||||
@@ -342,6 +359,7 @@ const runOpsTopology: RunOpsTopology = singleton("runOpsTopology", () => {
|
||||
clientType,
|
||||
poolTimeout: env.RUN_OPS_LEGACY_DATABASE_READ_REPLICA_POOL_TIMEOUT,
|
||||
connectTimeout: env.RUN_OPS_LEGACY_DATABASE_READ_REPLICA_CONNECTION_TIMEOUT,
|
||||
useDriverAdapter: env.RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER === "1",
|
||||
})
|
||||
)
|
||||
)
|
||||
@@ -415,9 +433,40 @@ function getClient() {
|
||||
clientType: "writer",
|
||||
poolTimeout: env.DATABASE_WRITER_POOL_TIMEOUT,
|
||||
connectTimeout: env.DATABASE_WRITER_CONNECTION_TIMEOUT,
|
||||
useDriverAdapter: env.CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER === "1",
|
||||
});
|
||||
}
|
||||
|
||||
function buildDriverAdapterPool(
|
||||
connectionString: string,
|
||||
clientType: string,
|
||||
poolTimeoutSeconds: number,
|
||||
connectionLimit: number
|
||||
): PrismaPg {
|
||||
const pool = new Pool({
|
||||
connectionString,
|
||||
max: connectionLimit,
|
||||
connectionTimeoutMillis: poolTimeoutSeconds * 1000,
|
||||
application_name: env.SERVICE_NAME,
|
||||
});
|
||||
pool.on("error", (error) => {
|
||||
logger.error("prisma driver adapter pool error", {
|
||||
clientType,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
ignoreError: true,
|
||||
});
|
||||
});
|
||||
|
||||
let schema: string | undefined;
|
||||
try {
|
||||
schema = new URL(connectionString).searchParams.get("schema") ?? undefined;
|
||||
} catch {
|
||||
schema = undefined;
|
||||
}
|
||||
|
||||
return new PrismaPg(pool, { schema, disposeExternalPool: true });
|
||||
}
|
||||
|
||||
// Generalized writer builder shared by the control-plane client and the run-ops
|
||||
// clients. Returns a RAW, untagged, un-wrapped PrismaClient — the
|
||||
// caller applies tagDatasource + captureInfrastructureErrors.
|
||||
@@ -426,11 +475,13 @@ export function buildWriterClient({
|
||||
clientType,
|
||||
poolTimeout,
|
||||
connectTimeout,
|
||||
useDriverAdapter = false,
|
||||
}: {
|
||||
url: string;
|
||||
clientType: string;
|
||||
poolTimeout?: number;
|
||||
connectTimeout?: number;
|
||||
useDriverAdapter?: boolean;
|
||||
}): PrismaClient {
|
||||
const databaseUrl = buildPrismaConnectionUrl(url, {
|
||||
connectionLimit: env.DATABASE_CONNECTION_LIMIT.toString(),
|
||||
@@ -439,66 +490,78 @@ export function buildWriterClient({
|
||||
applicationName: env.SERVICE_NAME,
|
||||
});
|
||||
|
||||
console.log(`🔌 setting up prisma client to ${redactUrlSecrets(databaseUrl)}`);
|
||||
console.log(
|
||||
`🔌 setting up prisma client to ${redactUrlSecrets(databaseUrl)}${
|
||||
useDriverAdapter ? " (pg driver adapter)" : ""
|
||||
}`
|
||||
);
|
||||
|
||||
const client = new PrismaClient({
|
||||
datasources: {
|
||||
db: {
|
||||
url: databaseUrl.href,
|
||||
},
|
||||
const logConfig = [
|
||||
// events
|
||||
{
|
||||
emit: "event",
|
||||
level: "error",
|
||||
},
|
||||
log: [
|
||||
// events
|
||||
{
|
||||
emit: "event",
|
||||
level: "error",
|
||||
},
|
||||
{
|
||||
emit: "event",
|
||||
level: "info",
|
||||
},
|
||||
{
|
||||
emit: "event",
|
||||
level: "warn",
|
||||
},
|
||||
// stdout
|
||||
...((process.env.PRISMA_LOG_TO_STDOUT === "1"
|
||||
? [
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "error",
|
||||
},
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "info",
|
||||
},
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "warn",
|
||||
},
|
||||
]
|
||||
: []) satisfies Prisma.LogDefinition[]),
|
||||
// Query performance monitoring
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
|
||||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
|
||||
? [
|
||||
{
|
||||
emit: "event",
|
||||
level: "query",
|
||||
},
|
||||
]
|
||||
: []) satisfies Prisma.LogDefinition[]),
|
||||
// verbose
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1"
|
||||
? [
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "query",
|
||||
},
|
||||
]
|
||||
: []) satisfies Prisma.LogDefinition[]),
|
||||
],
|
||||
});
|
||||
{
|
||||
emit: "event",
|
||||
level: "info",
|
||||
},
|
||||
{
|
||||
emit: "event",
|
||||
level: "warn",
|
||||
},
|
||||
// stdout
|
||||
...((process.env.PRISMA_LOG_TO_STDOUT === "1"
|
||||
? [
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "error",
|
||||
},
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "info",
|
||||
},
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "warn",
|
||||
},
|
||||
]
|
||||
: []) satisfies Prisma.LogDefinition[]),
|
||||
// Query performance monitoring
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
|
||||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
|
||||
? [
|
||||
{
|
||||
emit: "event",
|
||||
level: "query",
|
||||
},
|
||||
]
|
||||
: []) satisfies Prisma.LogDefinition[]),
|
||||
// verbose
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1"
|
||||
? [
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "query",
|
||||
},
|
||||
]
|
||||
: []) satisfies Prisma.LogDefinition[]),
|
||||
] satisfies Prisma.LogDefinition[];
|
||||
|
||||
const client = useDriverAdapter
|
||||
? new PrismaClient({
|
||||
adapter: buildDriverAdapterPool(
|
||||
url,
|
||||
clientType,
|
||||
poolTimeout ?? env.DATABASE_POOL_TIMEOUT,
|
||||
env.DATABASE_CONNECTION_LIMIT
|
||||
),
|
||||
log: logConfig,
|
||||
})
|
||||
: new PrismaClient({
|
||||
datasources: { db: { url: databaseUrl.href } },
|
||||
log: logConfig,
|
||||
});
|
||||
|
||||
// Only use structured logging if we're not already logging to stdout
|
||||
if (process.env.PRISMA_LOG_TO_STDOUT !== "1") {
|
||||
@@ -571,6 +634,7 @@ function getReplicaClient() {
|
||||
clientType: "reader",
|
||||
poolTimeout: env.DATABASE_READ_REPLICA_POOL_TIMEOUT,
|
||||
connectTimeout: env.DATABASE_READ_REPLICA_CONNECTION_TIMEOUT,
|
||||
useDriverAdapter: env.CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER === "1",
|
||||
});
|
||||
}
|
||||
|
||||
@@ -582,11 +646,13 @@ export function buildReplicaClient({
|
||||
clientType,
|
||||
poolTimeout,
|
||||
connectTimeout,
|
||||
useDriverAdapter = false,
|
||||
}: {
|
||||
url: string;
|
||||
clientType: string;
|
||||
poolTimeout?: number;
|
||||
connectTimeout?: number;
|
||||
useDriverAdapter?: boolean;
|
||||
}): PrismaClient {
|
||||
const replicaUrl = buildPrismaConnectionUrl(url, {
|
||||
connectionLimit: env.DATABASE_CONNECTION_LIMIT.toString(),
|
||||
@@ -595,66 +661,78 @@ export function buildReplicaClient({
|
||||
applicationName: env.SERVICE_NAME,
|
||||
});
|
||||
|
||||
console.log(`🔌 setting up read replica connection to ${redactUrlSecrets(replicaUrl)}`);
|
||||
console.log(
|
||||
`🔌 setting up read replica connection to ${redactUrlSecrets(replicaUrl)}${
|
||||
useDriverAdapter ? " (pg driver adapter)" : ""
|
||||
}`
|
||||
);
|
||||
|
||||
const replicaClient = new PrismaClient({
|
||||
datasources: {
|
||||
db: {
|
||||
url: replicaUrl.href,
|
||||
},
|
||||
const logConfig = [
|
||||
// events
|
||||
{
|
||||
emit: "event",
|
||||
level: "error",
|
||||
},
|
||||
log: [
|
||||
// events
|
||||
{
|
||||
emit: "event",
|
||||
level: "error",
|
||||
},
|
||||
{
|
||||
emit: "event",
|
||||
level: "info",
|
||||
},
|
||||
{
|
||||
emit: "event",
|
||||
level: "warn",
|
||||
},
|
||||
// stdout
|
||||
...((process.env.PRISMA_LOG_TO_STDOUT === "1"
|
||||
? [
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "error",
|
||||
},
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "info",
|
||||
},
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "warn",
|
||||
},
|
||||
]
|
||||
: []) satisfies Prisma.LogDefinition[]),
|
||||
// Query performance monitoring
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
|
||||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
|
||||
? [
|
||||
{
|
||||
emit: "event",
|
||||
level: "query",
|
||||
},
|
||||
]
|
||||
: []) satisfies Prisma.LogDefinition[]),
|
||||
// verbose
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1"
|
||||
? [
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "query",
|
||||
},
|
||||
]
|
||||
: []) satisfies Prisma.LogDefinition[]),
|
||||
],
|
||||
});
|
||||
{
|
||||
emit: "event",
|
||||
level: "info",
|
||||
},
|
||||
{
|
||||
emit: "event",
|
||||
level: "warn",
|
||||
},
|
||||
// stdout
|
||||
...((process.env.PRISMA_LOG_TO_STDOUT === "1"
|
||||
? [
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "error",
|
||||
},
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "info",
|
||||
},
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "warn",
|
||||
},
|
||||
]
|
||||
: []) satisfies Prisma.LogDefinition[]),
|
||||
// Query performance monitoring
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
|
||||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
|
||||
? [
|
||||
{
|
||||
emit: "event",
|
||||
level: "query",
|
||||
},
|
||||
]
|
||||
: []) satisfies Prisma.LogDefinition[]),
|
||||
// verbose
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1"
|
||||
? [
|
||||
{
|
||||
emit: "stdout",
|
||||
level: "query",
|
||||
},
|
||||
]
|
||||
: []) satisfies Prisma.LogDefinition[]),
|
||||
] satisfies Prisma.LogDefinition[];
|
||||
|
||||
const replicaClient = useDriverAdapter
|
||||
? new PrismaClient({
|
||||
adapter: buildDriverAdapterPool(
|
||||
url,
|
||||
clientType,
|
||||
poolTimeout ?? env.DATABASE_POOL_TIMEOUT,
|
||||
env.DATABASE_CONNECTION_LIMIT
|
||||
),
|
||||
log: logConfig,
|
||||
})
|
||||
: new PrismaClient({
|
||||
datasources: { db: { url: replicaUrl.href } },
|
||||
log: logConfig,
|
||||
});
|
||||
|
||||
// Only use structured logging if we're not already logging to stdout
|
||||
if (process.env.PRISMA_LOG_TO_STDOUT !== "1") {
|
||||
@@ -714,9 +792,11 @@ export function buildReplicaClient({
|
||||
function buildRunOpsWriterClient({
|
||||
url,
|
||||
clientType,
|
||||
useDriverAdapter = false,
|
||||
}: {
|
||||
url: string;
|
||||
clientType: string;
|
||||
useDriverAdapter?: boolean;
|
||||
}): RunOpsPrismaClient {
|
||||
const databaseUrl = buildPrismaConnectionUrl(url, {
|
||||
connectionLimit: env.DATABASE_CONNECTION_LIMIT.toString(),
|
||||
@@ -727,20 +807,42 @@ function buildRunOpsWriterClient({
|
||||
applicationName: env.SERVICE_NAME,
|
||||
});
|
||||
|
||||
console.log(`🔌 setting up run-ops prisma client to ${redactUrlSecrets(databaseUrl)}`);
|
||||
console.log(
|
||||
`🔌 setting up run-ops prisma client to ${redactUrlSecrets(databaseUrl)}${
|
||||
useDriverAdapter ? " (pg driver adapter)" : ""
|
||||
}`
|
||||
);
|
||||
|
||||
const client = new RunOpsPrismaClient({
|
||||
datasources: { db: { url: databaseUrl.href } },
|
||||
log: [
|
||||
{ emit: "event", level: "error" },
|
||||
{ emit: "event", level: "info" },
|
||||
{ emit: "event", level: "warn" },
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
|
||||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
|
||||
? [{ emit: "event", level: "query" }]
|
||||
: []) as { emit: "event"; level: "query" }[]),
|
||||
],
|
||||
});
|
||||
const client = useDriverAdapter
|
||||
? new RunOpsPrismaClient({
|
||||
adapter: buildDriverAdapterPool(
|
||||
url,
|
||||
clientType,
|
||||
env.RUN_OPS_DATABASE_WRITER_POOL_TIMEOUT ?? env.DATABASE_POOL_TIMEOUT,
|
||||
env.DATABASE_CONNECTION_LIMIT
|
||||
),
|
||||
log: [
|
||||
{ emit: "event", level: "error" },
|
||||
{ emit: "event", level: "info" },
|
||||
{ emit: "event", level: "warn" },
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
|
||||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
|
||||
? [{ emit: "event", level: "query" }]
|
||||
: []) as { emit: "event"; level: "query" }[]),
|
||||
],
|
||||
})
|
||||
: new RunOpsPrismaClient({
|
||||
datasources: { db: { url: databaseUrl.href } },
|
||||
log: [
|
||||
{ emit: "event", level: "error" },
|
||||
{ emit: "event", level: "info" },
|
||||
{ emit: "event", level: "warn" },
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
|
||||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
|
||||
? [{ emit: "event", level: "query" }]
|
||||
: []) as { emit: "event"; level: "query" }[]),
|
||||
],
|
||||
});
|
||||
|
||||
if (process.env.PRISMA_LOG_TO_STDOUT !== "1") {
|
||||
client.$on("info", (log) => logger.info("RunOpsPrismaClient info", { clientType, event: log }));
|
||||
@@ -767,9 +869,11 @@ function buildRunOpsWriterClient({
|
||||
function buildRunOpsReplicaClient({
|
||||
url,
|
||||
clientType,
|
||||
useDriverAdapter = false,
|
||||
}: {
|
||||
url: string;
|
||||
clientType: string;
|
||||
useDriverAdapter?: boolean;
|
||||
}): RunOpsPrismaClient {
|
||||
const replicaUrl = buildPrismaConnectionUrl(url, {
|
||||
connectionLimit: (
|
||||
@@ -784,20 +888,42 @@ function buildRunOpsReplicaClient({
|
||||
applicationName: env.SERVICE_NAME,
|
||||
});
|
||||
|
||||
console.log(`🔌 setting up run-ops read replica connection to ${redactUrlSecrets(replicaUrl)}`);
|
||||
console.log(
|
||||
`🔌 setting up run-ops read replica connection to ${redactUrlSecrets(replicaUrl)}${
|
||||
useDriverAdapter ? " (pg driver adapter)" : ""
|
||||
}`
|
||||
);
|
||||
|
||||
const client = new RunOpsPrismaClient({
|
||||
datasources: { db: { url: replicaUrl.href } },
|
||||
log: [
|
||||
{ emit: "event", level: "error" },
|
||||
{ emit: "event", level: "info" },
|
||||
{ emit: "event", level: "warn" },
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
|
||||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
|
||||
? [{ emit: "event", level: "query" }]
|
||||
: []) as { emit: "event"; level: "query" }[]),
|
||||
],
|
||||
});
|
||||
const client = useDriverAdapter
|
||||
? new RunOpsPrismaClient({
|
||||
adapter: buildDriverAdapterPool(
|
||||
url,
|
||||
clientType,
|
||||
env.RUN_OPS_DATABASE_READ_REPLICA_POOL_TIMEOUT ?? env.DATABASE_POOL_TIMEOUT,
|
||||
env.RUN_OPS_DATABASE_READ_REPLICA_CONNECTION_LIMIT ?? env.DATABASE_CONNECTION_LIMIT
|
||||
),
|
||||
log: [
|
||||
{ emit: "event", level: "error" },
|
||||
{ emit: "event", level: "info" },
|
||||
{ emit: "event", level: "warn" },
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
|
||||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
|
||||
? [{ emit: "event", level: "query" }]
|
||||
: []) as { emit: "event"; level: "query" }[]),
|
||||
],
|
||||
})
|
||||
: new RunOpsPrismaClient({
|
||||
datasources: { db: { url: replicaUrl.href } },
|
||||
log: [
|
||||
{ emit: "event", level: "error" },
|
||||
{ emit: "event", level: "info" },
|
||||
{ emit: "event", level: "warn" },
|
||||
...((process.env.VERBOSE_PRISMA_LOGS === "1" ||
|
||||
process.env.VERY_SLOW_QUERY_THRESHOLD_MS !== undefined
|
||||
? [{ emit: "event", level: "query" }]
|
||||
: []) as { emit: "event"; level: "query" }[]),
|
||||
],
|
||||
});
|
||||
|
||||
if (process.env.PRISMA_LOG_TO_STDOUT !== "1") {
|
||||
client.$on("info", (log) => logger.info("RunOpsPrismaClient info", { clientType, event: log }));
|
||||
|
||||
@@ -238,6 +238,12 @@ const EnvironmentSchema = z
|
||||
)
|
||||
.optional(),
|
||||
CONTROL_PLANE_DATABASE_READ_REPLICA_URL: z.string().optional(),
|
||||
CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER: z.string().default("0"),
|
||||
CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER: z.string().default("0"),
|
||||
RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER: z.string().default("0"),
|
||||
RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER: z.string().default("0"),
|
||||
RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER: z.string().default("0"),
|
||||
RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER: z.string().default("0"),
|
||||
// Control-plane cache relax knobs. Unset -> defaults (DEFAULT_CP_CACHE_TTL_MS / _MAX_ENTRIES).
|
||||
CONTROL_PLANE_CACHE_TTL_MS: z.coerce.number().int().optional(),
|
||||
CONTROL_PLANE_CACHE_MAX_ENTRIES: z.coerce.number().int().optional(),
|
||||
|
||||
@@ -14,7 +14,12 @@ export async function loader({ request }: LoaderFunctionArgs) {
|
||||
}
|
||||
|
||||
// We need to remove empty lines from the prisma metrics, grafana doesn't like them
|
||||
const prismaMetrics = (await prisma.$metrics.prometheus()).replace(/^\s*[\r\n]/gm, "");
|
||||
let prismaMetrics = "";
|
||||
try {
|
||||
prismaMetrics = (await prisma.$metrics.prometheus()).replace(/^\s*[\r\n]/gm, "");
|
||||
} catch {
|
||||
prismaMetrics = "";
|
||||
}
|
||||
const coreMetrics = await metricsRegister.metrics();
|
||||
|
||||
// Order matters, core metrics end with `# EOF`, prisma metrics don't
|
||||
|
||||
@@ -27,6 +27,25 @@ const INFRASTRUCTURE_PRISMA_CODES = new Set([
|
||||
* (which both scrubs the message and is retryable by the SDK) instead of
|
||||
* folding `.message` into a client-facing error.
|
||||
*/
|
||||
const CONNECTIVITY_ERRNO = new Set([
|
||||
"ECONNREFUSED",
|
||||
"ENOTFOUND",
|
||||
"ETIMEDOUT",
|
||||
"ECONNRESET",
|
||||
"EHOSTUNREACH",
|
||||
"EPIPE",
|
||||
]);
|
||||
const CONNECTIVITY_MESSAGE =
|
||||
/ECONNREFUSED|ENOTFOUND|ETIMEDOUT|ECONNRESET|EHOSTUNREACH|database not reachable|can't reach database|connection terminated|server has closed the connection|timed out fetching a new connection/i;
|
||||
|
||||
export function looksLikeConnectivityError(error: unknown): boolean {
|
||||
const e = error as { code?: unknown; message?: unknown };
|
||||
if (typeof e?.code === "string" && CONNECTIVITY_ERRNO.has(e.code)) {
|
||||
return true;
|
||||
}
|
||||
return typeof e?.message === "string" && CONNECTIVITY_MESSAGE.test(e.message);
|
||||
}
|
||||
|
||||
export function isInfrastructureError(error: unknown): boolean {
|
||||
if (
|
||||
error instanceof Prisma.PrismaClientInitializationError ||
|
||||
@@ -37,10 +56,13 @@ export function isInfrastructureError(error: unknown): boolean {
|
||||
}
|
||||
|
||||
if (error instanceof Prisma.PrismaClientKnownRequestError) {
|
||||
return INFRASTRUCTURE_PRISMA_CODES.has(error.code);
|
||||
if (INFRASTRUCTURE_PRISMA_CODES.has(error.code)) {
|
||||
return true;
|
||||
}
|
||||
return error.code === "P2010" && looksLikeConnectivityError(error);
|
||||
}
|
||||
|
||||
return false;
|
||||
return looksLikeConnectivityError(error);
|
||||
}
|
||||
|
||||
// One-shot marker so a single infra error is logged exactly once: the client
|
||||
|
||||
@@ -7,6 +7,7 @@ import type {
|
||||
MollifierDrainerTerminalFailureHandler,
|
||||
} from "@trigger.dev/redis-worker";
|
||||
import { logger } from "~/services/logger.server";
|
||||
import { looksLikeConnectivityError } from "~/utils/prismaErrors";
|
||||
import { recordRunDebugLog } from "~/v3/eventRepository/index.server";
|
||||
import { PerformTaskRunAlertsService } from "~/v3/services/alerts/performTaskRunAlerts.server";
|
||||
import { startSpan } from "~/v3/tracing.server";
|
||||
@@ -29,6 +30,7 @@ export function isRetryablePgError(err: unknown): boolean {
|
||||
if (msg.includes("Can't reach database server")) return true;
|
||||
if (msg.includes("Connection lost")) return true;
|
||||
if (msg.includes("ECONNRESET")) return true;
|
||||
if (looksLikeConnectivityError(err)) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
|
||||
@@ -551,7 +551,13 @@ function configurePrismaMetrics({ meter }: { meter: Meter }) {
|
||||
|
||||
meter.addBatchObservableCallback(
|
||||
async (res) => {
|
||||
const { counters, gauges, histograms } = await readPrismaMetrics();
|
||||
let prismaMetrics: Awaited<ReturnType<typeof readPrismaMetrics>>;
|
||||
try {
|
||||
prismaMetrics = await readPrismaMetrics();
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
const { counters, gauges, histograms } = prismaMetrics;
|
||||
|
||||
// Observe counters
|
||||
res.observe(queriesTotal, counters.queriesTotal);
|
||||
|
||||
@@ -175,6 +175,7 @@
|
||||
"p-retry": "^4.6.1",
|
||||
"parse-duration": "^2.1.0",
|
||||
"pg": "8.15.6",
|
||||
"@prisma/adapter-pg": "6.14.0",
|
||||
"posthog-js": "^1.93.3",
|
||||
"posthog-node": "5.35.6",
|
||||
"prism-react-renderer": "^2.3.1",
|
||||
@@ -243,6 +244,7 @@
|
||||
"@types/marked": "^4.0.3",
|
||||
"@types/morgan": "^1.9.3",
|
||||
"@types/node-fetch": "^2.6.2",
|
||||
"@types/pg": "^8.11.10",
|
||||
"@types/prismjs": "^1.26.0",
|
||||
"@types/qs": "^6.9.7",
|
||||
"@types/react": "18.2.69",
|
||||
|
||||
@@ -8,7 +8,7 @@ generator client {
|
||||
provider = "prisma-client-js"
|
||||
output = "../generated/prisma"
|
||||
binaryTargets = ["native", "debian-openssl-1.1.x"]
|
||||
previewFeatures = ["metrics"]
|
||||
previewFeatures = ["metrics", "driverAdapters"]
|
||||
}
|
||||
|
||||
model User {
|
||||
|
||||
@@ -37,12 +37,15 @@ export function isPrismaKnownError(error: unknown): error is PrismaClientKnownRe
|
||||
*/
|
||||
const retryCodes = ["P2024", "P2028", "P2034"];
|
||||
|
||||
const ADAPTER_ACQUIRE_TIMEOUT = /timeout exceeded when trying to connect/i;
|
||||
|
||||
export function isPrismaRetriableError(error: unknown): boolean {
|
||||
if (!isPrismaKnownError(error)) {
|
||||
return false;
|
||||
if (isPrismaKnownError(error) && retryCodes.includes(error.code)) {
|
||||
return true;
|
||||
}
|
||||
|
||||
return retryCodes.includes(error.code);
|
||||
const message = (error as { message?: unknown })?.message;
|
||||
return typeof message === "string" && ADAPTER_ACQUIRE_TIMEOUT.test(message);
|
||||
}
|
||||
|
||||
/*
|
||||
@@ -90,15 +93,15 @@ export async function $transaction<R>(
|
||||
try {
|
||||
return await (prisma as PrismaClient).$transaction(fn, options);
|
||||
} catch (error) {
|
||||
if (isPrismaKnownError(error)) {
|
||||
if (
|
||||
retryCodes.includes(error.code) &&
|
||||
typeof options?.maxRetries === "number" &&
|
||||
attempt < options.maxRetries
|
||||
) {
|
||||
return $transaction(prisma, fn, prismaError, options, attempt + 1);
|
||||
}
|
||||
if (
|
||||
isPrismaRetriableError(error) &&
|
||||
typeof options?.maxRetries === "number" &&
|
||||
attempt < options.maxRetries
|
||||
) {
|
||||
return $transaction(prisma, fn, prismaError, options, attempt + 1);
|
||||
}
|
||||
|
||||
if (isPrismaKnownError(error)) {
|
||||
prismaError(error);
|
||||
|
||||
if (options?.swallowPrismaErrors) {
|
||||
|
||||
@@ -7,7 +7,7 @@ generator client {
|
||||
provider = "prisma-client-js"
|
||||
output = "../generated/run-ops"
|
||||
binaryTargets = ["native", "debian-openssl-1.1.x"]
|
||||
previewFeatures = ["metrics"]
|
||||
previewFeatures = ["metrics", "driverAdapters"]
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
Generated
+25
-6
@@ -391,6 +391,9 @@ importers:
|
||||
'@popperjs/core':
|
||||
specifier: ^2.11.8
|
||||
version: 2.11.8
|
||||
'@prisma/adapter-pg':
|
||||
specifier: 6.14.0
|
||||
version: 6.14.0
|
||||
'@prisma/instrumentation':
|
||||
specifier: ^6.14.0
|
||||
version: 6.14.0(@opentelemetry/api@1.9.1)
|
||||
@@ -854,6 +857,9 @@ importers:
|
||||
'@types/node-fetch':
|
||||
specifier: ^2.6.2
|
||||
version: 2.6.2
|
||||
'@types/pg':
|
||||
specifier: ^8.11.10
|
||||
version: 8.11.14
|
||||
'@types/prismjs':
|
||||
specifier: ^1.26.0
|
||||
version: 1.26.0
|
||||
@@ -5679,6 +5685,9 @@ packages:
|
||||
'@posthog/types@1.376.4':
|
||||
resolution: {integrity: sha512-EoDEvA925lf6yxPpbP4wozlXgu4b9WEqxZlFBUDd4k2akP5R/RWyHpvQT8aYyfY6BtSLn8TnVwxPQOM4b90isA==}
|
||||
|
||||
'@prisma/adapter-pg@6.14.0':
|
||||
resolution: {integrity: sha512-heUCNPZ3f2Iv/JId280HCoN7NECWkYckC3K1cOKpTRRjiJv0mN0w99V91vBq7aHHTtlOVpfDPQbUMNAjMegIWg==}
|
||||
|
||||
'@prisma/client@6.14.0':
|
||||
resolution: {integrity: sha512-8E/Nk3eL5g7RQIg/LUj1ICyDmhD053STjxrPxUtCRybs2s/2sOEcx9NpITuAOPn07HEpWBfhAVe1T/HYWXUPOw==}
|
||||
engines: {node: '>=18.18'}
|
||||
@@ -5700,6 +5709,9 @@ packages:
|
||||
'@prisma/debug@6.14.0':
|
||||
resolution: {integrity: sha512-j4Lf+y+5QIJgQD4sJWSbkOD7geKx9CakaLp/TyTy/UDu9Wo0awvWCBH/BAxTHUaCpIl9USA5VS/KJhDqKJSwug==}
|
||||
|
||||
'@prisma/driver-adapter-utils@6.14.0':
|
||||
resolution: {integrity: sha512-On9vTNiJ7J/O1kVqedLtfdhhrfRYprkUOhxjlmmWEv12WNdG6v5x4PsrfZXdBtZqRZaqK1i1TigO6IdtYh8z+A==}
|
||||
|
||||
'@prisma/engines-version@6.14.0-25.717184b7b35ea05dfa71a3236b7af656013e1e49':
|
||||
resolution: {integrity: sha512-EgN9ODJpiX45yvwcngoStp3uQPJ3l+AEVoQ6dMMO2QvmwIlnxfApzKmJQExzdo7/hqQANrz5txHJdGYHzOnGHA==}
|
||||
|
||||
@@ -13324,9 +13336,6 @@ packages:
|
||||
pg-protocol@1.10.3:
|
||||
resolution: {integrity: sha512-6DIBgBQaTKDJyxnXaLiLR8wBpQQcGWuAESkRBX/t6OwA8YsqP+iVSiond2EDy6Y/dsGk8rh/jtax3js5NeV7JQ==}
|
||||
|
||||
pg-protocol@1.9.5:
|
||||
resolution: {integrity: sha512-DYTWtWpfd5FOro3UnAfwvhD8jh59r2ig8bPtc9H8Ds7MscE/9NYruUQWFAOuraRl29jwcT2kyMFQ3MxeaVjUhg==}
|
||||
|
||||
pg-types@2.2.0:
|
||||
resolution: {integrity: sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==}
|
||||
engines: {node: '>=4'}
|
||||
@@ -20315,6 +20324,14 @@ snapshots:
|
||||
|
||||
'@posthog/types@1.376.4': {}
|
||||
|
||||
'@prisma/adapter-pg@6.14.0':
|
||||
dependencies:
|
||||
'@prisma/driver-adapter-utils': 6.14.0
|
||||
pg: 8.15.6
|
||||
postgres-array: 3.0.4
|
||||
transitivePeerDependencies:
|
||||
- pg-native
|
||||
|
||||
'@prisma/client@6.14.0(prisma@6.14.0(magicast@0.3.5)(typescript@7.0.2))(typescript@7.0.2)':
|
||||
optionalDependencies:
|
||||
prisma: 6.14.0(magicast@0.3.5)(typescript@7.0.2)
|
||||
@@ -20340,6 +20357,10 @@ snapshots:
|
||||
|
||||
'@prisma/debug@6.14.0': {}
|
||||
|
||||
'@prisma/driver-adapter-utils@6.14.0':
|
||||
dependencies:
|
||||
'@prisma/debug': 6.14.0
|
||||
|
||||
'@prisma/engines-version@6.14.0-25.717184b7b35ea05dfa71a3236b7af656013e1e49': {}
|
||||
|
||||
'@prisma/engines@6.14.0':
|
||||
@@ -23169,7 +23190,7 @@ snapshots:
|
||||
'@types/pg@8.11.14':
|
||||
dependencies:
|
||||
'@types/node': 24.13.3
|
||||
pg-protocol: 1.9.5
|
||||
pg-protocol: 1.10.3
|
||||
pg-types: 4.0.2
|
||||
|
||||
'@types/pg@8.6.1':
|
||||
@@ -29209,8 +29230,6 @@ snapshots:
|
||||
|
||||
pg-protocol@1.10.3: {}
|
||||
|
||||
pg-protocol@1.9.5: {}
|
||||
|
||||
pg-types@2.2.0:
|
||||
dependencies:
|
||||
pg-int8: 1.0.1
|
||||
|
||||
Reference in New Issue
Block a user