Files
Eric Allam 951d8e8d7b feat(webapp): per-client database pool metrics that survive the driver adapter (#4541)
## What

Follow-up to #4539. The driver-adapter work is inert until a client
flips to the pg driver adapter, but the moment one does, our database
observability degrades: the OTel metrics pipeline reads pool stats from
Prisma's `$metrics`, which is owned by the Rust engine's `quaint` pool.
Under the adapter, `pg.Pool` owns the pool, so those gauges read zero.
The pipeline also only ever scraped a single client (the control-plane
writer singleton).

This PR makes database metrics driver-agnostic and per-client:

- Every configured client registers a metrics source: control-plane
writer/replica, run-ops writer/replica, legacy writer/replica.
Previously only the control-plane writer singleton was scraped.
- Each OTel instrument is observed per client with `db_client` and
`db_driver` (`quaint` | `pg-adapter`) attributes. `db_client` uses our
canonical datasource-role labels (`control-plane-writer`,
`control-plane-replica`, `run-ops-writer`, `run-ops-replica`,
`legacy-run-ops-writer`, `legacy-run-ops-replica`) — the same strings
used for the `db.datasource` span attribute, so a metric and a trace
point at the same pool.
- Pool figures come from the authoritative source per driver:
- **pg-adapter**: `pg.Pool` (`totalCount`/`idleCount`/`waitingCount`,
plus cumulative opened/closed from `connect`/`remove` events).
- **quaint**: the Rust engine's `$metrics` pool gauges/counters, exactly
as before.
- Query counters and duration histograms still come from `$metrics` for
both drivers (the Rust engine executes queries in both cases).
- New `db.pool.connections.waiting` gauge (pg.Pool exposes this; quaint
reports 0).
- Stops exporting Prisma metrics from the Prometheus `/metrics` route.
Pool observability now lives entirely in the OTel pipeline, per driver,
per client.

## Why

So we can flip any client (including the control-plane writer, the
primary desync-fix target) to the driver adapter without losing pool
visibility. Existing dashboards keyed on the same metric names keep
working; they gain a per-client dimension.

## Testing

Unit (`apps/webapp/app/utils/databaseMetrics.server.test.ts`): the pure
normalizer — quaint reads pool from `$metrics`; adapter reads pool from
`pg.Pool` and keeps engine query metrics; `busy` never goes negative;
graceful zeroing when `$metrics` is unavailable (adapter still reports
live pool figures).

Live smoke test against a prod-shaped local stack: three
physically-distinct Postgres DBs (control-plane, run-ops, legacy) behind
dual PgBouncers, split mode on, with a mix of adapter and quaint
clients. Reading the actual emitted OTel metrics, every pool shows up as
its own series:

```
db.pool.connections.total{db_client="control-plane-writer",  db_driver="pg-adapter"} = 1
db.pool.connections.total{db_client="control-plane-replica", db_driver="quaint"}     = 1
db.pool.connections.total{db_client="run-ops-writer",        db_driver="pg-adapter"} = 1
db.pool.connections.total{db_client="run-ops-replica",       db_driver="quaint"}     = 1
db.pool.connections.total{db_client="legacy-run-ops-writer", db_driver="quaint"}     = 1
db.pool.connections.total{db_client="legacy-run-ops-replica",db_driver="quaint"}     = 1
db.client.queries.total{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing
db.client.queries.duration.count{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing
```

Confirms: metrics are attributed per pool with the correct driver;
adapter pools' figures come from `pg.Pool`; and query counters/duration
histograms keep incrementing under the pg adapter. Also verified
`/metrics` (Prometheus) now returns zero `prisma_*` series while still
serving the app's own metrics.

`pnpm run typecheck --filter webapp` passes.

## Notes

- `/metrics` (Prometheus) no longer includes `prisma_*` series. Anything
scraping that endpoint for Prisma metrics should read the equivalent
`db.*` metrics from the OTel exporter instead.
- **PgBouncer + `?schema=` gotcha (separate from this PR, worth flagging
for rollout):** since #4539 parses `?schema=` from the DSN and passes `{
schema }` to the adapter, node-postgres sends `search_path` as a startup
parameter. A transaction-mode PgBouncer rejects that with `FATAL:
unsupported startup parameter: search_path`. Our prod control-plane DSNs
use the default `public` schema with no `?schema=` param, so this is
latent, but any client we flip to the adapter must not carry `?schema=`
in its DSN (or the pooler needs `ignore_startup_parameters =
search_path`).

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-10 13:54:06 +01:00

153 lines
4.5 KiB
TypeScript

import { singleton } from "./singleton";
export type MetricHistogramValue = {
buckets: [number, number][];
sum: number;
count: number;
};
type PrismaMetricsJson = {
counters: Array<{ key: string; value: number }>;
gauges: Array<{ key: string; value: number }>;
histograms: Array<{ key: string; value: MetricHistogramValue }>;
};
type MetricsCapableClient = {
$metrics: { json: () => Promise<PrismaMetricsJson> };
};
type PoolLike = {
totalCount: number;
idleCount: number;
waitingCount: number;
};
export type DatabaseMetricsSource = {
clientType: string;
usesDriverAdapter: boolean;
client: MetricsCapableClient;
pool?: PoolLike;
poolCounters?: { opened: () => number; closed: () => number };
};
export type NormalizedPoolMetrics = {
open: number;
busy: number;
idle: number;
waiting: number;
openedTotal: number;
closedTotal: number;
};
export type NormalizedDatabaseMetrics = {
clientType: string;
driver: "pg-adapter" | "quaint";
engineMetricsAvailable: boolean;
pool?: NormalizedPoolMetrics;
counters?: { queriesTotal: number; datasourceQueriesTotal: number };
gauges?: { queriesActive: number; queriesWait: number };
histograms: {
queriesWait?: MetricHistogramValue;
queriesDuration?: MetricHistogramValue;
datasourceQueriesDuration?: MetricHistogramValue;
};
};
const sources = singleton("databaseMetricsSources", () => new Map<string, DatabaseMetricsSource>());
export function registerDatabaseMetricsSource(source: DatabaseMetricsSource): void {
sources.set(source.clientType, source);
}
export function listDatabaseMetricsSources(): ReadonlyArray<DatabaseMetricsSource> {
return Array.from(sources.values());
}
export function resetDatabaseMetricsSources(): void {
sources.clear();
}
function indexByKey(entries: Array<{ key: string; value: number }>): Record<string, number> {
const out: Record<string, number> = {};
for (const entry of entries) {
out[entry.key] = entry.value;
}
return out;
}
export function normalizeDatabaseMetrics(
source: DatabaseMetricsSource,
json: PrismaMetricsJson | undefined
): NormalizedDatabaseMetrics {
const driver = source.usesDriverAdapter ? ("pg-adapter" as const) : ("quaint" as const);
const counters = json ? indexByKey(json.counters) : undefined;
const gauges = json ? indexByKey(json.gauges) : undefined;
let pool: NormalizedPoolMetrics | undefined;
if (source.usesDriverAdapter && source.pool) {
const total = source.pool.totalCount;
const idle = source.pool.idleCount;
pool = {
open: total,
idle,
busy: Math.max(0, total - idle),
waiting: source.pool.waitingCount,
openedTotal: source.poolCounters?.opened() ?? 0,
closedTotal: source.poolCounters?.closed() ?? 0,
};
} else if (counters && gauges) {
pool = {
open: gauges["prisma_pool_connections_open"] ?? 0,
busy: gauges["prisma_pool_connections_busy"] ?? 0,
idle: gauges["prisma_pool_connections_idle"] ?? 0,
waiting: 0,
openedTotal: counters["prisma_pool_connections_opened_total"] ?? 0,
closedTotal: counters["prisma_pool_connections_closed_total"] ?? 0,
};
}
const result: NormalizedDatabaseMetrics = {
clientType: source.clientType,
driver,
engineMetricsAvailable: json !== undefined,
pool,
histograms: {},
};
if (json && counters && gauges) {
const histograms: Record<string, MetricHistogramValue> = {};
for (const histogram of json.histograms) {
histograms[histogram.key] = histogram.value;
}
result.counters = {
queriesTotal: counters["prisma_client_queries_total"] ?? 0,
datasourceQueriesTotal: counters["prisma_datasource_queries_total"] ?? 0,
};
result.gauges = {
queriesActive: gauges["prisma_client_queries_active"] ?? 0,
queriesWait: gauges["prisma_client_queries_wait"] ?? 0,
};
result.histograms = {
queriesWait: histograms["prisma_client_queries_wait_histogram_ms"],
queriesDuration: histograms["prisma_client_queries_duration_histogram_ms"],
datasourceQueriesDuration: histograms["prisma_datasource_queries_duration_histogram_ms"],
};
}
return result;
}
export async function collectDatabaseClientMetrics(): Promise<NormalizedDatabaseMetrics[]> {
return Promise.all(
Array.from(sources.values()).map(async (source) => {
let json: PrismaMetricsJson | undefined;
try {
json = await source.client.$metrics.json();
} catch {
json = undefined;
}
return normalizeDatabaseMetrics(source, json);
})
);
}