951d8e8d7b
## What Follow-up to #4539. The driver-adapter work is inert until a client flips to the pg driver adapter, but the moment one does, our database observability degrades: the OTel metrics pipeline reads pool stats from Prisma's `$metrics`, which is owned by the Rust engine's `quaint` pool. Under the adapter, `pg.Pool` owns the pool, so those gauges read zero. The pipeline also only ever scraped a single client (the control-plane writer singleton). This PR makes database metrics driver-agnostic and per-client: - Every configured client registers a metrics source: control-plane writer/replica, run-ops writer/replica, legacy writer/replica. Previously only the control-plane writer singleton was scraped. - Each OTel instrument is observed per client with `db_client` and `db_driver` (`quaint` | `pg-adapter`) attributes. `db_client` uses our canonical datasource-role labels (`control-plane-writer`, `control-plane-replica`, `run-ops-writer`, `run-ops-replica`, `legacy-run-ops-writer`, `legacy-run-ops-replica`) — the same strings used for the `db.datasource` span attribute, so a metric and a trace point at the same pool. - Pool figures come from the authoritative source per driver: - **pg-adapter**: `pg.Pool` (`totalCount`/`idleCount`/`waitingCount`, plus cumulative opened/closed from `connect`/`remove` events). - **quaint**: the Rust engine's `$metrics` pool gauges/counters, exactly as before. - Query counters and duration histograms still come from `$metrics` for both drivers (the Rust engine executes queries in both cases). - New `db.pool.connections.waiting` gauge (pg.Pool exposes this; quaint reports 0). - Stops exporting Prisma metrics from the Prometheus `/metrics` route. Pool observability now lives entirely in the OTel pipeline, per driver, per client. ## Why So we can flip any client (including the control-plane writer, the primary desync-fix target) to the driver adapter without losing pool visibility. Existing dashboards keyed on the same metric names keep working; they gain a per-client dimension. ## Testing Unit (`apps/webapp/app/utils/databaseMetrics.server.test.ts`): the pure normalizer — quaint reads pool from `$metrics`; adapter reads pool from `pg.Pool` and keeps engine query metrics; `busy` never goes negative; graceful zeroing when `$metrics` is unavailable (adapter still reports live pool figures). Live smoke test against a prod-shaped local stack: three physically-distinct Postgres DBs (control-plane, run-ops, legacy) behind dual PgBouncers, split mode on, with a mix of adapter and quaint clients. Reading the actual emitted OTel metrics, every pool shows up as its own series: ``` db.pool.connections.total{db_client="control-plane-writer", db_driver="pg-adapter"} = 1 db.pool.connections.total{db_client="control-plane-replica", db_driver="quaint"} = 1 db.pool.connections.total{db_client="run-ops-writer", db_driver="pg-adapter"} = 1 db.pool.connections.total{db_client="run-ops-replica", db_driver="quaint"} = 1 db.pool.connections.total{db_client="legacy-run-ops-writer", db_driver="quaint"} = 1 db.pool.connections.total{db_client="legacy-run-ops-replica",db_driver="quaint"} = 1 db.client.queries.total{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing db.client.queries.duration.count{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing ``` Confirms: metrics are attributed per pool with the correct driver; adapter pools' figures come from `pg.Pool`; and query counters/duration histograms keep incrementing under the pg adapter. Also verified `/metrics` (Prometheus) now returns zero `prisma_*` series while still serving the app's own metrics. `pnpm run typecheck --filter webapp` passes. ## Notes - `/metrics` (Prometheus) no longer includes `prisma_*` series. Anything scraping that endpoint for Prisma metrics should read the equivalent `db.*` metrics from the OTel exporter instead. - **PgBouncer + `?schema=` gotcha (separate from this PR, worth flagging for rollout):** since #4539 parses `?schema=` from the DSN and passes `{ schema }` to the adapter, node-postgres sends `search_path` as a startup parameter. A transaction-mode PgBouncer rejects that with `FATAL: unsupported startup parameter: search_path`. Our prod control-plane DSNs use the default `public` schema with no `?schema=` param, so this is latent, but any client we flip to the adapter must not carry `?schema=` in its DSN (or the pooler needs `ignore_startup_parameters = search_path`). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
153 lines
4.5 KiB
TypeScript
153 lines
4.5 KiB
TypeScript
import { singleton } from "./singleton";
|
|
|
|
export type MetricHistogramValue = {
|
|
buckets: [number, number][];
|
|
sum: number;
|
|
count: number;
|
|
};
|
|
|
|
type PrismaMetricsJson = {
|
|
counters: Array<{ key: string; value: number }>;
|
|
gauges: Array<{ key: string; value: number }>;
|
|
histograms: Array<{ key: string; value: MetricHistogramValue }>;
|
|
};
|
|
|
|
type MetricsCapableClient = {
|
|
$metrics: { json: () => Promise<PrismaMetricsJson> };
|
|
};
|
|
|
|
type PoolLike = {
|
|
totalCount: number;
|
|
idleCount: number;
|
|
waitingCount: number;
|
|
};
|
|
|
|
export type DatabaseMetricsSource = {
|
|
clientType: string;
|
|
usesDriverAdapter: boolean;
|
|
client: MetricsCapableClient;
|
|
pool?: PoolLike;
|
|
poolCounters?: { opened: () => number; closed: () => number };
|
|
};
|
|
|
|
export type NormalizedPoolMetrics = {
|
|
open: number;
|
|
busy: number;
|
|
idle: number;
|
|
waiting: number;
|
|
openedTotal: number;
|
|
closedTotal: number;
|
|
};
|
|
|
|
export type NormalizedDatabaseMetrics = {
|
|
clientType: string;
|
|
driver: "pg-adapter" | "quaint";
|
|
engineMetricsAvailable: boolean;
|
|
pool?: NormalizedPoolMetrics;
|
|
counters?: { queriesTotal: number; datasourceQueriesTotal: number };
|
|
gauges?: { queriesActive: number; queriesWait: number };
|
|
histograms: {
|
|
queriesWait?: MetricHistogramValue;
|
|
queriesDuration?: MetricHistogramValue;
|
|
datasourceQueriesDuration?: MetricHistogramValue;
|
|
};
|
|
};
|
|
|
|
const sources = singleton("databaseMetricsSources", () => new Map<string, DatabaseMetricsSource>());
|
|
|
|
export function registerDatabaseMetricsSource(source: DatabaseMetricsSource): void {
|
|
sources.set(source.clientType, source);
|
|
}
|
|
|
|
export function listDatabaseMetricsSources(): ReadonlyArray<DatabaseMetricsSource> {
|
|
return Array.from(sources.values());
|
|
}
|
|
|
|
export function resetDatabaseMetricsSources(): void {
|
|
sources.clear();
|
|
}
|
|
|
|
function indexByKey(entries: Array<{ key: string; value: number }>): Record<string, number> {
|
|
const out: Record<string, number> = {};
|
|
for (const entry of entries) {
|
|
out[entry.key] = entry.value;
|
|
}
|
|
return out;
|
|
}
|
|
|
|
export function normalizeDatabaseMetrics(
|
|
source: DatabaseMetricsSource,
|
|
json: PrismaMetricsJson | undefined
|
|
): NormalizedDatabaseMetrics {
|
|
const driver = source.usesDriverAdapter ? ("pg-adapter" as const) : ("quaint" as const);
|
|
const counters = json ? indexByKey(json.counters) : undefined;
|
|
const gauges = json ? indexByKey(json.gauges) : undefined;
|
|
|
|
let pool: NormalizedPoolMetrics | undefined;
|
|
if (source.usesDriverAdapter && source.pool) {
|
|
const total = source.pool.totalCount;
|
|
const idle = source.pool.idleCount;
|
|
pool = {
|
|
open: total,
|
|
idle,
|
|
busy: Math.max(0, total - idle),
|
|
waiting: source.pool.waitingCount,
|
|
openedTotal: source.poolCounters?.opened() ?? 0,
|
|
closedTotal: source.poolCounters?.closed() ?? 0,
|
|
};
|
|
} else if (counters && gauges) {
|
|
pool = {
|
|
open: gauges["prisma_pool_connections_open"] ?? 0,
|
|
busy: gauges["prisma_pool_connections_busy"] ?? 0,
|
|
idle: gauges["prisma_pool_connections_idle"] ?? 0,
|
|
waiting: 0,
|
|
openedTotal: counters["prisma_pool_connections_opened_total"] ?? 0,
|
|
closedTotal: counters["prisma_pool_connections_closed_total"] ?? 0,
|
|
};
|
|
}
|
|
|
|
const result: NormalizedDatabaseMetrics = {
|
|
clientType: source.clientType,
|
|
driver,
|
|
engineMetricsAvailable: json !== undefined,
|
|
pool,
|
|
histograms: {},
|
|
};
|
|
|
|
if (json && counters && gauges) {
|
|
const histograms: Record<string, MetricHistogramValue> = {};
|
|
for (const histogram of json.histograms) {
|
|
histograms[histogram.key] = histogram.value;
|
|
}
|
|
result.counters = {
|
|
queriesTotal: counters["prisma_client_queries_total"] ?? 0,
|
|
datasourceQueriesTotal: counters["prisma_datasource_queries_total"] ?? 0,
|
|
};
|
|
result.gauges = {
|
|
queriesActive: gauges["prisma_client_queries_active"] ?? 0,
|
|
queriesWait: gauges["prisma_client_queries_wait"] ?? 0,
|
|
};
|
|
result.histograms = {
|
|
queriesWait: histograms["prisma_client_queries_wait_histogram_ms"],
|
|
queriesDuration: histograms["prisma_client_queries_duration_histogram_ms"],
|
|
datasourceQueriesDuration: histograms["prisma_datasource_queries_duration_histogram_ms"],
|
|
};
|
|
}
|
|
|
|
return result;
|
|
}
|
|
|
|
export async function collectDatabaseClientMetrics(): Promise<NormalizedDatabaseMetrics[]> {
|
|
return Promise.all(
|
|
Array.from(sources.values()).map(async (source) => {
|
|
let json: PrismaMetricsJson | undefined;
|
|
try {
|
|
json = await source.client.$metrics.json();
|
|
} catch {
|
|
json = undefined;
|
|
}
|
|
return normalizeDatabaseMetrics(source, json);
|
|
})
|
|
);
|
|
}
|