951d8e8d7b
## What Follow-up to #4539. The driver-adapter work is inert until a client flips to the pg driver adapter, but the moment one does, our database observability degrades: the OTel metrics pipeline reads pool stats from Prisma's `$metrics`, which is owned by the Rust engine's `quaint` pool. Under the adapter, `pg.Pool` owns the pool, so those gauges read zero. The pipeline also only ever scraped a single client (the control-plane writer singleton). This PR makes database metrics driver-agnostic and per-client: - Every configured client registers a metrics source: control-plane writer/replica, run-ops writer/replica, legacy writer/replica. Previously only the control-plane writer singleton was scraped. - Each OTel instrument is observed per client with `db_client` and `db_driver` (`quaint` | `pg-adapter`) attributes. `db_client` uses our canonical datasource-role labels (`control-plane-writer`, `control-plane-replica`, `run-ops-writer`, `run-ops-replica`, `legacy-run-ops-writer`, `legacy-run-ops-replica`) — the same strings used for the `db.datasource` span attribute, so a metric and a trace point at the same pool. - Pool figures come from the authoritative source per driver: - **pg-adapter**: `pg.Pool` (`totalCount`/`idleCount`/`waitingCount`, plus cumulative opened/closed from `connect`/`remove` events). - **quaint**: the Rust engine's `$metrics` pool gauges/counters, exactly as before. - Query counters and duration histograms still come from `$metrics` for both drivers (the Rust engine executes queries in both cases). - New `db.pool.connections.waiting` gauge (pg.Pool exposes this; quaint reports 0). - Stops exporting Prisma metrics from the Prometheus `/metrics` route. Pool observability now lives entirely in the OTel pipeline, per driver, per client. ## Why So we can flip any client (including the control-plane writer, the primary desync-fix target) to the driver adapter without losing pool visibility. Existing dashboards keyed on the same metric names keep working; they gain a per-client dimension. ## Testing Unit (`apps/webapp/app/utils/databaseMetrics.server.test.ts`): the pure normalizer — quaint reads pool from `$metrics`; adapter reads pool from `pg.Pool` and keeps engine query metrics; `busy` never goes negative; graceful zeroing when `$metrics` is unavailable (adapter still reports live pool figures). Live smoke test against a prod-shaped local stack: three physically-distinct Postgres DBs (control-plane, run-ops, legacy) behind dual PgBouncers, split mode on, with a mix of adapter and quaint clients. Reading the actual emitted OTel metrics, every pool shows up as its own series: ``` db.pool.connections.total{db_client="control-plane-writer", db_driver="pg-adapter"} = 1 db.pool.connections.total{db_client="control-plane-replica", db_driver="quaint"} = 1 db.pool.connections.total{db_client="run-ops-writer", db_driver="pg-adapter"} = 1 db.pool.connections.total{db_client="run-ops-replica", db_driver="quaint"} = 1 db.pool.connections.total{db_client="legacy-run-ops-writer", db_driver="quaint"} = 1 db.pool.connections.total{db_client="legacy-run-ops-replica",db_driver="quaint"} = 1 db.client.queries.total{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing db.client.queries.duration.count{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing ``` Confirms: metrics are attributed per pool with the correct driver; adapter pools' figures come from `pg.Pool`; and query counters/duration histograms keep incrementing under the pg adapter. Also verified `/metrics` (Prometheus) now returns zero `prisma_*` series while still serving the app's own metrics. `pnpm run typecheck --filter webapp` passes. ## Notes - `/metrics` (Prometheus) no longer includes `prisma_*` series. Anything scraping that endpoint for Prisma metrics should read the equivalent `db.*` metrics from the OTel exporter instead. - **PgBouncer + `?schema=` gotcha (separate from this PR, worth flagging for rollout):** since #4539 parses `?schema=` from the DSN and passes `{ schema }` to the adapter, node-postgres sends `search_path` as a startup parameter. A transaction-mode PgBouncer rejects that with `FATAL: unsupported startup parameter: search_path`. Our prod control-plane DSNs use the default `public` schema with no `?schema=` param, so this is latent, but any client we flip to the adapter must not carry `?schema=` in its DSN (or the pooler needs `ignore_startup_parameters = search_path`). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
142 lines
4.4 KiB
TypeScript
142 lines
4.4 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import {
|
|
normalizeDatabaseMetrics,
|
|
type DatabaseMetricsSource,
|
|
type MetricHistogramValue,
|
|
} from "./databaseMetrics.server";
|
|
|
|
const durationHistogram: MetricHistogramValue = {
|
|
buckets: [
|
|
[1, 10],
|
|
[10, 5],
|
|
],
|
|
sum: 1234,
|
|
count: 15,
|
|
};
|
|
|
|
function quaintJson() {
|
|
return {
|
|
counters: [
|
|
{ key: "prisma_client_queries_total", value: 100 },
|
|
{ key: "prisma_datasource_queries_total", value: 250 },
|
|
{ key: "prisma_pool_connections_opened_total", value: 12 },
|
|
{ key: "prisma_pool_connections_closed_total", value: 3 },
|
|
],
|
|
gauges: [
|
|
{ key: "prisma_client_queries_active", value: 4 },
|
|
{ key: "prisma_client_queries_wait", value: 2 },
|
|
{ key: "prisma_pool_connections_open", value: 9 },
|
|
{ key: "prisma_pool_connections_busy", value: 4 },
|
|
{ key: "prisma_pool_connections_idle", value: 5 },
|
|
],
|
|
histograms: [{ key: "prisma_client_queries_duration_histogram_ms", value: durationHistogram }],
|
|
};
|
|
}
|
|
|
|
const stubClient = { $metrics: { json: async () => quaintJson() } };
|
|
|
|
describe("normalizeDatabaseMetrics", () => {
|
|
it("reads pool figures from $metrics for a quaint (Rust) client", () => {
|
|
const source: DatabaseMetricsSource = {
|
|
clientType: "writer",
|
|
usesDriverAdapter: false,
|
|
client: stubClient,
|
|
};
|
|
|
|
const result = normalizeDatabaseMetrics(source, quaintJson());
|
|
|
|
expect(result.driver).toBe("quaint");
|
|
expect(result.clientType).toBe("writer");
|
|
expect(result.engineMetricsAvailable).toBe(true);
|
|
expect(result.pool).toEqual({
|
|
open: 9,
|
|
busy: 4,
|
|
idle: 5,
|
|
waiting: 0,
|
|
openedTotal: 12,
|
|
closedTotal: 3,
|
|
});
|
|
expect(result.counters).toEqual({ queriesTotal: 100, datasourceQueriesTotal: 250 });
|
|
expect(result.gauges).toEqual({ queriesActive: 4, queriesWait: 2 });
|
|
expect(result.histograms.queriesDuration).toEqual(durationHistogram);
|
|
});
|
|
|
|
it("reads pool figures from pg.Pool for a driver-adapter client and keeps engine query metrics", () => {
|
|
const source: DatabaseMetricsSource = {
|
|
clientType: "control-plane-writer",
|
|
usesDriverAdapter: true,
|
|
client: stubClient,
|
|
pool: { totalCount: 8, idleCount: 3, waitingCount: 6 },
|
|
poolCounters: { opened: () => 20, closed: () => 12 },
|
|
};
|
|
|
|
const result = normalizeDatabaseMetrics(source, quaintJson());
|
|
|
|
expect(result.driver).toBe("pg-adapter");
|
|
expect(result.engineMetricsAvailable).toBe(true);
|
|
expect(result.pool).toEqual({
|
|
open: 8,
|
|
busy: 5,
|
|
idle: 3,
|
|
waiting: 6,
|
|
openedTotal: 20,
|
|
closedTotal: 12,
|
|
});
|
|
expect(result.counters).toEqual({ queriesTotal: 100, datasourceQueriesTotal: 250 });
|
|
});
|
|
|
|
it("does not report negative busy when idle exceeds total for an adapter pool", () => {
|
|
const source: DatabaseMetricsSource = {
|
|
clientType: "reader",
|
|
usesDriverAdapter: true,
|
|
client: stubClient,
|
|
pool: { totalCount: 2, idleCount: 5, waitingCount: 0 },
|
|
poolCounters: { opened: () => 0, closed: () => 0 },
|
|
};
|
|
|
|
const result = normalizeDatabaseMetrics(source, quaintJson());
|
|
|
|
expect(result.pool?.busy).toBe(0);
|
|
});
|
|
|
|
it("omits engine-derived metrics and pool when $metrics is unavailable for a quaint client", () => {
|
|
const source: DatabaseMetricsSource = {
|
|
clientType: "writer",
|
|
usesDriverAdapter: false,
|
|
client: stubClient,
|
|
};
|
|
|
|
const result = normalizeDatabaseMetrics(source, undefined);
|
|
|
|
expect(result.engineMetricsAvailable).toBe(false);
|
|
expect(result.pool).toBeUndefined();
|
|
expect(result.counters).toBeUndefined();
|
|
expect(result.gauges).toBeUndefined();
|
|
expect(result.histograms.queriesDuration).toBeUndefined();
|
|
});
|
|
|
|
it("keeps pg.Pool figures but omits engine metrics when $metrics is unavailable for an adapter client", () => {
|
|
const source: DatabaseMetricsSource = {
|
|
clientType: "control-plane-writer",
|
|
usesDriverAdapter: true,
|
|
client: stubClient,
|
|
pool: { totalCount: 7, idleCount: 2, waitingCount: 1 },
|
|
poolCounters: { opened: () => 9, closed: () => 2 },
|
|
};
|
|
|
|
const result = normalizeDatabaseMetrics(source, undefined);
|
|
|
|
expect(result.engineMetricsAvailable).toBe(false);
|
|
expect(result.pool).toEqual({
|
|
open: 7,
|
|
busy: 5,
|
|
idle: 2,
|
|
waiting: 1,
|
|
openedTotal: 9,
|
|
closedTotal: 2,
|
|
});
|
|
expect(result.counters).toBeUndefined();
|
|
expect(result.gauges).toBeUndefined();
|
|
});
|
|
});
|