-
[OPIK-6904] Add ClickHouse partition-health observability metrics (#7458)
frostbyte_neo 发布于
2026-07-14 13:29:54 +00:00 - [OPIK-6904] Add ClickHouse partition-health observability metrics
Add a config-gated Quartz job (disabled by default) that polls system.parts and
the lightweight-delete mask and publishes opik.clickhouse.partition.* OpenTelemetry
gauges: partition count per table, per-partition size, largest active part, parts,
rows, last activity, and LWD-masked rows. Modeled on the existing retention
observability jobs: a distributed lock keeps a single instance polling per interval
and clears its snapshot when it loses the lock, so exactly one series is reported
per (table, partition) and aged-out partitions drop cleanly.The metrics work on the current unpartitioned tables (one partition_id = 'all'
series) and fan out to real per-partition series once partitioning lands, with no
code change.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- Address PR review: move partition-metrics DAO to infra, template SQL, config fixes
- Move ClickHousePartitionMetricsDAO (+ test) from com.comet.opik.domain to
com.comet.opik.infrastructure.metrics: it reports on storage-engine internals,
not an application domain entity. - Render the LWD query via the StringTemplate engine () instead of %s
interpolation, matching the DAO idiom elsewhere; validated identifiers only. - Add SETTINGS log_comment to both ClickHouse queries so they are attributable
in query_log / the slow-query dashboard. - Reject-and-warn on invalid LWD table names instead of silently dropping them,
and guard against null entries. - Bind lwdTables as a comma-separated scalar with a computed getter so a
PARTITION_METRICS_LWD_TABLES env override binds cleanly (Dropwizard substitutes
${...} into raw YAML before parsing, so a comma value cannot bind to a List). - Extract registerPartitionGauge/registerLwdGauge helpers to de-duplicate the
observable-gauge registration, and single-quote log placeholders. - refactor: defer partition-metrics refresh until lock is held
- fix: isolate LWD scan failure from partition-health gauges
- OPIK-6904: address review — modern job pattern, Duration config
- Job: drop the run LongCounter; replace the try/catch around .block()
with reactive onErrorResume + fire-and-forget subscribe; hold the
subscription Disposable so interrupt() can cancel the in-flight poll;
build the Snapshot via @Builder(toBuilder=true). - PartitionMetricsConfig: drop Java field defaults (config.yml is the
single source of truth); model the interval as io.dropwizard Duration
with @MinDuration/@MaxDuration; strip() instead of trim(). - config.yml / config-test.yml: interval as a Duration (5m); propagate
the partitionMetrics block to the test config. - Test: delete traces through the API (the prod lightweight-delete path)
instead of a direct DELETE query.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Wrap the refresh Mono in Mono.defer so the DAO calls (getPartitionStats /
getLwdRowCounts) — and their eager query rendering and config split — run only
once bestEffortLock subscribes, i.e. after the distributed lock is acquired.
Also move the Snapshot.EMPTY sentinel inside the Snapshot record.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Wrap the getLwdRowCounts arm in onErrorResume so a failing LWD mask scan
(the expensive, read-write-user arm) degrades to an empty list and logs a
warning, instead of erroring the Mono.zip and leaving the reliable
system.parts gauges stale/EMPTY. The LWD gauges simply stop reporting until
the next successful poll, matching the lock-loss semantics.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Rework the partition-metrics job to match the AlertProjectMigrationJob
conventions raised in review:Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Co-authored-by: Claude Opus 4.8 (1M context) noreply@anthropic.com
下载附件