-
[OPIK-6813] [BE] feat: add distributed lock (semaphore) metrics (#7118)
frostbyte_neo 发布于
2026-06-17 08:49:01 +00:00 - [OPIK-6813] [BE] feat: add distributed lock (semaphore) metrics
Instrument RedissonLockService with OpenTelemetry metrics so the
acquire-waiter queue (held in the pod heap as CompletableFutures) is
observable. lock_waiting is the OPIK-6813 signal: a runaway per-project
acquire queue is invisible in Redis and surfaces only as heap growth.Metrics (tagged by a low-cardinality lock name, UUID ids collapsed):
- lock_waiting / lock_held (gauges)
- lock_acquired_total / lock_acquire_failed_total (counters)
- lock_acquire_wait_milliseconds (histogram)
Best-effort: a telemetry failure never breaks locking.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
-
test(backend): add LockMetrics unit tests for label() and best-effort recording
-
fix(backend): cap LockMetrics label cardinality to prevent non-UUID key blowup
-
refactor(backend): make LockMetrics stateless, move label to Lock.metricName
Addresses review: lock_waiting/lock_held are now OTel UpDownCounters (no
in-class state), and the UUID-collapsing label lives on LockService.Lock as
metricName() instead of a regex in LockMetrics. Removes the per-lock AtomicLong
maps and the cardinality cap.- refactor(backend): address lock-metrics review — DI, multidimensional histogram, precise emission
- Inject LockMetrics as a Guice singleton instead of instantiating it in RedissonLockService.
- Drop lock_acquired_total/lock_acquire_failed_total; the lock_acquire_wait_milliseconds
histogram already counts, so outcomes are now an outcome=success|failure attribute on it. - Encapsulate the lock_held start/end pair in instrumentHeld() so it brackets the action.
- Emit acquire failure precisely: a timed-out (empty) acquire now records a failure via
doOnSuccess null-check, not only on error.
Co-authored-by: Claude Opus 4.8 (1M context) noreply@anthropic.com
下载附件