Files
帝皇侠 b3fd6634cc [ISSUE #15279]feat(lock): implement distributed reentrant/non-reentrant lock (#15280)
* feat(lock): implement distributed reentrant/non-reentrant lock with watchdog, wait queue and gRPC push notification

Phase 1 of the Nacos distributed lock system:
- Reentrant and non-reentrant lock types with owner-based identity
- Watchdog auto-renewal mechanism (30s timeout, 10s renewal interval)
- Wait queue with FIFO semantics and gRPC server-push notifications
- Raft consensus integration for strong consistency across cluster
- Connection disconnect detection with automatic lock release via Raft
- Lock expiration scanner with Raft-path release
- Defensive copy for waitQueue access and null-owner guard in scanner
- Client SDK implementing JUC Lock interface (lock/tryLock/unlock)
- 44 unit tests + 35 integration tests covering core flows and edge cases

* feat(lock): implement distributed reentrant/non-reentrant lock with watchdog, wait queue and gRPC push notification

Phase 1 of the Nacos distributed lock system:
- Reentrant and non-reentrant lock types with owner-based identity
- Watchdog auto-renewal mechanism (30s timeout, 10s renewal interval)
- Wait queue with FIFO semantics and gRPC server-push notifications
- Raft consensus integration for strong consistency across cluster
- Connection disconnect detection with automatic lock release via Raft
- Lock expiration scanner with Raft-path release
- Defensive copy for waitQueue access and null-owner guard in scanner
- Client SDK implementing JUC Lock interface (lock/tryLock/unlock)
- 44 unit tests + 35 integration tests covering core flows and edge cases

* fix(lock): prevent self-deadlock on non-reentrant lock reentry from same thread

Non-reentrant lock reentry on the same thread previously sent the request to
the server, which rejected it and placed the thread in the wait queue, causing
self-deadlock (thread waits for itself to release the lock). Add a client-side
guard (checkReentrantGuard) that throws IllegalMonitorStateException before any
network call when the current thread already holds the lock.

* fix(lock): connection disconnect fully releases reentrant locks via force release

releaseLocksByConnection() previously called unLock() once per lock, which only
decremented reentrantCount by 1. For locks acquired N times, the dead connection
held the lock with count N-1 until expiry.

Add forceRelease flag to MutexLockRequest. When set, onApply calls
forceRelease() which clears reentrantCount/owner/expiry in a single Raft
round-trip. releaseLocksByConnection() now uses this path.

* fix(lock): enforce FIFO wait queue semantics to prevent queue-jumping

Previously, acquireLock() allowed any new request to acquire a released
lock even when waiters were queued, breaking FIFO ordering. This fix:

- Add waiterRetry flag (LockInstance/LockInfo) so the server can
  distinguish queue re-entries from new requests
- Force-release and re-enqueue new requests when the wait queue is
  non-empty, preventing queue-jumping
- Change pollFirstWaiter() to peekFirstWaiter() on unlock/expire so
  entries stay in the queue until the waiter actually re-acquires
- Add removeStaleWaiter() for cleaning up entries on non-head retry
- Add FIFO integration tests (JUC-021, JUC-022)

Fixes: FIFO wait queue semantics not enforced in acquireLock()

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(lock): deduplicate wait queue entries by owner and connection

addWaiter() now checks for an existing entry with the same
owner+connectionId before inserting. If found, it updates the
deadline instead of adding a duplicate. This prevents queue
bloat when LockGrpcClient retries ACQUIRE during notification
wait, and avoids duplicate notifications for the same waiter.

Fixes: blocking lock retry causes duplicate entries in wait queue

* test(lock): add unit tests for distributed lock module

- NonReentrantAtomicLockTest: lock/unlock, reentry rejection, forceRelease, autoExpire
- AbstractAtomicLockTest: renew, isClear, hasWaiters, removeStaleWaiter, drainAllWaiters, removeExpiredWaiters, peekFirstWaiter with expired entries, null input handling, dedup
- MutexAtomicLockTest: migrateFromLegacy (FULL/EMPTY/null/idempotent)
- LockOperationServiceImplTest: acquireLock FIFO enforcement, releaseLock normal/force, renewLock, expireLock
- WaitEntryTest: isExpired, constructor, setters
- LockKeyTest: equals, hashCode, toString, getters/setters
- LockService: add @Since("3.3.0") to renew method

* fix(lock): use Raft CLEANUP_CONNECTION for connection disconnect cleanup

Replace direct forceUnLock with a Raft-consensus CLEANUP_CONNECTION
operation to ensure cluster-wide consistency for lock releases and
wait queue cleanup when a connection disconnects.

Also async-ize pushWithoutAck calls via notificationExecutor to avoid
blocking the Raft FSM thread on push failures.

* fix(lock): fix doUnLock semantics, endTime NPE risk, and showLocks encapsulation

- ReentrantAtomicLock/MutexAtomicLock/NonReentrantAtomicLock: doUnLock
  now returns false when the lock is not held, instead of silently
  succeeding and corrupting state
- LockInfo.endTime: Long → long to eliminate auto-unboxing NPE in
  AbstractAtomicLock.renew()
- LockManager.showLocks() returns Collections.unmodifiableMap();
  NacosLockManager.getRawLockMap() added for snapshot internals
- NacosLockSnapshotOperation uses getRawLockMap() for putAll()
- Add test for unlocking an unheld lock (null-owner bypass path)

* fix(lock): remove unused import, fix Javadoc placement, and add missing license header

* fix(lock): fix FIFO notification gap, ThreadLocal leak, batch cleanup, and metrics

- Add notifyFirstWaiter() after FIFO force-enqueue in acquireLock()
- Clean up ThreadLocal entries in NacosLock.unlock() when reentrantCount reaches 0
- Batch connection cleanup into single Raft consensus instead of N separate writes
- Add RENEW metrics (grpcRenewSuccess/grpcRenewTotal) to LockMetricsMonitor
- Add LockNotificationType enum replacing raw strings for notification types
- Add try-catch in async push callbacks for consistent error handling
- Copy lock map before snapshot serialization to avoid UnmodifiableMap issues
- Add owner null check in LockExpireScanner to reduce unnecessary Raft submissions

* fix(lock): enforce strict FIFO in acquireLock by skipping tryLock when queue has waiters

New requests no longer acquire-then-forceRelease when the wait queue is
non-empty. Instead, acquireLock() checks hasWaiters() before calling
tryLock(), and directly enqueues non-retry requests behind existing
waiters. This eliminates the unnecessary acquire-release dance while
maintaining strict FIFO ordering.

* test(lock): add cluster JUC lock concurrency stability IT

* Format distributed lock code

* test(lock): add lock waiter retry reproduction tests

Add lock-test coverage for non-head waiterRetry acquisition and stale waiter behavior.

* fix(lock): fix lock wait queue cancellation handling

Add server-side wait cancellation for interrupted lock acquisition and cover FIFO retry/cancel paths.

* fix(lock): improve stability and edge case handling for distributed lock

- Fix ThreadLocal leak in unlock() when server exception occurs
- Fix tryLock(time,unit) interrupt/timeout server queue cleanup
- Add null-safe response parsing in LockGrpcClient
- Add closed-state guard in NacosLockService and LockGrpcClient
- Add setResultCode in LockOperationResponse factory methods
- Clear stale state on LockResult.setSuccess(true)
- Remove empty shell locks in LockExpireScanner
- Remove HashMap copy in LockExpireScanner iteration

* fix(lock): improve lock stability, fix bugs, and add unit tests

- Remove redundant lock acquisition in AbstractAtomicLock subclass methods
- Fix deadline reset bug in tryLockAsQueueHead that prevented waiter expiration
- Fix watchdog NacosException not triggering unregister
- Change LockInstance.expiredTime from Long to long to avoid NPE risk
- Fix forceRelease not clearing connectionId field
- Add unit tests for AbstractAtomicLock, LockRequestHandler, LockExpireScanner,
  LockInfo, NacosLockWatchdog, NacosLock, and NacosLockInterrupt

* Fix CI failures - spotless formatting

* fix(lock): change lock model fields from primitive to boxed types and rename waitTimeMs to waitTime

LockInstance: expiredTime long→Long, waitTimeMs→waitTime Long
LockInfo: endTime long→Long, waitTimeMs→waitTime Long
Update all callers and unit tests accordingly.

* test: improve lock coverage

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-09 10:12:53 +08:00
..