-
fix(core): resolve all lock ordering deadlocks in metrics collector (#33513)
发布于
2025-11-17 22:02:52 +00:00 Summary
This PR fixes a critical deadlock issue in the metrics collector that
occurred due to inconsistent lock acquisition order between the
collection thread and registration threads. The fix involved
restructuring lock scopes across multiple functions to maintain a
consistent lock hierarchy.Changes
- Fixed lock acquisition order in 4 registration functions
(register_main_cli_process, register_main_cli_subprocess,
register_task_process, register_batch) - Restructured collect_metrics() to minimize system lock scope and
release it before acquiring other locks - Fixed collection helper methods to read PIDs in scoped blocks without
holding system lock - Added comprehensive trace logging for debugging lock contentions
- Added concurrent test case to verify no deadlocks occur under stress
Testing
- All 12 metrics tests pass
- Comprehensive concurrent stress test added:
test_concurrent_group_creation_with_subprocess_updates - Lock ordering consistency test added:
test_lock_order_consistency_across_registration_threads
Lock Ordering Rule
Established and enforced this hierarchy across all threads:
- Acquire system lock first
- Release system lock
- Then acquire registration/PID locks
This prevents circular wait conditions (A→B / B→A) that cause deadlocks.
Co-authored-by: nx-cloud[bot] <71083854+nx-cloud[bot]@users.noreply.github.com>
下载附件
- Fixed lock acquisition order in 4 registration functions