发布

  • fix(core): resolve all lock ordering deadlocks in metrics collector (#33513)

    frostbyte_neo 发布于 2025-11-17 22:02:52 +00:00

    Summary

    This PR fixes a critical deadlock issue in the metrics collector that
    occurred due to inconsistent lock acquisition order between the
    collection thread and registration threads. The fix involved
    restructuring lock scopes across multiple functions to maintain a
    consistent lock hierarchy.

    Changes

    • Fixed lock acquisition order in 4 registration functions
      (register_main_cli_process, register_main_cli_subprocess,
      register_task_process, register_batch)
    • Restructured collect_metrics() to minimize system lock scope and
      release it before acquiring other locks
    • Fixed collection helper methods to read PIDs in scoped blocks without
      holding system lock
    • Added comprehensive trace logging for debugging lock contentions
    • Added concurrent test case to verify no deadlocks occur under stress

    Testing

    • All 12 metrics tests pass
    • Comprehensive concurrent stress test added:
      test_concurrent_group_creation_with_subprocess_updates
    • Lock ordering consistency test added:
      test_lock_order_consistency_across_registration_threads

    Lock Ordering Rule

    Established and enforced this hierarchy across all threads:

    1. Acquire system lock first
    2. Release system lock
    3. Then acquire registration/PID locks

    This prevents circular wait conditions (A→B / B→A) that cause deadlocks.


    Co-authored-by: nx-cloud[bot] <71083854+nx-cloud[bot]@users.noreply.github.com>

    下载附件