Files
Dhruv Gupta 741f2d29e4 fix(host,runner): reconnect host + all sessions promptly on laptop wake (#5054)
On macOS sleep the OS freezes every Omnigent process and drops the network,
killing the host control-channel WebSocket and every runner (session) tunnel.
On wake nothing reconnected promptly: the only liveness signal was the
websockets keepalive ping (30s interval / 90s timeout), so a half-open
post-sleep socket took up to ~120s to be noticed — and the server had already
deregistered the host, so the desktop app showed "disconnected" that whole
time while the terminal (a one-time startup banner) still read "connected".

Add omnigent/suspend_watch.py: watch_for_resume() detects a resume by polling
a short interval and comparing wall-clock vs monotonic-clock drift. The
monotonic clock freezes during sleep on macOS/Linux while the realtime clock
keeps counting, so a resume shows up as a large divergence; a merely-blocked
event loop advances both equally, so this never false-fires on CPU stalls.
Uses time.monotonic (never loop.time, which under uvloop includes sleep on
macOS and would zero out the divergence).

Wire it into both reconnect loops:
- Host (connect.py): a watcher aborts the live tunnel (ws.transport.abort())
  on wake and flags a prompt reconnect, so run() reattaches at the base
  backoff instead of the escalated one (required on a loopback server, where
  an abrupt close is not auto-classified as a benign recycle).
- Runner (serve.py): a per-connection watcher aborts the tunnel and notes the
  resume so serve_tunnel reconnects promptly; each session self-heals on its
  own event loop, so no host orchestration is needed.

Result: opening the laptop reconnects the host and all its sessions within
~5s instead of up to ~2 minutes. Windows degrades to todays keepalive
behavior (its monotonic clock counts suspend) with no regression.

Tests: unit tests for the detector (fires once on divergence, never on a
blocked loop, survives a raising callback) plus host and runner integration
tests (a simulated wake aborts the live tunnel and forces a prompt reconnect).

Co-authored-by: Isaac

Signed-off-by: Dhruv Gupta <dhruv.gupta@databricks.com>
2026-08-19 14:04:09 -07:00

156 lines
5.0 KiB
Python

"""Unit tests for :mod:`omnigent.suspend_watch`.
The suspend watcher must fire exactly once when the wall clock jumps ahead of
the monotonic clock (a real system sleep), must never fire when both clocks
advance together (a merely-blocked event loop), and must survive a callback
that raises. Clocks and the sleeper are injected so the loop is driven
deterministically without touching real time or patching ``asyncio.sleep``
globally (which the ``no-global-asyncio-patch`` lint forbids in tests).
"""
from __future__ import annotations
import asyncio
import contextlib
from collections.abc import Callable
from omnigent.suspend_watch import (
SUSPEND_GAP_THRESHOLD_S,
SUSPEND_POLL_INTERVAL_S,
watch_for_resume,
)
def _clamping_clock(values: list[float]) -> Callable[[], float]:
"""A clock stub that returns *values* in order, repeating the last.
Clamping (rather than raising ``StopIteration``) keeps the watcher's
``while True`` loop from crashing on an over-read after the values that
matter for the assertion have been consumed.
:param values: Successive readings the clock should return.
:returns: A zero-arg callable yielding each value then the final one.
"""
index = {"i": 0}
def read() -> float:
value = values[min(index["i"], len(values) - 1)]
index["i"] += 1
return value
return read
async def _run_watcher(
*,
wall: list[float],
mono: list[float],
stop_after_sleeps: int,
on_resume: Callable[[float], None],
) -> None:
"""Drive :func:`watch_for_resume` for a fixed number of polls.
The injected sleeper raises :class:`asyncio.CancelledError` on the
*stop_after_sleeps*-th call to end the otherwise-infinite loop, matching
the idiom used in ``tests/runner/transports/ws_tunnel/test_serve.py``.
:param wall: Wall-clock readings (clamped).
:param mono: Monotonic-clock readings (clamped).
:param stop_after_sleeps: Sleep call on which to cancel the loop.
:param on_resume: Callback forwarded to the watcher.
:returns: None.
"""
calls = {"n": 0}
async def fake_sleep(_delay: float) -> None:
calls["n"] += 1
if calls["n"] >= stop_after_sleeps:
raise asyncio.CancelledError
with contextlib.suppress(asyncio.CancelledError):
await watch_for_resume(
on_resume,
wall_clock=_clamping_clock(wall),
mono_clock=_clamping_clock(mono),
sleep=fake_sleep,
)
async def test_fires_once_on_wall_monotonic_divergence() -> None:
"""A wall jump the monotonic clock did not share is reported once.
Models a real sleep: the wall clock advanced 3600 s across one poll while
the monotonic clock advanced only 5 s (it froze while suspended). The gap
(~3595 s) exceeds the threshold, so ``on_resume`` fires exactly once with
that duration.
:returns: None.
"""
fired: list[float] = []
await _run_watcher(
wall=[1000.0, 4600.0],
mono=[0.0, 5.0],
stop_after_sleeps=2,
on_resume=lambda gap: fired.append(round(gap, 1)),
)
assert fired == [3595.0]
async def test_does_not_fire_when_clocks_advance_together() -> None:
"""A blocked event loop (both clocks jump equally) never fires.
This is the load-bearing false-positive guard: a long synchronous call or
GC pause advances the wall AND monotonic clocks by the same amount, so the
divergence stays ~0 and no resume is reported — only genuinely suspended
time (which the monotonic clock excludes) can trip the watcher.
:returns: None.
"""
fired: list[float] = []
await _run_watcher(
wall=[1000.0, 4600.0],
mono=[0.0, 3600.0],
stop_after_sleeps=2,
on_resume=lambda gap: fired.append(gap),
)
assert fired == []
async def test_survives_raising_callback() -> None:
"""A callback that raises is swallowed so the watcher keeps running.
``on_resume`` aborts a socket and flags a reconnect; a bug there must not
kill the watcher (which would silently disable wake detection for the rest
of the process's life).
:returns: None.
"""
calls = {"n": 0}
def raising(_gap: float) -> None:
calls["n"] += 1
raise RuntimeError("boom")
# Loop runs two polls: the first diverges (fires -> raises -> swallowed),
# the second is quiet, then the sleeper cancels. Reaching the cancel proves
# the raise did not propagate out of the watcher.
await _run_watcher(
wall=[1000.0, 4600.0, 4605.0],
mono=[0.0, 5.0, 10.0],
stop_after_sleeps=3,
on_resume=raising,
)
assert calls["n"] == 1
def test_default_constants_are_sane() -> None:
"""The shipped threshold sits above the poll interval and jitter.
A threshold at or below the interval would risk firing on ordinary
scheduling delay; keeping it well above documents the intended margin.
:returns: None.
"""
assert SUSPEND_GAP_THRESHOLD_S > SUSPEND_POLL_INTERVAL_S
assert SUSPEND_POLL_INTERVAL_S > 0