-
fix: cancel pending delayed snapshots when the run completes or disconnects
frostbyte_neo 发布于
2026-06-10 16:13:18 +00:00 The compute suspend flow delays snapshots by snapshotDelayMs to avoid
wasted work on short-lived waitpoints, with the intent that a run which
continues before the delay expires cancels the pending snapshot. But the
only cancel() call site is the /continue workload action, which runners
only invoke when restoring from an already-taken snapshot - so a pending
snapshot is never actually cancelled (zero snapshot.canceled events in
prod). When a run resumes and completes within the delay window, the
stale snapshot fires anyway and fcrun pauses the VM for ~6-13s while its
controller is mid warm-start long-poll. The frozen guest can't fire its
abort timer or send a FIN, so firestarter keeps the connection claimable
past the client deadline and dispatches runs into it - each one a ~300s
stall (TRI-10293).Cancel the pending snapshot when the attempt completes and when the run
socket disconnects. Genuine waitpoint suspensions keep the runner socket
connected and the attempt incomplete, so neither hook cancels a snapshot
that is still wanted. Cancellation is guarded by runnerId so a stale
duplicate runner for a reassigned run can't cancel the new runner's
pending snapshot.下载附件