github-featured / triggerdotdev--trigger.dev

发布

0 发布 4,184 Git 标签

RSS 订阅源
  • re2-prod-supervisor-tri-10293 d0b2d79b3b
    比较

    fix(supervisor): cancel pending delayed snapshots when the run completes or disconnects (#3894)

    frostbyte_neo 发布于 2026-06-11 16:29:54 +00:00

    The compute suspend flow delays snapshots by snapshotDelayMs (~30s) so
    short-lived waitpoints skip the snapshot entirely, with the intent that
    a run continuing before the delay expires cancels the pending snapshot.
    But the only cancel() call site was the /continue action, which
    runners only invoke when restoring from an already-taken snapshot — so
    pending snapshots were never cancelled (zero snapshot.canceled events
    ever emitted in prod). When a run resumed and completed inside the
    window, the stale snapshot fired ~30s later anyway, pausing the VM 6–13s
    mid warm-start long-poll; the frozen guest couldn't fire its abort timer
    or send a FIN, causing stalls and run-engine driven retries.

    Change

    • Cancel the pending snapshot on attempt.complete — after the platform
      accepts the completion, before the HTTP reply (so it can't reorder with
      the runner's next /suspend).
    • Cancel on runDisconnected (crash, exit, or run replaced on the
      socket).
    • Both cancels are guarded by a runnerId match (new
      TimerWheel.peek()): a stale duplicate runner for a reassigned run must
      not cancel the fresh runner's pending snapshot. A missing runnerId falls
      through to an unconditional cancel (the pre-existing /continue
      behavior is unchanged).

    Waitpoint suspensions keep the runner socket connected and the attempt
    incomplete, so neither hook touches a snapshot that is still wanted.

    Known limitation (fail-safe direction): socket.data.runnerId is frozen
    at the websocket handshake, so after a same-supervisor restore the
    disconnect-path guard refuses the cancel. The attempt.complete path
    uses the runner's current header id and is unaffected.

    下载附件