github-featured / triggerdotdev--trigger.dev

发布

0 发布 4,184 Git 标签

RSS 订阅源
  • re2-test-warm-start-verify 002b8458d5
    比较

    feat(supervisor): verify warm-start delivery, cold-start silently lost dispatches (#3918)

    frostbyte_neo 发布于 2026-06-16 13:14:53 +00:00 | 667 次提交 在此版本后已推送到 main

    Problem

    Firestarter's didWarmStart: true means the response was written to a
    long-poll socket — not that the runner received it. A silently dead
    poller (no FIN, e.g. a VM torn down mid-poll) leaves the dispatched run
    stuck in PENDING_EXECUTING until the run engine's heartbeat redrive,
    and each redrive burns a queue redelivery toward
    TASK_RUN_DEQUEUED_MAX_RETRIES.

    Change

    After a warm-start hit, the supervisor retains the DequeuedMessage
    (TimerWheel, default 10s), then probes the existing getLatestSnapshot
    API. If the run is still on the exact dequeued snapshot, no runner ever
    acted — it falls through to the regular cold-create path. Recovery: ~10s

    • cold start, no new APIs, no CLI changes.
    • Double-start safe: startRunAttempt runs under a per-run lock and
      409s stale snapshot ids, so a reviving runner and the fallback workload
      can't both execute; the loser exits before running anything.
    • Probe errors → do nothing: healthy runners legitimately act late
      during platform brownouts (nested attempt-start retries), so falling
      back on uncertainty would stampede duplicates. The heartbeat redrive
      stays as the backstop (also covers supervisor restarts dropping timers).
    • Off by default: TRIGGER_WARM_START_VERIFY_ENABLED (+
      TRIGGER_WARM_START_VERIFY_DELAY_MS, 1–60s, default 10s). Disabled =
      complete no-op. Works for all workload managers (compute/k8s/docker)
      since it hooks the shared dequeue path.
    • Emits warmstart.verify wide events (outcome: delivered | fallback | probe_error), making the silent-loss rate directly measurable.
    下载附件