## Summary
A run's finish commit and its follow-up side effects (completing the
associated waitpoint, waking blocked parents, releasing the queue slot,
nudging batch completion) are separate writes across Postgres and Redis.
If a database error landed between them, the child run was already
finished, so the runner's retries hit the "Run is already finished"
guard and the completion signal was lost for good. A parent blocked on
`triggerAndWait` or `batchTriggerAndWait` then stayed waiting forever.
TTL expiry had the same shape: its worker retry returned early on a
non-pending run, and the batch expiry path swallowed a failed
waitpoint-job enqueue.
## Fix
Every finalizing path (attempt success, permanent failure, cancellation,
TTL expiry) now enqueues a durable `ensureRunFinalized` job before the
finish commit, and acks it once the inline side effects all succeed. In
steady state the guard never executes; the cost is one Redis enqueue and
ack per completion.
When the inline path dies in between, the guard fires after a short
delay and re-derives everything from current state: it releases the
run's queue message and concurrency slot, completes a still-pending
associated waitpoint from the run row's output or error, re-runs the
blocked-run fan-out (covering a lost unblock enqueue even after the
waitpoint committed), and re-schedules the batch completion check. Every
leg is idempotent, so racing the inline path is a no-op. The job retries
with a capped backoff for roughly five weeks before dead-lettering, so
it outlives any database outage while a genuinely poisoned item still
becomes visible.
Cancellation gets special handling: CANCELED is the only terminal run
status where execution can still be in flight, so the guard only
re-delivers for a canceled run once its execution snapshot is FINISHED,
re-arming itself until then rather than resuming the parent while the
child is still winding down.
A `finalization_rederivations` counter increments whenever the guard
actually re-delivers a lost signal; it should stay at zero in a healthy
system.
Tests cover six shapes: waitpoint completion lost after the finish
commit, unblock fan-out lost after the waitpoint completed, a failed
guard enqueue failing the completion request with nothing committed, a
stale guard held back during an in-flight cancellation, waitpoint
completion lost during TTL expiry, and the happy path where the guard is
acked and never runs.
Known accepted edge: a guard re-run after a partial inline completion
can re-emit a cached-run completion event for the same span; this only
happens during failure recovery and is bounded to duplicate trace
events.
Matt Aitken
ef7b3aaf38