Files
triggerdotdev--trigger.dev/internal-packages/run-store
Daniel Sutton e4ae8cbcd4 fix(run-engine,run-store,webapp): stop split-mode waits hanging on resume (#4164)
## Summary

On the run-ops database split, a run that waits (`triggerAndWait`,
`batchTriggerAndWait`, `wait.forToken`) could hang forever after its
wait had already completed. The runner reads a resume from
`/snapshots/since` exactly once: if that read returned the resume
snapshot without its completed-waitpoints, the runner logged "executing
without completed waitpoints", advanced its cursor, and never re-read
it, so the awaiting run never continued.

## Root cause

The resume snapshot and its completed-waitpoint rows were written as two
separate commits. This regressed when the split replaced Prisma's atomic
nested `connect` with an FK-free insert (in
[#4163](https://github.com/triggerdotdev/trigger.dev/pull/4163)), and
`/snapshots/since` is served from a read replica. A fetch landing in the
sub-millisecond gap between the two commits, or a multi-reader replica
serving the snapshot from a different point in time than its join rows,
delivered an empty resume. Because the runner consumes each snapshot
once and treats an empty resume as terminal, a single stale read was
fatal and produced a permanent, nondeterministic hang.

## Fixes

- Commit a snapshot and its completed-waitpoint links in one
transaction, restoring the atomicity the split removed.
- Repair the completed-waitpoints from the owning primary when a
multi-reader replica serves the snapshot without its join rows. This
covers single-waitpoint resumes, which carry no
`completedWaitpointOrder` and so were missed by the count-based repair.
- Read the primary in the checkpoint `WAIT_FOR_BATCH` pre-check, so a
batch that already resumed is not re-suspended into a stall.
- Fall back to the primary when a waitpoint token misses both read
replicas, so a token completed immediately after it was minted no longer
returns a spurious 404.
- Route batch-item creation by `batchTaskRunId`, consistent with the
batch-completion count and the row's foreign key.
- Reject control-plane-only relation selects on the dedicated schema
with a clear error instead of an opaque Prisma failure, and stop
`createDateTimeWaitpoint` bypassing residency routing through a caller
transaction.

Verified against the deployed split topology: a resume snapshot and its
completed-waitpoints are now always delivered together, so the runner
can no longer drop a resume.
2026-07-06 10:55:02 +00:00
..