Commit Graph

2 Commits

Author SHA1 Message Date
Eric Allam 14e081e17d release concurrency system reliability improvements (#2081)
* Refill release concurrency tokens when a run is cancelled

* Improved release concurrency accounting system + a sweeper to auto-refill tokens for snapshots that are no longer the latest snapshot on a run (e.g. the run has moved to a new snapshot state)

* Fix order of arguments to the releasings sweeper

* Add a heartbeat for SUSPENDED snapshots, where when stalled will attempt to continue the run if unblocked
2025-05-21 13:21:17 +01:00
Eric Allam 7b1159eb45 MarQS reserve concurrency system & queue priority for resuming/retrying (#1715)
* run engine v1: orgs are no longer considered for concurrency

* Add reserve concurrency concept to allow waiting to resume parent tasks to release concurrency at the env level for child tasks to use (or else there is a deadlock). WIP recursive tasks

* child tasks inherit the queue timestamp from their parent tasks to prioritize completing child tasks based on when their parent started

* handle reserve concurrency with recursive deadlocks

* Finish docs update for concurrency

* Some fixes from badge conflict resolution

* WIP priority queues

* Implement MarQS priority queues

* Fix the migrations
2025-02-19 11:43:34 +00:00