Commit Graph

12 Commits

Author SHA1 Message Date
nicktrn fa7f4b1fed feat(k8s): add placement tags for flexible node selection (#2390)
* add tier scheduling support to supervisor

* add billing info to dequeued message w/o cache

* add cache with best effort invalidation

* fix invalidate circular dep

* add changeset

* use new plan type on runs as fallback during dequeue

* tidy up

* be more explicit with plan type fallback

* remove additional billing check from hot path

* switch to placement tags

* update changeset

* update platform package

* start using new entitlement response

* ensure skipChecks optimization validates at batch level

* add optional items to add to queue manager limits

* make the bool env helper only accept boolean defaults

* remove redundant private field

* update placement tag helper to prevent unsupported tags
2025-08-15 09:52:14 +01:00
Eric Allam 1f4434035e latest @opentelemetry packages and correlate external traces (#2334) 2025-08-06 08:10:46 +01:00
Eric Allam 6ff0b923e1 New internal idempotency implementation for trigger and batch trigger (#2256)
* Introduce request idempotency to prevent duplicate triggers

* Implement request idempotency on trigger

* Use x-trigger-request-idempotency-key header instead

Plus cleanup shared logic in the routes

* Add changeset

* Oops, lets not hardcode a 408

* A couple of improvements
2025-07-10 16:42:32 +01:00
Eric Allam bdaa2ed15a Move batch trigger processing jobs to a dedicated redis worker (#2233) 2025-07-04 12:16:05 +01:00
Eric Allam ea6f66bdfc Improved schedule engine (#2174) 2025-06-17 06:48:05 +01:00
Eric Allam 48e1e7250b v4: fix stuck batch issue when processing happens before transaction commits (#2163)
* v4: fix stuck batch issue when processing happens before transaction commits

* Fix flaky test
2025-06-10 17:02:34 +01:00
Eric Allam 261dce102a v4: fix batchTriggerAndWait completion issues by processing batch chunks sequentially (#2152)
* v4: fix batchTriggerAndWait completion issues by processing batch chunks sequentially

* Remove unnecessary test
2025-06-05 17:56:56 +01:00
Eric Allam 13d378f6df v4: batches can only complete when all their runs have been created (#2101)
* v4: batches can only complete when all their runs have been created

* Fix flaky runs replication test
2025-05-23 16:53:50 +01:00
Matt Aitken 8b0e4f2367 Fix for completing batches (when there are idempotent runs) (#1985) 2025-04-25 18:41:20 +01:00
Eric Allam e837500486 feat: v4 deadlock detection (#1970)
* Locked task runs will now require queues and tasks to be in the locked version

* Client errors caught in a run function now will skip retrying

* Extracted out the trigger queues logic

* extract validation, idempotency keys, payloads to concerns

* Extracted out a bunch of more stuff and getting trigger tests to work

* Add queue and locked version tests

* Deadlock detection WIP

* more deadlock detection

* Only detect deadlocks when the parent run is waiting on the child run

* Improve the error experience around deadlocks

* A couple tweaks to make CodeRabbit happy and fixing the tests in CI

* Fixed failing test

* Changeset

* wip

* Make sure to scope queries to the runtime env
2025-04-25 12:56:54 +01:00
Matt Aitken bbf397e661 Completed batch waitpoints when we completed the BatchTaskRun (#1945)
* Completed batch waitpoints when we completed the BatchTaskRun

* Try complete the batch faster now it’s being used operationally

* Fix for tests that were using the old engine.unblockRunForCreatedBatch() function
2025-04-17 21:01:58 +01:00
Eric Allam d6fd7f559e Run engine: Using root queue timestamp to prioritize completing runs (#1818)
* Improve the new run engine Trigger/Batch trigger service class names

* centralize queue timestamp logic in EnqueueSystem, adding queueTimestamp support and propagation

* Fixing the create tags in the new run engine trigger task service
2025-03-26 10:25:49 +00:00