Commit Graph

80 Commits

Author SHA1 Message Date
nicktrn 15816e9955 Fix long retry delays (#2020)
* make testcontainers wait until container has stopped

* require unit tests for publishing again

* add failing test case

* make it pass

* add retry threshold ms env var and use it
2025-05-02 16:57:21 +01:00
Matt Aitken 280a6d0297 V4 dequeue test fixes (#1991)
* Fix priority tests by dequeuing in sequence

* Expect this test to fail until we support parallel dequeuing of an env with streaming

* Fix one and expect fail from the other

* Fix for batch trigger test dequeuing

* Fixed pending version test
2025-04-29 12:40:22 +01:00
Matt Aitken f579afb4cc V4 dequeue performance (return faster) (#1989)
* Loop through tenants only once, return as soon as we have enough.

* Hello world deadlock task set back to triggerAndWait
2025-04-28 14:49:38 +01:00
Eric Allam e837500486 feat: v4 deadlock detection (#1970)
* Locked task runs will now require queues and tasks to be in the locked version

* Client errors caught in a run function now will skip retrying

* Extracted out the trigger queues logic

* extract validation, idempotency keys, payloads to concerns

* Extracted out a bunch of more stuff and getting trigger tests to work

* Add queue and locked version tests

* Deadlock detection WIP

* more deadlock detection

* Only detect deadlocks when the parent run is waiting on the child run

* Improve the error experience around deadlocks

* A couple tweaks to make CodeRabbit happy and fixing the tests in CI

* Fixed failing test

* Changeset

* wip

* Make sure to scope queries to the runtime env
2025-04-25 12:56:54 +01:00
Matt Aitken bb606a1fcd Don’t allowed canceled delayed runs to be put into the queue (#1981) 2025-04-24 15:10:47 +01:00
Matt Aitken 18f77660e0 Runlock telemetry (#1974)
* Added error throwing tests to runlock

* Added some more run lock tests

* Added customConditions so the test can use the Logger

* Fix for JSON error

* Added a span to the run lock with a name
2025-04-24 13:37:33 +01:00
Matt Aitken aa1027969b Increase v4 visibility timeouts & don't try continue finished runs (#1978)
* Increase v4 visibility timeouts

* Added some logging for #continueRunIfUnblocked: run has no checkpoint

* If a run is finished or pending cancel, don’t try and continue it
2025-04-24 13:36:18 +01:00
Eric Allam 006951aa53 Set a maximum number of times the dequeue loop can run (#1950) 2025-04-18 13:36:52 +01:00
Eric Allam 863ecf4830 fix: prevent unbounded looping while dequeueing by exiting early when no queues have messages to dequeue (#1946) 2025-04-18 09:41:09 +01:00
Matt Aitken bbf397e661 Completed batch waitpoints when we completed the BatchTaskRun (#1945)
* Completed batch waitpoints when we completed the BatchTaskRun

* Try complete the batch faster now it’s being used operationally

* Fix for tests that were using the old engine.unblockRunForCreatedBatch() function
2025-04-17 21:01:58 +01:00
Eric Allam 4b42f778ce fix: release concurrency token exhaustion no longer happens on batchTriggerAndWait (#1937) 2025-04-17 15:07:06 +01:00
Eric Allam e501113a73 fix: refill release concurrency token bucket queue when runs resume before checkpoints are created (#1933)
* fix: refill release concurrency token bucket queue when runs resume before checkpoints are created

* Fix the engine package.json tests

* Remove log
2025-04-16 20:18:38 +01:00
Matt Aitken 10f78cbad4 Delete project (#1913)
* Delete project

- Don’t schedule tasks if the project is deleted
- Delete queues from the master queues
- Add the old delete project UI back in

* Mark the project as deleted last

* Fix for overriding local variable

* Added a todo for deleting env queues

* Remove todo
2025-04-16 13:31:22 +01:00
nicktrn 2957ee9af9 Remove batch ID carryover for non-batch waits (#1930)
* add failing test case

* do not carry over previous batch id when blocking with waitpoint

* delete irrelevant test
2025-04-16 11:01:09 +01:00
nicktrn 3f97cf4a6b Managed run controller revamp (#1927)
* update nypm to support text-based bun lockfiles

* fix retry spans

* only download debug logs if admin

* add nypm changeset

* pull out env override logic

* use runner env gather helper

* handle dev flushing failures gracefully

* fix path normalization for init.ts

* add logger

* add execution heartbeat service

* add snapshot poller service

* fix poller

* add changesets

* create socket in constructor

* enable strictPropertyInitialization

* deprecate dequeue from version

* start is not async

* dependency injection in prep for tests

* add warm start count to all controller logs

* add restore count

* pull out run execution logic

* temp disable pre

* add a controller log when starting an execution

* refactor execution and squash some bugs

* cleanup completed docker containers by default

* execution fixes and logging improvements

* don't throw afet abort cleanup

* poller should use private interval

* rename heartbeat service file

* rename HeartbeatService to IntervalService

* restore old heartbeat service but deprecate it

* use the new interval service everywhere

* Revert "temp disable pre"

This reverts commit e03f4179de6a731c17253b68a6e00bcb7ac1736b.

* add changeset

* replace all run engine find uniques with find first
2025-04-15 16:08:51 +01:00
nicktrn 1816115f0f Fix current worker deployment getter (#1924)
* only return last v1 deployment in the shared queue consumer

* be explicit about only returning managed deployments
2025-04-14 14:55:27 +01:00
nicktrn faf5d01b2c Both run engines will only lock to versions they can handle (#1922)
* run engine v1 will only lock to v1 deployments

* run engine v2 will only lock to managed v2 deployments

* test: create background worker and deployment with correct engine version
2025-04-13 21:51:42 +01:00
nicktrn ebb71932f9 snapshot status should remain EXECUTING for short retry delays (#1902) 2025-04-09 13:23:39 +01:00
nicktrn 72d98778eb Fix node options and OOM retries (#1897)
* don't hardcode max old space size in deployment images

* flags: treat underscores as hyphens

* append attempt number to runner name if >1

* improve retry spans for oom errors
2025-04-08 13:07:45 +01:00
Matt Aitken 56a880f1bd Fix dequeue snapshot batch ids (#1893)
* Pass the batchId through from previous snapshots when dequeuing

* Fix and test for resuming batch after checkpoint
2025-04-07 17:22:56 +01:00
Matt Aitken fc0a331c42 We weren’t copying the batch info when suspending a run (#1890) 2025-04-07 16:00:07 +01:00
Matt Aitken 32d2992802 Security update of nanoid from 3.3.4 to 3.3.8 (#1887) 2025-04-07 13:56:37 +01:00
Eric Allam e7bca6fa1b fix: release concurrency system only consumes tokens when releasings are executed successfully (#1883)
* fix: release concurrency system only consumes tokens when releasings are executed successfully

* Add release concurrency metrics sending to otel
2025-04-04 17:20:33 +01:00
Eric Allam 7b9ab1781a re2: add ability to selectively disable release concurrency queue consumers via env var (#1881)
* run engine: add ability to selectively disable release concurrency queue consumers via env var

Also added some additional logging

* actually pass the disableConsumers option through to the release concurrency system

* Fixed CLI e2e failing tests
2025-04-04 12:28:39 +01:00
Eric Allam e539c7c941 v4: allow disabling of the run engine worker via the RUN_ENGINE_WORKER_ENABLED env var (#1865) 2025-04-02 11:39:55 +01:00
Eric Allam 0e82698548 v4: Waitpoint PAT completion, fix dequeuing without retry config, fixed metadata (#1826)
* WIP

* Completing waitpoints using public access tokens

* fixed example

* adding agent-loops reference, adding useWaitToken react hook, adding a way to throw an error to output from the run, adding an icon option to logger.trace

* Fixed metadata system

* Check if the waitpoint is COMPLETED already in the complete endpoint and return true
2025-03-27 10:04:12 +00:00
Matt Aitken 00586ffaaf Waitpoint tokens page, wait.listTokens() and wait.retrieveToken() (#1824)
* Added waitpoints/tokens to the sidebar

* Added indexes to the Waitpoint time for filtering

* Begun work on `WaitpointTokenListPresenter`, the pag is a copy of the Queues page for now

* MVP of waitpoint token page

* Added status

* Expiry of timeout/ttl

* Improvements to the waitpoint table

* Improved columns and icon

* Changes from the RunTag copy on hover branch

* Fix for nested button error

* Added waitpoint tags to the DB/table

* Applied Eric’s task run tag fix (it’s live on prod in the legacy run engine branch)

* Added tags to waitpoints

* Removed todos that have been done

* Added token support for releaseConcurrency. Also added a ton of JSDocs

* Added releaseConcurrency to the API token endpoint…

* WIP on waitpoint page filters

* Fix for tags filtering

* Waitpoint filters working

* Fix for badly named function

* WaitpointPresenter used from SpanPresenter

* Waitpoint detail panel WIP

* Fix for server client hydration issue with CodeBlock

* Selected waitpoint panel

* Added a blank state

* Added waitpoint docs link

* Fix for animated number going past the target

* Fix for the queue list pagination and upgrade status

* Engine version error for waitpoint token list

* RunTag component doesn’t get squished and hover behaviour is nicer

* Associating runs with waitpoints

* Added triggered icon

* Link directly to the waitpoint

* Fix for TS error on waitpoint retrieve

* Added CopyableText component, used for waitpoint id in the table

* Removed the confetti 🎊

* Deleted some old images

* Moved some schemas/types to core. Use `id` instead of `friendlyId`

* Added wait.listTokens() function. Made some changes to the types to make it nicer

* WIP wait.retrieveToken()

* wait.retrieveToken working

* Added data to retrieve token

* Separate ApiWaitpointPresenter completely

* Added completed time to the waitpoint detail panel

* Fix for the Avatar component having SSR issues. Specify the size in rems and removed the useLayoutEffect

* Fix for applied idempotency key filter dropdown showing the id field

* Use parentheses to make sure the token list query respects idempotency key correctly

* Use the proper logger, and have a decent message with info to track the bug down

* Pass the org title into the Avatar

* Better error when failing to creating a manual waitpoint after X attempts
2025-03-26 18:05:26 +00:00
Eric Allam d6fd7f559e Run engine: Using root queue timestamp to prioritize completing runs (#1818)
* Improve the new run engine Trigger/Batch trigger service class names

* centralize queue timestamp logic in EnqueueSystem, adding queueTimestamp support and propagation

* Fixing the create tags in the new run engine trigger task service
2025-03-26 10:25:49 +00:00
Eric Allam 7f804f381c middleware working and tests passing 2025-03-25 15:55:42 +00:00
Eric Allam f1d5886c11 Another attempt at getting CI to pass tests 2025-03-21 15:50:51 +00:00
Eric Allam 941e03b3eb Using the new PENDING_VERSION status now in the UI 2025-03-21 15:06:24 +00:00
Eric Allam 7e411ac162 New PENDING_VERSION system which now requires queues to exist at dequeue time 2025-03-21 15:06:24 +00:00
Eric Allam 7a58439728 WIP queue indexing 2025-03-21 15:06:24 +00:00
nicktrn 49a3f72e13 Publish redis-worker and add graceful shutdown manager (#1810)
* add shutdown manager

* update ai test instructions

* add shutdown timeout to redis-worker

* move redis worker to packages

* add unregister method

* prep for publishing package

* fix types

* update ai files

* fix cursor terminal links

* prevent overly friendly ids

* use structured logger

* use unique shutdown handler names

* rework suspend completion

* add trycatch util

* rework suspend restore

* add http server metrics

* add missing prom-client to core

* add prom metrics to redis worker

* bundle redis-worker

* fix esm/cjs interop

* remove proxy from changeset ignore and add supervisor

* add pause to prerelease script for any manual edits

* unregister the correct handler and add early detection

* small change to http handler return

* fix worker tests

* fix shutdown manager tests
2025-03-21 14:53:27 +00:00
Eric Allam e5ea9cb2d7 move the release concurrency queue into the release concurrency system, make it disabled by default, configure the run engine in the webapp with env vars 2025-03-19 17:55:29 +00:00
Eric Allam 28b3ed0496 Implement release concurrency system 2025-03-19 17:55:29 +00:00
Eric Allam 7eaf81abbe Use releaserId in case we don't end up using run IDs 2025-03-19 17:54:47 +00:00
Eric Allam f9c7e95981 Added updated execution states chart and updated readme 2025-03-19 17:54:47 +00:00
Eric Allam b170a62671 fixed the create checkpoint valid snapshot logic 2025-03-19 17:54:47 +00:00
Eric Allam e7c8f94447 implement the QUEUED_EXECUTING dequeuing, and creating a checkpoint while the run is in QUEUED_EXECUTING state by saving the EXECUTING_WITH_WAITPOINTS snapshotId as the previousSnapshotId on the QUEUED_EXECUTING snapshot 2025-03-19 17:54:47 +00:00
Eric Allam 29371e950b Implement checkpoint tests, handle dequeuing QUEUED_EXECUTING runs 2025-03-19 17:54:47 +00:00
Eric Allam 42fb5d05ff Fix more tests 2025-03-19 17:54:47 +00:00
Eric Allam 67d74f066c More tests passing, fixed the heartbeat issue 2025-03-19 17:54:47 +00:00
Eric Allam 7866e95636 waiting for worker system 2025-03-19 17:54:47 +00:00
Eric Allam a4581f1597 ttl system 2025-03-19 17:54:47 +00:00
Eric Allam 363f0668b5 Delayed run system 2025-03-19 17:54:47 +00:00
Eric Allam 63b43ff357 more system work 2025-03-19 17:54:47 +00:00
Eric Allam 717cec879b move startRunAttempt to RunAttemptSystem 2025-03-19 17:54:47 +00:00
Eric Allam 1b61b95c57 WIP run engine systems 2025-03-19 17:54:47 +00:00
Eric Allam 7d11e827d9 Remove reserve concurrency system from run engine 2025-03-19 17:54:47 +00:00