Commit Graph

104 Commits

Author SHA1 Message Date
nicktrn ebb71932f9 snapshot status should remain EXECUTING for short retry delays (#1902) 2025-04-09 13:23:39 +01:00
nicktrn 72d98778eb Fix node options and OOM retries (#1897)
* don't hardcode max old space size in deployment images

* flags: treat underscores as hyphens

* append attempt number to runner name if >1

* improve retry spans for oom errors
2025-04-08 13:07:45 +01:00
Matt Aitken 56a880f1bd Fix dequeue snapshot batch ids (#1893)
* Pass the batchId through from previous snapshots when dequeuing

* Fix and test for resuming batch after checkpoint
2025-04-07 17:22:56 +01:00
Matt Aitken fc0a331c42 We weren’t copying the batch info when suspending a run (#1890) 2025-04-07 16:00:07 +01:00
Matt Aitken 32d2992802 Security update of nanoid from 3.3.4 to 3.3.8 (#1887) 2025-04-07 13:56:37 +01:00
Eric Allam e7bca6fa1b fix: release concurrency system only consumes tokens when releasings are executed successfully (#1883)
* fix: release concurrency system only consumes tokens when releasings are executed successfully

* Add release concurrency metrics sending to otel
2025-04-04 17:20:33 +01:00
Eric Allam 7b9ab1781a re2: add ability to selectively disable release concurrency queue consumers via env var (#1881)
* run engine: add ability to selectively disable release concurrency queue consumers via env var

Also added some additional logging

* actually pass the disableConsumers option through to the release concurrency system

* Fixed CLI e2e failing tests
2025-04-04 12:28:39 +01:00
Eric Allam e539c7c941 v4: allow disabling of the run engine worker via the RUN_ENGINE_WORKER_ENABLED env var (#1865) 2025-04-02 11:39:55 +01:00
Eric Allam 31bef8d7cd fix: update migration with the "Drop/Add" index of SecretStore key (#1864) 2025-04-02 10:43:02 +01:00
nicktrn bed834b589 Misc v4 checkpoint fixes (#1859)
* logs for optional services

* print env vars on startup in debug mode

* routes need to explicitly ask to keep connection alive

* log indicators for now

* make workload api listen host configurable

* expose supervisor metrics and make more configurable

* configurable pull secrets, no defaults

* remove restore route

* run controller to handle queued executing

* fix v3 deploys in v4 project

* update admin worker route

* only start pod cleaner et al in k8s mode

* set new worker group as default if none yet

* make image ref optional

* checkpoint image ref is optional for output as well

* export feature flag const

* one last image ref type fix

* make runner intervals configurable

* ability to set arbitrary env vars on new runners

* set default runtime back to node 21

* move all runner env vars to the same section
2025-04-01 10:08:15 +01:00
Matt Aitken 174484fb32 Added default time period to Runs, Waitpoint and Batch list pages. Improved dev presence (#1832)
* New time period filter (permanently displayed)

* Batches, and fix for blank state

* Waitpoint token filtering

* Tags query: remove things we’re not using

* Run tag and waitpoint tags use startsWith for faster search

* Fix for run page on wrong env. Added schedule last triggered column

* Removed the Redis pubsub, just use the presence key

* Improve the dev presence responsiveness

* The CLI presence connection recovers when the webapp is restarted

* Dev schedules are now working for v4

* Refactored to make the dev presence stuff

* Got rid of stupid extra /dev, added a connecting state with icon

* Remove unused Redis client
2025-03-28 09:32:42 +00:00
Eric Allam 0e82698548 v4: Waitpoint PAT completion, fix dequeuing without retry config, fixed metadata (#1826)
* WIP

* Completing waitpoints using public access tokens

* fixed example

* adding agent-loops reference, adding useWaitToken react hook, adding a way to throw an error to output from the run, adding an icon option to logger.trace

* Fixed metadata system

* Check if the waitpoint is COMPLETED already in the complete endpoint and return true
2025-03-27 10:04:12 +00:00
Matt Aitken 00586ffaaf Waitpoint tokens page, wait.listTokens() and wait.retrieveToken() (#1824)
* Added waitpoints/tokens to the sidebar

* Added indexes to the Waitpoint time for filtering

* Begun work on `WaitpointTokenListPresenter`, the pag is a copy of the Queues page for now

* MVP of waitpoint token page

* Added status

* Expiry of timeout/ttl

* Improvements to the waitpoint table

* Improved columns and icon

* Changes from the RunTag copy on hover branch

* Fix for nested button error

* Added waitpoint tags to the DB/table

* Applied Eric’s task run tag fix (it’s live on prod in the legacy run engine branch)

* Added tags to waitpoints

* Removed todos that have been done

* Added token support for releaseConcurrency. Also added a ton of JSDocs

* Added releaseConcurrency to the API token endpoint…

* WIP on waitpoint page filters

* Fix for tags filtering

* Waitpoint filters working

* Fix for badly named function

* WaitpointPresenter used from SpanPresenter

* Waitpoint detail panel WIP

* Fix for server client hydration issue with CodeBlock

* Selected waitpoint panel

* Added a blank state

* Added waitpoint docs link

* Fix for animated number going past the target

* Fix for the queue list pagination and upgrade status

* Engine version error for waitpoint token list

* RunTag component doesn’t get squished and hover behaviour is nicer

* Associating runs with waitpoints

* Added triggered icon

* Link directly to the waitpoint

* Fix for TS error on waitpoint retrieve

* Added CopyableText component, used for waitpoint id in the table

* Removed the confetti 🎊

* Deleted some old images

* Moved some schemas/types to core. Use `id` instead of `friendlyId`

* Added wait.listTokens() function. Made some changes to the types to make it nicer

* WIP wait.retrieveToken()

* wait.retrieveToken working

* Added data to retrieve token

* Separate ApiWaitpointPresenter completely

* Added completed time to the waitpoint detail panel

* Fix for the Avatar component having SSR issues. Specify the size in rems and removed the useLayoutEffect

* Fix for applied idempotency key filter dropdown showing the id field

* Use parentheses to make sure the token list query respects idempotency key correctly

* Use the proper logger, and have a decent message with info to track the bug down

* Pass the org title into the Avatar

* Better error when failing to creating a manual waitpoint after X attempts
2025-03-26 18:05:26 +00:00
Eric Allam d6fd7f559e Run engine: Using root queue timestamp to prioritize completing runs (#1818)
* Improve the new run engine Trigger/Batch trigger service class names

* centralize queue timestamp logic in EnqueueSystem, adding queueTimestamp support and propagation

* Fixing the create tags in the new run engine trigger task service
2025-03-26 10:25:49 +00:00
Eric Allam 7f804f381c middleware working and tests passing 2025-03-25 15:55:42 +00:00
Eric Allam f1d5886c11 Another attempt at getting CI to pass tests 2025-03-21 15:50:51 +00:00
Eric Allam de5e0c8e8b try to fix flaky tests 2025-03-21 15:06:55 +00:00
Eric Allam 02eeefd647 Increase maxRetriesPerRequest in CI 2025-03-21 15:06:24 +00:00
Eric Allam 941e03b3eb Using the new PENDING_VERSION status now in the UI 2025-03-21 15:06:24 +00:00
Eric Allam 7e411ac162 New PENDING_VERSION system which now requires queues to exist at dequeue time 2025-03-21 15:06:24 +00:00
Eric Allam 7a58439728 WIP queue indexing 2025-03-21 15:06:24 +00:00
nicktrn 49a3f72e13 Publish redis-worker and add graceful shutdown manager (#1810)
* add shutdown manager

* update ai test instructions

* add shutdown timeout to redis-worker

* move redis worker to packages

* add unregister method

* prep for publishing package

* fix types

* update ai files

* fix cursor terminal links

* prevent overly friendly ids

* use structured logger

* use unique shutdown handler names

* rework suspend completion

* add trycatch util

* rework suspend restore

* add http server metrics

* add missing prom-client to core

* add prom metrics to redis worker

* bundle redis-worker

* fix esm/cjs interop

* remove proxy from changeset ignore and add supervisor

* add pause to prerelease script for any manual edits

* unregister the correct handler and add early detection

* small change to http handler return

* fix worker tests

* fix shutdown manager tests
2025-03-21 14:53:27 +00:00
Eric Allam 38e1887495 Fixed failing redis worker tests 2025-03-19 17:58:53 +00:00
Eric Allam e5ea9cb2d7 move the release concurrency queue into the release concurrency system, make it disabled by default, configure the run engine in the webapp with env vars 2025-03-19 17:55:29 +00:00
Eric Allam 28b3ed0496 Implement release concurrency system 2025-03-19 17:55:29 +00:00
Eric Allam 7eaf81abbe Use releaserId in case we don't end up using run IDs 2025-03-19 17:54:47 +00:00
Eric Allam f9c7e95981 Added updated execution states chart and updated readme 2025-03-19 17:54:47 +00:00
Eric Allam b170a62671 fixed the create checkpoint valid snapshot logic 2025-03-19 17:54:47 +00:00
Eric Allam e7c8f94447 implement the QUEUED_EXECUTING dequeuing, and creating a checkpoint while the run is in QUEUED_EXECUTING state by saving the EXECUTING_WITH_WAITPOINTS snapshotId as the previousSnapshotId on the QUEUED_EXECUTING snapshot 2025-03-19 17:54:47 +00:00
Eric Allam 29371e950b Implement checkpoint tests, handle dequeuing QUEUED_EXECUTING runs 2025-03-19 17:54:47 +00:00
Eric Allam 42fb5d05ff Fix more tests 2025-03-19 17:54:47 +00:00
Eric Allam 67d74f066c More tests passing, fixed the heartbeat issue 2025-03-19 17:54:47 +00:00
Eric Allam 7866e95636 waiting for worker system 2025-03-19 17:54:47 +00:00
Eric Allam a4581f1597 ttl system 2025-03-19 17:54:47 +00:00
Eric Allam 363f0668b5 Delayed run system 2025-03-19 17:54:47 +00:00
Eric Allam 63b43ff357 more system work 2025-03-19 17:54:47 +00:00
Eric Allam 717cec879b move startRunAttempt to RunAttemptSystem 2025-03-19 17:54:47 +00:00
Eric Allam 1b61b95c57 WIP run engine systems 2025-03-19 17:54:47 +00:00
Eric Allam 7d11e827d9 Remove reserve concurrency system from run engine 2025-03-19 17:54:47 +00:00
Eric Allam 8c66ec3189 Remove reserve concurrency system from run queue 2025-03-19 17:54:47 +00:00
Eric Allam 063651c7b8 Upgrade vitest in the run-engine package 2025-03-19 17:54:47 +00:00
Eric Allam 2048b72288 go to QUEUE_EXECUTING state if reacquiring concurrency doesn't work 2025-03-19 17:54:47 +00:00
Eric Allam 61c0834aae Implement reserve concurrency clearing when the child run is acked 2025-03-19 17:54:47 +00:00
Eric Allam fd9b0bf676 WIP new reacquire concurrency system 2025-03-19 17:54:47 +00:00
Eric Allam cf3b23862e remove project and task current concurrency tracking 2025-03-19 17:54:47 +00:00
Eric Allam de9e296551 Add configuration for the release concurrency queue 2025-03-19 17:54:47 +00:00
Eric Allam cce402dfe7 If max tokens is 0, then don't do releasings 2025-03-19 17:54:47 +00:00
Eric Allam b7ddf20d62 Correctly use the new release concurrency queue in the run engine 2025-03-19 17:54:47 +00:00
Eric Allam bf41703dfd improve the consumer of the concurrency queue 2025-03-19 17:54:47 +00:00
Eric Allam c8b99728b9 Get all the release concurrency queue tests passing 2025-03-19 17:54:47 +00:00