Commit Graph

237 Commits

Author SHA1 Message Date
Eric Allam 02eeefd647 Increase maxRetriesPerRequest in CI 2025-03-21 15:06:24 +00:00
Eric Allam 941e03b3eb Using the new PENDING_VERSION status now in the UI 2025-03-21 15:06:24 +00:00
Eric Allam 7e411ac162 New PENDING_VERSION system which now requires queues to exist at dequeue time 2025-03-21 15:06:24 +00:00
Eric Allam 7a58439728 WIP queue indexing 2025-03-21 15:06:24 +00:00
nicktrn 49a3f72e13 Publish redis-worker and add graceful shutdown manager (#1810)
* add shutdown manager

* update ai test instructions

* add shutdown timeout to redis-worker

* move redis worker to packages

* add unregister method

* prep for publishing package

* fix types

* update ai files

* fix cursor terminal links

* prevent overly friendly ids

* use structured logger

* use unique shutdown handler names

* rework suspend completion

* add trycatch util

* rework suspend restore

* add http server metrics

* add missing prom-client to core

* add prom metrics to redis worker

* bundle redis-worker

* fix esm/cjs interop

* remove proxy from changeset ignore and add supervisor

* add pause to prerelease script for any manual edits

* unregister the correct handler and add early detection

* small change to http handler return

* fix worker tests

* fix shutdown manager tests
2025-03-21 14:53:27 +00:00
Eric Allam 38e1887495 Fixed failing redis worker tests 2025-03-19 17:58:53 +00:00
Eric Allam e5ea9cb2d7 move the release concurrency queue into the release concurrency system, make it disabled by default, configure the run engine in the webapp with env vars 2025-03-19 17:55:29 +00:00
Eric Allam 28b3ed0496 Implement release concurrency system 2025-03-19 17:55:29 +00:00
Eric Allam 7eaf81abbe Use releaserId in case we don't end up using run IDs 2025-03-19 17:54:47 +00:00
Eric Allam f9c7e95981 Added updated execution states chart and updated readme 2025-03-19 17:54:47 +00:00
Eric Allam b170a62671 fixed the create checkpoint valid snapshot logic 2025-03-19 17:54:47 +00:00
Eric Allam e7c8f94447 implement the QUEUED_EXECUTING dequeuing, and creating a checkpoint while the run is in QUEUED_EXECUTING state by saving the EXECUTING_WITH_WAITPOINTS snapshotId as the previousSnapshotId on the QUEUED_EXECUTING snapshot 2025-03-19 17:54:47 +00:00
Eric Allam 29371e950b Implement checkpoint tests, handle dequeuing QUEUED_EXECUTING runs 2025-03-19 17:54:47 +00:00
Eric Allam 42fb5d05ff Fix more tests 2025-03-19 17:54:47 +00:00
Eric Allam 67d74f066c More tests passing, fixed the heartbeat issue 2025-03-19 17:54:47 +00:00
Eric Allam 7866e95636 waiting for worker system 2025-03-19 17:54:47 +00:00
Eric Allam a4581f1597 ttl system 2025-03-19 17:54:47 +00:00
Eric Allam 363f0668b5 Delayed run system 2025-03-19 17:54:47 +00:00
Eric Allam 63b43ff357 more system work 2025-03-19 17:54:47 +00:00
Eric Allam 717cec879b move startRunAttempt to RunAttemptSystem 2025-03-19 17:54:47 +00:00
Eric Allam 1b61b95c57 WIP run engine systems 2025-03-19 17:54:47 +00:00
Eric Allam 7d11e827d9 Remove reserve concurrency system from run engine 2025-03-19 17:54:47 +00:00
Eric Allam 8c66ec3189 Remove reserve concurrency system from run queue 2025-03-19 17:54:47 +00:00
Eric Allam 063651c7b8 Upgrade vitest in the run-engine package 2025-03-19 17:54:47 +00:00
Eric Allam 2048b72288 go to QUEUE_EXECUTING state if reacquiring concurrency doesn't work 2025-03-19 17:54:47 +00:00
Eric Allam 61c0834aae Implement reserve concurrency clearing when the child run is acked 2025-03-19 17:54:47 +00:00
Eric Allam fd9b0bf676 WIP new reacquire concurrency system 2025-03-19 17:54:47 +00:00
Eric Allam cf3b23862e remove project and task current concurrency tracking 2025-03-19 17:54:47 +00:00
Eric Allam de9e296551 Add configuration for the release concurrency queue 2025-03-19 17:54:47 +00:00
Eric Allam cce402dfe7 If max tokens is 0, then don't do releasings 2025-03-19 17:54:47 +00:00
Eric Allam b7ddf20d62 Correctly use the new release concurrency queue in the run engine 2025-03-19 17:54:47 +00:00
Eric Allam bf41703dfd improve the consumer of the concurrency queue 2025-03-19 17:54:47 +00:00
Eric Allam c8b99728b9 Get all the release concurrency queue tests passing 2025-03-19 17:54:47 +00:00
Eric Allam d11d491215 WIP release concurrency queue 2025-03-19 17:54:47 +00:00
Eric Allam 1362d8296d Add batchId to the delayed enqueueRun call 2025-03-19 17:54:47 +00:00
Eric Allam 2b6ce169be remove unused method 2025-03-19 17:54:47 +00:00
Eric Allam e9fa4ce3a2 Update delays to use a redis worker and work with the new reserve concurrency system 2025-03-19 17:54:47 +00:00
Eric Allam 1a233efd42 the run engine now works with the new reserve concurrency system 2025-03-19 17:54:47 +00:00
Eric Allam 555d9912ec fixed some key producer tests 2025-03-19 17:54:47 +00:00
Eric Allam 880538ff92 improve the dead letter queue stuff 2025-03-19 17:54:47 +00:00
Eric Allam a5b2b39201 ack 2025-03-19 17:54:47 +00:00
Eric Allam 16546f144f dequeue message 2025-03-19 17:54:47 +00:00
Eric Allam 8b3551c9e2 wip reserve concurrency system 2025-03-19 17:54:47 +00:00
Matt Aitken ec4511b629 Add paused status to RuntimeEnvironment and TaskQueue 2025-03-18 09:11:39 +00:00
Matt Aitken d5ee760b9d Get values for the queue and running 2025-03-17 18:49:19 +00:00
Matt Aitken 7b862b9438 Major dashboard improvements (environment centric) (#1796)
* Delete the proxy app (was v2)

* Delete RunPresenterElectric

* Select the best proj/org/env

* Storing current proj/env in DB. Initial selection logic working with tasks page

* 2sm needed to be in the Tailwind merge list

* Move the task stream route (although we don’t actually use the env for now)

* Alerts moved from /v3

* API keys page moved from /v3

* Concurrency page moved from /v3

* WIP on side menu sections

* Improved the accordion animation

* Moved schedules from /v3

* More pages moved

* Move pages working

* Run page working

* Schedules working

* Moved deployments

* Alert pages moved

* Delete electric hooks, not used

* Started setting up blank states

* Test page working

* Removed “Select task” from the test page

* Some work on deployment page

* Style tweaks

* Redirect from project root to approriate env

* Improved env selector styling

* Fix for jsx errors

* Better min width on env selector

* Improved the env switching logic

* Added deployments to env routing

* Redirect deployments to the correct env

* Redirect run from proj to env

* JSX icon fix

* Only allow single env schedules from now on

* Remove env var count from the API keys page

* Move improvements and redirects

* Project settings moved

* Fix for scroll area on test page

* Tweaked the test design

* Made recent payloads column narrower

* Improved the test layout some more

* Added org icon, new project selector menu

* WIP on org switching menu

* Org switching is working

* New menu working well, removed old side menu items

* Buttons can now have a component name or an actual component for their icons

* Removed the Projects page, instead redirect appropriately

* Fix for broken blank states

* Minor run table improvements

* Removed unused switcher log and logic

* Concurrency page fix for invalid html, improved layout

* Minor improvements

* Moved the side menu to the project level

* Improved account styling

* Moved org settings pages (with redirects)

* Add current plan to billing side menu link

* Upgrade to get staging from env dropdown

* New env badge on concurrency limits page

* Show Run Engine version in span presenter

* New promote icon

* Concurrency limits page is the sum of engine v1 + v2 queues

* Fix for missing batch import

* Added currentConcurrencyOfEnvQueue function

* Basic avatar setting working

* Avatar setting is working

* You can change the color of your icon

* Avatar improvements

* Bugfix for mising prop

* Removed some old env badges

* Fixed replaying

* Removed EnvironmentLabel

* Old env badge deleted, changed everywhere to the new one

* Fix for Slack integration paths

* Fix for waitpoint completion form moving

* Bulk replay/cancel env fix

* Fix for alert webhook path

* Redirect projects/v3/* to project/*

* Fixes for CLI redirect routes

* Remove welcome email (unused)

* Change how we count schedules towards your limits

* Use new schedules limits when checking a schedule

* Added projectId back in to task queries (indexes)

* WIP dev presence

* CLI modal

* Moved things around and use Context

* Fix for p inside p

* Dev connected status on run page

* Correct dev env (not a teammates)

* Show disconnected message at the end

* Minor tweak on project dropdown icon padding

* Fix for inconsistent date format for presence

* Added a message when pushing to the billing page

* Center the team page

* Project settings page centered

* Improvements to the dev presence
2025-03-17 13:32:31 +00:00
Eric Allam d855d55ea0 re2: env based queue selection algo (#1775)
* re2: fix @trigger.dev/core exports

* re2: WIP env based queue selection algo

* more wip

* WIP

* Get run engine tests to pass

* Adding tests for the fair dequeueing strat in the run engine

* Configure the new queue selection strategy in the webapp and get it all building and typechecks passing

* webapp now uses built packages, building redis-worker, run-engine, database, using better tsconfig setups for tests, moving isomorphic code into core/v3/isomorphic

* Fixed webapp typechecks

* dev now depends on build, fixed supervisor typecheck

* Fixed run engine tests

* Fixed e2e tests
2025-03-07 14:30:19 +00:00
Matt Aitken 4dbf11aed1 Run Engine 2: More robust attempt failing/retrying (inc. OOM retrying) (#1773)
* Added describe to tests that were missing it

* Added a function to get the maxOldSpaceSize

* Make it easy to take `NODE_OPTIONS` and set the old space flag

* Added a zed task to rebuild the packages

* Moved isOOMRunError and added SIBABRT condition

* Deduplication flags function with tests

* Export flags file

* On TaskRunProcess, set max old space and deduplicate the flags with priority order

* Move retrying logic to a separate function, it was getting very messy

* Created new test file for attempt failures

* Allow setting retry settings for tests

* Some retrying tests, including OOM

* More failure condition tests

* Fix for OOM retrying

* Complete the attempt span if it was an OOM error

* Remove old broken import

* Fixed order of exports

* Fix for docker-provider checkpoints import
2025-03-07 09:07:49 +00:00
Eric Allam cf4c4d7b65 Add attempt metrics in dev (prod WIP). Added max concurrent runs setting to dev using p-limit (#1766) 2025-03-06 11:57:01 +00:00
Matt Aitken 722fae566c Reschedule heartbeats and added a test to check it works (#1765) 2025-03-06 10:44:53 +00:00