Commit Graph

381 Commits

Author SHA1 Message Date
github-actions[bot] 64ca9dc9a8 Release 4.0.0-v4-beta.6 (#1951)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.6

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-18 16:50:22 +01:00
nicktrn aa74260646 Dev command engine URL now configurable via the webapp (#1948)
* return dev engine url from config call

* set and use engine url from dev config reply
2025-04-18 11:11:11 +01:00
github-actions[bot] 4a8303212c Release 4.0.0-v4-beta.5 (#1943)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.5

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-04-17 17:25:07 +01:00
github-actions[bot] 59eea654ac Release 4.0.0-v4-beta.4 (#1939)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.4

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
2025-04-17 17:05:52 +01:00
github-actions[bot] 543540581d Release 4.0.0-v4-beta.3 (#1932)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.3

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-16 19:57:42 +01:00
nicktrn 8b182ce8cb Fix stalled run detection (#1934)
* drive run heartbeats from child process

* track last heartbeat

* add changeset

* use v4-specific heartbeat interval env var and decrease to 20s

* delete old heartbeat service

* remove heartbeat handler during execution cleanup
2025-04-16 19:48:51 +01:00
nicktrn 1f6a283ddf Improve usage flushing (#1931)
* add flush to global usage api

* enable controller debug logs

* initialize usage manager after env overrides

* add previous run id to more debug logs

* add changeset
2025-04-16 14:34:20 +01:00
github-actions[bot] bcef9696e8 Release 4.0.0-v4-beta.2 (#1928)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.2

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-15 16:19:04 +01:00
nicktrn 3f97cf4a6b Managed run controller revamp (#1927)
* update nypm to support text-based bun lockfiles

* fix retry spans

* only download debug logs if admin

* add nypm changeset

* pull out env override logic

* use runner env gather helper

* handle dev flushing failures gracefully

* fix path normalization for init.ts

* add logger

* add execution heartbeat service

* add snapshot poller service

* fix poller

* add changesets

* create socket in constructor

* enable strictPropertyInitialization

* deprecate dequeue from version

* start is not async

* dependency injection in prep for tests

* add warm start count to all controller logs

* add restore count

* pull out run execution logic

* temp disable pre

* add a controller log when starting an execution

* refactor execution and squash some bugs

* cleanup completed docker containers by default

* execution fixes and logging improvements

* don't throw afet abort cleanup

* poller should use private interval

* rename heartbeat service file

* rename HeartbeatService to IntervalService

* restore old heartbeat service but deprecate it

* use the new interval service everywhere

* Revert "temp disable pre"

This reverts commit e03f4179de6a731c17253b68a6e00bcb7ac1736b.

* add changeset

* replace all run engine find uniques with find first
2025-04-15 16:08:51 +01:00
github-actions[bot] 1d920d5b3f Release 4.0.0-v4-beta.1 (#1916)
* chore: Update version for release (v4-beta)

* Release 4.0.0-v4-beta.1

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-04-11 20:19:03 +01:00
nicktrn cc553d3392 Fix bun detection, dev flushing, and init command (#1914)
* update nypm to support text-based bun lockfiles

* add nypm changeset

* handle dev flushing failures gracefully

* fix path normalization for init.ts

* add changesets

* chore: remove pre.json after exiting pre mode

* init command to install v4-beta packages

* Revert "chore: remove pre.json after exiting pre mode"

This reverts commit f5694fde9314114c74a220c2213d19667bca1a6c.

* make init default to cli version for all packages
2025-04-11 20:11:13 +01:00
github-actions[bot] b949b5d257 chore: Update version for release (v4-beta) (#1755)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-04-09 16:07:14 +01:00
nicktrn 98a3cfbfa1 Clear runner state of previous executions before warm starts (#1905)
* attach all run controller logs to the run

* make controller-level pre-suspend wait duration configurable

* snapshot status should remain EXECUTING for short retry delays

* add suspended process error

* ensure clean slate before waiting for next run

* treat immediate retries as warm starts

* fix for finished runs waiting forever
2025-04-09 15:53:42 +01:00
nicktrn 72d98778eb Fix node options and OOM retries (#1897)
* don't hardcode max old space size in deployment images

* flags: treat underscores as hyphens

* append attempt number to runner name if >1

* improve retry spans for oom errors
2025-04-08 13:07:45 +01:00
nicktrn 58567bfe44 Add v4 timeline metrics to prod runs (#1884)
* set correct worker name on create

* dev: kill run process on failed cancel attempt

* add missing prod timeline events

* add more debug logs after api call failures

* import events are always visible

* set dequeued at on new runners

* skip fork event test

* fix timeline test for import event
2025-04-04 18:39:13 +01:00
nicktrn 7644d92efa Fix restored runner ID and various small enhancements (#1880)
* make worker nodetype configurable

* make max dequeue count configurable from supervisor

* deterministic runner ids

* update runner id on restore

* disable resource monitor

* explicit run controller type imports

* enhance debug logs
2025-04-04 12:35:23 +01:00
Eric Allam 7b9ab1781a re2: add ability to selectively disable release concurrency queue consumers via env var (#1881)
* run engine: add ability to selectively disable release concurrency queue consumers via env var

Also added some additional logging

* actually pass the disableConsumers option through to the release concurrency system

* Fixed CLI e2e failing tests
2025-04-04 12:28:39 +01:00
Eric Allam 25c1abac8e v4: eagerly fork child processes before warm start initiates (#1879)
* v4: eagerly fork child process before warm start

Also fixes an issue where the attempt span events weren't coming through in the partial spans

* Making flushing a safer operation

* Add changeset

* Improve attempt metrics when in a warm start, and pass the STYLE_VARIANT for a warm or cold start on attempt spans
2025-04-04 11:07:36 +01:00
Eric Allam 18ad897171 fix: silence erroneous prisma package.json warnings (#1875) 2025-04-02 22:39:29 +01:00
nicktrn 7591b6e619 Fix v4 restore race condition (#1870) 2025-04-02 18:05:43 +01:00
nicktrn dcacc3a9e2 Fixes for v4 waits and restores (#1868)
* remove dummy metrics from heartbeat

* fix heartbeat timeouts..

* reset clock when resolving waitpoints

* optionally set metadata url when scheduling run

* don't use global vitest as it's outdated

* don't run test files in parallel

* handle special graceful shutdown code

* exit codes for success and failure can now be set and overridden

* ensure immediate cleanup in tests

* update lockfile after conflict
2025-04-02 15:27:01 +01:00
Eric Allam 51bb3ee464 v4 ai improvements (#1863)
* AI SDK sql generator and approval example WIP

* Create some nice spans and extract the tools out so we can get the type of the streams

* Enable AI SDK telemetry

* Adding ai.tool and removing toolTask, 3rd party telemetry spans now wil create partials, better ai SDK telemetry icons

* Created a separate d3-chat example, split out from the openai-agents example

* Fixed the tool options being passed to metadata in `ai.tool`

* Add a link to the run

* Slightly improved design

* Add a crawler task using crawl4ai

* Use a tool to get the userId

* Couple of tweaks

* Adding markdown rendering to assistant messages and added an e2b based chart rendering task

* Backup to anthropic

* Add changeset
2025-04-02 12:00:02 +01:00
nicktrn bed834b589 Misc v4 checkpoint fixes (#1859)
* logs for optional services

* print env vars on startup in debug mode

* routes need to explicitly ask to keep connection alive

* log indicators for now

* make workload api listen host configurable

* expose supervisor metrics and make more configurable

* configurable pull secrets, no defaults

* remove restore route

* run controller to handle queued executing

* fix v3 deploys in v4 project

* update admin worker route

* only start pod cleaner et al in k8s mode

* set new worker group as default if none yet

* make image ref optional

* checkpoint image ref is optional for output as well

* export feature flag const

* one last image ref type fix

* make runner intervals configurable

* ability to set arbitrary env vars on new runners

* set default runtime back to node 21

* move all runner env vars to the same section
2025-04-01 10:08:15 +01:00
Eric Allam f320e14f38 cli: add dev lock file to prevent 2 dev processes running at the same time in the same dir (#1854)
* cli: add dev lock file to prevent 2 dev processes running at the same time in the same dir

* Make sure the .trigger dir exists before creating the dev.lock file
2025-03-31 16:27:11 +01:00
nicktrn 394f1decd3 Misc v4 fixes and improved logging (#1831)
* logs for optional services

* print env vars on startup in debug mode

* routes need to explicitly ask to keep connection alive

* log indicators for now

* make workload api listen host configurable

* expose supervisor metrics and make more configurable

* configurable pull secrets, no defaults

* remove restore route

* run controller to handle queued executing

* fix v3 deploys in v4 project

* update admin worker route

* only start pod cleaner et al in k8s mode

* set new worker group as default if none yet
2025-03-31 14:06:17 +01:00
Matt Aitken 174484fb32 Added default time period to Runs, Waitpoint and Batch list pages. Improved dev presence (#1832)
* New time period filter (permanently displayed)

* Batches, and fix for blank state

* Waitpoint token filtering

* Tags query: remove things we’re not using

* Run tag and waitpoint tags use startsWith for faster search

* Fix for run page on wrong env. Added schedule last triggered column

* Removed the Redis pubsub, just use the presence key

* Improve the dev presence responsiveness

* The CLI presence connection recovers when the webapp is restarted

* Dev schedules are now working for v4

* Refactored to make the dev presence stuff

* Got rid of stupid extra /dev, added a connecting state with icon

* Remove unused Redis client
2025-03-28 09:32:42 +00:00
Eric Allam 0e82698548 v4: Waitpoint PAT completion, fix dequeuing without retry config, fixed metadata (#1826)
* WIP

* Completing waitpoints using public access tokens

* fixed example

* adding agent-loops reference, adding useWaitToken react hook, adding a way to throw an error to output from the run, adding an icon option to logger.trace

* Fixed metadata system

* Check if the waitpoint is COMPLETED already in the complete endpoint and return true
2025-03-27 10:04:12 +00:00
nicktrn 5f48b9c7f9 Fix worker deployment engine version (#1820)
* enable re2 cli flag by default

* background worker engine determines deployment type
2025-03-26 10:22:13 +00:00
Eric Allam 6d5d10a45a Support other init file names 2025-03-25 16:33:12 +00:00
Eric Allam 3ca5985277 Hooks now all use the new types, and adding some spans 2025-03-25 15:55:42 +00:00
Eric Allam df9cad6667 Get it all working 2025-03-25 15:55:42 +00:00
Eric Allam ba520fcdd8 Make sure and register all the config hooks 2025-03-25 15:55:42 +00:00
Eric Allam 23ae77afff handle max duration timeout errors better 2025-03-25 15:55:42 +00:00
Eric Allam 6ba2b0476b more tests 2025-03-25 15:55:42 +00:00
Eric Allam dc6e659e94 new locals API 2025-03-25 15:55:42 +00:00
Eric Allam 1777ff4cfb move onStart to the new lifecycle hook system 2025-03-25 15:55:42 +00:00
Eric Allam 2c44e59d07 init.ts at the root of the trigger dir is now automatically loaded 2025-03-25 15:55:42 +00:00
Eric Allam 0215a8cfa8 vibes 2025-03-25 15:55:42 +00:00
Eric Allam f59bad9227 Revamping the lifecycle hooks, starting with init 2025-03-25 15:55:42 +00:00
Eric Allam f46a409f0e Fixed typescript error 2025-03-21 15:06:55 +00:00
Eric Allam bee5e75931 Fixed cli e2e tests 2025-03-21 15:06:55 +00:00
Eric Allam cbe651e7e4 Fix useTaskTrigger queue option params and remove exportName from e2e tests 2025-03-21 15:06:24 +00:00
Eric Allam 941e03b3eb Using the new PENDING_VERSION status now in the UI 2025-03-21 15:06:24 +00:00
Eric Allam 1bec4e1432 Getting the queues indexing to work by passing them to the create background worker request bodies 2025-03-21 15:06:24 +00:00
Eric Allam 7a58439728 WIP queue indexing 2025-03-21 15:06:24 +00:00
nicktrn 49a3f72e13 Publish redis-worker and add graceful shutdown manager (#1810)
* add shutdown manager

* update ai test instructions

* add shutdown timeout to redis-worker

* move redis worker to packages

* add unregister method

* prep for publishing package

* fix types

* update ai files

* fix cursor terminal links

* prevent overly friendly ids

* use structured logger

* use unique shutdown handler names

* rework suspend completion

* add trycatch util

* rework suspend restore

* add http server metrics

* add missing prom-client to core

* add prom metrics to redis worker

* bundle redis-worker

* fix esm/cjs interop

* remove proxy from changeset ignore and add supervisor

* add pause to prerelease script for any manual edits

* unregister the correct handler and add early detection

* small change to http handler return

* fix worker tests

* fix shutdown manager tests
2025-03-21 14:53:27 +00:00
Eric Allam fd9b0bf676 WIP new reacquire concurrency system 2025-03-19 17:54:47 +00:00
nicktrn 7842e9dff0 Warm start and restore improvements (#1793)
* handle warm start service failure on supervisor side

* export zodfetch wrapper

* add warm start client

* rename to keepalive

* add missing warm start header

* make heartbeat and snapshot poll interval configurable

* create warm start client in constructor

* add warm start run debug log

* re-enable checkpoints and improve error messages

* reduce run pod container name cardinality

* move runner id generator into core

* fix runner id import

* log when no checkpoint client and we try to restore

* move run controller constructor to the top

* fix import

* support env overrides after restore
2025-03-14 18:37:53 +00:00
saadi 7d65aa9299 Describe the MCP server feature in the readme 2025-03-11 19:17:46 +01:00
saadi 603c903fbe Add MCP tool to list all available tasks and enable fuzzy task matching
This enables LLMs to figure out which task you are referring to, without needing to specify the full task ID
2025-03-11 18:34:17 +01:00