Commit Graph

14 Commits

Author SHA1 Message Date
nicktrn 72d98778eb Fix node options and OOM retries (#1897)
* don't hardcode max old space size in deployment images

* flags: treat underscores as hyphens

* append attempt number to runner name if >1

* improve retry spans for oom errors
2025-04-08 13:07:45 +01:00
nicktrn 58567bfe44 Add v4 timeline metrics to prod runs (#1884)
* set correct worker name on create

* dev: kill run process on failed cancel attempt

* add missing prod timeline events

* add more debug logs after api call failures

* import events are always visible

* set dequeued at on new runners

* skip fork event test

* fix timeline test for import event
2025-04-04 18:39:13 +01:00
nicktrn 7644d92efa Fix restored runner ID and various small enhancements (#1880)
* make worker nodetype configurable

* make max dequeue count configurable from supervisor

* deterministic runner ids

* update runner id on restore

* disable resource monitor

* explicit run controller type imports

* enhance debug logs
2025-04-04 12:35:23 +01:00
nicktrn dcacc3a9e2 Fixes for v4 waits and restores (#1868)
* remove dummy metrics from heartbeat

* fix heartbeat timeouts..

* reset clock when resolving waitpoints

* optionally set metadata url when scheduling run

* don't use global vitest as it's outdated

* don't run test files in parallel

* handle special graceful shutdown code

* exit codes for success and failure can now be set and overridden

* ensure immediate cleanup in tests

* update lockfile after conflict
2025-04-02 15:27:01 +01:00
nicktrn bed834b589 Misc v4 checkpoint fixes (#1859)
* logs for optional services

* print env vars on startup in debug mode

* routes need to explicitly ask to keep connection alive

* log indicators for now

* make workload api listen host configurable

* expose supervisor metrics and make more configurable

* configurable pull secrets, no defaults

* remove restore route

* run controller to handle queued executing

* fix v3 deploys in v4 project

* update admin worker route

* only start pod cleaner et al in k8s mode

* set new worker group as default if none yet

* make image ref optional

* checkpoint image ref is optional for output as well

* export feature flag const

* one last image ref type fix

* make runner intervals configurable

* ability to set arbitrary env vars on new runners

* set default runtime back to node 21

* move all runner env vars to the same section
2025-04-01 10:08:15 +01:00
nicktrn 394f1decd3 Misc v4 fixes and improved logging (#1831)
* logs for optional services

* print env vars on startup in debug mode

* routes need to explicitly ask to keep connection alive

* log indicators for now

* make workload api listen host configurable

* expose supervisor metrics and make more configurable

* configurable pull secrets, no defaults

* remove restore route

* run controller to handle queued executing

* fix v3 deploys in v4 project

* update admin worker route

* only start pod cleaner et al in k8s mode

* set new worker group as default if none yet
2025-03-31 14:06:17 +01:00
nicktrn 4fe1d49f94 Add v4 pod lifecycle handlers (#1819)
* patch k8s client to allow field selector on informers

* add pod cleaner and tests

* add failed pod handler and tests

* remove supervisor test script for now
2025-03-26 10:22:24 +00:00
nicktrn 49a3f72e13 Publish redis-worker and add graceful shutdown manager (#1810)
* add shutdown manager

* update ai test instructions

* add shutdown timeout to redis-worker

* move redis worker to packages

* add unregister method

* prep for publishing package

* fix types

* update ai files

* fix cursor terminal links

* prevent overly friendly ids

* use structured logger

* use unique shutdown handler names

* rework suspend completion

* add trycatch util

* rework suspend restore

* add http server metrics

* add missing prom-client to core

* add prom metrics to redis worker

* bundle redis-worker

* fix esm/cjs interop

* remove proxy from changeset ignore and add supervisor

* add pause to prerelease script for any manual edits

* unregister the correct handler and add early detection

* small change to http handler return

* fix worker tests

* fix shutdown manager tests
2025-03-21 14:53:27 +00:00
nicktrn 7842e9dff0 Warm start and restore improvements (#1793)
* handle warm start service failure on supervisor side

* export zodfetch wrapper

* add warm start client

* rename to keepalive

* add missing warm start header

* make heartbeat and snapshot poll interval configurable

* create warm start client in constructor

* add warm start run debug log

* re-enable checkpoints and improve error messages

* reduce run pod container name cardinality

* move runner id generator into core

* fix runner id import

* log when no checkpoint client and we try to restore

* move run controller constructor to the top

* fix import

* support env overrides after restore
2025-03-14 18:37:53 +00:00
nicktrn 7f2569721e Fix supervisor builds (#1790)
* fix and restructure dockerignore

* switch to using pnpm deploy

* pass webapp node image as build arg

* ensure pnpm is downloaded at build, not runtime
2025-03-12 16:10:37 +00:00
Eric Allam d855d55ea0 re2: env based queue selection algo (#1775)
* re2: fix @trigger.dev/core exports

* re2: WIP env based queue selection algo

* more wip

* WIP

* Get run engine tests to pass

* Adding tests for the fair dequeueing strat in the run engine

* Configure the new queue selection strategy in the webapp and get it all building and typechecks passing

* webapp now uses built packages, building redis-worker, run-engine, database, using better tsconfig setups for tests, moving isomorphic code into core/v3/isomorphic

* Fixed webapp typechecks

* dev now depends on build, fixed supervisor typecheck

* Fixed run engine tests

* Fixed e2e tests
2025-03-07 14:30:19 +00:00
nicktrn 38ddd830d7 Add supervisor split service controls (#1774)
* dockerignore node_modules in subdirectories

* image tag action should handle re2 tags

* add supervisor containerfile

* add publish worker re2 workflow

* fix copypasta

* require branch check

* add more granular service control to supervisor session

* fix supervisor api domain for split setups

* remove default workload api domain

* option to disable workload api

* fix bool env var coercion
2025-03-07 12:40:11 +00:00
nicktrn e297c7f7f0 Add supervisor build pipeline (#1772)
* dockerignore node_modules in subdirectories

* image tag action should handle re2 tags

* add supervisor containerfile

* add publish worker re2 workflow

* fix copypasta

* require branch check
2025-03-06 18:22:36 +00:00
nicktrn 94fb532f5e Add RE2 supervisor (#1769)
* add new core server only export

* exit after run if warm starts disabled

* add supervisor
2025-03-06 13:20:45 +00:00