5 Commits

Author	SHA1	Message	Date
nicktrn	14c2bdf89b	v3: checkpoint and reliability improvements (#1198) ... * only checkpoint retries with delays greater than threshold * rename checkpoint threshold env var * log task monitor ignores * crash runs with unbounded attempts * fix retry check in shared queue consumer * add missing stop for env var sync spinner * prod entry point refactor * missing awaits * more verbose prod flush and exit logs * reduce checkpoint support logs * heartbeat while checkpointing between retries * dynamic coordinator config * measure lazy attempt creation time in prod * simplify delay threshold * heartbeat clarifications * crash run if it doesn't reach checkpointable state * require dynamic config threshold * fix retry prep, await previous worker kill * unify wait mechanics * fix prod worker without tasks error * ensure worker is ready to be checkpointed for dependency waits * improve worker attempt creation logging * prevent crashes caused by failed socket schema parsing * fix dynamic imports in v3 catalog * clarify attempt retry mechanics * move backoff helper to core-apps * remove core-apps barrel file * add backoff execute with callback * deprecate non-lazy attempt messages * update socket.io-client to v4.7.5 * fix socket.io types for emits with timeout * retry all the things * remove todo * fix retry restores * improve index failure logs * retry incomplete dependency waits * fix checkpoint in-progress detection * prevent losing messages during reconnect * checkpoint when greater or equal to threshold * improve handling of duration wait edge cases * add ready for lazy attempt replay * retry attempt completion * allow failing runs with unfriendly run id * fix min max jitter * cancel checkpoints on run failure * improve attempt creation errors * prevent crashing run on failed cleanup * handle at-least-once execute lazy attempt delivery * log exit code on prepare for retry * fix timeout promise * mark some things * chaos monkey superpowers * refactor checkpointer * set chaos monkey defaults * less chaos * fix backoff * handle uncaught entry point exceptions * only replay rpcs on true reconnects * allow resume unless final run status * add changeset * small fixes	2024-07-03 16:54:09 +01:00
nicktrn	9ecf07731a	v3: small features and fixes for prod infra (#968) ... * add checkpoint restore events * fix retries.enabledInDev * fix tsconfig paths * fix provider build and dev * update kubernetes provider and fix builds again * upgrade prod base to node 20 * update infra publish workflow * rethrow k8s errors after logging * shorten index container names * fix error type assertion * rename type assertion * remove resource limits for now * add missing run id on create * add push to deploy command for self-hosting * checkpointing fixes * update coordinator image * ensure valid registry login * delete checkpoint archive after successful push * log options on error * structured logs for socket connections * fix structured log merge * exit process after checkpointing * update restore pull secret name * append shortcode to restore names * log handler payload * disable post start lifecycle hook * pass in coordinator host via volume * replace dapi with taskinfo * add missing restore label * don't restart restored containers * remove init container from create * atomic post-completion checkpoints * switch to run id for container names * improve wait accuracy * measure basic checkpoint perf * always log disconnect reason * use system clock to end wait spans * checkpoint readiness and cancel signals * restore from checkpoint events and fix statuses * remove attempt id env var * restore dependencies from events * reconnect wip * lifecycle hooks are back * fix hooks and improve reconnect * make docker send postStart hook * only checkpoint for retry if large delay * fix a few more resume issues * lifecycle hook fixes * skip connection handler when waiting for post start hook --------- Co-authored-by: Eric Allam <eric@trigger.dev>	2024-03-25 16:28:56 +00:00
Eric Allam	478ce006cb	v3: task run status and canceling runs (#941) ... * WIP task run status, revamped resuming task dependencies * Don’t select a span when toggling collapsing * Cancel runs and attempts, in prod and dev	2024-03-13 15:51:43 +00:00
nicktrn	9b6f8f9238	v3: provider and coordinator updates (#927) ... * add zod socket * start using zod socket * use zod socket for shared queue * use shared provider shell * update k8s provider task ops * set otlp endpoint * update docker actions and fix builds * remove unused types * update injected secret key env var name * complete socket.io types to schemas migration * update env example to new v3 key var * fix coordinator build * set task run label * ensure attempt id is always set * increase prod worker timeout * set otlp endpoint for on runs * pass otlp env var to prod worker * auto-remove completed index jobs	2024-03-06 16:01:59 +00:00
nicktrn	52c9d485f8	v3: cli build command, prod runs, checkpoints (#919) ... * fix trailing slash in api url * add maybe platform down error * shorten config path * extend protobuf compiler install instructions * build command and image model * new image triggers task indexing * index support for prod builds * shorten example provider secret * multi-stage task prod build * lock prod tasks to node 18 image * extract shared build and dev command libs * pull out more shared deps * add coordinator and providers * fix core-apps build * add dev builds for new apps * fix cwd * enable corepack * some build fixes * enable buildkit for old docker versions * coordinator image fixes * update provider containerfile * build dev images in parallel * upgrade pgadmin * fix prod facade build * prod runs * fix merge * don't knock out platform on invalid attempt id * fix prod facade * rename to build.ts * fix prod builds * set to executing after fetching payload * make prod worker listen on random port if in use * remove experimental warnings in dev * prod resume * prevent execution after completion * exit prod worker after completion * always restart otel collector * docker checkpoints and prod runtime messaging * don't retry indexing without chance of success * make platform checkpoint aware * deploy with existing hash sets latest worker * log restore requests * only try to checkpoint long waits * tidying up * lockfile * fix build * prod worker merge fixes * fix prod complete and cancel * fix lua nil checks * socket namespace abstraction * cleanup * make all build args optional * add build script * don't require env vars for dev * fix schema * prod merge * small fix * bind correct logger * fix v3 ref catalog entry * resume prod batch * pass socket to error and disconnect handlers * fix non-batch resume * fix batch resume * send connection env vars when not in dev * create worker via socket * move api client back into v3 cli * fix lockfile * fix resume with failures * marqs replace message * typecheck prior to build * don't define api url in prod builds * support prod retries after resume * skip typecheck option	2024-03-04 13:10:07 +00:00