Commit Graph

16 Commits

Author SHA1 Message Date
nicktrn bce398b892 increase indexing specs 2024-05-03 09:36:16 +01:00
nicktrn d0ef36260a stop trying to pull init image when already present 2024-04-29 15:33:46 +01:00
nicktrn 75a54540a4 use full image refs for init containers 2024-04-24 10:13:28 +01:00
nicktrn 584c7da5df v3: prod worker graceful shutdown (#1034)
* graceful exit with timeout

* handle and display graceful timeout errors

* fix for very long waits

* changeset

* increase termination grace period to an hour
2024-04-18 15:46:14 +01:00
nicktrn c9e1a3e9c5 v3: fix dependency trigger and wait (#1030)
* fix dev dependecy resumes

* add catalog entry with multiple trigger waits

* update docker provider lifecycle command

* switch to x for clearing run filters

* Revert "fix dev dependecy resumes"

This reverts commit b4061f2ae71f21889adcf061928e2e77c436f0ed.

* fix dependency resumes, properly this time

* add catalog entry for dependency waits in loops

* advice in docs re parallel dependency waits

* fix link from v3 to v2 docs

* move lifecycle command logging to debug only

* Removed batchOptions from the trigger options and the docs

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-04-18 14:48:43 +01:00
nicktrn 7a9bd18ba2 v3: stop swallowing deployment errors and display them better (#1020)
* exit worker with code 111 for handled errors

* task monitor will ignore previously handled errors

* fix logging of index errors

* deployment error component

* changeset
2024-04-10 17:58:51 +01:00
nicktrn f2894c177a switch to busybox and only call post start hook for restore (#1005) 2024-04-05 15:47:58 +01:00
nicktrn 1207efbbad v3: prod image upgrade and fixes (#1003)
🚢 Publish Infra Images / build (coordinator) (push) Has been cancelled
🚢 Publish Infra Images / build (kubernetes-provider) (push) Has been cancelled
* fix shutdown after final attempt

* fix workdir permissions

* correctly handle self-hosted deploy errors

* switch to node:20-bookworm-slim

* add binaries catalog file

* changeset

* skip recommended packages

* post start hook retries

* comment and improve lifecycle retry
2024-04-05 13:09:07 +01:00
nicktrn 7ea8532cce v3: crash observability (#1002)
* indexing resource limits

* refactor and storage limits

* handle and display worker crashes

* enable prod cancellation

* fix reconnect delay

* improve crash messages

* log and display crash events

* changeset
2024-04-05 13:00:20 +01:00
nicktrn b66d5525ef v3: machine config (#978)
* add and use machine config

* assign tasks to worker nodes only

* add secure flag to zod connection

* changeset

* add pre stop hook

* don't use secure connection by default

* pass more identifiers to provider and apply labels
2024-03-28 14:21:49 +00:00
Eric Allam 74236ae7c1 Fix changeset workflow (thought I’d already fixed this?) 2024-03-27 14:47:42 +00:00
nicktrn 9ecf07731a v3: small features and fixes for prod infra (#968)
* add checkpoint restore events

* fix retries.enabledInDev

* fix tsconfig paths

* fix provider build and dev

* update kubernetes provider and fix builds again

* upgrade prod base to node 20

* update infra publish workflow

* rethrow k8s errors after logging

* shorten index container names

* fix error type assertion

* rename type assertion

* remove resource limits for now

* add missing run id on create

* add push to deploy command for self-hosting

* checkpointing fixes

* update coordinator image

* ensure valid registry login

* delete checkpoint archive after successful push

* log options on error

* structured logs for socket connections

* fix structured log merge

* exit process after checkpointing

* update restore pull secret name

* append shortcode to restore names

* log handler payload

* disable post start lifecycle hook

* pass in coordinator host via volume

* replace dapi with taskinfo

* add missing restore label

* don't restart restored containers

* remove init container from create

* atomic post-completion checkpoints

* switch to run id for container names

* improve wait accuracy

* measure basic checkpoint perf

* always log disconnect reason

* use system clock to end wait spans

* checkpoint readiness and cancel signals

* restore from checkpoint events and fix statuses

* remove attempt id env var

* restore dependencies from events

* reconnect wip

* lifecycle hooks are back

* fix hooks and improve reconnect

* make docker send postStart hook

* only checkpoint for retry if large delay

* fix a few more resume issues

* lifecycle hook fixes

* skip connection handler when waiting for post start hook

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2024-03-25 16:28:56 +00:00
nicktrn 20aa1cbe9b v3: checkpoint and restore with sim for local dev (#933)
* zod ipc connection with acks

* pass sender through to ipc handlers

* bits and bobs

* add host networking back in

* disable verbose logs

* restore after tasks and batches.. almost

* restore and resume all the things

* fix for systems without checkpoint support

* Get deployment indexing errors to work with the new ZodIpc stuff

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2024-03-11 15:20:33 +00:00
nicktrn 9b6f8f9238 v3: provider and coordinator updates (#927)
* add zod socket

* start using zod socket

* use zod socket for shared queue

* use shared provider shell

* update k8s provider task ops

* set otlp endpoint

* update docker actions and fix builds

* remove unused types

* update injected secret key env var name

* complete socket.io types to schemas migration

* update env example to new v3 key var

* fix coordinator build

* set task run label

* ensure attempt id is always set

* increase prod worker timeout

* set otlp endpoint for on runs

* pass otlp env var to prod worker

* auto-remove completed index jobs
2024-03-06 16:01:59 +00:00
Eric Allam b7845685eb v3: no bundle builds for the dev CLI (#923)
* WIP no bundling

* Convert dev CLI to use an unbundled build process to support otel instrumentation

* Fix pnpm lock file

* A couple of fixes to get typechecking to work
2024-03-04 15:36:09 +00:00
nicktrn 52c9d485f8 v3: cli build command, prod runs, checkpoints (#919)
* fix trailing slash in api url

* add maybe platform down error

* shorten config path

* extend protobuf compiler install instructions

* build command and image model

* new image triggers task indexing

* index support for prod builds

* shorten example provider secret

* multi-stage task prod build

* lock prod tasks to node 18 image

* extract shared build and dev command libs

* pull out more shared deps

* add coordinator and providers

* fix core-apps build

* add dev builds for new apps

* fix cwd

* enable corepack

* some build fixes

* enable buildkit for old docker versions

* coordinator image fixes

* update provider containerfile

* build dev images in parallel

* upgrade pgadmin

* fix prod facade build

* prod runs

* fix merge

* don't knock out platform on invalid attempt id

* fix prod facade

* rename to build.ts

* fix prod builds

* set to executing after fetching payload

* make prod worker listen on random port if in use

* remove experimental warnings in dev

* prod resume

* prevent execution after completion

* exit prod worker after completion

* always restart otel collector

* docker checkpoints and prod runtime messaging

* don't retry indexing without chance of success

* make platform checkpoint aware

* deploy with existing hash sets latest worker

* log restore requests

* only try to checkpoint long waits

* tidying up

* lockfile

* fix build

* prod worker merge fixes

* fix prod complete and cancel

* fix lua nil checks

* socket namespace abstraction

* cleanup

* make all build args optional

* add build script

* don't require env vars for dev

* fix schema

* prod merge

* small fix

* bind correct logger

* fix v3 ref catalog entry

* resume prod batch

* pass socket to error and disconnect handlers

* fix non-batch resume

* fix batch resume

* send connection env vars when not in dev

* create worker via socket

* move api client back into v3 cli

* fix lockfile

* fix resume with failures

* marqs replace message

* typecheck prior to build

* don't define api url in prod builds

* support prod retries after resume

* skip typecheck option
2024-03-04 13:10:07 +00:00