* support named capture groups
* write crash errors to attempt.error
* make restored pod names unique per checkpoint
* use last eight characters of checkpoint id instead
* add more chaos monkey env vars
* Ignore unfreezable states
* prevent excessive queue config parsing errors
* handle dependency resume edge case
* better entry point logging
* ignore checkpoint cancellation timeouts
* add missing idempotency keys to wait for dep replays
* remove checkpoints between attempts
* fix retry container names on kubernetes
* add changeset
* fix types
* bring back internal duration timers
* add amin email regex env var
* fix displayed init command for self-hosted setups
* shared env var to disable telemetry in cli and webapp
* pin sdk version during init
* if specified, add api url to dev command shown after init
* improve checkpoint support detection
* control forced checkpoint simulation via env var
* add public init to providers
* better checkpoint support check for coordinator
* add docker to coordinator image
* update docker provider containerfile
* bump remaining containers to node 20
* add infra image build to default publish workflow
* lockfile
* remove concurrency group from infra workflow
* add docker provider to build matrix
* fix var subst
* checkpoint test is docker specific
* enable v3 projects by default on self-hosted instances
* fix v3 setup command again
* add default posthog key
* self-hosting docs
* add latest tags to versioned infra and webapp builds
* some checkpoint errors should skip retrying
* add changeset
* shorten paragraph
* some docs updates
* update tunnelling section
* add registry setup section
* use correct cli push flag
* add checkout to v3 branch
* update the worker machine setup steps
* fix infra build
* small docs update
* remove unused feature function
* Revert "remove unused feature function"
This reverts commit cfe07887a12b6893dca8ce499964481a9b3dc9db.
* fix self-hosted v3 feature gate
* add note about missing arm support
* simplify helper script syntax
* WIP worker TaskRunAttempt creation
* Handling failing task runs that cannot create an attempt for whatever reason
* Move the visibility queue stuff into a graphile job
* Fixed task runs with unsanitized queue names
* “Borrow” the code from alerts PR to get self hosted deployments working
* Add an admin API endpoint to get info about the shared marqs queue
* Allow admins to view any project metrics
* start adding lazy attempts to prod
* lazy attempt creation for prod workers
* resurrect prod stack traces
* add exception event to failed run spans
* simplify dependency resumes
* fix typecheck
* fix merge
* fresh process for all attempts
* always try sigterm first
* stop heartbeat timeout on non-inplace replace message
* add missing ack on checkpoint creation service failure
* bypass dequeue for retries with running worker
* respect retry delays
* crash runs with invalid run status for execution
* remove debug logs
* fix nack message
* fix version locking
* fresh attempt processes in dev and prod
* improve handling of ipc timeouts
* consider checkpoint failures on cancellation
* add basic chaos monkey to checkpointer
* changeset
* control forced checkpoint simulation via env var
* fix merge
* kill old attempt processes before checkpointing
* detailed perf logging for checkpointing
* add coordinator otlp endpoint example
* improve prod run cancellation
* rename supports lazy attempts migration
* fix graceful exit
* fix retry mechanics
* clear paused state before retry
* remove checkpoint image after push
* crash worker on unrecoverable errors
* refactor unrecoverable error emit
* switch to do hosted busybox image
* increase wait for duration ipc timeout
* add changeset for misc fixes
* fix merge
* fix retry delay span runId
* fix dev retries
* improve prod worker logging
* log checkpoint sizes
* add lazy attempts catalog entries
* Fixed merge issue: use zodFetch, not wrapZodFetch
* Revert "Fixed merge issue: use zodFetch, not wrapZodFetch"
This reverts commit d137e4e1fe.
* importEnvVars uses wrapZodFetch now
* add backwards compat for retries without checkpoints
* handle more cases of unrecoverable runs
* don't kill the child process if it shouldn't be killed
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* fix dev dependecy resumes
* add catalog entry with multiple trigger waits
* update docker provider lifecycle command
* switch to x for clearing run filters
* Revert "fix dev dependecy resumes"
This reverts commit b4061f2ae71f21889adcf061928e2e77c436f0ed.
* fix dependency resumes, properly this time
* add catalog entry for dependency waits in loops
* advice in docs re parallel dependency waits
* fix link from v3 to v2 docs
* move lifecycle command logging to debug only
* Removed batchOptions from the trigger options and the docs
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* add and use machine config
* assign tasks to worker nodes only
* add secure flag to zod connection
* changeset
* add pre stop hook
* don't use secure connection by default
* pass more identifiers to provider and apply labels
* add checkpoint restore events
* fix retries.enabledInDev
* fix tsconfig paths
* fix provider build and dev
* update kubernetes provider and fix builds again
* upgrade prod base to node 20
* update infra publish workflow
* rethrow k8s errors after logging
* shorten index container names
* fix error type assertion
* rename type assertion
* remove resource limits for now
* add missing run id on create
* add push to deploy command for self-hosting
* checkpointing fixes
* update coordinator image
* ensure valid registry login
* delete checkpoint archive after successful push
* log options on error
* structured logs for socket connections
* fix structured log merge
* exit process after checkpointing
* update restore pull secret name
* append shortcode to restore names
* log handler payload
* disable post start lifecycle hook
* pass in coordinator host via volume
* replace dapi with taskinfo
* add missing restore label
* don't restart restored containers
* remove init container from create
* atomic post-completion checkpoints
* switch to run id for container names
* improve wait accuracy
* measure basic checkpoint perf
* always log disconnect reason
* use system clock to end wait spans
* checkpoint readiness and cancel signals
* restore from checkpoint events and fix statuses
* remove attempt id env var
* restore dependencies from events
* reconnect wip
* lifecycle hooks are back
* fix hooks and improve reconnect
* make docker send postStart hook
* only checkpoint for retry if large delay
* fix a few more resume issues
* lifecycle hook fixes
* skip connection handler when waiting for post start hook
---------
Co-authored-by: Eric Allam <eric@trigger.dev>
* fix basic resume mechanics
* fix dependency resume
* fix deploy page sort
* remove task list console log
* checkpoint restore between attempts
* add checkpoint requirement checks to docker provider
* switch to raw execa for long statements
* catch and report pre-connection index errors
* graceful worker shutdown after completion
* fix final completion
* use same task process for each attempt
* update retry catalog entry
* add immediate return task
---------
Co-authored-by: Eric Allam <eric@trigger.dev>
* zod ipc connection with acks
* pass sender through to ipc handlers
* bits and bobs
* add host networking back in
* disable verbose logs
* restore after tasks and batches.. almost
* restore and resume all the things
* fix for systems without checkpoint support
* Get deployment indexing errors to work with the new ZodIpc stuff
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
* WIP proxy/deploy
* WIP, registry proxy in express working
* A couple of notes, preparing for indexing
* Move the changes to prod-worker into the new file
* Deploy command working with indexing and runs (docker provider only for now)
* Removed ts-expect-error directive
* Fixed build command
* add zod socket
* start using zod socket
* use zod socket for shared queue
* use shared provider shell
* update k8s provider task ops
* set otlp endpoint
* update docker actions and fix builds
* remove unused types
* update injected secret key env var name
* complete socket.io types to schemas migration
* update env example to new v3 key var
* fix coordinator build
* set task run label
* ensure attempt id is always set
* increase prod worker timeout
* set otlp endpoint for on runs
* pass otlp env var to prod worker
* auto-remove completed index jobs
nicktrn