Commit Graph

47 Commits

Author SHA1 Message Date
nicktrn 62700245a3 v3: fix consecutive waits (#1073)
* fix spacing for delete hints

* don't try to resume deps on wait resume

* sending duration wait resumes is not an error anymore

* set correct status with new wait resume flow

* cancel checkpoint schema v2

* don't mix messages and schemas

* prevent unintended case fallthrough in tree view

* completely switch to platform-led duration wait resumes

* prevent infinite restores

* some entries for the catalog

* add pg to additional packages

* add checkpoint safe timeout

* prevent duplicate spans after restore

* wait for post start

* add sdk version to deploy tab

* fail on impossible checkpoint scenarios

* remove debug logs
2024-05-01 10:33:30 +01:00
Eric Allam b68012f81c v3: Various fixes for Next.js projects and projects that use v2 and v3 together (#1051)
* Fixes an issue that was treating v2 trigger directories as v3

* Make msw a normal dependency (for now) to fix Module Not Found error in Next.js.

* Extract out all the zod* stuff from core so the SDK does not import it

* Add a changeset

* Fixing typecheck errors in the webapp

* Export the Task and TaskOptions types

* Extract additional exports from core/v3 that aren’t used in the SDK

* Move to our global system from AsyncLocalStorage for the current task context storage

* Update the esbuild core bundling plugin for the new core v3 exports

* Fix v3 CLI telemetry

* Add support for tasks located in subdirectories inside trigger dirs

* Remove the env var check during deploy (too many false negatives)
2024-04-24 10:09:36 +01:00
nicktrn 51315fc3c8 v3: shared queue pool (#994)
* configurable retry delay checkpoint threshold

* shared queue consumer pool

* add more attributes to marqs spans

* inject trace context into queued messages
2024-04-02 10:22:52 +01:00
nicktrn b66d5525ef v3: machine config (#978)
* add and use machine config

* assign tasks to worker nodes only

* add secure flag to zod connection

* changeset

* add pre stop hook

* don't use secure connection by default

* pass more identifiers to provider and apply labels
2024-03-28 14:21:49 +00:00
nicktrn 9ecf07731a v3: small features and fixes for prod infra (#968)
* add checkpoint restore events

* fix retries.enabledInDev

* fix tsconfig paths

* fix provider build and dev

* update kubernetes provider and fix builds again

* upgrade prod base to node 20

* update infra publish workflow

* rethrow k8s errors after logging

* shorten index container names

* fix error type assertion

* rename type assertion

* remove resource limits for now

* add missing run id on create

* add push to deploy command for self-hosting

* checkpointing fixes

* update coordinator image

* ensure valid registry login

* delete checkpoint archive after successful push

* log options on error

* structured logs for socket connections

* fix structured log merge

* exit process after checkpointing

* update restore pull secret name

* append shortcode to restore names

* log handler payload

* disable post start lifecycle hook

* pass in coordinator host via volume

* replace dapi with taskinfo

* add missing restore label

* don't restart restored containers

* remove init container from create

* atomic post-completion checkpoints

* switch to run id for container names

* improve wait accuracy

* measure basic checkpoint perf

* always log disconnect reason

* use system clock to end wait spans

* checkpoint readiness and cancel signals

* restore from checkpoint events and fix statuses

* remove attempt id env var

* restore dependencies from events

* reconnect wip

* lifecycle hooks are back

* fix hooks and improve reconnect

* make docker send postStart hook

* only checkpoint for retry if large delay

* fix a few more resume issues

* lifecycle hook fixes

* skip connection handler when waiting for post start hook

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2024-03-25 16:28:56 +00:00
Eric Allam dc5eb68a0b Fix typecheck errors because of resolveJsonModule 2024-03-19 15:33:09 +00:00
nicktrn 1f190fe680 v3: fix dependency resume (#946)
* fix basic resume mechanics

* fix dependency resume

* fix deploy page sort

* remove task list console log

* checkpoint restore between attempts

* add checkpoint requirement checks to docker provider

* switch to raw execa for long statements

* catch and report pre-connection index errors

* graceful worker shutdown after completion

* fix final completion

* use same task process for each attempt

* update retry catalog entry

* add immediate return task

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
2024-03-18 12:21:40 +00:00
Eric Allam 478ce006cb v3: task run status and canceling runs (#941)
* WIP task run status, revamped resuming task dependencies

* Don’t select a span when toggling collapsing

* Cancel runs and attempts, in prod and dev
2024-03-13 15:51:43 +00:00
nicktrn 20aa1cbe9b v3: checkpoint and restore with sim for local dev (#933)
* zod ipc connection with acks

* pass sender through to ipc handlers

* bits and bobs

* add host networking back in

* disable verbose logs

* restore after tasks and batches.. almost

* restore and resume all the things

* fix for systems without checkpoint support

* Get deployment indexing errors to work with the new ZodIpc stuff

---------

Co-authored-by: Eric Allam <eallam@icloud.com>
2024-03-11 15:20:33 +00:00
Eric Allam 83304f86f8 Capture uncaugh exceptions during indexing and fail deploys, with error output (#934) 2024-03-10 20:57:46 +00:00
Eric Allam c385db63be v3: cli deploy command (#931)
* WIP proxy/deploy

* WIP, registry proxy in express working

* A couple of notes, preparing for indexing

* Move the changes to prod-worker into the new file

* Deploy command working with indexing and runs (docker provider only for now)

* Removed ts-expect-error directive

* Fixed build command
2024-03-08 10:49:42 +00:00
nicktrn 9b6f8f9238 v3: provider and coordinator updates (#927)
* add zod socket

* start using zod socket

* use zod socket for shared queue

* use shared provider shell

* update k8s provider task ops

* set otlp endpoint

* update docker actions and fix builds

* remove unused types

* update injected secret key env var name

* complete socket.io types to schemas migration

* update env example to new v3 key var

* fix coordinator build

* set task run label

* ensure attempt id is always set

* increase prod worker timeout

* set otlp endpoint for on runs

* pass otlp env var to prod worker

* auto-remove completed index jobs
2024-03-06 16:01:59 +00:00
nicktrn 52c9d485f8 v3: cli build command, prod runs, checkpoints (#919)
* fix trailing slash in api url

* add maybe platform down error

* shorten config path

* extend protobuf compiler install instructions

* build command and image model

* new image triggers task indexing

* index support for prod builds

* shorten example provider secret

* multi-stage task prod build

* lock prod tasks to node 18 image

* extract shared build and dev command libs

* pull out more shared deps

* add coordinator and providers

* fix core-apps build

* add dev builds for new apps

* fix cwd

* enable corepack

* some build fixes

* enable buildkit for old docker versions

* coordinator image fixes

* update provider containerfile

* build dev images in parallel

* upgrade pgadmin

* fix prod facade build

* prod runs

* fix merge

* don't knock out platform on invalid attempt id

* fix prod facade

* rename to build.ts

* fix prod builds

* set to executing after fetching payload

* make prod worker listen on random port if in use

* remove experimental warnings in dev

* prod resume

* prevent execution after completion

* exit prod worker after completion

* always restart otel collector

* docker checkpoints and prod runtime messaging

* don't retry indexing without chance of success

* make platform checkpoint aware

* deploy with existing hash sets latest worker

* log restore requests

* only try to checkpoint long waits

* tidying up

* lockfile

* fix build

* prod worker merge fixes

* fix prod complete and cancel

* fix lua nil checks

* socket namespace abstraction

* cleanup

* make all build args optional

* add build script

* don't require env vars for dev

* fix schema

* prod merge

* small fix

* bind correct logger

* fix v3 ref catalog entry

* resume prod batch

* pass socket to error and disconnect handlers

* fix non-batch resume

* fix batch resume

* send connection env vars when not in dev

* create worker via socket

* move api client back into v3 cli

* fix lockfile

* fix resume with failures

* marqs replace message

* typecheck prior to build

* don't define api url in prod builds

* support prod retries after resume

* skip typecheck option
2024-03-04 13:10:07 +00:00
Eric Allam f5ea64ef4f Renamed the coordinator app to wss (web socket server) 2023-01-05 16:30:02 +00:00
Eric Allam 87c781ca16 Clients can now reconnect if they lose access to the coordinator server 2023-01-05 16:19:03 +00:00
Eric Allam 03f6c32051 Fixed some issues with disconnected runs and started the renaming process 2023-01-05 11:07:45 +00:00
Eric Allam f00e3c8633 The host now publishes the START_WORKFLOW_RUN message so there isn’t a race condition 2023-01-05 10:39:46 +00:00
Eric Allam edb1d9c95f Added new INTERRUPTION step type to be able to display it in the UI 2023-01-04 17:23:09 +00:00
Eric Allam 17683d2377 Workflow runs can now be interrupted and retried 2023-01-04 15:26:15 +00:00
Eric Allam 22e02bcef4 Cleanup TriggerServer instances when the connection is lost 2023-01-04 14:25:35 +00:00
Eric Allam 48bacb0e60 Make workflow run steps idempotent 2023-01-04 13:39:28 +00:00
Eric Allam d660c581f4 Add support for durable delays (either durations or scheduled times) 2023-01-02 16:17:28 +00:00
Eric Allam 1fcc27b990 Resolve is the better name here again 2023-01-02 14:14:22 +00:00
Eric Allam ae63b49251 Initial work on the delay functionality 2023-01-02 13:41:11 +00:00
Eric Allam 8967d5e216 “Resolve” is a better word for that message, and postMessage response schema is always success 2023-01-02 11:50:15 +00:00
Eric Allam d5e695dcfa integration requests respond with just output 2022-12-30 19:03:14 +00:00
Eric Allam 8b7dafcf2d WIP sending/receiving requests 2022-12-29 15:29:49 +00:00
Eric Allam b9bf7033a7 Big overhaul to unify how events works across the different types
Workflow runs are still TODO but are pretty much ready to go
2022-12-23 16:34:42 +00:00
Matt Aitken dd6937c84f Deleted the coordinator specific readme 2022-12-22 12:39:36 +00:00
Eric Allam 67612a8e0c Registering webhooks with existing connections (and github only) 2022-12-21 16:56:32 +00:00
Eric Allam 234f6a00bb Add support for firing custom events in workflow runs 2022-12-20 14:47:37 +00:00
Eric Allam 15e6af4f36 Added support for log messages 2022-12-20 13:44:22 +00:00
Eric Allam d457085f7b Workflow runs, with a workflow step for output 2022-12-20 11:14:03 +00:00
Eric Allam 6bf2ac423c Got a trigger firing end-to-end 2022-12-18 21:58:40 +00:00
Eric Allam a0bb1a235c Moves pulsar back into the docker compose file
Pulsar now starts/stops when running docker:db
2022-12-16 16:21:20 +00:00
Eric Allam 36090aea80 Triggering workflows from custom events 2022-12-16 14:28:49 +00:00
Eric Allam d8c22b61ef Workflows are now being registered in the webapp, along with their trigger 2022-12-15 16:41:08 +00:00
Eric Allam afc35caea6 Restructured internal-messages into internal-platform, to be used in the webapp 2022-12-14 15:58:42 +00:00
Eric Allam 210f2e5ea3 Coordinator now publishing the Initialize Workflow message and it’s being consumed in the webapp 2022-12-14 14:40:20 +00:00
Eric Allam c1e7e80bf9 Reworked how the coordinator publishes and subscribes to messages in Pulsar 2022-12-14 14:21:16 +00:00
James Ritchie b8b558ef88 2 readme updates from when i ran the project for the first time 2022-12-14 13:30:41 +00:00
Eric Allam 57a8c0a234 Doing real authorization of API keys now so we can get back the organization ID
I’ve also started to add pulsar integration with the webapp, to get messages from coordinators
2022-12-13 15:51:52 +00:00
Eric Allam 49d35a7a34 A couple of things in here
- Improved starting and stopping pulsar locally
- Starting to explore how the connections stuff in workflows (including how the types will work)
- Authorization now returns an organization id (to properly scope the pulsar topics to a unique workflow/org pair)
- Better handling of closing host connections in the coordinator
- Upgrade all zod to 3.20
2022-12-13 14:46:08 +00:00
Eric Allam d5a59051f3 Added pulsar docker commands and getting the message broker working in the coordinator 2022-12-09 20:23:44 +00:00
Matt Aitken 85d86af33b Updated the instructions for installing the coordinator for the first time 2022-12-09 10:55:47 +00:00
Eric Allam 575d434240 Adding the pulsar client (with installation instructions) to the coordinator 2022-12-08 14:39:10 +00:00
Eric Allam fbad4d3118 Moving the sdk and coordinator work into the monorepo 2022-12-08 13:53:35 +00:00