cbc84e2011
measure basic checkpoint perf
2024-03-21 18:36:12 +00:00
e2b10f42a0
improve wait accuracy
2024-03-21 16:10:22 +00:00
9fc62f7f10
switch to run id for container names
2024-03-21 14:56:23 +00:00
5180aacd4a
atomic post-completion checkpoints
2024-03-21 14:14:46 +00:00
18bd7f4623
exit process after checkpointing
2024-03-20 14:38:21 +00:00
8fda3d7b36
structured logs for socket connections
2024-03-20 13:17:41 +00:00
c734db3951
log options on error
2024-03-20 10:05:03 +00:00
8ca21c1dd4
delete checkpoint archive after successful push
2024-03-20 10:00:15 +00:00
fdcb06a173
ensure valid registry login
2024-03-20 09:54:54 +00:00
5ff738a18c
checkpointing fixes
2024-03-19 19:21:12 +00:00
1f190fe680
v3: fix dependency resume ( #946 )
...
* fix basic resume mechanics
* fix dependency resume
* fix deploy page sort
* remove task list console log
* checkpoint restore between attempts
* add checkpoint requirement checks to docker provider
* switch to raw execa for long statements
* catch and report pre-connection index errors
* graceful worker shutdown after completion
* fix final completion
* use same task process for each attempt
* update retry catalog entry
* add immediate return task
---------
Co-authored-by: Eric Allam <eric@trigger.dev >
2024-03-18 12:21:40 +00:00
478ce006cb
v3: task run status and canceling runs ( #941 )
...
* WIP task run status, revamped resuming task dependencies
* Don’t select a span when toggling collapsing
* Cancel runs and attempts, in prod and dev
2024-03-13 15:51:43 +00:00
20aa1cbe9b
v3: checkpoint and restore with sim for local dev ( #933 )
...
* zod ipc connection with acks
* pass sender through to ipc handlers
* bits and bobs
* add host networking back in
* disable verbose logs
* restore after tasks and batches.. almost
* restore and resume all the things
* fix for systems without checkpoint support
* Get deployment indexing errors to work with the new ZodIpc stuff
---------
Co-authored-by: Eric Allam <eallam@icloud.com >
2024-03-11 15:20:33 +00:00
83304f86f8
Capture uncaugh exceptions during indexing and fail deploys, with error output ( #934 )
2024-03-10 20:57:46 +00:00
c385db63be
v3: cli deploy command ( #931 )
...
* WIP proxy/deploy
* WIP, registry proxy in express working
* A couple of notes, preparing for indexing
* Move the changes to prod-worker into the new file
* Deploy command working with indexing and runs (docker provider only for now)
* Removed ts-expect-error directive
* Fixed build command
2024-03-08 10:49:42 +00:00
9b6f8f9238
v3: provider and coordinator updates ( #927 )
...
* add zod socket
* start using zod socket
* use zod socket for shared queue
* use shared provider shell
* update k8s provider task ops
* set otlp endpoint
* update docker actions and fix builds
* remove unused types
* update injected secret key env var name
* complete socket.io types to schemas migration
* update env example to new v3 key var
* fix coordinator build
* set task run label
* ensure attempt id is always set
* increase prod worker timeout
* set otlp endpoint for on runs
* pass otlp env var to prod worker
* auto-remove completed index jobs
2024-03-06 16:01:59 +00:00
52c9d485f8
v3: cli build command, prod runs, checkpoints ( #919 )
...
* fix trailing slash in api url
* add maybe platform down error
* shorten config path
* extend protobuf compiler install instructions
* build command and image model
* new image triggers task indexing
* index support for prod builds
* shorten example provider secret
* multi-stage task prod build
* lock prod tasks to node 18 image
* extract shared build and dev command libs
* pull out more shared deps
* add coordinator and providers
* fix core-apps build
* add dev builds for new apps
* fix cwd
* enable corepack
* some build fixes
* enable buildkit for old docker versions
* coordinator image fixes
* update provider containerfile
* build dev images in parallel
* upgrade pgadmin
* fix prod facade build
* prod runs
* fix merge
* don't knock out platform on invalid attempt id
* fix prod facade
* rename to build.ts
* fix prod builds
* set to executing after fetching payload
* make prod worker listen on random port if in use
* remove experimental warnings in dev
* prod resume
* prevent execution after completion
* exit prod worker after completion
* always restart otel collector
* docker checkpoints and prod runtime messaging
* don't retry indexing without chance of success
* make platform checkpoint aware
* deploy with existing hash sets latest worker
* log restore requests
* only try to checkpoint long waits
* tidying up
* lockfile
* fix build
* prod worker merge fixes
* fix prod complete and cancel
* fix lua nil checks
* socket namespace abstraction
* cleanup
* make all build args optional
* add build script
* don't require env vars for dev
* fix schema
* prod merge
* small fix
* bind correct logger
* fix v3 ref catalog entry
* resume prod batch
* pass socket to error and disconnect handlers
* fix non-batch resume
* fix batch resume
* send connection env vars when not in dev
* create worker via socket
* move api client back into v3 cli
* fix lockfile
* fix resume with failures
* marqs replace message
* typecheck prior to build
* don't define api url in prod builds
* support prod retries after resume
* skip typecheck option
2024-03-04 13:10:07 +00:00
f5ea64ef4f
Renamed the coordinator app to wss (web socket server)
2023-01-05 16:30:02 +00:00
87c781ca16
Clients can now reconnect if they lose access to the coordinator server
2023-01-05 16:19:03 +00:00
03f6c32051
Fixed some issues with disconnected runs and started the renaming process
2023-01-05 11:07:45 +00:00
f00e3c8633
The host now publishes the START_WORKFLOW_RUN message so there isn’t a race condition
2023-01-05 10:39:46 +00:00
edb1d9c95f
Added new INTERRUPTION step type to be able to display it in the UI
2023-01-04 17:23:09 +00:00
17683d2377
Workflow runs can now be interrupted and retried
2023-01-04 15:26:15 +00:00
22e02bcef4
Cleanup TriggerServer instances when the connection is lost
2023-01-04 14:25:35 +00:00
48bacb0e60
Make workflow run steps idempotent
2023-01-04 13:39:28 +00:00
d660c581f4
Add support for durable delays (either durations or scheduled times)
2023-01-02 16:17:28 +00:00
1fcc27b990
Resolve is the better name here again
2023-01-02 14:14:22 +00:00
ae63b49251
Initial work on the delay functionality
2023-01-02 13:41:11 +00:00
8967d5e216
“Resolve” is a better word for that message, and postMessage response schema is always success
2023-01-02 11:50:15 +00:00
d5e695dcfa
integration requests respond with just output
2022-12-30 19:03:14 +00:00
8b7dafcf2d
WIP sending/receiving requests
2022-12-29 15:29:49 +00:00
b9bf7033a7
Big overhaul to unify how events works across the different types
...
Workflow runs are still TODO but are pretty much ready to go
2022-12-23 16:34:42 +00:00
67612a8e0c
Registering webhooks with existing connections (and github only)
2022-12-21 16:56:32 +00:00
234f6a00bb
Add support for firing custom events in workflow runs
2022-12-20 14:47:37 +00:00
15e6af4f36
Added support for log messages
2022-12-20 13:44:22 +00:00
d457085f7b
Workflow runs, with a workflow step for output
2022-12-20 11:14:03 +00:00
6bf2ac423c
Got a trigger firing end-to-end
2022-12-18 21:58:40 +00:00
a0bb1a235c
Moves pulsar back into the docker compose file
...
Pulsar now starts/stops when running docker:db
2022-12-16 16:21:20 +00:00
36090aea80
Triggering workflows from custom events
2022-12-16 14:28:49 +00:00
d8c22b61ef
Workflows are now being registered in the webapp, along with their trigger
2022-12-15 16:41:08 +00:00
afc35caea6
Restructured internal-messages into internal-platform, to be used in the webapp
2022-12-14 15:58:42 +00:00
210f2e5ea3
Coordinator now publishing the Initialize Workflow message and it’s being consumed in the webapp
2022-12-14 14:40:20 +00:00
c1e7e80bf9
Reworked how the coordinator publishes and subscribes to messages in Pulsar
2022-12-14 14:21:16 +00:00
57a8c0a234
Doing real authorization of API keys now so we can get back the organization ID
...
I’ve also started to add pulsar integration with the webapp, to get messages from coordinators
2022-12-13 15:51:52 +00:00
49d35a7a34
A couple of things in here
...
- Improved starting and stopping pulsar locally
- Starting to explore how the connections stuff in workflows (including how the types will work)
- Authorization now returns an organization id (to properly scope the pulsar topics to a unique workflow/org pair)
- Better handling of closing host connections in the coordinator
- Upgrade all zod to 3.20
2022-12-13 14:46:08 +00:00
d5a59051f3
Added pulsar docker commands and getting the message broker working in the coordinator
2022-12-09 20:23:44 +00:00
fbad4d3118
Moving the sdk and coordinator work into the monorepo
2022-12-08 13:53:35 +00:00
nicktrn