Commit Graph

21 Commits

Author SHA1 Message Date
Eric Allam 7d4708839f Upgrade zod to 3.23.8 (#1484)
* Upgrade to zod 3.23.8

* Add changeset
2024-11-19 15:51:37 +00:00
Eric Allam af427aa32d Proxy no longer rewrites hostname, instead uses worker origin request forwarding 2024-10-24 17:02:57 +01:00
Eric Allam f9ec66c562 v3: new build system (#1265)
* upgrade @opentelemetry packages to the latest versions

* remove v2 only packages, will be moved to a dedicated repo

* remove more v2 code and run pnpm install

* use the npm yalt package in the webapp

* convert @trigger.dev/core to tshy

* Switch from jest to vitest in @trigger.dev/core

* Fixed core test

* move core-backend code into core subpath export

* convert @trigger.dev/sdk to tshy

* Removed hono

* move core-apps to core/v3/apps, remove core-apps, start converting cli-v3

* Fix up some of the commands

* cli now building and loadable

* using package-json-from-dist to get package version now in core and cli

* dev command WIP

* cleaned up some repetition and structure of the entry point stuff

* bringing back the background worker stuff

* Indexing of the v3 catalog

* getting closer to executing dev runs...

* centralize dev logging using event emitter

* Move indexing to it’s own entry point, simplify code

* dev runs working

* Get instrumentation to work with openai

* debugging achieved internally

* provide worker files as part of the worker creation on the server

* support for cjs and esm javascript

* Fixed timeout

* worker manifest now has the config path

* auto-upgrade config to non-deprecated alternatives

* Adding package preview release

* deployment WIP

* improve the syncEnvVars output and adapt resolveEnvVars

* WIP bun runtime

* WIP bun support

* seed tasks with the machine preset if listed in the config

* deploy run executions WIP, extracted TaskRunProcess into 1 place

* deployed tasks running and executing 🎉

* support for waits and better flushing & process cleanup

* Fixed the heartbeating

* Better warning messages

* Improve and unify the indexing between dev and deploy

* Support for external deps that need node-gyp to build

* build extensions can now install custom packages and run instructions in the image. Also prisma extension now works and also works with multiple schema files

* Add back in the main/types/module to sdk

* dev no longer is Ink/React, grace period for disconnections in dev

* Fix the changeset config

* More changeset fixes

* Remove config packages

* More changeset fixes

* Fixed typescript issues (needed to revert back to zod 3.22.3

* Fix pr_checks workflow

* Remove the prepare script

* Fixed tests and package versions

* Remove cli test script

* Remove packages from tailwind watch paths

* Add repo to public packages

* Just commit the generated files and do the building at dev time

* Try and get pkg.pr.new working

* Try again

* Fix emitDecoratorMetadata importing named export from typescript

* config file backwards compat with export const config

* Fixed issue where import errors weren’t coming through

* p-retry is a prod dep

* typescript needs to be a prod dependency for emitDecoratorMetadata

* Add better debug logging to help track down import-in-the-middle bug

* An external is only considered resolvable if it resolves to the same path as the collected external

* Fix runtime checks to allow >=18.20

* Move extensions to a new build package

* Fixed building packages in dockerfile

* Remove the e2e test from publish workflow for now

* Don’t treat pkg.pr.new versions has needing upgrading

* making sure config handleError works, and discovered path aliases don’t work in config files

* Strip empty string env vars so they accidentally override real values

* Couple of things

* Update version to use preview instead of beta

* Hopefully fix re-attempts with >30s delay

* Match socket emit messages to current latest in main

* Initial guide

* Go back to beta

* Go back to the preview, and update guide to use pr preview tags

* Go back to beta

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
2024-08-23 13:10:15 +01:00
nicktrn d5772e57bd Remove rogue zod error 2024-07-03 16:57:13 +01:00
nicktrn 14c2bdf89b v3: checkpoint and reliability improvements (#1198)
* only checkpoint retries with delays greater than threshold

* rename checkpoint threshold env var

* log task monitor ignores

* crash runs with unbounded attempts

* fix retry check in shared queue consumer

* add missing stop for env var sync spinner

* prod entry point refactor

* missing awaits

* more verbose prod flush and exit logs

* reduce checkpoint support logs

* heartbeat while checkpointing between retries

* dynamic coordinator config

* measure lazy attempt creation time in prod

* simplify delay threshold

* heartbeat clarifications

* crash run if it doesn't reach checkpointable state

* require dynamic config threshold

* fix retry prep, await previous worker kill

* unify wait mechanics

* fix prod worker without tasks error

* ensure worker is ready to be checkpointed for dependency waits

* improve worker attempt creation logging

* prevent crashes caused by failed socket schema parsing

* fix dynamic imports in v3 catalog

* clarify attempt retry mechanics

* move backoff helper to core-apps

* remove core-apps barrel file

* add backoff execute with callback

* deprecate non-lazy attempt messages

* update socket.io-client to v4.7.5

* fix socket.io types for emits with timeout

* retry all the things

* remove todo

* fix retry restores

* improve index failure logs

* retry incomplete dependency waits

* fix checkpoint in-progress detection

* prevent losing messages during reconnect

* checkpoint when greater or equal to threshold

* improve handling of duration wait edge cases

* add ready for lazy attempt replay

* retry attempt completion

* allow failing runs with unfriendly run id

* fix min max jitter

* cancel checkpoints on run failure

* improve attempt creation errors

* prevent crashing run on failed cleanup

* handle at-least-once execute lazy attempt delivery

* log exit code on prepare for retry

* fix timeout promise

* mark some things

* chaos monkey superpowers

* refactor checkpointer

* set chaos monkey defaults

* less chaos

* fix backoff

* handle uncaught entry point exceptions

* only replay rpcs on true reconnects

* allow resume unless final run status

* add changeset

* small fixes
2024-07-03 16:54:09 +01:00
Matt Aitken b6de469d07 Proxy rate limit (#1131)
* Updated worker types

* Cloudflare rate limiter applied to sendEvent/sendEvents

* Latest wrangler

* Updated to the latest compatibility_date and nest the unsafe bindings properly

* Added some types from a Discord members

* Better logging and added a reset header so the SDK can use it from inside the run function

* Set staging proxy rate limit to 100/60s
2024-05-28 12:09:05 +01:00
github-actions[bot] daae6df428 chore: Update version for release (#794)
* chore: Update version for release

* Update lockfile

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2023-12-13 10:23:36 +00:00
github-actions[bot] d42ba80108 chore: Update version for release (#785)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-08 10:19:57 +00:00
github-actions[bot] 750c1ff1e3 chore: Update version for release (#783)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-07 12:59:53 +00:00
github-actions[bot] 9fb8dceef4 chore: Update version for release (#781)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-06 20:05:29 +00:00
github-actions[bot] 08e6cad28a chore: Update version for release (#779)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-06 15:42:36 +00:00
github-actions[bot] 33d555d00e chore: Update version for release (#776)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-12-06 11:28:06 +00:00
github-actions[bot] 1567239718 chore: Update version for release (#771)
* chore: Update version for release

* Release lockfile

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2023-12-04 14:14:41 +00:00
github-actions[bot] 00b0c3e02e chore: Update version for release (#765)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-11-30 21:54:13 +00:00
github-actions[bot] 419b93809d chore: Update version for release (#757)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-11-29 11:24:58 +00:00
github-actions[bot] a5b49cb61f chore: Update version for release (#753)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-11-28 14:53:06 +00:00
github-actions[bot] 742d16087b chore: Update version for release (#739)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2023-11-16 16:31:41 +00:00
Matt Aitken 5dd1bcc589 sendEvents proxy uses batches of 10 (max for SQS) 2023-11-14 21:34:08 +00:00
Matt Aitken ed5ee16de5 Better errors now get to users of sendEvent 2023-11-14 21:33:47 +00:00
Matt Aitken f406e59c45 Configuration and docs for deploying the proxy to Cloudflare 2023-11-14 14:04:42 +00:00
Matt Aitken 2b48e6d04b Feature: sendEvent can use an AWS SQS queue to increase reliability and throughput (#729)
* Initial commit (by create-cloudflare CLI)

* Changed the prettier rules

* Started writing Readme and got dev working

* Validate the request, parse the event and respond with the correct format

* Use ulidx instead of ulid in core

* Removed some boilerplate and use new function for sendEvent

* Removed old ulid package

* When handling a webhook, it’s not an error if the HTTP endpoint environment isn’t found

* If the env vars aren’t set, log the request and pass it through

* Pass requests through. Setup the env vars

* Flipped the API key detection because we can enforce that it’s the shape of a private API key

* Progress with sendEvent proxy

* Events are being put on the SQS queue

* The SQS event queue is working

* Make sure there’s a timestamp before enqueuing the evnet

* Log the sqs_event, not API key

* Added a bit more detail to the readme

* Improved the Cloudflare logs

* We don’t need the global.window hack for AWS, just globalThis

* Removed commented out wrangler.toml values

* Reworked the proxy to make it easier to add more endpoints

* Use json utility to send Responses

* Start work on proxying bulk events
2023-11-13 15:08:01 +00:00