Commit Graph

7 Commits

Author SHA1 Message Date
nicktrn d5772e57bd Remove rogue zod error 2024-07-03 16:57:13 +01:00
nicktrn 14c2bdf89b v3: checkpoint and reliability improvements (#1198)
* only checkpoint retries with delays greater than threshold

* rename checkpoint threshold env var

* log task monitor ignores

* crash runs with unbounded attempts

* fix retry check in shared queue consumer

* add missing stop for env var sync spinner

* prod entry point refactor

* missing awaits

* more verbose prod flush and exit logs

* reduce checkpoint support logs

* heartbeat while checkpointing between retries

* dynamic coordinator config

* measure lazy attempt creation time in prod

* simplify delay threshold

* heartbeat clarifications

* crash run if it doesn't reach checkpointable state

* require dynamic config threshold

* fix retry prep, await previous worker kill

* unify wait mechanics

* fix prod worker without tasks error

* ensure worker is ready to be checkpointed for dependency waits

* improve worker attempt creation logging

* prevent crashes caused by failed socket schema parsing

* fix dynamic imports in v3 catalog

* clarify attempt retry mechanics

* move backoff helper to core-apps

* remove core-apps barrel file

* add backoff execute with callback

* deprecate non-lazy attempt messages

* update socket.io-client to v4.7.5

* fix socket.io types for emits with timeout

* retry all the things

* remove todo

* fix retry restores

* improve index failure logs

* retry incomplete dependency waits

* fix checkpoint in-progress detection

* prevent losing messages during reconnect

* checkpoint when greater or equal to threshold

* improve handling of duration wait edge cases

* add ready for lazy attempt replay

* retry attempt completion

* allow failing runs with unfriendly run id

* fix min max jitter

* cancel checkpoints on run failure

* improve attempt creation errors

* prevent crashing run on failed cleanup

* handle at-least-once execute lazy attempt delivery

* log exit code on prepare for retry

* fix timeout promise

* mark some things

* chaos monkey superpowers

* refactor checkpointer

* set chaos monkey defaults

* less chaos

* fix backoff

* handle uncaught entry point exceptions

* only replay rpcs on true reconnects

* allow resume unless final run status

* add changeset

* small fixes
2024-07-03 16:54:09 +01:00
Matt Aitken b6de469d07 Proxy rate limit (#1131)
* Updated worker types

* Cloudflare rate limiter applied to sendEvent/sendEvents

* Latest wrangler

* Updated to the latest compatibility_date and nest the unsafe bindings properly

* Added some types from a Discord members

* Better logging and added a reset header so the SDK can use it from inside the run function

* Set staging proxy rate limit to 100/60s
2024-05-28 12:09:05 +01:00
Matt Aitken 5dd1bcc589 sendEvents proxy uses batches of 10 (max for SQS) 2023-11-14 21:34:08 +00:00
Matt Aitken ed5ee16de5 Better errors now get to users of sendEvent 2023-11-14 21:33:47 +00:00
Matt Aitken f406e59c45 Configuration and docs for deploying the proxy to Cloudflare 2023-11-14 14:04:42 +00:00
Matt Aitken 2b48e6d04b Feature: sendEvent can use an AWS SQS queue to increase reliability and throughput (#729)
* Initial commit (by create-cloudflare CLI)

* Changed the prettier rules

* Started writing Readme and got dev working

* Validate the request, parse the event and respond with the correct format

* Use ulidx instead of ulid in core

* Removed some boilerplate and use new function for sendEvent

* Removed old ulid package

* When handling a webhook, it’s not an error if the HTTP endpoint environment isn’t found

* If the env vars aren’t set, log the request and pass it through

* Pass requests through. Setup the env vars

* Flipped the API key detection because we can enforce that it’s the shape of a private API key

* Progress with sendEvent proxy

* Events are being put on the SQS queue

* The SQS event queue is working

* Make sure there’s a timestamp before enqueuing the evnet

* Log the sqs_event, not API key

* Added a bit more detail to the readme

* Improved the Cloudflare logs

* We don’t need the global.window hack for AWS, just globalThis

* Removed commented out wrangler.toml values

* Reworked the proxy to make it easier to add more endpoints

* Use json utility to send Responses

* Start work on proxying bulk events
2023-11-13 15:08:01 +00:00