Eric Allam
ae46e3f7c8
This PR implements a new run TTL system and queue size limits to prevent unbounded queue growth which should help prevent situations where queues enter a "death spiral" where the queue will never be able to catch up. The main/correct way to battle this situation is to enforce a maximum TTL on all runs (e.g. up to 14 days) where runs that have been queued for that maximum TTL will get auto-expired, making room for newer runs to execute. This required creating a new TTL system that can handle higher workloads and is now deeply integrated into the RunQueue. When runs are enqueued with a TTL, they are added to their normal queue as well as to the TTL queue. When runs are dequeued, they are removed from both their normal queue and the TTL queue. If runs are dequeued by the TTL system, they are removed from their normal queue. Both these dequeues happen automatically so there is no race condition. The TTL expiration system is also made reliable by expiring runs via a Redis worker, which is enqueued to atomically inside the TTL dequeue lua script. ### Optional associated waitpoints Additionally, this PR implements an optimization where runs that aren't triggered with a dependent parent run will no longer create an associated waitpoint. Associated waitpoints are then lazily created if a dependent run wants to wait for the child run post-facto (via debounce or idempotency), which is a rare situation but is possible. This means fewer waitpoint creations but also fewer waitpoint completions for runs with no dependencies. ### Environment Queue Limits Prevents any single queue growing too large by enforcing queue size limits at trigger time. - Queue size checks happen at trigger time - runs are rejected if queue would exceed limit - Dashboard UI shows queue limits on both the Queues page and a new Limits page - In-memory caching for queue size checks to reduce Redis load ### Batch trigger fixes Currently when a batch item cannot be created for whatever reason (e.g. queue limits) the run will never get created, which means a stalled run if using `batchTriggerAndWait`. We've updated the system to handle this differently: now when a batch item cannot be triggered and converted into a run, we will eventually (after retrying 8 times up to 30s) we will create a "pre-failed" run with the error details, correctly resolving the batchTriggerAndWait.
Trigger.dev References
Contains code that tests or uses the @trigger.dev/* packages in some way, either by using them to test out a framework adapter, an integration, or parts of the main SDK.
All the dependencies to the @trigger.dev/* packages will be both referenced in the package.json dependencies as workspace:*, as well as using a direct path from the tsconfig.json file like so:
{
"extends": "@trigger.dev/tsconfig/node18.json",
"include": ["./src/**/*.ts"],
"compilerOptions": {
"baseUrl": ".",
"lib": ["DOM", "DOM.Iterable"],
"paths": {
"@/*": ["./src/*"],
"@trigger.dev/sdk": ["../../packages/trigger-sdk/src/index"],
"@trigger.dev/sdk/*": ["../../packages/trigger-sdk/src/*"],
"@trigger.dev/express": ["../../packages/express/src/index"],
"@trigger.dev/express/*": ["../../packages/express/src/*"],
"@trigger.dev/core": ["../../packages/core/src/index"],
"@trigger.dev/core/*": ["../../packages/core/src/*"],
"@trigger.dev/integration-kit": ["../../packages/integration-kit/src/index"],
"@trigger.dev/integration-kit/*": ["../../packages/integration-kit/src/*"],
"@trigger.dev/github": ["../../integrations/github/src/index"],
"@trigger.dev/github/*": ["../../integrations/github/src/*"],
"@trigger.dev/slack": ["../../integrations/slack/src/index"],
"@trigger.dev/slack/*": ["../../integrations/slack/src/*"],
"@trigger.dev/openai": ["../../integrations/openai/src/index"],
"@trigger.dev/openai/*": ["../../integrations/openai/src/*"],
"@trigger.dev/resend": ["../../integrations/resend/src/index"],
"@trigger.dev/resend/*": ["../../integrations/resend/src/*"],
"@trigger.dev/typeform": ["../../integrations/typeform/src/index"],
"@trigger.dev/typeform/*": ["../../integrations/typeform/src/*"],
"@trigger.dev/plain": ["../../integrations/plain/src/index"],
"@trigger.dev/plain/*": ["../../integrations/plain/src/*"],
"@trigger.dev/supabase": ["../../integrations/supabase/src/index"],
"@trigger.dev/supabase/*": ["../../integrations/supabase/src/*"],
"@trigger.dev/stripe": ["../../integrations/stripe/src/index"],
"@trigger.dev/stripe/*": ["../../integrations/stripe/src/*"],
"@trigger.dev/sendgrid": ["../../integrations/sendgrid/src/index"],
"@trigger.dev/sendgrid/*": ["../../integrations/sendgrid/src/*"],
"@trigger.dev/airtable": ["../../integrations/airtable/src/index"],
"@trigger.dev/airtable/*": ["../../integrations/airtable/src/*"]
}
}
}
Creating a New Reference Project
This guide assumes that you have followed the Contributing.md instructions to set up a local trigger.dev instance. If not, please complete the setup before continuing.
Step-by-Step Instructions
- Run an HTTP tunnel: You will need to run an HTTP tunnel to expose your local webapp, it is required for some API calls during building the image to deploy on your local instance. This is optional if you do not plan to test deployment on your local instance.
- Download the ngrok CLI. This can be done by following the instructions on ngrok's website.
- Create an account on ngrok to obtain the authtoken and add it to the CLI.
ngrok config add-authtoken <your-auth-token>
Replace the with the token you obtain from ngrok.
- Run the tunnel.
ngrok http <your-app-port>
Replace the with the webapp port, default is 3030.
-
Add your tunnel URL to the env: After running the ngrok tunnel, you will see URL in your terminal, it will look something like
https://<your-tunnel-address>.ngrok-free.app. Replace theAPP_ORIGINvariable with this URL in your.envfile in the root of the trigger.dev project. -
Run the webapp on localhost:
pnpm run dev --filter webapp --filter coordinator --filter docker-provider
- Build the CLI in a new terminal window:
# Build the CLI
pnpm run build --filter trigger.dev
# Make it accessible to `pnpm exec`
pnpm i
- Set up a new project in the webapp:
- Open the webapp running on
localhost:3030. - Create a new project in the webapp UI.
- Go to the Project Settings page and copy the project reference id from there.
- Copy the hello-world project as a template:
cp -r references/hello-world references/<new-project>
Replace <new-project> with your desired project name.
- Update project details:
-
Open
<new-project>/package.jsonand change the name field. (Tip: Use the same name as in the webapp to avoid confusion.) -
Open
<new-project>/trigger.config.tsand update the project field with the project reference you copied from the webapp. -
Run
pnpm iin your<new-project>directory to sync the dependencies.
- Authorize the CLI for your project:
pnpm exec trigger login -a http://localhost:3030 --profile local
- Run the new project: You can now run your project using the CLI with the following command:
pnpm exec trigger dev --profile local
You can also deploy them against your local instance with the following command:
pnpm exec trigger deploy --self-hosted --load-image --profile local