Eric Allam
a7ff6de388
* Improve dynamic flush scheduler for otel data The changes introduce a more flexible and adaptive dynamic flush scheduler to address production issues where the system wasn't flushing data fast enough, causing memory growth and crashes. This issue arises from the existing scheduler handling only a single flush at a time, limiting concurrency and failing to cope with the influx of logs. - Added configuration options for setting minimum and maximum concurrency levels, maximum batch size, and memory pressure threshold. These parameters ensure that flush operations adjust dynamically based on workload and pressure. - Implemented `pLimit` to facilitate concurrent flush operations, with adjustments made according to batch queue length and memory pressure. - Metrics reporting improvements were added to monitor the dynamic behavior of the flush scheduler, aiding in identifying performance issues and optimizing the operation accordingly. * Implement load shedding for TaskEvent records This change introduces load shedding mechanisms to manage TaskEvent records, particularly those of kind LOG, when the system experiences high volumes and is unable to flush to the database in a timely manner. The addition aims to prevent overwhelming the system and ensure critical tasks are prioritized. - Added configuration options for `loadSheddingThreshold` and `loadSheddingEnabled` in multiple modules to activate load shedding. - Introduced `isDroppableEvent` function to allow specific events to be dropped when load shedding is enabled. - Ensured metrics are updated to reflect dropped events and load shedding status, providing visibility into system performance during high load conditions. - Updated loggers to inform about load shedding state changes, ensuring timely awareness of load management activities. * Fix undefined 'queuePressure' variable in DynamicFlushScheduler The 'queuePressure' variable was being used without being defined in the DynamicFlushScheduler class, causing potential runtime errors. This commit adds the missing definition and ensures that the variable is correctly calculated based on the 'totalQueuedItems' and 'memoryPressureThreshold'. - Addressed code inconsistencies and improved formatting. - Defined 'queuePressure' in the 'adjustConcurrency' method to prevent potential undefined errors. - Enhanced readability by maintaining consistent spacing and format across the file, contributing to the stability and maintainability of the code. - Adjusted batch size logic based on the newly defined 'queuePressure' variable. * Refactor concurrency adjustment logic in scheduler The concurrency adjustment logic in the dynamic flush scheduler has been refactored to improve clarity and maintainability. This change moves the calculation of pressure metrics outside of the conditional blocks to ensure they are always determined prior to decision-making. - The queue pressure and time since last flush calculations were moved up in the code to be independent of the 'backOff' condition. - This refactor sets up the groundwork for more reliable concurrency scaling and better performance monitoring capabilities. The overall logic of adjusting concurrency based on system pressure metrics remains unchanged. This adjustment addresses ongoing issues with the scheduler that were not resolved by previous changes. * Some tweaks
About Trigger.dev
Trigger.dev is an open source platform and SDK which allows you to create long-running background jobs. Write normal async code, deploy, and never hit a timeout.
Key features:
- JavaScript and TypeScript SDK
- No timeouts
- Retries (with exponential backoff)
- Queues and concurrency controls
- Schedules and crons
- Full Observability; logs, live trace views, advanced filtering
- React hooks to interact with the Trigger API from your React app
- Pipe LLM streams straight to your users through the Realtime API
- Trigger tasks and display the run status and metadata anywhere in your app
- Custom alerts, get notified by email, Slack or webhooks
- No infrastructure to manage
- Elastic (scaling)
- Works with your existing tech stack
In your codebase
Create tasks where they belong: in your codebase. Version control, localhost, test and review like you're already used to.
import { task } from "@trigger.dev/sdk/v3";
//1. You need to export each task
export const helloWorld = task({
//2. Use a unique id for each task
id: "hello-world",
//3. The run function is the main function of the task
run: async (payload: { message: string }) => {
//4. You can write code that runs for a long time here, there are no timeouts
console.log(payload.message);
},
});
Deployment
Use our SDK to write tasks in your codebase. There's no infrastructure to manage, your tasks automatically scale and connect to our cloud. Or you can always self-host.
Environments
We support Development, Staging, and Production environments, allowing you to test your tasks before deploying them to production.
Full visibility of every job run
View every task in every run so you can tell exactly what happened. We provide a full trace view of every task run so you can see what happened at every step.
Getting started
The quickest way to get started is to create an account and project in our web app, and follow the instructions in the onboarding. Build and deploy your first task in minutes.
Useful links:
- Quick start - get up and running in minutes
- How it works - understand how Trigger.dev works under the hood
- Guides and examples - walk-through guides and code examples for popular frameworks and use cases
Self-hosting
If you prefer to self-host Trigger.dev, you can follow our self-hosting guide.
We also have a dedicated self-hosting channel in our Discord server for support.
Development
To setup and develop locally or contribute to the open source project, follow our development guide.