Eric Allam
b98dd79fe4
## What
Makes two transaction-resilience behaviors real and env-var
configurable, defaults set to the good values, so we can tune during and
after the Aug 15 database patch window without a redeploy:
- **maxWait 2s → 10s** (TRI-12982): how long Prisma waits to borrow a
connection before it can `BEGIN`. A restart freeze holds the pool full,
and the only thing that errored was transaction starts giving up at 2s.
- **Retry transaction-start P2028-at-acquisition** (TRI-12984): when
Prisma can't borrow a connection within `maxWait` it raises P2028
(`Unable to start a transaction in the given time`) and **no SQL ran**,
so retrying is safe. Scoped narrowly: only that error (never P2024
pool-exhaustion), 2 attempts, jittered backoff, and a token-bucket
budget so a mass freeze can't amplify into a retry storm.
## Env vars (`DATABASE_*` convention)
Generic defaults:
| var | default |
|---|---|
| `DATABASE_TRANSACTION_MAX_WAIT_MS` | `10000` |
| `DATABASE_TRANSACTION_START_RETRY_ENABLED` | `true` (kill switch) |
| `DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS` | `2` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS` | `250` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST` | `100` |
Per-writer-pool overrides, each falling back to the generic when unset
(same pattern as the per-client pool/connect-timeout work):
`RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all 7 knobs each). Transactions
only open on writer pools, so those are the only pools with their own
knobs. Each pool gets its **own** token bucket, so a storm on one pool
can't drain another's retry budget.
## Design
- The retry primitives live in `internal-packages/database` and never
read `process.env` (IoC): a P2028-at-acquisition classifier, a
`TokenBucketRetryBudget`, and `withTransactionStartRetry`, folded into
the `$transaction` helper via a new `startRetry` option. Config is
resolved at the app boundary and threaded in.
- The `$transaction` helper is the chokepoint (wraps the whole
transaction), not the per-statement `$allOperations` extension.
- The run engine's writes go through `PostgresRunStore`'s own
`.$transaction(...)`, not the webapp helper, so both the helper and the
two `PostgresRunStore` sites apply maxWait + retry (sharing the per-pool
config). Builds on the `options?: { timeout, maxWait }` seam added in
#4514.
- Webapp `$transaction` call sites get the default `maxWait` + retry
injected at one merge point, so no call site needed editing.
## Evidence
- Unit red/green in `internal-packages/database`: reverting the helper
wiring turned the acquisition-retry test red (`Unable to start a
transaction in the given time`), re-applying it green. Full package
suite 25/25. Covers: classifier (P2028-acq yes, P2024 no, in-tx P2028
no), retry (retry-then-succeed, no-retry P2024, stop at maxAttempts,
disabled, budget-exhausted, jitter bounds), token bucket, and
`$transaction` wiring.
- Typecheck clean: webapp, run-store, run-engine.
- Full-stack run: bounded queue-ay pass (15 projects, real dev runs
through the run-engine `PostgresRunStore` transaction path). 13 pass;
the 2 failures are one documented known-failure and one
stale-worker-state flake that passes 2/2 with this change active on a
fresh app.
- Boots cleanly with per-pool overrides set.
## Configuration & rollout
Ship **inert** first (zero behavior change), then flip to the good
values **live via env** — no redeploy needed for either.
### Inert — behaves exactly as today
```
DATABASE_TRANSACTION_MAX_WAIT_MS=2000 # Prisma's built-in default (change defaults to 10000)
DATABASE_TRANSACTION_START_RETRY_ENABLED=false # disable the new retry entirely
```
`maxWait=2000` is what every path used before (Prisma's default; the
run-store sites and the helper passed no maxWait). `retry=false`
short-circuits `withTransactionStartRetry` to a single run and makes the
serialization-retry exclusion a no-op. Verified on the pooler-freeze
rig: identical fail-fast P2028 at ~2003ms with zero retries —
byte-for-byte current behavior, across all pools.
### Production ("good") — the baked defaults
Rely on defaults (nothing to set) or set explicitly:
```
DATABASE_TRANSACTION_MAX_WAIT_MS=10000
DATABASE_TRANSACTION_START_RETRY_ENABLED=true
DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS=3 # 3 attempts (2 retries); ~30s acquisition tolerance covers a ~20-25s freeze
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS=50
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS=250
DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC=50
DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST=100
```
Per-pool overrides `RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all seven knobs each) are
optional and fall back to the generic set — not needed for v1; the
generic set covers the control-plane, run-ops, and run-ops-legacy writer
pools. Readers open no transactions and take nothing.
**Guardrail:** the retry only engages when a pool's `pool_timeout` >
`maxWait`. Prod is fine (`DATABASE_POOL_TIMEOUT=60` >> 10). Do not set
any writer pool's `pool_timeout` at or under `maxWait`, or saturation
failures flip from retryable P2028 to non-retryable P2024 and the retry
silently stops helping.
### Rollback
Env flip (set inert) or revert. Retry only fires where no SQL ran, and
the per-pool token bucket caps a storm. No migration.
refs TRI-13295, TRI-12982, TRI-12984
Build and deploy fully‑managed AI agents and workflows
Website | Docs | Issues | Example projects | Feature requests | Public roadmap | Self-hosting
About Trigger.dev
Trigger.dev is the open-source platform for building AI workflows in TypeScript. Long-running tasks with retries, queues, observability, and elastic scaling.
The platform designed for building AI agents
Build AI agents using all the frameworks, services and LLMs you're used to, deploy them to Trigger.dev and get durable, long-running tasks with retries, queues, observability, and elastic scaling out of the box.
-
Long-running without timeouts: Execute your tasks with absolutely no timeouts, unlike AWS Lambda, Vercel, and other serverless platforms.
-
Durability, retries & queues: Build rock solid agents and AI applications using our durable tasks, retries, queues and idempotency.
-
True runtime freedom: Customize your deployed tasks with system packages – run browsers, Python scripts, FFmpeg and more.
-
Human-in-the-loop: Programmatically pause your tasks until a human can approve, reject or give feedback.
-
Realtime apps & streaming: Move your background jobs to the foreground by subscribing to runs or streaming AI responses to your app.
-
Observability & monitoring: Each run has full tracing and logs. Configure error alerts to catch bugs fast.
Key features:
- JavaScript and TypeScript SDK - Build background tasks using familiar programming models
- Long-running tasks - Handle resource-heavy tasks without timeouts
- Durable cron schedules - Create and attach recurring schedules of up to a year
- Trigger.dev Realtime - Trigger, subscribe to, and get real-time updates for runs, with LLM streaming support
- Build extensions - Hook directly into the build system and customize the build process. Run Python scripts, FFmpeg, browsers, and more.
- React hooks - Interact with the Trigger.dev API on your frontend using our React hooks package
- Batch triggering - Use batchTrigger() to initiate multiple runs of a task with custom payloads and options
- Structured inputs / outputs - Define precise data schemas for your tasks with runtime payload validation
- Waits - Add waits to your tasks to pause execution for a specified duration
- Preview branches - Create isolated environments for testing and development. Integrates with Vercel and git workflows
- Waitpoints - Add human-in-the-loop judgment at critical decision points without disrupting workflow
- Concurrency & queues - Set concurrency rules to manage how multiple tasks execute
- Multiple environments - Support for DEV, PREVIEW, STAGING, and PROD environments
- No infrastructure to manage - Auto-scaling infrastructure that eliminates timeouts and server management
- Automatic retries - If your task encounters an uncaught error, we automatically attempt to run it again
- Checkpointing - Tasks are inherently durable, thanks to our checkpointing feature
- Versioning - Atomic versioning allows you to deploy new versions without affecting running tasks
- Machines - Configure the number of vCPUs and GBs of RAM you want the task to use
- Observability & monitoring - Monitor every aspect of your tasks' performance with comprehensive logging and visualization tools
- Logging & tracing - Comprehensive logging and tracing for all your tasks
- Tags - Attach up to ten tags to each run, allowing you to filter via the dashboard, realtime, and the SDK
- Run metadata - Attach metadata to runs which updates as the run progresses and is available to use in your frontend for live updates
- Bulk actions - Perform actions on multiple runs simultaneously, including replaying and cancelling
- Real-time alerts - Choose your preferred notification method for run failures and deployments
Write tasks in your codebase
Create tasks where they belong: in your codebase. Version control, localhost, test and review like you're already used to.
import { task } from "@trigger.dev/sdk";
//1. You need to export each task
export const helloWorld = task({
//2. Use a unique id for each task
id: "hello-world",
//3. The run function is the main function of the task
run: async (payload: { message: string }) => {
//4. You can write code that runs for a long time here, there are no timeouts
console.log(payload.message);
},
});
Deployment
Use our SDK to write tasks in your codebase. There's no infrastructure to manage, your tasks automatically scale and connect to our cloud. Or you can always self-host.
Environments
We support Development, Staging, Preview, and Production environments, allowing you to test your tasks before deploying them to production.
Full visibility of every job run
View every task in every run so you can tell exactly what happened. We provide a full trace view of every task run so you can see what happened at every step.
Getting started
The quickest way to get started is to create an account and project in our web app, and follow the instructions in the onboarding. Build and deploy your first task in minutes.
Useful links:
- Quick start - get up and running in minutes
- How it works - understand how Trigger.dev works under the hood
- Guides and examples - walk-through guides and code examples for popular frameworks and use cases
Self-hosting
If you prefer to self-host Trigger.dev, you can follow our self-hosting guides:
- Docker self-hosting guide - use Docker Compose to spin up a Trigger.dev instance
- Kubernetes self-hosting guide - use our official Helm chart to deploy Trigger.dev to your Kubernetes cluster
Support and community
We have a large active community in our official Discord server for support, including a dedicated channel for self-hosting.
Development
To setup and develop locally or contribute to the open source project, follow our development guide.