## Summary
Several boundary catches and customer-input validation paths were
logging at `error` level for failures the system already handles
gracefully — disconnect on auth failure, return undefined, skip retries,
etc. This batch routes them to `warn` (which stays in stdout) or counts
them as OTel metrics, so visibility is preserved without surfacing them
as alerts.
## Changes
**New helper / pattern:**
- `apiBuilder.server.ts` — `logBoundaryError(message, error, url)`
inspects the inner error type at loader/action boundary catches;
downgrades to `warn` for `AbortError`, `ServiceValidationError`, and
`EngineServiceValidationError`.
- `platform.v3.server.ts` — `platform_client.failures_total` OTel
counter with `{function, kind}` labels; helper
`recordPlatformFailure(fn, kind)` replaces the previous error-level
logging across all `BillingClient` wrappers.
**Log-level downgrades:**
- `handleSocketIo.server.ts` — `Worker authentication failed` → warn
(system disconnects on failure; refs TRI-8863)
- `waitpointSystem.ts` — when `runStatus === "CANCELED"` in the
suspended-without-checkpoint branch, skip the throw and warn instead
(benign cancel-vs-resume race, nothing to resume)
- `runAttemptSystem.ts` — `flushedMetadata` parse/validate failures →
warn (customer-side data shape, system returns gracefully)
- `batch-queue/index.ts` — final-attempt failures with
`result.skipRetries` → warn (callbacks already opted out of retry, e.g.
queue size limit hit)
- `queryPerformanceMonitor.server.ts` — slow queries → warn
(observability signal, not an application error)
- `timeoutDeployment.server.ts` — deployment-state mismatch in the
timeout job → warn (timeout-vs-completion race)
**Inner error preservation:**
- `waitpointCompletionPacket.server.ts` — `logger.error(uploadError)`
before throwing the `ServiceValidationError` wrapper, so the underlying
upload error stays visible.
## Why
The pattern across all of these is the same: a boundary log treated any
thrown/returned error as `error` regardless of cause, even when the
cause was an expected, system-handled condition (client disconnect,
customer quota, race condition, schema validation of customer data).
That made the logs noisy and made it harder to spot real bugs.
Where the underlying signal is still useful operationally (slow queries,
billing call failures), we route it to OTel metrics with low-cardinality
labels so dashboards and alerts can be tuned independently of error
logs.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run build --filter @internal/run-engine`
- [ ] Trigger a run on hello-world and verify task lifecycle is
unaffected
- [ ] Cancel a suspended run and verify the cancel-while-suspended
branch in `waitpointSystem.ts` returns `{status: "skipped"}` instead of
throwing
- [ ] Confirm `platform_client.failures_total` counter shows up in
metrics with `{function, kind}` labels when the billing client errors
## Changes
Three changes in
`internal-packages/run-engine/src/engine/systems/debounceSystem.ts`, in
order of impact:
1. **Fast-path skip before the lock.** In `handleExistingRun`, do an
unlocked read of `delayUntil` (and `createdAt` for the max-duration
check) from the run row before entering `runLock.lock("handleDebounce",
...)`. If `newDelayUntil <= currentDelayUntil` and the run is still
within its max-duration window, return the existing run immediately
without taking the lock. Safe because debounce is monotonic-forward only
— a stale read either matches reality or undershoots, both of which
decay correctly (re-checked properly inside the lock by whichever caller
is actually pushing forward). Trailing-mode triggers carrying
`updateData` still take the lock so the data update is applied.
2. **Quantize `newDelayUntil`.** Round the computed `newDelayUntil` to
1-second buckets (configurable via `quantizeNewDelayUntilMs`, set to 0
to disable). Without quantization, every call has a slightly larger
`newDelayUntil` than the last and they all pass the fast-path check.
With it, concurrent callers on the same key share a target time and ~95%
short-circuit. User-visible effect: a debounced run might fire up to 1s
earlier than the strict spec — non-issue for typical debounce use cases
(chat summarization, batched notifications, etc.).
3. **Graceful lock-contention fallback.** Wrap the `runLock.lock(...)`
call so `LockAcquisitionTimeoutError` and Redlock `ExecutionError` /
`ResourceLockedError` return the existing run id with success instead of
propagating a 5xx. Debounce is best-effort: if we can't take the lock,
the herd is already updating it for us; fall in line. This kills the 5xx
→ SDK-retry feedback loop. With (1)+(2) this rarely fires; without them
it's the difference between 5xx and 200.
Defaults preserve current behaviour aside from quantization (1s) and
fast-path (on). Both are configurable via `RunEngineOptions.debounce`.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Changelog
Reduce 5xx feedback loops on hot debounce keys by quantizing
`delayUntil`, adding an unlocked fast-path skip before the redlock, and
gracefully handling redlock contention in `handleDebounce` so the SDK no
longer retries into a herd.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Summary
Adds `isReplay` boolean to the run context (`ctx.run.isReplay`),
following the same pattern as the existing `isTest`. The value is
derived from the existing `replayedFromTaskRunFriendlyId` database
field, so no schema migration is needed.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- Verified `@trigger.dev/core` builds successfully
- Verified `webapp` typechecks successfully
- All new fields use `default(false)` for backwards compatibility
---
## Changelog
- Added `isReplay` to `TaskRun` and `V3TaskRun` schemas in `common.ts`
- Added `RUN_IS_REPLAY` semantic attribute and wired it in `taskContext`
- Propagated `isReplay` through the dequeue system, run attempt system,
and all execution context construction paths (V1 + V2)
- Added `isReplay` to `DequeuedMessage` and
`TaskRunExecutionLazyAttemptPayload` schemas
- Added patch changeset for `@trigger.dev/core`
- Updated docs: added `isReplay` to context reference, added "Detecting
replays" section to replaying page
---
💯
Link to Devin session:
https://app.devin.ai/sessions/1d6f1b3cc39a4623b72d05bf00f2d70c
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: nick <55853254+nicktrn@users.noreply.github.com>
## Summary
- The `processMasterQueueForEnvironment` call in the priority test was
racing against background `processQueueForWorkerQueue` jobs scheduled
50ms after each trigger
- With a 50ms debounce (`processWorkerQueueDebounceMs: 50`) and runs
triggered sequentially, the RunQueue Worker could process those jobs
mid-sequence, pushing partial batches to the worker queue in the wrong
overall priority order
- `masterQueueConsumersDisabled: true` only blocks the shard-level
polling loops — it does not prevent the RunQueue's own Worker from
processing these debounced jobs
- Fix: add `worker.disabled: true` to the test 1 engine config, which
propagates to `workerOptions.disabled` in the RunQueue constructor and
prevents the Worker from starting
## Test plan
- [x] Both priority tests pass: `pnpm run test
./src/engine/tests/priority.test.ts --run`
- [x] Test 1 log confirms no `✅ Starting run engine worker` or worker
loop messages — workers fully disabled
- [x] Test 2 unaffected (uses master queue consumers for automatic
promotion, no `disabled` flag added)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Adds `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` (default `"0"`).
When enabled, the Prisma reads inside `RunEngine.getSnapshotsSince` run
against the read-only replica client instead of the primary. Offloads
the snapshot-polling queries fired by every running task runner off the
writer.
## Why
`getSnapshotsSince` is called from the managed runner's
fetch-and-process loop (once per poll interval, plus on every
snapshot-change notification). It runs four sequential reads per call —
one `findFirst` by snapshot id, one `findMany` on snapshots with
`createdAt > X`, one raw SQL against `_completedWaitpoints`, and chunked
`findMany` on `waitpoint`. Per concurrent run, every few seconds. It's
read-only, tolerates a small amount of staleness, and is an obvious
candidate for the replica.
## Replica-lag considerations
- **Step 1 "since snapshot not found"**: if the runner just received a
snapshot id from the primary and asks the replica before it replicates,
the function throws and the caller treats the response as an error
(runner falls back to a metadata refresh). Self-correcting, not silent.
- **Step 2 missing newly-created snapshots**: the next poll's `createdAt
> sinceSnapshot.createdAt` filter still picks them up once the replica
catches up.
- **Waitpoint junction race**: the riskiest path — if a latest snapshot
is replicated but its `_completedWaitpoints` join rows aren't yet, the
runner could advance past that snapshot with `completedWaitpoints: []`.
WAL/storage-level replication replays commits in order, so in practice
both should appear atomically on the reader, but the race window is why
the flag ships disabled.
Aurora reader shrinks all three windows to single-digit ms in typical
conditions, and its storage-level replication gives atomic visibility of
committed transactions on the reader.
## Test plan
- [ ] Flip the flag on in a non-prod environment, confirm snapshot
polling behaves normally and `getSnapshotsSince` errors in Sentry stay
flat.
- [ ] Verify writer query volume drops and reader query volume rises on
the snapshot-polling queries.
- [ ] Keep an eye on `AuroraReplicaLag` (or equivalent) during rollout.
The cancel button was missing from the runs list for runs in `DEQUEUED`
status. The runs list gates the button on `run.isCancellable`, which
goes through `isCancellableRunStatus` -> `CANCELLABLE_RUN_STATUSES` =
`NON_FINAL_RUN_STATUSES`. `DEQUEUED` was never added to that list when
it was introduced in the run engine.
The single run page uses a separate check (`!run.isFinished`, i.e. the
inverse of `FINAL_RUN_STATUSES`), so cancellation already worked there -
only the list was affected.
Adding `DEQUEUED` to `NON_FINAL_RUN_STATUSES` also flips
`isCrashableRunStatus` and `isFailableRunStatus`, but:
- The crash path is the right behaviour - a `DEQUEUED` run (worker has
claimed but not yet executing) can legitimately crash before
`EXECUTING`, same as `PENDING`/`DELAYED` already do.
- The fail path (`failedTaskRun.server.ts`) is only reached from V1 code
paths (marqs consumers, v1 heartbeat handler). `DEQUEUED` is a
V2-engine-only status, so V1 consumers never see it.
When cancelling a `DEQUEUED` run the execution snapshot goes to
`PENDING_CANCEL` (worker must ack) but `TaskRun.status` flips to
`CANCELED` immediately - the UI reflects cancellation without waiting
for the worker. Added an integration test in
`run-engine/src/engine/tests/cancelling.test.ts` covering the full
trigger -> dequeue -> cancel -> worker-ack flow.
## Stall safety
The stall recovery path (PENDING_EXECUTING heartbeat miss ->
nack-and-requeue -> back to QUEUED) lives entirely inside
`@internal/run-engine` and never touches the webapp's `taskStatus.ts`
helpers - the engine has zero imports from `~/v3/taskStatus` and doesn't
know `CrashTaskRunService` / `FailedTaskRunService` exist. A stalled
DEQUEUED run still goes back to the queue for retry; this change cannot
cause stalls to crash or fail.
The only realistic impact is the intended UI fix - the theoretical V1
crash/fail branches for DEQUEUED are unreachable in practice because V1
runs never have DEQUEUED status.
## Summary
Nests the `TaskRunExecutionSnapshot` creation inside the
`taskRun.update()` Prisma call in the dequeue flow, reducing **2 DB
commits → 1** per dequeue operation. This is the highest-volume of the
five unmerged flows identified in TRI-8450 (~9,200 commits/sec on the
engine service).
**Pattern**: Follows the same nested-write approach already used in the
completion path (`runAttemptSystem.ts:735`) and trigger path
(`engine/index.ts:674`).
**Changes**:
- `dequeueSystem.ts`: Moved snapshot creation into `executionSnapshots:
{ create: {...} }` within the existing `taskRun.update()`. Pre-generates
the snapshot ID via `generateInternalId()` (plain cuid, matching what
Prisma's `@default(cuid())` produces) so the event emission, heartbeat
enqueue, and return value can all be constructed from data already in
scope — **no extra DB read needed** after the merged write.
`SnapshotId.toFriendlyId()` is used only for the return value's
`friendlyId` field, matching the original `createExecutionSnapshot`
behavior.
- `executionSnapshotSystem.ts`: Added public
`enqueueHeartbeatIfNeeded()` method that exposes the heartbeat
scheduling logic (previously only available internally via
`createExecutionSnapshot`). This is needed because `PENDING_EXECUTING`
requires a heartbeat, unlike the `FINISHED` status in the completion
reference pattern. This method is reusable by future merge targets
(retry-immediate, checkpoint, cancel, requeue).
**Net DB change per dequeue**: eliminates 1 write transaction (the
separate `TaskRunExecutionSnapshot.create`). No extra reads added — the
snapshot ID is pre-generated and the `executionSnapshotCreated` event
payload is constructed inline from values already available in the
closure.
## Review & Testing Checklist for Human
- [ ] **Verify manually-constructed event payload matches DB state**:
The `executionSnapshotCreated` event is now built inline (not read back
from DB). Confirm the field values (`runStatus: "PENDING"`,
`attemptNumber`, `checkpointId`, `workerId`, `runnerId`,
`completedWaitpointIds`) match what Prisma actually writes. A mismatch
here would be silent — event consumers would get stale/wrong data.
- [ ] **Verify `attemptNumber` source is equivalent**: Old code used
`lockedTaskRun.attemptNumber` (post-update result). New code uses
`result.run.attemptNumber` (pre-update). The `taskRun.update()` data
payload does NOT include `attemptNumber`, so they should be identical —
but confirm this assumption holds for all dequeue scenarios (e.g.
retried runs).
- [ ] **Verify `isValid` defaults to `true` in schema**: The old
`createExecutionSnapshot` explicitly set `isValid: error ? false :
true`. The nested create omits `isValid` (no error in the dequeue happy
path). Confirm the Prisma schema default for
`TaskRunExecutionSnapshot.isValid` is `true`.
- [ ] **Verify `runStatus: "PENDING"` hardcoding matches the mapping**:
The old code passed `lockedTaskRun.status` ("DEQUEUED") to
`createExecutionSnapshot`, which mapped it to "PENDING" via `run.status
=== "DEQUEUED" ? "PENDING" : run.status`. The new code hardcodes
`"PENDING"` directly. This is correct but brittle if `status` ever
changes from "DEQUEUED" to something else upstream.
- [ ] **Spot-check `completedWaitpoints` connect + order logic**: The
nested create replicates the connect/order logic from
`createExecutionSnapshot` (lines 387-393). Verify the
`snapshot.completedWaitpoints` type provides `id` and `index` fields
compatible with this usage.
- [ ] **Verify `checkpoint` in return value**: The return now uses
`snapshot.checkpoint` (from the *previous* snapshot) instead of reading
the newly-created snapshot's checkpoint relation. Since `checkpointId`
is passed through unchanged, they should be identical — but worth a
sanity check.
**Recommended test plan**: deploy to staging, run the
`sample_pg_activity.py` sampler for a 5-minute window, and verify the
COMMIT count drop on the engine service + proportional `IO:XactSync`
reduction.
### Notes
- This only covers the **dequeue** flow (flow #1 from TRI-8450). The
remaining four flows (retry-immediate, checkpoint, requeue, cancel) are
separate follow-ups.
- The new `enqueueHeartbeatIfNeeded` method is deliberately designed for
reuse by those follow-up PRs.
- CI note: the `priority.test.ts` failure in shard 7 is a flaky ordering
assertion unrelated to this change (it compares `friendlyId` values in
dequeue order). The `audit` check is also pre-existing/unrelated.
Link to Devin session:
https://app.devin.ai/sessions/034fe0e7224f49278a2de260203e1377
Requested by: @ericallam
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
The TaskRun.runTags string array already stores tag names, making the
TaskRunTag M2M relation redundant write overhead. Remove createTags
calls, connect: tags, and join table writes from both V1 and V2 trigger
paths. Simplify the add-tags API to just push to runTags directly.
## Summary
Currently, every triggered run follows a two-step path through Redis:
1. **Enqueue** — A Lua script atomically adds the message to a queue
sorted set (ordered by priority-adjusted timestamp)
2. **Dequeue** — A debounced `processQueueForWorkerQueue` job fires
~500ms later, checks concurrency limits, removes the message from the
sorted set, and pushes it to a worker queue (Redis list) where workers
pick it up via `BLPOP`
This means every run pays at least ~500ms of latency between being
triggered and being available for a worker to execute, even when the
queue is empty and concurrency is wide open.
### What changed
The enqueue Lua scripts now atomically decide whether to **skip the
queue sorted set entirely** and push directly to the worker queue. This
happens inside the same Lua script that handles normal enqueue, so the
decision is atomic with respect to concurrency bookkeeping.
A run takes the **fast path** when all of these are true:
- **Fast path is enabled** for this worker queue (gated per
`WorkerInstanceGroup`)
- **No available messages** in the queue (`ZRANGEBYSCORE` finds nothing
with score ≤ now) — this respects priority ordering and allows fast path
even when the queue has future-scored messages (e.g. nacked retries with
delay)
- **Environment concurrency** has capacity
- **Queue concurrency** has capacity (including per-concurrency-key
limits for CK queues)
When the fast path is taken:
- The message is stored and pushed directly to the worker queue
(`RPUSH`)
- Concurrency slots are claimed (`SADD` to the same sets used by the
normal dequeue path)
- The `processQueueForWorkerQueue` job is **not scheduled** (no work to
do)
- TTL sorted set is skipped (the `expireRun` worker job handles TTL
independently)
When any condition fails, the existing slow path runs unchanged.
### Rollout gating
- **Development environments**: Fast path is always enabled
- **Production environments**: Gated by a new `enableFastPath` boolean
on `WorkerInstanceGroup` (defaults to `false`), allowing
region-by-region rollout
### Rolling deploy safety
Each process registers its own Lua scripts via `defineCommand`
(identified by SHA hash). Old and new processes never share scripts. The
Redis data structures are fully compatible in both directions — ack,
nack, and release operations work identically regardless of which path a
message took.
## Test plan
- [x] Fast path taken when queue is empty and concurrency available
- [x] Slow path when `enableFastPath` is false
- [x] Slow path when queue has available messages (respects priority
ordering)
- [x] Fast path when queue only has future-scored messages
- [x] Slow path when env concurrency is full
- [x] Fast-path message can be acknowledged correctly
- [x] Fast-path message can be nacked and re-enqueued to the queue
sorted set
- [x] Run all existing run-queue tests (ack, nack, CK, concurrency
sweeper, dequeue) to verify no regressions
- [x] Typecheck passes for run-engine and webapp
Adds three new top-level columns to the ClickHouse task_runs_v2 table
primarily for analytics:
- `trigger_source` / `root_trigger_source` - extracted from the existing
TaskRun.annotations JSON during WAL
replication
- `is_warm_start` - new nullable boolean on TaskRun in Postgres, set in
the existing taskRun.update() at attempt
start (no additional write). null until the first attempt starts.
Run region is already available via the existing `worker_queue` column
in ClickHouse.
Scheduled runs create predictable hourly spikes that compete with
on-demand runs for node capacity. Runs triggered "on-demand" via the
SDK, API, or dashboard, are more sensitive to cold start latency since
users are typically
waiting on the result. When a burst of scheduled runs lands at the top
of the hour, it can saturate the shared pool resources causing
contention, affecting cold starts across the board.
The idea in this change is to absorb these periodic spikes in a
dedicated pool without affecting the cold starts of on-demand runs.
Scheduled runs are inherently less sensitive to cold starts.
### Changes in this PR
Follows up on run annotations (#3241), which made trigger origin
available on every run in the tree. This PR exposes
annotations at dequeue time to the supervisor. This enables scheduling
decisions based on trigger source.
The affinities are soft preferences at schedule time, so runs fall back
gracefully if the target pool is out out of capacity.
Queue limit ServiceValidationErrors were being logged at error level.
These are
expected validation rejections, not bugs.
- Add logLevel property to ServiceValidationError (webapp + run-engine)
- Set logLevel: warn on all queue limit throws
- Schedule engine: detect queue limit failures and log as warn
- Redis-worker: respect logLevel on thrown errors
Adds an `annotations` JSONB column to task runs that captures where and
how each run was triggered.
This enables filtering and analyzing trigger origins without querying up
the run tree. Also enables making scheduling decisions based on the
trigger source, e.g., use separate affinities for scheduled runs.
Each run records:
- **triggerSource**: who initiated it (sdk, api, dashboard, cli, mcp,
schedule)
- **triggerAction**: what kind of action (trigger, replay, test)
- **rootTriggerSource**: the trigger source of the root ancestor,
propagated through the entire run
tree
- **rootScheduleId**: schedule id, in case the run tree was triggered
from a schedule
Currently the main motivation for annotations it to determine whether a
run is part of a schedule-originated tree without traversing ancestors.
### A couple of design considerations
- **Decoupled source from method**: triggerSource and triggerAction are
separate fields to avoid
combinatorial explosion (every new source × every new action)
- **Server-side first**: all annotation values are primarily determined
on the server, only a minor SDK change needed
- **Forward-compatible**: annotation fields use
`z.enum([...]).or(anyString)` so new values can be
added without breaking validation; we currently don't need an explicit
version field for annotations.
Note: `metadata` would have been a more fitting name for the db column,
as it is consistent with other tables where we store this type of
information. It is already in use to store user metadata though, so we
go with `annotations` instead.
- Full prompt management UI: list, detail, override, and version
management for AI prompts defined with `prompts.define()`
- Rich AI span inspectors for all AI SDK operations with token usage,
messages, and prompt context
- Real-time generation tracking with live polling and filtering
## Prompt management
Define prompts in your code with `prompts.define()`, then manage
versions and overrides from the dashboard without redeploying:
```typescript
import { task, prompts } from "@trigger.dev/sdk";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { z } from "zod";
const supportPrompt = prompts.define({
id: "customer-support",
model: "gpt-4o",
variables: z.object({
customerName: z.string(),
plan: z.string(),
issue: z.string(),
}),
content: `You are a support agent for Acme SaaS.
Customer: {{customerName}} ({{plan}} plan)
Issue: {{issue}}
Respond with empathy and precision.`,
});
export const supportTask = task({
id: "handle-support",
run: async (payload) => {
const resolved = await supportPrompt.resolve({
customerName: payload.name,
plan: payload.plan,
issue: payload.issue,
});
const result = await generateText({
model: openai(resolved.model ?? "gpt-4o"),
system: resolved.text,
prompt: payload.issue,
...resolved.toAISDKTelemetry(),
});
return { response: result.text };
},
});
```
The prompts list page shows each prompt with its current version, model,
override status, and a usage sparkline over the last 24 hours.
From the prompt detail page you can:
- **Create overrides** to change the prompt template or model without
redeploying. Overrides take priority over the deployed version when
`prompt.resolve()` is called.
- **Promote** any code-deployed version to be the current version
- **Browse generations** across all versions with infinite scroll and
live polling for new results
- **Filter** by version, model, operation type, and provider
- **View metrics** (total generations, avg tokens, avg cost, latency)
broken down by version
## AI span inspectors
Every AI SDK operation now gets a custom inspector in the run trace
view:
- **`ai.generateText` / `ai.streamText`** — Shows model, token usage,
cost, the full message thread (system prompt, user message, assistant
response), and linked prompt details
- **`ai.generateObject` / `ai.streamObject`** — Same as above plus the
JSON schema and structured output
- **`ai.toolCall`** — Shows tool name, call ID, and input arguments
- **`ai.embed`** — Shows model and the text being embedded
For generation spans linked to a prompt, a "Prompt" tab shows the prompt
metadata, the input variables passed to `resolve()`, and the template
content from the prompt version.
All AI span inspectors include a compact timestamp and duration header.
## Other improvements
- Resizable panel sizes now persist across page refreshes (patched
`@window-splitter/state` to fix snapshot restoration)
- Run page panels also persist their sizes
- Fixed `<div>` inside `<p>` DOM nesting warnings in span titles and
chat messages
- Added Operations and Providers filters to the AI metrics dashboard
## Screenshots
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 14
17@2x"
src="https://github.com/user-attachments/assets/f3e59989-a2fa-4990-a9d0-3cacda431868"
/>
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15
37@2x"
src="https://github.com/user-attachments/assets/2f2d02df-2d2b-44fb-ac6f-9153f6a6c387"
/>
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15
54@2x"
src="https://github.com/user-attachments/assets/baa161e0-ef91-4fa4-a55f-986b71cccdf0"
/>
When processing batchTriggerAndWait items, each batch item was acquiring
a
Redis lock on the parent run to insert a TaskRunWaitpoint row. With high
concurrency (processingConcurrency=50), this caused
LockAcquisitionTimeoutError
(880 errors/24h in prod), orphaned runs, and stuck parent runs.
Since blockRunWithCreatedBatch already transitions the parent to
EXECUTING_WITH_WAITPOINTS before items are processed, the per-item lock
is
unnecessary. The new blockRunWithWaitpointLockless method performs only
the
idempotent CTE insert and timeout scheduling without acquiring the lock.
Queues with concurrency keys now appear as a single entry in the master
queue instead of one entry per key. This prevents high-CK-count tenants
from consuming the entire `parentQueueLimit` window and starving other
tenants on the same shard.
A new per-queue **CK index** (sorted set) tracks active concurrency key
sub-queues. The master queue gets one `:ck:*` wildcard entry per base
queue. Dequeuing from that entry round-robins across sub-queues,
maintaining per-CK concurrency tracking and fairness.
All existing operations (enqueue, dequeue, ack, nack, DLQ, TTL expiry)
are CK-index-aware and keep the index consistent. Old-format entries
drain naturally during rollout — no migration step needed, single
deploy.
When the dev CLI exits (e.g. ctrl+c via pnpm), runs that were
mid-execution
previously stayed stuck in EXECUTING status for up to 5 minutes until
the
heartbeat timeout fired. Now they are cancelled within seconds.
The dev CLI spawns a lightweight detached watchdog process at startup.
The
watchdog monitors the CLI process ID and, when it detects the CLI has
exited,
calls a new POST /engine/v1/dev/disconnect endpoint to cancel all
in-flight
runs immediately (skipping PENDING_CANCEL since the worker is known to
be dead).
Watchdog design:
- Fully detached (detached: true, stdio: ignore, unref()) so it survives
even when pnpm sends SIGKILL to the process tree
- Active run IDs maintained via atomic file write
(.trigger/active-runs.json)
- Single-instance guarantee via PID file (.trigger/watchdog.pid)
- Safety timeout: exits after 24 hours to prevent zombie processes
- On clean shutdown, the watchdog is killed (no disconnect needed)
Disconnect endpoint:
- Rate-limited: 5 calls/min per environment
- Capped at 500 runs per call
- Small counts (<= 25): cancelled inline with pMap concurrency 10
- Large counts: delegated to the bulk action system
- Uses finalizeRun: true to skip PENDING_CANCEL and go straight to
FINISHED
Run engine change:
- cancelRun() now respects finalizeRun when the run is in EXECUTING
status,
skipping the PENDING_CANCEL waiting state and going directly to FINISHED
The global rate limiter was being applied at the FairQueue claim phase,
consuming 1 token per queue-claim-attempt rather than per item
processed.
With many small queues (each batch is its own queue), consumers burned
through tokens on empty or single-item queues, causing aggressive
throttling well below the intended items/sec limit.
Changes:
- Move rate limiter from FairQueue claim phase to BatchQueue worker
queue
consumer loop (before blockingPop), so each token = 1 item processed
- Replace the FairQueue rate limiter with a worker queue depth cap to
prevent unbounded growth that could cause visibility timeouts
- Add BATCH_QUEUE_WORKER_QUEUE_MAX_DEPTH env var (optional, disabled by
default)
Gracefully handle oversized batch items instead of aborting the stream.
When an NDJSON batch item exceeds the maximum size, the parser now emits
an error marker instead of throwing, allowing the batch to seal
normally. The oversized item becomes a pre-failed run with
`PAYLOAD_TOO_LARGE` error code, while other items in the batch process
successfully. This prevents `batchTriggerAndWait` from seeing connection
errors and retrying with exponential backoff.
Also fixes the NDJSON parser not consuming the remainder of an oversized
line split across multiple chunks, which caused "Invalid JSON" errors on
subsequent lines.
Split the CTE in blockRunWithWaitpoint so the pending waitpoint check
is a separate SQL statement. In READ COMMITTED isolation, each statement
gets its own snapshot, so a separate SELECT sees the latest committed
state from concurrent completeWaitpoint calls.
Previously, the CTE did INSERT + pending check in one statement (one
snapshot). If completeWaitpoint committed between the CTE start and
the SELECT, the SELECT would still see PENDING due to the stale
snapshot. Neither side would enqueue continueRunIfUnblocked, leaving
the run stuck forever.
Fix slow fair queue processing by removing spurious cooloff on
concurrency blocks and fixing a race condition where retry attempt
counts were not atomically updated during message re-queue.
Removed cooloff entirely from the batch queue
This PR implements a new run TTL system and queue size limits to prevent
unbounded queue growth which should help prevent situations where queues
enter a "death spiral" where the queue will never be able to catch up.
The main/correct way to battle this situation is to enforce a maximum
TTL on all runs (e.g. up to 14 days) where runs that have been queued
for that maximum TTL will get auto-expired, making room for newer runs
to execute. This required creating a new TTL system that can handle
higher workloads and is now deeply integrated into the RunQueue. When
runs are enqueued with a TTL, they are added to their normal queue as
well as to the TTL queue. When runs are dequeued, they are removed from
both their normal queue and the TTL queue. If runs are dequeued by the
TTL system, they are removed from their normal queue. Both these
dequeues happen automatically so there is no race condition.
The TTL expiration system is also made reliable by expiring runs via a
Redis worker, which is enqueued to atomically inside the TTL dequeue lua
script.
### Optional associated waitpoints
Additionally, this PR implements an optimization where runs that aren't
triggered with a dependent parent run will no longer create an
associated waitpoint. Associated waitpoints are then lazily created if a
dependent run wants to wait for the child run post-facto (via debounce
or idempotency), which is a rare situation but is possible. This means
fewer waitpoint creations but also fewer waitpoint completions for runs
with no dependencies.
### Environment Queue Limits
Prevents any single queue growing too large by enforcing queue size
limits at trigger time.
- Queue size checks happen at trigger time - runs are rejected if queue
would exceed limit
- Dashboard UI shows queue limits on both the Queues page and a new
Limits page
- In-memory caching for queue size checks to reduce Redis load
### Batch trigger fixes
Currently when a batch item cannot be created for whatever reason (e.g.
queue limits) the run will never get created, which means a stalled run
if using `batchTriggerAndWait`. We've updated the system to handle this
differently: now when a batch item cannot be triggered and converted
into a run, we will eventually (after retrying 8 times up to 30s) we
will create a "pre-failed" run with the error details, correctly
resolving the batchTriggerAndWait.
Moving usage updates into the run engine to prevent inefficient &
additional incremental updates to the TaskRun table. Read/Modify/Write
pattern is safe inside of the run engine because of the run lock. We can
also now cap the usageDurationMs value from overflowing and causing an
error.
## Why?
This is preventing at least one update per TaskRun and instead updating
these values piggybacking on other updates.
## Aurora PostgreSQL Reader Consistency Notes
### TL;DR
Aurora readers share the same storage as the writer, but maintain
separate in-memory page caches. This means:
- **Storage is always consistent** - writes are synchronously committed
to shared storage
- **Page cache can lag** - typically <100ms, but can cause stale reads
if data is cached
### How It Works
1. Writer commits to shared storage (synchronous 4/6 quorum)
2. Writer sends cache invalidation messages to readers (asynchronous)
3. If reader has data in cache → returns cached (potentially stale)
value
4. If reader has cache miss → fetches from shared storage (always
current)
### Monitoring
```sql
SELECT server_id,
CASE WHEN session_id = 'MASTER_SESSION_ID' THEN 'Writer' ELSE 'Reader' END AS role,
replica_lag_in_msec
FROM aurora_replica_status();
```
Move expensive findMany queries for PENDING_VERSION and
WAITING_FOR_DEPLOY
runs to read replicas to avoid blocking migrations on the primary
database.
Changes:
- Add readOnlyPrisma to SystemResources type
- Pass readOnlyPrisma to systems in RunEngine constructor
- Update pendingVersionSystem to use readOnlyPrisma for findMany
- Update executeTasksWaitingForDeploy to use _replica for findMany
## Summary
- Store the original user-provided idempotency key and scope alongside
the hash
- Expose `ctx.run.idempotencyKey` as the user-provided key (not the
hash)
- Add `ctx.run.idempotencyKeyScope` to show the scope ("run", "attempt",
or "global")
<img width="539" height="450" alt="CleanShot 2026-01-19 at 11 40 46"
src="https://github.com/user-attachments/assets/b6f42991-697e-4314-a164-aef77b8fd25c"
/>
## Problem
Idempotency keys were hashed (SHA-256) before storage, making debugging
difficult since users couldn't see the value they originally set or
search for runs by idempotency key.
## Solution
Attach metadata to the `String` object returned by
`idempotencyKeys.create()` using a Symbol, extract it in the SDK before
the API call, and store it in the database alongside the hash.
```typescript
const key = await idempotencyKeys.create("my-key", { scope: "global" });
await childTask.triggerAndWait(payload, { idempotencyKey: key });
// In child task:
ctx.run.idempotencyKey // "my-key" (previously showed the hash)
ctx.run.idempotencyKeyScope // "global"
```
Test plan
- Trigger task with idempotencyKeys.create() using different scopes (run, attempt, global)
- Verify ctx.run.idempotencyKey returns user-provided key
- Verify ctx.run.idempotencyKeyScope returns correct scope
- Verify PostgreSQL stores idempotencyKeyOptions JSON
- Verify ClickHouse receives idempotency_key_user and idempotency_key_scope via replication
---------
Co-authored-by: James Ritchie <james@trigger.dev>
This PR fixes some issues with the new BatchQueue by implementing the
full two-phase dequeue process in the FairQueue, and moving the
responsibility of consuming the worker queue to the BatchQueue and
independently enabling it via the `BATCH_QUEUE_WORKER_QUEUE_ENABLED` env
var. We've also introduced the `BATCH_QUEUE_SHARD_COUNT` env var to
control the count of master queue shards in the FairQueue. We can also
control how many queues are considered in each iteration of the master
queue consumer via the `BATCH_QUEUE_MASTER_QUEUE_LIMIT` env var.
This PR will also now skip trying to dequeue from tenants that are at
concurrency capacity, which should lead to fewer issues with low
concurrency tenants blocking higher concurrency tenants from processing.
Adds support for **debounced task runs** - when triggering a task with a
debounce key, subsequent triggers with the same key will reschedule the
existing delayed run instead of creating new runs. This continues until
no new triggers occur within the delay window.
## Usage
```typescript
await myTask.trigger({ userId: "123" }, {
debounce: {
key: "user-123-update",
delay: "5s",
mode: "leading", // default
}
});
```
- **key**: Scoped to the task identifier
- **delay**: How long to wait before executing (supports duration
strings like `"5s"`, `"1m"`)
- **mode**: Either `"leading"` or `"trailing"`. Leading debounce will
use the payload and options from the first run created with the debounce
key. Trailing will use payload and options from the last run.
### "trailing" mode overrides
When using `mode: "trailing"` with debounce, the following options are
updated from the **last** trigger:
- **`payload`** - The task input data
- **`metadata`** - Run metadata
- **`tags`** - Run tags (replaces existing tags)
- **`maxAttempts`** - Maximum retry attempts
- **`maxDuration`** - Maximum compute time
- **`machine`** - Machine preset (cpu/memory)
## Behavior
- **First run wins**: The first trigger creates the run, subsequent
triggers push its execution time later
- **Idempotency keys take precedence**: If both are specified,
idempotency is checked first
- **Max duration**: Configurable via `DEBOUNCE_MAX_DURATION_MS` env var
(default: 10 minutes)
Works with `triggerAndWait` - parent runs correctly block on the
debounced run.
New batch trigger system with larger payloads, streaming ingestion,
larger batch sizes, and a fair processing system.
This PR introduces a new `FairQueue` abstraction inspired by our own
`RunQueue` that enables multi-tenant fair queueing with concurrency
limits. The new `BatchQueue` is built on top of the `FairQueue`, and
handles processing Batch triggers in a fair manner with per-environment
concurrency limits defined per-org. Additionally, there is a global
concurrency limit to prevent the BatchQueue system from creating too
many runs too quickly, which can cause downstream issues.
For this new BatchQueue system we have a completely new batch trigger
creation and ingestion system. Previously this was a single endpoint
with a single JSON body that defined details about the batch as well as
all the items in the batch.
We're introducing a two-phase batch trigger ingestion system. In the
first phase, the BatchTaskRun record is created (and possibly rate
limited). The second phase is another endpoint that accepts an NDJSON
body with each line being a single item/run with payload and options.
At ingestion time all items are added to a queue, in order, and then
processed by the BatchQueue system.
## New batch trigger rate limits
This PR implements a new batch trigger specific rate limit, configured
on the `Organization.batchRateLimitConfig` column, and defaults using
these environment variables:
- `BATCH_RATE_LIMIT_REFILL_RATE` defaults to 10
- `BATCH_RATE_LIMIT_REFILL_INTERVAL` the duration interval, defaults to
`"10s"`
- `BATCH_RATE_LIMIT_MAX` defaults to 1200
This rate limiter is scoped to the environment ID and controls how many
runs can be submitted via batch triggers per interval. The SDK handles
the retrying side.
## Batch queue concurrency limits
The new column `Organization.batchQueueConcurrencyConfig` now defines an
org specific `processingConcurrency` value, with a backup of the env var
`BATCH_CONCURRENCY_LIMIT_DEFAULT` which defaults to 10. This controls
how many batch queue items are processed concurrently per environment.
There is also a global rate limit for the batch queue set via the
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` which defaults to being disabled. If
set, the entire batch queue system won't process more than
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` items per second. This allows
controlling the maximum number of runs created per second via batch
triggers.
## Batch trigger settings
- `STREAMING_BATCH_MAX_ITEMS` controls the maximum number of items in a
single batch
- `STREAMING_BATCH_ITEM_MAXIMUM_SIZE` controls the maximum size of each
item in a batch
- `BATCH_CONCURRENCY_DEFAULT_CONCURRENCY` controls the default
environment concurrency
- `BATCH_QUEUE_DRR_QUANTUM` how many credits each environment gets each
round for the DRR scheduler
- `BATCH_QUEUE_MAX_DEFICIT` the maximum deficit for the DRR scheduler
- `BATCH_QUEUE_CONSUMER_COUNT` how many queue consumers to run
- `BATCH_QUEUE_CONSUMER_INTERVAL_MS` how frequently they poll for items
in the queue
### Configuration Recommendations by Use Case
**High-throughput priority (fairness acceptable at 0.98+):**
```env
BATCH_QUEUE_DRR_QUANTUM=25
BATCH_QUEUE_MAX_DEFICIT=100
BATCH_QUEUE_CONSUMER_COUNT=10
BATCH_QUEUE_CONSUMER_INTERVAL_MS=50
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=25
```
**Strict fairness priority (throughput can be lower):**
```env
BATCH_QUEUE_DRR_QUANTUM=5
BATCH_QUEUE_MAX_DEFICIT=25
BATCH_QUEUE_CONSUMER_COUNT=3
BATCH_QUEUE_CONSUMER_INTERVAL_MS=100
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=5
```
This PR makes our image builds deterministic and reproducible by
ensuring that identical source code always produces the same image
layers and image digest. This means that deployments where nothing has
changed will no longer invalidate the image cache in our worker cluster
nodes, thus avoid making the cold starts for runs worse.
**Context**
New deployments currently increase the cold start times for runs, as
they generate a new image which needs to be pulled in the worker cluster
where runs are executed. It happens also when the source code for the
deployment has not changed due to non-deterministic steps in our build
system. This addresses the latter issue by making builds reproducible.
**Main changes**
- Avoided baking `TRIGGER_DEPLOYMENT_ID` and
`TRIGGER_DEPLOYMENT_VERSION` in the image, we now pass these via the
supervisor instead.
- Used `json-stable-stringify` for consistent key ordering in the files
we generate for the build, e.g., `package.json`, `build.json`,
`index.json`.
- Removed `metafile.json` from the image contents as it is not actually
used in the container. This is only relevant for the `analyze` command.
- Added `SOURCE_DATE_EPOCH=0` and `rewrite-timestamp=true` to Docker
builds to normalize file timestamps.
- Removed some `timings` and `outputHashes` from build outputs and
manifests.
The builds are now reproducible for both native build server and Depot
paths. This should also lead to better image layer cache reuse in
general.
Eric Allam
