Gracefully handle oversized batch items instead of aborting the stream.
When an NDJSON batch item exceeds the maximum size, the parser now emits
an error marker instead of throwing, allowing the batch to seal
normally. The oversized item becomes a pre-failed run with
`PAYLOAD_TOO_LARGE` error code, while other items in the batch process
successfully. This prevents `batchTriggerAndWait` from seeing connection
errors and retrying with exponential backoff.
Also fixes the NDJSON parser not consuming the remainder of an oversized
line split across multiple chunks, which caused "Invalid JSON" errors on
subsequent lines.
Replace flat master queue index with two-level tenant dispatch to fix
noisy neighbor problem. When a tenant has many queues at capacity, the
scheduler now iterates tenants (Level 1) not queues, then fetches
per-tenant queues (Level 2) only for eligible tenants.
Single-deploy migration: new enqueues write to dispatch indexes only,
consumer drains old master queue alongside new dispatch path until
empty.
Clarifies in the Query docs that run metadata is not available on the
Query page and that the output column is JSON, so dot notation (e.g.
output.externalId) should be used for selecting and filtering. Adds an
example that filters by an output field in WHERE
Split the CTE in blockRunWithWaitpoint so the pending waitpoint check
is a separate SQL statement. In READ COMMITTED isolation, each statement
gets its own snapshot, so a separate SELECT sees the latest committed
state from concurrent completeWaitpoint calls.
Previously, the CTE did INSERT + pending check in one statement (one
snapshot). If completeWaitpoint committed between the CTE start and
the SELECT, the SELECT would still see PENDING due to the stale
snapshot. Neither side would enqueue continueRunIfUnblocked, leaving
the run stuck forever.
Closes #
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
Verified the pricing definition displays the correct tier amount on the
plan selection page.
---
## Changelog
Fixed incorrect pricing tier for additional realtime connections from
$10/month per 100 to $10/month per 1000.
---
## Screenshots
N/A
💯https://claude.ai/code/session_015QrZZJHPWta3QCBnhX2Pff
Co-authored-by: Claude <noreply@anthropic.com>
Fix slow fair queue processing by removing spurious cooloff on
concurrency blocks and fixing a race condition where retry attempt
counts were not atomically updated during message re-queue.
Removed cooloff entirely from the batch queue
- Fix for series color assignment being out of sync with the graph
(ensures added series colors match their graph representation)
- Reload widgets when returning to screen if props changed (prevents
stale widgets after filtering and scrolling)
- Add .server-changes/ convention for tracking server-only changes
- Create scripts/enhance-release-pr.mjs to deduplicate and categorize
changeset PR body
- Create scripts/generate-github-release.mjs to format unified GitHub
release body
- Change release.yml to create one unified GitHub release instead of
per-package releases
- Add update-release job to patch Docker image link after images are
pushed to GHCR
- Update changesets-pr.yml to trigger on .server-changes, enhance PR
body, and clean up consumed files
- Document server changes in CLAUDE.md, CONTRIBUTING.md, CHANGESETS.md,
and RELEASE.md
What changed
- Fixed some functions like dateAdd, toString, ifNotFinite
- Removed all functions that accept lambdas as they're not supported
(yet)
- Added tests for all TRQL functions that use ClickHouse
Adds OpenAPI specs and sidebar pages for four previously undocumented
public endpoints: retrieve
run result, per-task batch trigger, retrieve batch, and retrieve batch
results.
This PR implements a new run TTL system and queue size limits to prevent
unbounded queue growth which should help prevent situations where queues
enter a "death spiral" where the queue will never be able to catch up.
The main/correct way to battle this situation is to enforce a maximum
TTL on all runs (e.g. up to 14 days) where runs that have been queued
for that maximum TTL will get auto-expired, making room for newer runs
to execute. This required creating a new TTL system that can handle
higher workloads and is now deeply integrated into the RunQueue. When
runs are enqueued with a TTL, they are added to their normal queue as
well as to the TTL queue. When runs are dequeued, they are removed from
both their normal queue and the TTL queue. If runs are dequeued by the
TTL system, they are removed from their normal queue. Both these
dequeues happen automatically so there is no race condition.
The TTL expiration system is also made reliable by expiring runs via a
Redis worker, which is enqueued to atomically inside the TTL dequeue lua
script.
### Optional associated waitpoints
Additionally, this PR implements an optimization where runs that aren't
triggered with a dependent parent run will no longer create an
associated waitpoint. Associated waitpoints are then lazily created if a
dependent run wants to wait for the child run post-facto (via debounce
or idempotency), which is a rare situation but is possible. This means
fewer waitpoint creations but also fewer waitpoint completions for runs
with no dependencies.
### Environment Queue Limits
Prevents any single queue growing too large by enforcing queue size
limits at trigger time.
- Queue size checks happen at trigger time - runs are rejected if queue
would exceed limit
- Dashboard UI shows queue limits on both the Queues page and a new
Limits page
- In-memory caching for queue size checks to reduce Redis load
### Batch trigger fixes
Currently when a batch item cannot be created for whatever reason (e.g.
queue limits) the run will never get created, which means a stalled run
if using `batchTriggerAndWait`. We've updated the system to handle this
differently: now when a batch item cannot be triggered and converted
into a run, we will eventually (after retrying 8 times up to 30s) we
will create a "pre-failed" run with the error details, correctly
resolving the batchTriggerAndWait.
Without doing an expensive query we can’t tell if it’s definitely a v3
projects – like getting run counts.
So let’s just assume if the project hasn’t been upgraded to v4 (by
running dev/deploy CLI with v4) AND the project is older than the v4
release then it’s v3.
Small fixes and improvements to the logs page:
- Clicking the Run ID didn't open inspector
- Swapped the "open link in tab" icon with Runs icon
- Prevent tooltip hovering on Level info
<img width="350" height="206" alt="CleanShot 2026-02-20 at 10 00 37@2x"
src="https://github.com/user-attachments/assets/3e82f24a-c0a1-4c01-a8e9-9e06a8af982a"
/>
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## @trigger.dev/build@4.4.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.1`
## trigger.dev@4.4.1
### Patch Changes
- Add OTEL metrics pipeline for task workers. Workers collect process
CPU/memory, Node.js runtime metrics (event loop utilization, event loop
delay, heap usage), and user-defined custom metrics via
`otel.metrics.getMeter()`. Metrics are exported to ClickHouse with
10-second aggregation buckets and 1m/5m rollups, and are queryable
through the dashboard query engine with typed attribute columns,
`prettyFormat()` for human-readable values, and AI query support.
([#3061](https://github.com/triggerdotdev/trigger.dev/pull/3061))
- Updated dependencies:
- `@trigger.dev/build@4.4.1`
- `@trigger.dev/core@4.4.1`
- `@trigger.dev/schema-to-json@4.4.1`
## @trigger.dev/python@4.4.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.4.1`
- `@trigger.dev/build@4.4.1`
- `@trigger.dev/core@4.4.1`
## @trigger.dev/react-hooks@4.4.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.1`
## @trigger.dev/redis-worker@4.4.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.1`
## @trigger.dev/rsc@4.4.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.1`
## @trigger.dev/schema-to-json@4.4.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.1`
## @trigger.dev/sdk@4.4.1
### Patch Changes
- Add OTEL metrics pipeline for task workers. Workers collect process
CPU/memory, Node.js runtime metrics (event loop utilization, event loop
delay, heap usage), and user-defined custom metrics via
`otel.metrics.getMeter()`. Metrics are exported to ClickHouse with
10-second aggregation buckets and 1m/5m rollups, and are queryable
through the dashboard query engine with typed attribute columns,
`prettyFormat()` for human-readable values, and AI query support.
([#3061](https://github.com/triggerdotdev/trigger.dev/pull/3061))
- Updated dependencies:
- `@trigger.dev/core@4.4.1`
## @trigger.dev/core@4.4.1
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Adds a direct Vercel Marketplace link, documents configuring build
options via the project config page, and adds a warning and workaround
for projects using a Vercel Root Directory
- Adds an end-to-end OTEL metrics pipeline: task workers collect and
export metrics via OpenTelemetry, the webapp ingests them into
ClickHouse, and they're queryable through the existing dashboard query
engine
- Workers emit process CPU/memory metrics (via
`@opentelemetry/host-metrics`) and Node.js runtime metrics (event loop
utilization, event loop delay, heap usage)
- Users can create custom metrics in their tasks via
`otel.metrics.getMeter()` from `@trigger.dev/sdk`
- Metrics are automatically tagged with run context (run ID, task slug,
machine, worker version) so they can be sliced per-run, per-task, or
per-machine
- The TSQL query engine gains metrics table support with typed attribute
columns, `prettyFormat()` for human-readable values, and per-schema time
bucket thresholds
- Includes reference tasks
(`references/hello-world/src/trigger/metrics.ts`) demonstrating
CPU-intensive, memory-ramp, bursty workload, and custom metrics patterns
## What changed
### Metrics collection (packages/core, packages/cli-v3)
- **Metrics export pipeline** — `TracingSDK` now sets up a
`MeterProvider` with a `PeriodicExportingMetricReader` that chains
through `TaskContextMetricExporter` (adds run context attributes) and
`BufferingMetricExporter` (batches exports to reduce overhead)
- **Host metrics** — Enabled `@opentelemetry/host-metrics` for process
CPU, memory, and system-level metrics
- **Node.js runtime metrics** — New `nodejsRuntimeMetrics.ts` module
using `performance.eventLoopUtilization()`, `monitorEventLoopDelay()`,
and `process.memoryUsage()` to emit 6 observable gauges
- File system and diskio metrics
- **Custom metrics** — Exposed `otel.metrics` from `@trigger.dev/sdk` so
users can create counters, histograms, and gauges in their tasks
- **Machine ID** — Stable per-worker machine identifier for grouping
metrics
- **Dev worker** — Drops `system.*` metrics to reduce noise, keeps
sending metrics between runs in warm workers
### Metrics ingestion (apps/webapp)
- **OTEL endpoint** — `otel.v1.metrics.ts` accepts OTEL metric export
requests (JSON and protobuf), converts to ClickHouse rows
- **ClickHouse schema** — `017_create_metrics_v1.sql` with 10-second
aggregation buckets, JSON attributes column, 60-day TTLs
### Query engine (internal-packages/tsql, apps/webapp)
- **Metrics query schema** — Typed columns for metric attributes
(`task_identifier`, `run_id`, `machine_name`, `worker_version`, etc.)
extracted from the JSON attributes column
- **`prettyFormat()`** — TSQL function that annotates columns with
format hints (`bytes`, `percent`, `durationSeconds`) for frontend
rendering without changing the underlying data
- **Per-schema time buckets** — Different tables can define their own
time bucket thresholds (metrics uses tighter intervals than runs)
- **AI query integration** — The AI query service knows about the
metrics table and can generate metric queries
- **Chart improvements** — Better formatting for byte values,
percentages, and durations in charts and tables
### Reference project
- **`references/hello-world/src/trigger/metrics.ts`** — 6 example tasks:
`cpu-intensive`, `memory-ramp`, `bursty-workload`, `sustained-workload`,
`concurrent-load`, `custom-metrics`
## Test plan
- [ ] Build all packages and webapp
- [ ] Start dev worker with hello-world reference project
- [ ] Run `cpu-intensive`, `memory-ramp`, and `custom-metrics` tasks
- [ ] Verify metrics in ClickHouse: `SELECT DISTINCT metric_name FROM
metrics_v1`
- [ ] Query via dashboard AI: "show me CPU utilization over time"
- [ ] Verify `prettyFormat` renders correctly in chart tooltips and
table cells
- [ ] Confirm dev worker drops `system.*` metrics but keeps `process.*`
and `nodejs.*`
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## @trigger.dev/sdk@4.4.0
### Minor Changes
- Added `query.execute()` which lets you query your Trigger.dev data
using TRQL (Trigger Query Language) and returns results as typed JSON
rows or CSV. It supports configurable scope (environment, project, or
organization), time filtering via `period` or `from`/`to` ranges, and a
`format` option for JSON or CSV output.
([#3060](https://github.com/triggerdotdev/trigger.dev/pull/3060))
```typescript
import { query } from "@trigger.dev/sdk";
import type { QueryTable } from "@trigger.dev/sdk";
// Basic untyped query
const result = await query.execute("SELECT run_id, status FROM runs
LIMIT 10");
// Type-safe query using QueryTable to pick specific columns
const typedResult = await query.execute<QueryTable<"runs", "run_id" |
"status" | "triggered_at">>(
"SELECT run_id, status, triggered_at FROM runs LIMIT 10"
);
typedResult.results.forEach((row) => {
console.log(row.run_id, row.status); // Fully typed
});
// Aggregation query with inline types
const stats = await query.execute<{ status: string; count: number }>(
"SELECT status, COUNT(*) as count FROM runs GROUP BY status",
{ scope: "project", period: "30d" }
);
// CSV export
const csv = await query.execute("SELECT run_id, status FROM runs", {
format: "csv",
period: "7d",
});
console.log(csv.results); // Raw CSV string
```
### Patch Changes
- Add `maxDelay` option to debounce feature. This allows setting a
maximum time limit for how long a debounced run can be delayed, ensuring
execution happens within a specified window even with continuous
triggers.
([#2984](https://github.com/triggerdotdev/trigger.dev/pull/2984))
```typescript
await myTask.trigger(payload, {
debounce: {
key: "my-key",
delay: "5s",
maxDelay: "30m", // Execute within 30 minutes regardless of continuous
triggers
},
});
```
- Aligned the SDK's `getRunIdForOptions` logic with the Core package to
handle semantic targets (`root`, `parent`) in root tasks.
([#2874](https://github.com/triggerdotdev/trigger.dev/pull/2874))
- Export `AnyOnStartAttemptHookFunction` type to allow defining
`onStartAttempt` hooks for individual tasks.
([#2966](https://github.com/triggerdotdev/trigger.dev/pull/2966))
- Fixed a minor issue in the deployment command on distinguishing
between local builds for the cloud vs local builds for self-hosting
setups.
([#3070](https://github.com/triggerdotdev/trigger.dev/pull/3070))
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
## @trigger.dev/build@4.4.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
## trigger.dev@4.4.0
### Patch Changes
- Fix runner getting stuck indefinitely when `execute()` is called on a
dead child process.
([#2978](https://github.com/triggerdotdev/trigger.dev/pull/2978))
- Add optional `timeoutInSeconds` parameter to the
`wait_for_run_to_complete` MCP tool. Defaults to 60 seconds. If the run
doesn't complete within the timeout, the current state of the run is
returned instead of waiting indefinitely.
([#3035](https://github.com/triggerdotdev/trigger.dev/pull/3035))
- Fixed a minor issue in the deployment command on distinguishing
between local builds for the cloud vs local builds for self-hosting
setups.
([#3070](https://github.com/triggerdotdev/trigger.dev/pull/3070))
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
- `@trigger.dev/build@4.4.0`
- `@trigger.dev/schema-to-json@4.4.0`
## @trigger.dev/core@4.4.0
### Patch Changes
- Add `maxDelay` option to debounce feature. This allows setting a
maximum time limit for how long a debounced run can be delayed, ensuring
execution happens within a specified window even with continuous
triggers.
([#2984](https://github.com/triggerdotdev/trigger.dev/pull/2984))
```typescript
await myTask.trigger(payload, {
debounce: {
key: "my-key",
delay: "5s",
maxDelay: "30m", // Execute within 30 minutes regardless of continuous
triggers
},
});
```
- Fixed a minor issue in the deployment command on distinguishing
between local builds for the cloud vs local builds for self-hosting
setups.
([#3070](https://github.com/triggerdotdev/trigger.dev/pull/3070))
- fix: vendor superjson to fix ESM/CJS compatibility
([#2949](https://github.com/triggerdotdev/trigger.dev/pull/2949))
Bundle superjson during build to avoid `ERR_REQUIRE_ESM` errors on
Node.js versions that don't support `require(ESM)` by default (<
22.12.0) and AWS Lambda which intentionally disables it.
- Add Vercel integration support to API schemas: `commitSHA` and
`integrationDeployments` on deployment responses, and `source` field for
environment variable imports.
([#2994](https://github.com/triggerdotdev/trigger.dev/pull/2994))
## @trigger.dev/python@4.4.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
- `@trigger.dev/sdk@4.4.0`
- `@trigger.dev/build@4.4.0`
## @trigger.dev/react-hooks@4.4.0
### Patch Changes
- Fix `onComplete` callback firing prematurely when the realtime stream
disconnects before the run finishes.
([#2929](https://github.com/triggerdotdev/trigger.dev/pull/2929))
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
## @trigger.dev/redis-worker@4.4.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
## @trigger.dev/rsc@4.4.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
## @trigger.dev/schema-to-json@4.4.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Closes #<issue>
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
N/A - Documentation and OpenAPI schema updates only.
---
## Changelog
Added comprehensive Queue Management API support:
**OpenAPI Endpoints:**
- `GET /api/v1/queues` - List all queues with pagination support
- `GET /api/v1/queues/{queueParam}` - Retrieve a specific queue by ID,
task ID, or custom queue name
- `POST /api/v1/queues/{queueParam}/pause` - Pause or resume a queue
- `POST /api/v1/queues/{queueParam}/concurrency/override` - Override
queue concurrency limits
- `POST /api/v1/queues/{queueParam}/concurrency/reset` - Reset
concurrency limits to base values
**Schema Definitions:**
- `QueueObject` - Complete queue representation with concurrency details
- `ListQueuesResult` - Paginated queue listing response
**Documentation:**
- Updated `queue-concurrency.mdx` with SDK usage examples for queue
management
- Added 5 new management API documentation pages for each endpoint
- Updated `docs.json` navigation structure with new "Queues API" section
All endpoints support flexible queue identification (by ID, task ID, or
custom queue name) and include TypeScript code samples.
---
## Screenshots
N/A
💯https://claude.ai/code/session_01LyrXwxHCbejvi34fykifPP
Co-authored-by: Claude <noreply@anthropic.com>
Adds a region selector to the Test task page and Replay run dialog, so
users can override the region from the dashboard. Disabled with a
placeholder for dev environments.
Closes#3016
This will prevent internal logs to be added to the
task_events_search_table
Closes #<issue>
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Ran the migration, deleted the old invalid rows and ran new tasks.
The undesired logs are not added to the table.
---
## Changelog
Updated the MATERIALIZED VIEW to also filter for `trace_id != ''`
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Expand documentation for the Vercel integration with detailed usage,
installation, environment variable sync, atomic deployments, and
environment mapping. Replace the previous "coming soon" placeholder with
complete instructions and UI flow for connecting via the Trigger.dev
dashboard or the Vercel Marketplace. Explain required GitHub
integration,
how env vars sync in both directions, which vars are excluded, and how
to control sync behavior. Describe atomic deployments (default for
production), how they gate Vercel deployments to ensure task/app
consistency, and note related configuration changes. Add tips and notes
to guide setup and troubleshooting.
This provides users with actionable guidance to connect Vercel, map
environments, and keep app and tasks in sync without custom CI scripts.
Iss
