- Adds an end-to-end OTEL metrics pipeline: task workers collect and
export metrics via OpenTelemetry, the webapp ingests them into
ClickHouse, and they're queryable through the existing dashboard query
engine
- Workers emit process CPU/memory metrics (via
`@opentelemetry/host-metrics`) and Node.js runtime metrics (event loop
utilization, event loop delay, heap usage)
- Users can create custom metrics in their tasks via
`otel.metrics.getMeter()` from `@trigger.dev/sdk`
- Metrics are automatically tagged with run context (run ID, task slug,
machine, worker version) so they can be sliced per-run, per-task, or
per-machine
- The TSQL query engine gains metrics table support with typed attribute
columns, `prettyFormat()` for human-readable values, and per-schema time
bucket thresholds
- Includes reference tasks
(`references/hello-world/src/trigger/metrics.ts`) demonstrating
CPU-intensive, memory-ramp, bursty workload, and custom metrics patterns
## What changed
### Metrics collection (packages/core, packages/cli-v3)
- **Metrics export pipeline** — `TracingSDK` now sets up a
`MeterProvider` with a `PeriodicExportingMetricReader` that chains
through `TaskContextMetricExporter` (adds run context attributes) and
`BufferingMetricExporter` (batches exports to reduce overhead)
- **Host metrics** — Enabled `@opentelemetry/host-metrics` for process
CPU, memory, and system-level metrics
- **Node.js runtime metrics** — New `nodejsRuntimeMetrics.ts` module
using `performance.eventLoopUtilization()`, `monitorEventLoopDelay()`,
and `process.memoryUsage()` to emit 6 observable gauges
- File system and diskio metrics
- **Custom metrics** — Exposed `otel.metrics` from `@trigger.dev/sdk` so
users can create counters, histograms, and gauges in their tasks
- **Machine ID** — Stable per-worker machine identifier for grouping
metrics
- **Dev worker** — Drops `system.*` metrics to reduce noise, keeps
sending metrics between runs in warm workers
### Metrics ingestion (apps/webapp)
- **OTEL endpoint** — `otel.v1.metrics.ts` accepts OTEL metric export
requests (JSON and protobuf), converts to ClickHouse rows
- **ClickHouse schema** — `017_create_metrics_v1.sql` with 10-second
aggregation buckets, JSON attributes column, 60-day TTLs
### Query engine (internal-packages/tsql, apps/webapp)
- **Metrics query schema** — Typed columns for metric attributes
(`task_identifier`, `run_id`, `machine_name`, `worker_version`, etc.)
extracted from the JSON attributes column
- **`prettyFormat()`** — TSQL function that annotates columns with
format hints (`bytes`, `percent`, `durationSeconds`) for frontend
rendering without changing the underlying data
- **Per-schema time buckets** — Different tables can define their own
time bucket thresholds (metrics uses tighter intervals than runs)
- **AI query integration** — The AI query service knows about the
metrics table and can generate metric queries
- **Chart improvements** — Better formatting for byte values,
percentages, and durations in charts and tables
### Reference project
- **`references/hello-world/src/trigger/metrics.ts`** — 6 example tasks:
`cpu-intensive`, `memory-ramp`, `bursty-workload`, `sustained-workload`,
`concurrent-load`, `custom-metrics`
## Test plan
- [ ] Build all packages and webapp
- [ ] Start dev worker with hello-world reference project
- [ ] Run `cpu-intensive`, `memory-ramp`, and `custom-metrics` tasks
- [ ] Verify metrics in ClickHouse: `SELECT DISTINCT metric_name FROM
metrics_v1`
- [ ] Query via dashboard AI: "show me CPU utilization over time"
- [ ] Verify `prettyFormat` renders correctly in chart tooltips and
table cells
- [ ] Confirm dev worker drops `system.*` metrics but keeps `process.*`
and `nodejs.*`
This will prevent internal logs to be added to the
task_events_search_table
Closes #<issue>
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Ran the migration, deleted the old invalid rows and ran new tasks.
The undesired logs are not added to the table.
---
## Changelog
Updated the MATERIALIZED VIEW to also filter for `trace_id != ''`
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
For now we’re going to always add FINAL to TRQL queries for data
correctness.
In the future we will implement an automated optimization where we use
`SELECT argMax(column, _version)` and `WHERE _is_deleted = 0`. But this
is a more complex change and needs more investigation of downsides.
## ✅ Checklist
- [X] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [X] The PR title follows the convention.
- [X] I ran and tested the code works
---
## Testing
Tested the migration on test env and locally.
Tested query and merge performance.
Generated tasks and observed the ingested data and searches.
---
## Changelog
* New ClickHouse table & MV (task_events_search_v1): A search-optimized
materialized view that filters out debug events, partial spans, and
empty span events at ingestion time.
* ClickHouse client updates: New getLogsSearchListQueryBuilder and
taskEventsSearch accessor on the ClickHouse class.
* LogsListPresenter: Switches to the new search table, uses
triggered_timestamp for cursor pagination instead of unixTimestamp.
* Spans route: Also switches to the new search query builder.
* Seed spanSpammer task: Adds a 10s trace with events and metadata
operations for testing.
Summary
- Implemented metrics dashboards with a built-in dashboard and custom
dashboards
- Added a "Big number” display type
What changed
- New data format for metric layouts and saving/editing layouts
(editing, saving, cancel revert)
- QueryWidget usable on Query page and Metrics dashboards
- Time filtering, auto-reloading and timeBucket() auto-bin support
- Filters added to metrics; widget popover/improved history and blank
states
- Side menu:
- Metrics/Insights section with icons, colors, padding, collapsible
behavior and reordering of custom dashboards
- Move action logic into service for reuse and API querying; refactor
reordering for reuse
<!-- devin-review-badge-begin -->
---
<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/3019"
target="_blank">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: James Ritchie <james@trigger.dev>
Bundle superjson and its dependency (copy-anything) during build to
avoid
ERR_REQUIRE_ESM errors on Node.js versions that don't support
require(ESM)
by default (< 22.12.0) and AWS Lambda which intentionally disables it.
- Add scripts/bundle-superjson.mjs to bundle superjson with esbuild
- Update build script to bundle vendor files before tshy compilation
- Move superjson from dependencies to devDependencies
- Update imports to use vendored bundles
Fixes#2937
<!-- devin-review-badge-begin -->
---
<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2949">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
Summary
- Query: add time limits, performance improvements, and styling updates
Changes
- Add ClickHouse output_text and error_text columns with indexes
- Automatically use _text columns for JSON based on query pattern;
support JSON column data prefixes
- Add idempotency key and scope columns
- Add enforcedWhereClause for tenant and time restrictions, instead of
the old tenant stuff.
- Implement basic time filter limiting and set default time period based
on plan; show message when results are clipped
- UX: resizable code area (including vertical splits), collapsible
sidebar, fix table/chart vertical sizing, max height for chart legend in
fullscreen
- Styling and UI tweaks: improved chart legend styling, more chart
colours, thinner line chart stroke, pricing callout color, improved
layout for callouts
- Features: generate and save AI titles
<!-- devin-review-badge-begin -->
---
<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2953">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
## ✨ Changes
### UI & UX
- Normalized log level display across table and detail view
- Fixed table header scroll behavior and sidebar positioning
- Improved loading state with taller segment and disabled resizing
- Added "no more logs" message with count
- Enhanced keyboard shortcuts
### Filtering & Search
- Streamlined filters: RunId and Task only (removed run filters)
- Side panel closes when filters change
- Fixed logs from previous search remaining in table
- Fixed table scroll position when changing filters
### Backend
- Added performance indexes on message and attributes
(`014_add_task_runs_v2_search_indexes.sql`)
- Added DEBUG level logging by default
- Removed internal logs from display
- Fixed ServiceValidationError forwarding to frontend
- Removed v1 logs API support
Moving usage updates into the run engine to prevent inefficient &
additional incremental updates to the TaskRun table. Read/Modify/Write
pattern is safe inside of the run engine because of the run lock. We can
also now cap the usageDurationMs value from overflowing and causing an
error.
## Why?
This is preventing at least one update per TaskRun and instead updating
these values piggybacking on other updates.
## Aurora PostgreSQL Reader Consistency Notes
### TL;DR
Aurora readers share the same storage as the writer, but maintain
separate in-memory page caches. This means:
- **Storage is always consistent** - writes are synchronously committed
to shared storage
- **Page cache can lag** - typically <100ms, but can cause stale reads
if data is cached
### How It Works
1. Writer commits to shared storage (synchronous 4/6 quorum)
2. Writer sends cache invalidation messages to readers (asynchronous)
3. If reader has data in cache → returns cached (potentially stale)
value
4. If reader has cache miss → fetches from shared storage (always
current)
### Monitoring
```sql
SELECT server_id,
CASE WHEN session_id = 'MASTER_SESSION_ID' THEN 'Writer' ELSE 'Reader' END AS role,
replica_lag_in_msec
FROM aurora_replica_status();
```
Move expensive findMany queries for PENDING_VERSION and
WAITING_FOR_DEPLOY
runs to read replicas to avoid blocking migrations on the primary
database.
Changes:
- Add readOnlyPrisma to SystemResources type
- Pass readOnlyPrisma to systems in RunEngine constructor
- Update pendingVersionSystem to use readOnlyPrisma for findMany
- Update executeTasksWaitingForDeploy to use _replica for findMany
## Summary
- Adds `IF NOT EXISTS` to the migration that adds
`idempotencyKeyOptions` column to prevent errors if the column already
exists
## Migration Checksum Fix
If you've already applied the previous version of this migration, you'll
need to update the checksum in your `_prisma_migrations` table to match
the new migration file.
**Previous checksum:**
`f8876e274e3f7735312275eb24a9c4b40f512ac12a286b2de3add47f66df5b27`
**New checksum:**
`0620a914ddbaf01279576274432e51c41f41502cd4c8de38621625380750e397`
### Fix instructions
Run this SQL command against your database:
```sql
UPDATE "_prisma_migrations"
SET checksum = '0620a914ddbaf01279576274432e51c41f41502cd4c8de38621625380750e397'
WHERE migration_name = '20260116154810_add_idempotency_key_options_to_task_run';
```
This updates the stored checksum to match the modified migration file,
allowing future migrations to proceed without checksum mismatch errors.
## Test plan
- [x] Verified migration applies cleanly on fresh database
- [ ] Verified checksum update works on database with previous migration
applied
🤖 Generated with [Claude Code](https://claude.com/claude-code)
What changed
- Upgraded recharts to 2.15.2
- Added multiple chart types and components: big number, line, stacked,
bar (including zoomable & reference line), big dataset bar, and usage
graph
- Implemented custom legend with animated values, tooltip showing x-axis
data, and hover/highlight behaviors for stacks and legend
- Added loading, no-data, and invalid chart states plus loading spinners
and improved loading animations/layout
- Storybook integration: initial charts setup, separate chart files,
alphabetized menu, chart state toggles, and story updates
- Interaction & UX improvements: zooming (drag/select), crosshair
pointer, show/select dates while zooming, prevent text selection on
drag, hide mouse wheel zoom, capped legend items, axis/legend styling
tweaks, better spacing, and min-height for charts
- Data & state handling: moved date data to route for unified zooming,
moved chartState to main Chart component, moved hard-coded/mock data out
of components, and set chart data when zooming to start/end dates
- Performance & animation: turned off/reduced chart animations, sped up
animated numbers, removed hover transitions for bars
- New UI primitives and layout: Card component, small card updates, SVG
icons, improved segmented control and popover variants, table
improvements (resizable columns, filtering, sorting, scrolling fixes)
- Various fixes and polish: tooltip style fixes, legend value updates,
hover/leave state resets, bar width fixes for small datasets,
type/import fixes, and numerous small style/typo tweaks
---------
Co-authored-by: James Ritchie <james@trigger.dev>
## Summary
- Store the original user-provided idempotency key and scope alongside
the hash
- Expose `ctx.run.idempotencyKey` as the user-provided key (not the
hash)
- Add `ctx.run.idempotencyKeyScope` to show the scope ("run", "attempt",
or "global")
<img width="539" height="450" alt="CleanShot 2026-01-19 at 11 40 46"
src="https://github.com/user-attachments/assets/b6f42991-697e-4314-a164-aef77b8fd25c"
/>
## Problem
Idempotency keys were hashed (SHA-256) before storage, making debugging
difficult since users couldn't see the value they originally set or
search for runs by idempotency key.
## Solution
Attach metadata to the `String` object returned by
`idempotencyKeys.create()` using a Symbol, extract it in the SDK before
the API call, and store it in the database alongside the hash.
```typescript
const key = await idempotencyKeys.create("my-key", { scope: "global" });
await childTask.triggerAndWait(payload, { idempotencyKey: key });
// In child task:
ctx.run.idempotencyKey // "my-key" (previously showed the hash)
ctx.run.idempotencyKeyScope // "global"
```
Test plan
- Trigger task with idempotencyKeys.create() using different scopes (run, attempt, global)
- Verify ctx.run.idempotencyKey returns user-provided key
- Verify ctx.run.idempotencyKeyScope returns correct scope
- Verify PostgreSQL stores idempotencyKeyOptions JSON
- Verify ClickHouse receives idempotency_key_user and idempotency_key_scope via replication
---------
Co-authored-by: James Ritchie <james@trigger.dev>
Closes #<issue>
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
_[Describe the steps you took to test this change]_
---
## Changelog
_[Short description of what has changed]_
---
## Screenshots
_[Screenshots]_
💯
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Summary
- Add nullable projectId field to TaskScheduleInstance.
- Create an index for TaskScheduleInstance.environmentId (added only if
it doesn’t exist, handled concurrently).
- Ensure TaskScheduleInstance.projectId is set everywhere in the
codebase.
Backfilling projectIds, once this is live
```sql
UPDATE "TaskScheduleInstance" tsi
SET "projectId" = ts."projectId"
FROM "TaskSchedule" ts
WHERE tsi."taskScheduleId" = ts."id";
```
* Removed EVENT_REPOSITORY_CLICKHOUSE_ROLLOUT_PERCENT
* Added hasLogsPageAccess featureFlag for logs page
* Replaced attributes with attributes_text for logs to reduce memory
usage and improve query performance
* Added support for event_v1 for logs, now depending on the settings the
logs are fetched either from `task_events_v1` or `task_events_v2`
* Show an error in the interface in cast the repository store is
`postgres`
It’s useful to know when they were modified for debugging and auditing.
For existing rows createdAt and updatedAt are set to now() during the
migration, to avoid a nullable column.
Don’t allow aliased columns to be queried – it was actually safe but
confusing. We call `created_at` -> `triggered_at` but we still allowed
created_at which was confusing.
Now we have nice errors if you try select columns that aren’t
selectable.
Also removed a ClickHouse setting `allow_experimental_object_type` which
worked fine locally but stopped all queries working on ClickHouse Cloud
🤦♂️
## Summary
Optimizes the runs replication service for better CPU efficiency and
throughput when inserting task runs into ClickHouse.
### Key Changes
- **Switch to compact array format** - Uses
`JSONCompactEachRowWithNames` instead of `JSONEachRow` for ClickHouse
inserts, reducing JSON serialization overhead
- **Type-safe tuple arrays** - Introduces `TaskRunInsertArray` and
`PayloadInsertArray` tuple types with compile-time column order
validation
- **Pre-sorted batch inserts** - Sorts inserts by primary key before
flushing for better ClickHouse insert performance
- **Programmatic index generation** - `TASK_RUN_INDEX` and
`PAYLOAD_INDEX` are generated from column arrays to prevent manual
synchronization errors
### Files Changed
- `runsReplicationService.server.ts` - Core optimization to use compact
array inserts
- `@internal/clickhouse` - Added `insertCompactRaw` method and tuple
types
- `taskRuns.ts` - Column definitions, index constants, and insert
functions
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Closes #<issue>
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- Verified log detail view displays correctly with message, metadata,
and attributes
- Tested search highlighting functionality in log messages (escapes
special regex characters)
- Confirmed tabs (Details/Run) switch properly with keyboard shortcuts
(d/r)
- Verified run information loads via async fetcher in Run tab
- Tested close button and Escape key for dismissing the panel
- Verified log details display correct information: level badges, kind
badges, timestamps, trace IDs, span IDs
- Confirmed links to parent spans and run pages work correctly
- Tested with various log levels (ERROR, WARN, INFO, DEBUG, TRACE) and
kinds (SPAN, SPAN_EVENT, LOG_*)
- Verified admin-only fields display correctly when user has admin
access
- Tested data loading states and error states (log not found, run not
found)
---
## Changelog
Created new Logs page.
The information shown is gathered from the spans from each run.
The feature supports all run filters with two new filters for level and
logs text search.
---
## Screenshots
<img width="2059" height="1196" alt="Logs page preview"
src="https://github.com/user-attachments/assets/70b667b4-98cc-4728-855a-2766dd5c1aa5"
/>
💯
---------
Co-authored-by: James Ritchie <james@trigger.dev>
Summary
- Improve query experience and safety across ClickHouse and TSQL.
Changes
- Display JSON columns when in non-pretty mode (no longer show [Object
Object]).
- Sanitize ClickHouse errors originating from TSQL.
- Remove tenant details from errors.
- Add AI-assisted error-fixing for queries.
- Improve code quality and readability.
- Provide autocomplete support for enum values.
- Enforce limits on ClickHouse queries (10s query limit).
- Add org-level and global concurrency limits.
- Warn and train AI to avoid SELECT *; when used, only return core
columns and show info.
- If AI suggests no time range, default to past 7 days.
- Format the default query for readability.
- Add an admin-only EXPLAIN button.
- Prevent impersonation queries from being saved to history.
TRQL (pronounced Treacle like the delicious British dark sweet syrup) is
the TRiggerQueryLanguage. It allows users to safely write queries on
their data. The queries are safely turned into ClickHouse queries which
are tenant-safe and not SQL injectable.
https://github.com/user-attachments/assets/bbfca473-b3fc-4150-8fe6-79e8840a2d29
This started out as a translation of HogQL by PostHog from Python to
TypeScript.
Features
- Tenant safe queries.
- Many underlying ClickHouse features including functions and
aggregations.
- Virtual columns, which are exposed to users as real columns but are
actually expressions.
- Transformations of data types and where clauses.
- Simple JSON path querying.
- Limits on execution time.
- Reporting of query statistics.
## Query page
There’s a new Query page (currently behind a feature flag) where you can
write TRQL queries and execute them against your environment, project or
organization.
Features
- Executing TRQL queries
- Syntax highlighting and errors
- Autocomplete
- AI generation/editing of queries
- Help and examples
- Table with auto-inferred data types from the table schema
- Table cell renderers for our special types like Run ids, environments,
machines, tasks, queues, etc.
- Copy/export as CSV/JSON
- Line and bar graphs with grouping and stacking
- History of queries
This PR fixes some issues with the new BatchQueue by implementing the
full two-phase dequeue process in the FairQueue, and moving the
responsibility of consuming the worker queue to the BatchQueue and
independently enabling it via the `BATCH_QUEUE_WORKER_QUEUE_ENABLED` env
var. We've also introduced the `BATCH_QUEUE_SHARD_COUNT` env var to
control the count of master queue shards in the FairQueue. We can also
control how many queues are considered in each iteration of the master
queue consumer via the `BATCH_QUEUE_MASTER_QUEUE_LIMIT` env var.
This PR will also now skip trying to dequeue from tenants that are at
concurrency capacity, which should lead to fewer issues with low
concurrency tenants blocking higher concurrency tenants from processing.
Adds support for **debounced task runs** - when triggering a task with a
debounce key, subsequent triggers with the same key will reschedule the
existing delayed run instead of creating new runs. This continues until
no new triggers occur within the delay window.
## Usage
```typescript
await myTask.trigger({ userId: "123" }, {
debounce: {
key: "user-123-update",
delay: "5s",
mode: "leading", // default
}
});
```
- **key**: Scoped to the task identifier
- **delay**: How long to wait before executing (supports duration
strings like `"5s"`, `"1m"`)
- **mode**: Either `"leading"` or `"trailing"`. Leading debounce will
use the payload and options from the first run created with the debounce
key. Trailing will use payload and options from the last run.
### "trailing" mode overrides
When using `mode: "trailing"` with debounce, the following options are
updated from the **last** trigger:
- **`payload`** - The task input data
- **`metadata`** - Run metadata
- **`tags`** - Run tags (replaces existing tags)
- **`maxAttempts`** - Maximum retry attempts
- **`maxDuration`** - Maximum compute time
- **`machine`** - Machine preset (cpu/memory)
## Behavior
- **First run wins**: The first trigger creates the run, subsequent
triggers push its execution time later
- **Idempotency keys take precedence**: If both are specified,
idempotency is checked first
- **Max duration**: Configurable via `DEBOUNCE_MAX_DURATION_MS` env var
(default: 10 minutes)
Works with `triggerAndWait` - parent runs correctly block on the
debounced run.
New batch trigger system with larger payloads, streaming ingestion,
larger batch sizes, and a fair processing system.
This PR introduces a new `FairQueue` abstraction inspired by our own
`RunQueue` that enables multi-tenant fair queueing with concurrency
limits. The new `BatchQueue` is built on top of the `FairQueue`, and
handles processing Batch triggers in a fair manner with per-environment
concurrency limits defined per-org. Additionally, there is a global
concurrency limit to prevent the BatchQueue system from creating too
many runs too quickly, which can cause downstream issues.
For this new BatchQueue system we have a completely new batch trigger
creation and ingestion system. Previously this was a single endpoint
with a single JSON body that defined details about the batch as well as
all the items in the batch.
We're introducing a two-phase batch trigger ingestion system. In the
first phase, the BatchTaskRun record is created (and possibly rate
limited). The second phase is another endpoint that accepts an NDJSON
body with each line being a single item/run with payload and options.
At ingestion time all items are added to a queue, in order, and then
processed by the BatchQueue system.
## New batch trigger rate limits
This PR implements a new batch trigger specific rate limit, configured
on the `Organization.batchRateLimitConfig` column, and defaults using
these environment variables:
- `BATCH_RATE_LIMIT_REFILL_RATE` defaults to 10
- `BATCH_RATE_LIMIT_REFILL_INTERVAL` the duration interval, defaults to
`"10s"`
- `BATCH_RATE_LIMIT_MAX` defaults to 1200
This rate limiter is scoped to the environment ID and controls how many
runs can be submitted via batch triggers per interval. The SDK handles
the retrying side.
## Batch queue concurrency limits
The new column `Organization.batchQueueConcurrencyConfig` now defines an
org specific `processingConcurrency` value, with a backup of the env var
`BATCH_CONCURRENCY_LIMIT_DEFAULT` which defaults to 10. This controls
how many batch queue items are processed concurrently per environment.
There is also a global rate limit for the batch queue set via the
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` which defaults to being disabled. If
set, the entire batch queue system won't process more than
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` items per second. This allows
controlling the maximum number of runs created per second via batch
triggers.
## Batch trigger settings
- `STREAMING_BATCH_MAX_ITEMS` controls the maximum number of items in a
single batch
- `STREAMING_BATCH_ITEM_MAXIMUM_SIZE` controls the maximum size of each
item in a batch
- `BATCH_CONCURRENCY_DEFAULT_CONCURRENCY` controls the default
environment concurrency
- `BATCH_QUEUE_DRR_QUANTUM` how many credits each environment gets each
round for the DRR scheduler
- `BATCH_QUEUE_MAX_DEFICIT` the maximum deficit for the DRR scheduler
- `BATCH_QUEUE_CONSUMER_COUNT` how many queue consumers to run
- `BATCH_QUEUE_CONSUMER_INTERVAL_MS` how frequently they poll for items
in the queue
### Configuration Recommendations by Use Case
**High-throughput priority (fairness acceptable at 0.98+):**
```env
BATCH_QUEUE_DRR_QUANTUM=25
BATCH_QUEUE_MAX_DEFICIT=100
BATCH_QUEUE_CONSUMER_COUNT=10
BATCH_QUEUE_CONSUMER_INTERVAL_MS=50
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=25
```
**Strict fairness priority (throughput can be lower):**
```env
BATCH_QUEUE_DRR_QUANTUM=5
BATCH_QUEUE_MAX_DEFICIT=25
BATCH_QUEUE_CONSUMER_COUNT=3
BATCH_QUEUE_CONSUMER_INTERVAL_MS=100
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=5
```
This PR makes our image builds deterministic and reproducible by
ensuring that identical source code always produces the same image
layers and image digest. This means that deployments where nothing has
changed will no longer invalidate the image cache in our worker cluster
nodes, thus avoid making the cold starts for runs worse.
**Context**
New deployments currently increase the cold start times for runs, as
they generate a new image which needs to be pulled in the worker cluster
where runs are executed. It happens also when the source code for the
deployment has not changed due to non-deterministic steps in our build
system. This addresses the latter issue by making builds reproducible.
**Main changes**
- Avoided baking `TRIGGER_DEPLOYMENT_ID` and
`TRIGGER_DEPLOYMENT_VERSION` in the image, we now pass these via the
supervisor instead.
- Used `json-stable-stringify` for consistent key ordering in the files
we generate for the build, e.g., `package.json`, `build.json`,
`index.json`.
- Removed `metafile.json` from the image contents as it is not actually
used in the container. This is only relevant for the `analyze` command.
- Added `SOURCE_DATE_EPOCH=0` and `rewrite-timestamp=true` to Docker
builds to normalize file timestamps.
- Removed some `timings` and `outputHashes` from build outputs and
manifests.
The builds are now reproducible for both native build server and Depot
paths. This should also lead to better image layer cache reuse in
general.
This PR applies a small change to the deployments table to keep track
of:
- where the deployment was triggered from
- build server metadata, if the build server was involved
Add validation for logical replication publication configuration. Helps
diagnose an issue where runs are no longer replicated to clickhouse
because of a configuration issue with the replication publication.
## Problem
The `LogicalReplicationClient` only checked if a publication existed,
not if it was correctly configured. This caused a silent failure where:
- Replication would start successfully
- Transaction boundaries (begin/commit) were received
- **But no actual data changes were replicated**
This happened when a publication existed but:
1. Had no tables associated with it
2. Was missing required actions (e.g., `delete`)
## Solution
Added `#validatePublicationConfiguration()` method that validates:
- ✅ Publication includes the expected table
- ✅ Publication has all required actions configured
When validation fails, error messages include the exact SQL command to
fix the issue:
**Missing table:**
```
Publication 'task_runs_to_clickhouse_v1_publication' exists but has NO TABLES configured.
Expected table: "public.TaskRun".
Run: ALTER PUBLICATION task_runs_to_clickhouse_v1_publication ADD TABLE "TaskRun";
```
**Missing actions:**
```
Publication 'task_runs_to_clickhouse_v1_publication' is missing required actions.
Expected: [insert, update, delete], Current: [insert, update], Missing: [delete].
Run: ALTER PUBLICATION task_runs_to_clickhouse_v1_publication SET (publish = 'insert, update, delete');
```
This prevents silent data loss and makes debugging configuration issues
much easier.
This PR adds support for CLI deployments using the native build server.
**Background**
The deployment command currently does the following:
- bundles the code
- submits the build context to our external build provider and waits for
the build
- triggers deployment state transitions using the platform API
Upstream build provider outages cause issue with deployments,
potentially blocking deployments entirely. We recently introduced the
`--force-local-build` flag as a fallback to enable deployment without a
dependency on the upstream build provider, though it requires users to
have docker in their systems. This PR continues that work by providing a
remote build path which uses our own build server and does not rely on
the external provider.
**Changes in this PR**
Introduced the new `--native-build-server` flag, which does the
following:
- scans all files relevant for the Trigger deployment and evaluates
ignore rules
- packages it up in an archive and uploads it as a deployment artifact
- queues the deployment and triggers the build
- streams logs from the build server
This no longer relies on external build services. Also deployment state
transitions happen on the server-side, giving us more flexibility to
evolve the flow and schemas of related deployment API endpoints. In
general it gives us better control of the whole build and deployment
process. This path will eventually become the default.
The `--detach` flag is also new, allowing to trigger deployments without
waiting for the result.
The deployment artifacts are uploaded via pre-signed URLs to avoid
unnecessary load on the platform. The new `/artifacts` endpoint
generates the pre-signed URLs; size limits are enforced on s3. This
endpoint is deliberately generic, we could extend it in the future to
upload other artifacts client-side in a similar way, e.g., large payload
packets.
* Don't use the organization max concurrency anymore
* Early draft of the concurrency page
* WIP adding a new stepper input component
* Move stepper to be alphabetical
* When max value is reached, disabled the + button
* Show placeholder if you delete all numbers
* Make all the html input values available to the component
* Adds size variants
* Move stepper into its own component
* Work on showing the extra concurrency
* The purchase form styling and functionality (minus actually purchasing)
* New style for outline input fields
* Concurrency purchasing working
* Purchasing concurrency and quota emails working
* Improvements to the modal
* Show cost breakdown in the modal
* Fix for allocated concurrency including DEV
* Improved types
* Allocating concurrency is working
* Live updates total env concurrency
* Implemented reset
* Fix for concurrency allocation editing across multiple projects
* Tabular numbers
* Added an error from allocating concurrency
* Fixes for allocating concurrency where it didn't calculate correctly
* "Increase limit" link to concurrency page
* Indent environments
* Added Preview limit when updating concurrency for an org
* Show error when changing plan fails
* Added maximumProjectCount column to Org
* Limit project count and display a rich error toast (with title and button now)
* Added title and button to toasts. Use it for new project error
* @trigger.dev/platform 1.0.20
* Allow submitting zero concurrency so you can downgrade back to nothing
* Use the server as the truth for omitted environments
* Updated the pricing panels
---------
Co-authored-by: James Ritchie <james@trigger.dev>
Eric Allam
