Commit Graph

373 Commits

Author SHA1 Message Date
Iss f0f4527655 docs: added startup_timeout_sec note (#3124) 2026-04-14 15:10:49 +01:00
nicktrn 0e63f8317e feat: add ttl support at task and config levels (#3196)
Add TTL (time-to-live) defaults at task-level and config-level, with
precedence: per-trigger > task > config > dev default (10m).

Docs PR: #3200 (merge after packages are released)
2026-03-30 23:25:07 +01:00
nicktrn 9cb3dcb07c feat(supervisor): compute workload manager (#3114)
Adds the `ComputeWorkloadManager` for routing task execution through the
compute gateway, including full checkpoint/restore support, OTel trace
integration, and template pre-warming.

## Changes

**Compute workload manager**
(`apps/supervisor/src/workloadManager/compute.ts`)
- Routes instance create, snapshot, delete, and restore through the
compute gateway API
- Wide event logging on create with full timing and context
- Configurable gateway timeout, auth token, image digest stripping

**Compute snapshot service**
(`apps/supervisor/src/services/computeSnapshotService.ts`)
- Timer wheel for delayed snapshot dispatch (avoids wasted work on
short-lived waitpoints)
- Configurable dispatch concurrency limit
(`COMPUTE_SNAPSHOT_DISPATCH_LIMIT`)
- Snapshot-complete callback handler with suspend completion reporting
- Trace context management and OTel span emission for snapshot
operations

**OTel trace service**
(`apps/supervisor/src/services/otlpTraceService.ts`)
- Fire-and-forget OTLP span emission for compute operations (provision,
restore, snapshot)
- BigInt nanosecond conversion preserving sub-ms precision for span
ordering

**Template creation**
(`apps/webapp/app/v3/services/computeTemplateCreation.server.ts`)
- Three-mode rollout: required (MICROVM projects), shadow (feature flag
/ percentage), skip
- Integrated into deploy finalize flow

**Shared compute package** (`internal-packages/compute/`)
- Gateway client with namespace-based API (instances, templates,
snapshots)
- Zod schemas for all gateway request/response types

**Database**
- `COMPUTE` variant added to `TaskRunCheckpointType` enum
- `WorkloadType` enum and column on `WorkerInstanceGroup`
- `hasComputeAccess` feature flag

**Env / config**
- Compute gateway URL, auth token, timeout
- Snapshot enable flag, delay, dispatch limit
- Dedicated OTLP endpoint for compute spans
(`COMPUTE_TRACE_OTLP_ENDPOINT`)
2026-03-29 22:03:59 +01:00
Oskar Otwinowski 8244ac6f84 feat(Notifications): Platform notifications for CLI and Dashboard (#3254)
For human reviewer:

- Check if Redis connection + code makes sense
- Check CLI methods (it's on a hotpath)
- Check DB Migrations and new tables


##  Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

Spawning new CLI / Dashboard notifications, check MVP, check if failures
not produce any problems with CLI/Dashboard

---

## Changelog

Added notifications mechanism for Dashboard and CLI

---

## Screenshots



💯
2026-03-26 14:19:17 +01:00
Eric Allam c00dae006b feat(mcp): add get_span_details tool (#3255)
Returns the fully detailed span with attributes and AI enrichment data
2026-03-24 13:18:49 +00:00
Eric Allam 774007e9b9 fix(mcp): get_run_details only caches completed runs (#3253) 2026-03-24 11:28:52 +00:00
Saadi Myftija d4772b5f60 feat: run annotations (#3241)
Adds an `annotations` JSONB column to task runs that captures where and
how each run was triggered.
This enables filtering and analyzing trigger origins without querying up
the run tree. Also enables making scheduling decisions based on the
trigger source, e.g., use separate affinities for scheduled runs.

Each run records:
- **triggerSource**: who initiated it (sdk, api, dashboard, cli, mcp,
schedule)
- **triggerAction**: what kind of action (trigger, replay, test)
- **rootTriggerSource**: the trigger source of the root ancestor,
propagated through the entire run
 tree
- **rootScheduleId**: schedule id, in case the run tree was triggered
from a schedule

Currently the main motivation for annotations it to determine whether a
run is part of a schedule-originated tree without traversing ancestors.

### A couple of design considerations
- **Decoupled source from method**: triggerSource and triggerAction are
separate fields to avoid
combinatorial explosion (every new source × every new action)
- **Server-side first**: all annotation values are primarily determined
on the server, only a minor SDK change needed
- **Forward-compatible**: annotation fields use
`z.enum([...]).or(anyString)` so new values can be
added without breaking validation; we currently don't need an explicit
version field for annotations.

Note: `metadata` would have been a more fitting name for the db column,
as it is consistent with other tables where we store this type of
information. It is already in use to store user metadata though, so we
go with `annotations` instead.
2026-03-23 16:07:30 +01:00
Eric Allam 54d95ee4b9 feat: AI prompt management dashboard and enhanced span inspectors (#3244)
- Full prompt management UI: list, detail, override, and version
management for AI prompts defined with `prompts.define()`
- Rich AI span inspectors for all AI SDK operations with token usage,
messages, and prompt context
- Real-time generation tracking with live polling and filtering

## Prompt management

Define prompts in your code with `prompts.define()`, then manage
versions and overrides from the dashboard without redeploying:

```typescript
import { task, prompts } from "@trigger.dev/sdk";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { z } from "zod";

const supportPrompt = prompts.define({
  id: "customer-support",
  model: "gpt-4o",
  variables: z.object({
    customerName: z.string(),
    plan: z.string(),
    issue: z.string(),
  }),
  content: `You are a support agent for Acme SaaS.
Customer: {{customerName}} ({{plan}} plan)
Issue: {{issue}}
Respond with empathy and precision.`,
});

export const supportTask = task({
  id: "handle-support",
  run: async (payload) => {
    const resolved = await supportPrompt.resolve({
      customerName: payload.name,
      plan: payload.plan,
      issue: payload.issue,
    });

    const result = await generateText({
      model: openai(resolved.model ?? "gpt-4o"),
      system: resolved.text,
      prompt: payload.issue,
      ...resolved.toAISDKTelemetry(),
    });

    return { response: result.text };
  },
});
```

The prompts list page shows each prompt with its current version, model,
override status, and a usage sparkline over the last 24 hours.

From the prompt detail page you can:

- **Create overrides** to change the prompt template or model without
redeploying. Overrides take priority over the deployed version when
`prompt.resolve()` is called.
- **Promote** any code-deployed version to be the current version
- **Browse generations** across all versions with infinite scroll and
live polling for new results
- **Filter** by version, model, operation type, and provider
- **View metrics** (total generations, avg tokens, avg cost, latency)
broken down by version

## AI span inspectors

Every AI SDK operation now gets a custom inspector in the run trace
view:

- **`ai.generateText` / `ai.streamText`** — Shows model, token usage,
cost, the full message thread (system prompt, user message, assistant
response), and linked prompt details
- **`ai.generateObject` / `ai.streamObject`** — Same as above plus the
JSON schema and structured output
- **`ai.toolCall`** — Shows tool name, call ID, and input arguments
- **`ai.embed`** — Shows model and the text being embedded

For generation spans linked to a prompt, a "Prompt" tab shows the prompt
metadata, the input variables passed to `resolve()`, and the template
content from the prompt version.

All AI span inspectors include a compact timestamp and duration header.

## Other improvements

- Resizable panel sizes now persist across page refreshes (patched
`@window-splitter/state` to fix snapshot restoration)
- Run page panels also persist their sizes
- Fixed `<div>` inside `<p>` DOM nesting warnings in span titles and
chat messages
- Added Operations and Providers filters to the AI metrics dashboard

## Screenshots

<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 14
17@2x"
src="https://github.com/user-attachments/assets/f3e59989-a2fa-4990-a9d0-3cacda431868"
/>

<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15
37@2x"
src="https://github.com/user-attachments/assets/2f2d02df-2d2b-44fb-ac6f-9153f6a6c387"
/>

<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15
54@2x"
src="https://github.com/user-attachments/assets/baa161e0-ef91-4fa4-a55f-986b71cccdf0"
/>
2026-03-23 06:23:19 +00:00
Eric Allam 1cfc296c6b feat(ai): LLM metrics tracking and AI span inspector (#3213)
- Automatic LLM cost enrichment for AI SDK spans (streamText,
generateText, generateObject) or any other spans that use semantic
gen_ai attributes with support for 145+ models
- New AI span inspector sidebar showing model, tokens, cost, messages,
tool calls, and response text
- LLM metrics dual-write to ClickHouse `llm_metrics_v1` table for
analytics
- LLM metrics built-in dashboard (unlinked at the moment)
- Provider cost fallback — uses gateway/OpenRouter reported costs from
`providerMetadata` when registry pricing is unavailable
- Prefix-stripping for gateway/OpenRouter model names (e.g.
`mistral/mistral-large-3` matches `mistral-large-3` pricing)
- Admin dashboard for managing LLM model pricing (list, create, edit,
delete, search, test pattern matching)
- Missing models detection page — queries ClickHouse for unpriced models
with sample spans and Claude Code-ready prompts for adding pricing
- AI span seed script (`pnpm run db:seed:ai-spans`) with 51 spans across
12 provider systems for local dev testing
- UI fixes: `completionTokens`/`promptTokens` aliases,
`ai.response.object` display for generateObject, cache read/write token
breakdown

## Screenshots:

<img width="1030" height="104" alt="CleanShot 2026-03-17 at 16 48 54@2x"
src="https://github.com/user-attachments/assets/bc8fccda-e48b-4d0c-bfb1-e620064e5979"
/>

<img width="1094" height="1512" alt="CleanShot 2026-03-17 at 16 49
23@2x"
src="https://github.com/user-attachments/assets/c2424569-d07e-4d67-a436-e8250043a1ee"
/>

<img width="1074" height="1412" alt="CleanShot 2026-03-17 at 16 49
18@2x"
src="https://github.com/user-attachments/assets/22342ac4-4769-45d1-a328-a24fb9a82a50"
/>

<img width="1012" height="2292" alt="CleanShot 2026-03-17 at 16 39
01@2x"
src="https://github.com/user-attachments/assets/59e327d1-6652-4293-8be0-bb8326e5fbc5"
/>

<img width="3680" height="2392" alt="CleanShot 2026-03-15 at 08 29
38@2x"
src="https://github.com/user-attachments/assets/1f77beb8-de67-495b-b890-bcdb8d7f1fe8"
/>

---------

Co-authored-by: James Ritchie <james@trigger.dev>
2026-03-17 18:26:43 +00:00
Eric Allam dbbe9f77f9 feat(cli): Expand and improve the MCP server and dev CLI command (#3224)
## Summary

Major expansion of the MCP server (14 → 25 tools), context efficiency
optimizations, new API endpoints, and a fix for the dev CLI leaking
build directories on disk.

### New MCP tools

- **Query & analytics**: `get_query_schema`, `query`, `list_dashboards`,
`run_dashboard_query` — query your data using TRQL directly from AI
assistants
- **Profile management**: `whoami`, `list_profiles`, `switch_profile` —
see and switch CLI profiles per-project (persisted to
`.trigger/mcp.json`)
- **Dev server control**: `start_dev_server`, `stop_dev_server`,
`dev_server_status` — start/stop `trigger dev` and stream build output
- **Task introspection**: `get_task_schema` — get payload schema for a
specific task (split out from `get_current_worker` to reduce context)

### New API endpoints

- `GET /api/v1/query/schema` — discover TRQL tables and columns
(server-driven, multi-table)
- `GET /api/v1/query/dashboards` — list built-in dashboard widgets and
their queries

### New features

- **`--readonly` flag** — hides write tools (`deploy`, `trigger_task`,
`cancel_run`) so agents can't make changes
- **`read:query` JWT scope** — new authorization scope for query
endpoints, with per-table granularity (`read:query:runs`,
`read:query:llm_metrics`, etc.)
- **Paginated trace output** — `get_run_details` now paginates trace
events via cursor, caching the full trace in a temp file so subsequent
pages don't re-fetch
- **MCP tool annotations** — all tools now have
`readOnlyHint`/`destructiveHint` annotations for clients that support
them
- **Project-scoped profile persistence** — `switch_profile` saves to
`.trigger/mcp.json` (gitignored), automatically loaded on next MCP
server start

### Context optimizations

- `get_query_schema` requires a table name — returns one table's schema
instead of all tables (60-80% fewer tokens)
- `get_current_worker` no longer inlines payload schemas — use
`get_task_schema` for specific tasks
- Query results formatted as text tables instead of JSON (~50% fewer
tokens for flat data)
- `cancel_run`, `list_deploys`, `list_preview_branches` formatted as
text instead of raw `JSON.stringify()`
- Schema and dashboard API responses cached (1hr and 5min respectively)

### Bug fixes

- Fixed `search_docs` failing due to renamed upstream Mintlify tool
(`SearchTriggerDev` → `search_trigger_dev`)
- Fixed `list_deploys` failing when deployments have null
`runtime`/`runtimeVersion` fields (fixes #3139)
- Fixed `list_preview_branches` crashing due to incorrect response shape
access
- Fixed `metrics` table column documented as `value` instead of
`metric_value` in query docs
- Fixed `/api/v1/query` not accepting JWT auth (added `allowJWT: true`)

### Dev CLI build directory fix

The dev CLI was leaking `build-*` directories in `.trigger/tmp/` on
every rebuild, accumulating hundreds of MB over time (842MB observed).
Three layers of protection added:

1. **During session**: deprecated workers are pruned (capped at 2
retained) when no active runs reference them, preventing unbounded
accumulation
2. **On SIGKILL/crash**: the watchdog process now cleans up
`.trigger/tmp/` when it detects the parent CLI was killed
3. **On next startup**: existing `clearTmpDirs()` wipes any remaining
orphans

## Test plan

- [ ] `pnpm run mcp:smoke` — 17 automated smoke tests for all read-only
MCP tools
- [ ] `pnpm run mcp:test list` — verify 25 tools registered (21 in
`--readonly` mode)
- [ ] `pnpm run mcp:test --readonly list` — verify write tools hidden
- [ ] Manual: start dev server, trigger task, rebuild multiple times,
verify build dirs stay capped at 4
- [ ] Manual: SIGKILL the dev CLI, verify watchdog cleans up
`.trigger/tmp/`
- [ ] Verify new API endpoints return correct data: `GET
/api/v1/query/schema`, `GET /api/v1/query/dashboards`

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-03-17 11:38:23 +00:00
Eric Allam 436f20efc6 feat(cli): auto-cancel dev runs on CLI exit via detached watchdog (#3191)
When the dev CLI exits (e.g. ctrl+c via pnpm), runs that were
mid-execution
previously stayed stuck in EXECUTING status for up to 5 minutes until
the
heartbeat timeout fired. Now they are cancelled within seconds.

The dev CLI spawns a lightweight detached watchdog process at startup.
The
watchdog monitors the CLI process ID and, when it detects the CLI has
exited,
calls a new POST /engine/v1/dev/disconnect endpoint to cancel all
in-flight
runs immediately (skipping PENDING_CANCEL since the worker is known to
be dead).

Watchdog design:
- Fully detached (detached: true, stdio: ignore, unref()) so it survives
  even when pnpm sends SIGKILL to the process tree
- Active run IDs maintained via atomic file write
(.trigger/active-runs.json)
- Single-instance guarantee via PID file (.trigger/watchdog.pid)
- Safety timeout: exits after 24 hours to prevent zombie processes
- On clean shutdown, the watchdog is killed (no disconnect needed)

Disconnect endpoint:
- Rate-limited: 5 calls/min per environment
- Capped at 500 runs per call
- Small counts (<= 25): cancelled inline with pMap concurrency 10
- Large counts: delegated to the bulk action system
- Uses finalizeRun: true to skip PENDING_CANCEL and go straight to
FINISHED

Run engine change:
- cancelRun() now respects finalizeRun when the run is in EXECUTING
status,
skipping the PENDING_CANCEL waiting state and going directly to FINISHED
2026-03-09 12:14:43 +00:00
Eric Allam 540e1c86a4 feat: Input Streams - Bidirectional task communication (#3146)
Input streams enable sending typed data to executing tasks from external
callers — backends, frontends, or other tasks. This unlocks interactive
use cases like approval UIs, cancel buttons, chat interfaces, and
human-in-the-loop AI workflows where the task needs to receive data
while running.

Three consumption patterns inside a task:

* `.wait()` — Suspend the task until data arrives (process freed, most
efficient)
* `.once()` — Wait for the next message (process stays alive)
* `.on()` — Subscribe to a continuous stream of messages

One send pattern from outside:

* `.send(runId, data)` — Send typed data to a specific run's input
stream

## User-facing API

### Define a typed input stream

```ts
import { streams, task } from "@trigger.dev/sdk";

const approval = streams.input<{ approved: boolean; reviewer: string }>({ id: "approval" });
```

### Consume inside a task

```ts
export const myTask = task({
  id: "my-task",
  run: async () => {
    // Pattern 1: Suspend until data arrives (most efficient — frees the process)
    const result = await approval.wait({ timeout: "5m" });

    // Pattern 2: Wait for next message (process stays alive)
    const data = await approval.once().unwrap();

    // Pattern 3: Subscribe to multiple messages
    approval.on((data) => { /* handle each message */ });
  },
});
```

### Send from outside

```ts
// From a backend (using secret API key)
await approval.send(runId, { approved: true, reviewer: "alice" });

// From a frontend (using public JWT token from trigger response)
const { send } = useInputStreamSend("approval", runId, { accessToken });
send({ approved: true, reviewer: "alice" });
```

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-03-02 16:49:54 +00:00
Eric Allam 23c327ea16 fix(metrics): disable collecting system and filesystem metrics in deployed workers (#3106) 2026-02-20 15:45:35 +00:00
Eric Allam 469b039090 feat: OTEL metrics pipeline for task workers (#3061)
- Adds an end-to-end OTEL metrics pipeline: task workers collect and
export metrics via OpenTelemetry, the webapp ingests them into
ClickHouse, and they're queryable through the existing dashboard query
engine
- Workers emit process CPU/memory metrics (via
`@opentelemetry/host-metrics`) and Node.js runtime metrics (event loop
utilization, event loop delay, heap usage)
- Users can create custom metrics in their tasks via
`otel.metrics.getMeter()` from `@trigger.dev/sdk`
- Metrics are automatically tagged with run context (run ID, task slug,
machine, worker version) so they can be sliced per-run, per-task, or
per-machine
- The TSQL query engine gains metrics table support with typed attribute
columns, `prettyFormat()` for human-readable values, and per-schema time
bucket thresholds
- Includes reference tasks
(`references/hello-world/src/trigger/metrics.ts`) demonstrating
CPU-intensive, memory-ramp, bursty workload, and custom metrics patterns

## What changed

### Metrics collection (packages/core, packages/cli-v3)
- **Metrics export pipeline** — `TracingSDK` now sets up a
`MeterProvider` with a `PeriodicExportingMetricReader` that chains
through `TaskContextMetricExporter` (adds run context attributes) and
`BufferingMetricExporter` (batches exports to reduce overhead)
- **Host metrics** — Enabled `@opentelemetry/host-metrics` for process
CPU, memory, and system-level metrics
- **Node.js runtime metrics** — New `nodejsRuntimeMetrics.ts` module
using `performance.eventLoopUtilization()`, `monitorEventLoopDelay()`,
and `process.memoryUsage()` to emit 6 observable gauges
- File system and diskio metrics
- **Custom metrics** — Exposed `otel.metrics` from `@trigger.dev/sdk` so
users can create counters, histograms, and gauges in their tasks
- **Machine ID** — Stable per-worker machine identifier for grouping
metrics
- **Dev worker** — Drops `system.*` metrics to reduce noise, keeps
sending metrics between runs in warm workers

### Metrics ingestion (apps/webapp)
- **OTEL endpoint** — `otel.v1.metrics.ts` accepts OTEL metric export
requests (JSON and protobuf), converts to ClickHouse rows
- **ClickHouse schema** — `017_create_metrics_v1.sql` with 10-second
aggregation buckets, JSON attributes column, 60-day TTLs

### Query engine (internal-packages/tsql, apps/webapp)
- **Metrics query schema** — Typed columns for metric attributes
(`task_identifier`, `run_id`, `machine_name`, `worker_version`, etc.)
extracted from the JSON attributes column
- **`prettyFormat()`** — TSQL function that annotates columns with
format hints (`bytes`, `percent`, `durationSeconds`) for frontend
rendering without changing the underlying data
- **Per-schema time buckets** — Different tables can define their own
time bucket thresholds (metrics uses tighter intervals than runs)
- **AI query integration** — The AI query service knows about the
metrics table and can generate metric queries
- **Chart improvements** — Better formatting for byte values,
percentages, and durations in charts and tables

### Reference project
- **`references/hello-world/src/trigger/metrics.ts`** — 6 example tasks:
`cpu-intensive`, `memory-ramp`, `bursty-workload`, `sustained-workload`,
`concurrent-load`, `custom-metrics`

## Test plan

- [ ] Build all packages and webapp
- [ ] Start dev worker with hello-world reference project
- [ ] Run `cpu-intensive`, `memory-ramp`, and `custom-metrics` tasks
- [ ] Verify metrics in ClickHouse: `SELECT DISTINCT metric_name FROM
metrics_v1`
- [ ] Query via dashboard AI: "show me CPU utilization over time"
- [ ] Verify `prettyFormat` renders correctly in chart tooltips and
table cells
- [ ] Confirm dev worker drops `system.*` metrics but keeps `process.*`
and `nodejs.*`
2026-02-20 13:16:34 +00:00
Saadi Myftija 7af789bc3e fix(deployments): external build token generation issue (#3070)
Fixes an issue introduced in #3024.

The behavior for local builds in older CLI versions relies on
`externalBuildData` to be defined to distinguish from the self-hosting
local build path, even though it doesn't actually use the token.
2026-02-16 19:05:40 +01:00
Eric Allam 062bcaece8 feat(mcp): add timeout parameter to wait_for_run_to_complete tool (#3035)
## Summary

- Adds an optional `timeoutInSeconds` parameter (default 60s) to the
`wait_for_run_to_complete` MCP tool
- If the run doesn't complete within the timeout, returns the current
run state instead of blocking indefinitely
- Uses `AbortSignal.timeout()` combined with the existing MCP signal

Fixes #3032
2026-02-12 16:14:25 +00:00
nicktrn 279102c17c fix(cli): reject execute() immediately when child process is dead (#2978)
## Summary
- When a child process crashes and a retry (`RETRY_IMMEDIATELY`) is
attempted on the same `TaskRunProcess`, `execute()` hangs forever
because the IPC send is silently skipped and the attempt promise can
never resolve
- This caused runner pods to stay up indefinitely with no heartbeats or
polls
- Fix: reject the attempt promise immediately when the child is not
connected, so the controller can proceed to warm start or exit

## Test plan
- [x] Added `taskRunProcess.test.ts` — verifies `execute()` rejects
promptly instead of hanging when the child process is dead
- [x] Deploy and verify no more stuck runner pods accumulate over time
2026-01-30 16:44:46 +00:00
nicktrn 7a54a843e2 fix(cli): update command should preserve existing package.json order (#2810)
This fixes a regression introduced in #2778 - stable sort is required
for deterministic builds, but we can safely preserve order for the user
package.json during package updates
2025-12-23 12:52:06 +00:00
Eric Allam 3875bb292a feat(engine): run debounce system (#2794)
Adds support for **debounced task runs** - when triggering a task with a
debounce key, subsequent triggers with the same key will reschedule the
existing delayed run instead of creating new runs. This continues until
no new triggers occur within the delay window.

## Usage

```typescript
await myTask.trigger({ userId: "123" }, {
  debounce: {
    key: "user-123-update",
    delay: "5s",
    mode: "leading", // default
  }
});
```

- **key**: Scoped to the task identifier
- **delay**: How long to wait before executing (supports duration
strings like `"5s"`, `"1m"`)
- **mode**: Either `"leading"` or `"trailing"`. Leading debounce will
use the payload and options from the first run created with the debounce
key. Trailing will use payload and options from the last run.

### "trailing" mode overrides

When using `mode: "trailing"` with debounce, the following options are
updated from the **last** trigger:

- **`payload`** - The task input data
- **`metadata`** - Run metadata
- **`tags`** - Run tags (replaces existing tags)
- **`maxAttempts`** - Maximum retry attempts
- **`maxDuration`** - Maximum compute time
- **`machine`**  - Machine preset (cpu/memory)

## Behavior

- **First run wins**: The first trigger creates the run, subsequent
triggers push its execution time later
- **Idempotency keys take precedence**: If both are specified,
idempotency is checked first
- **Max duration**: Configurable via `DEBOUNCE_MAX_DURATION_MS` env var
(default: 10 minutes)

Works with `triggerAndWait` - parent runs correctly block on the
debounced run.
2025-12-18 16:04:43 +00:00
Saadi Myftija 6d6ed471d1 feat(cli): deterministic image builds for deployments (#2778)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
This PR makes our image builds deterministic and reproducible by
ensuring that identical source code always produces the same image
layers and image digest. This means that deployments where nothing has
changed will no longer invalidate the image cache in our worker cluster
nodes, thus avoid making the cold starts for runs worse.

**Context**
New deployments currently increase the cold start times for runs, as
they generate a new image which needs to be pulled in the worker cluster
where runs are executed. It happens also when the source code for the
deployment has not changed due to non-deterministic steps in our build
system. This addresses the latter issue by making builds reproducible.

**Main changes**
- Avoided baking `TRIGGER_DEPLOYMENT_ID` and
`TRIGGER_DEPLOYMENT_VERSION` in the image, we now pass these via the
supervisor instead.
- Used `json-stable-stringify` for consistent key ordering in the files
we generate for the build, e.g., `package.json`, `build.json`,
`index.json`.
- Removed `metafile.json` from the image contents as it is not actually
used in the container. This is only relevant for the `analyze` command.
- Added `SOURCE_DATE_EPOCH=0` and `rewrite-timestamp=true` to Docker
builds to normalize file timestamps.
- Removed some `timings` and `outputHashes` from build outputs and
manifests.

The builds are now reproducible for both native build server and Depot
paths. This should also lead to better image layer cache reuse in
general.
2025-12-12 09:48:04 +01:00
nicktrn 74e9246bfa feat(cli): enable zstd compression for deployment images (#2773)
This will speed up ice cold starts (*) for two reasons:
- better compression ratio
- faster decompression

This is a minor release because zstd compression will now be enabled by
default for all deployments.

(*) ice cold starts happen when deploy images are not cached on the
worker node yet. These cold start durations are highly dependent on
image size and as it turns out, also the type of compression used.
2025-12-10 14:49:44 +00:00
Eric Allam c021d1db63 fix(dev): stop max listeners exceeded warning messages when running more than 10 runs concurrently (#2771)
<!-- CURSOR_SUMMARY -->
> [!NOTE]
> Centralizes SIGTERM handling in `DevSupervisor` and removes per-run
SIGTERM listeners in `DevRunController` to avoid
MaxListenersExceededWarning under high concurrency.
> 
> - **Dev runtime**:
> - **SIGTERM handling**: Add centralized handler in
`packages/cli-v3/src/dev/devSupervisor.ts` to gracefully stop all run
controllers; unregisters on `shutdown()`.
> - **Cleanup**: Remove per-controller `SIGTERM` listener and handler
from `packages/cli-v3/src/entryPoints/dev-run-controller.ts` to reduce
event listeners and warnings.
> - **Changeset**: Add patch note in
`.changeset/fuzzy-ghosts-admire.md`.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
5ad2f5341829cebf6fd37a3c616a2db5e4ad936a. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
2025-12-09 09:54:28 +00:00
Saadi Myftija 7fba9e9f6b feat(deployments): add build server meta and trigger source info (#2767)
This PR applies a small change to the deployments table to keep track
of:
- where the deployment was triggered from
- build server metadata, if the build server was involved
2025-12-08 16:29:26 +00:00
nicktrn 0d764d4c46 feat(cli): upgrade bun deployments to v1.3.3 (#2756)
New deployments with `runtime: "bun"` will now use Bun v1.3.3

Link to Bun release: https://bun.com/blog/bun-v1.3.3
2025-12-05 13:31:27 +00:00
Saadi Myftija 4dc956470d fix(deployments): misc fixes for the native build server deployment flow (#2748)
Improved a couple of error messages.

Also fixed an issue with the s2 token caching.
2025-12-04 17:01:47 +01:00
Eric Allam e7fec4097f fix(dev): CLI now properly cleans up the store dir on dev CLI exit (#2744) 2025-12-04 13:13:01 +01:00
Eric Allam d279988e38 fix(workers): prevent ERR_IPC_CHANNEL_CLOSED errors from causing an unhandled exception on TaskRunProcess (#2743) 2025-12-04 12:02:30 +00:00
Saadi Myftija 255a73a2fe feat(deployments): --native-build-server support for the deploy command (#2702)
This PR adds support for CLI deployments using the native build server.

**Background**

The deployment command currently does the following:
- bundles the code
- submits the build context to our external build provider and waits for
the build
- triggers deployment state transitions using the platform API

Upstream build provider outages cause issue with deployments,
potentially blocking deployments entirely. We recently introduced the
`--force-local-build` flag as a fallback to enable deployment without a
dependency on the upstream build provider, though it requires users to
have docker in their systems. This PR continues that work by providing a
remote build path which uses our own build server and does not rely on
the external provider.

**Changes in this PR**

Introduced the new `--native-build-server` flag, which does the
following:
- scans all files relevant for the Trigger deployment and evaluates
ignore rules
- packages it up in an archive and uploads it as a deployment artifact
- queues the deployment and triggers the build
- streams logs from the build server

This no longer relies on external build services. Also deployment state
transitions happen on the server-side, giving us more flexibility to
evolve the flow and schemas of related deployment API endpoints. In
general it gives us better control of the whole build and deployment
process. This path will eventually become the default.

The `--detach` flag is also new, allowing to trigger deployments without
waiting for the result.

The deployment artifacts are uploaded via pre-signed URLs to avoid
unnecessary load on the platform. The new `/artifacts` endpoint
generates the pre-signed URLs; size limits are enforced on s3. This
endpoint is deliberately generic, we could extend it in the future to
upload other artifacts client-side in a similar way, e.g., large payload
packets.
2025-12-03 16:40:21 +01:00
Eric Allam 5b7dfe23b5 feat(cli): implements content-addressable store for the dev CLI build outputs, reducing disk usage (#2725)
* feat(cli): implements content-addressable store for the dev CLI build outputs, reducing disk usage

* fix a few things
2025-12-03 10:17:35 +00:00
nicktrn af9b3e1c99 fix(cli): header will always print the correct profile (#2728) 2025-12-02 17:45:35 +00:00
Eric Allam 72e286af2f feat(otel): support for custom resource attributes via config#telemetry.resource and OTEL_RESOURCE_ATTRIBUTES env var (#2704) 2025-11-24 14:33:07 +00:00
Eric Allam 5e5c97ea4c fix(cli): stop dev runs stuck in dequeued status (fix #2639) (#2699)
* stop deleting the first dev version files on the first change, prevents system failures

* prevent dev runs getting stuck in dequeued status by deleting workers

* add changeset
2025-11-21 15:24:07 +00:00
Saadi Myftija 53047ab648 feat(deployments): external cache support for local builds (#2682)
* Add registry cache support for local builds

* Add changeset
2025-11-14 10:46:31 +01:00
Eric Allam 536d9fa217 feat(realtime): Realtime streams v2 (#2632) 2025-11-11 14:54:00 +00:00
Matt Aitken a342332146 Fix for the MCP tool that gets logs for debugging runs (#2653)
* Fix for the MCP tool that gets logs for debugging runs

This was broken when we changed the data on the backend that returns 
log/span data from runs. We changed the data structured and the internal
API that the MCP client uses was failing to parse with the Zod schema

* add changeset

* Revert "add changeset"

This reverts commit 86eca836d5907fa0d0f8ac595d4d5ebade140514.

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2025-11-11 13:26:07 +00:00
Saadi Myftija 27376df903 fix(cli): show a useful error message when config file is missing (#2650)
* fix(cli): show a useful error message when config file is missing

* Add changeset
2025-10-31 16:34:48 +01:00
Saadi Myftija 2f3f82f3a9 feat(deployments): show local build hint during depot outages (#2646)
* Add an API endpoint to query remote build provider status

* Show local build hint for failed deployments when Depot is down

* Show the local build flag in the help output

* Add changeset

* Fix import

* Fix docs link
2025-10-30 18:09:54 +01:00
nicktrn ae8e83b2d0 chore(runner): move max duration logic into parent process (#2637)
* chore(runner): move max duration logic into parent process

* chore(rsc): remove type-marker package.json

* add changeset

* chore(core): remove irrelevant test after our changes

* chore(core): clarify we don't care about the timeout promise
2025-10-28 13:49:59 +00:00
Saadi Myftija 7f25e82299 feat(deployments): support local builds in cloud (#2628)
* Enable skipping image push during deployment finalization step

* Add endpoint to generate registry credentials for a deployment

* Add a --force-local-build flag to the deployment command to skip remote build

* Do not show the new flag in the help output

* Add changeset

* Remove registry login logs from onLog, not useful

* Rename var

* Update platform package to the latest version
2025-10-23 10:57:36 +02:00
nicktrn 8917478d3c fix(runner): SIGTERM handling during warm start long poll (#2593) 2025-10-08 12:51:10 +01:00
Eric Allam 128bc437f6 feat(otel): Add support for storing run spans and log data in Clickhouse (#2567) 2025-10-01 12:41:18 -07:00
Saadi Myftija cc94d121f2 feat: installing status for deployments (#2544)
* Add installing status to the deployment db schema

* Replace the deployments /start endpoint with /progress

* Show the installing status in the dashboard

* Add installing status to the api schema and cli

* Add changeset
2025-09-24 10:27:43 +02:00
nicktrn db87295049 fix(runner): reduce restore recovery time and deprecated runner false positives (#2523)
* fix(runner): improve restore detection

* chore(supervisor): skip schema parsing when debug logs disabled

* fix(runner): deprecation race condition

* add changeset
2025-09-18 16:59:44 +01:00
nicktrn 885ae5e06f fix(runner): immediate poll to decrease restore time (#2516)
* fix(runner): immediate poll to decrease restore time

* chore: bump default prerelease tag

* fix(cli): s is not a function

* add changeset
2025-09-16 16:17:04 +01:00
Saadi Myftija 7d333e5b3c feat(cli): attach to existing deployments in the build server (#2501)
* Add external build data and image platform to the get deployment endpoint

* If provided, attach to an existing deployment in the deploy command

* Check status for existing deployments

* Add changeset
2025-09-15 17:16:26 +02:00
Eric Allam 6483a0f1c6 fix(core): Improves our schema to JSON Schema conversion (fix for zod 4) (#2483) 2025-09-15 14:11:40 +01:00
Eric Allam f077d49291 feat(engine): Improve execution stalls troubleshooting, align dev and prod behavior, adding heartbeats.yield utility (#2489)
* feat(engine): Improve execution stalls troubleshooting, align dev and prod behavior, adding heartbeats.yield utility

* A few improvements via the 🐇 review

* Allow treating EXECUTION stalls as OOM errors, improve the error message, add more information to the docs, improve resource monitor and add it to the docs

* Add changeset
2025-09-12 14:54:39 +01:00
Saadi Myftija 12bef0a938 feat: env command in the cli (#2485)
* Add a CLI command to list and view env vars

* Add changeset

* Restrict pemissions on env files created with `env pull`

* Escape env vars when exporting to file

* Switch changeset to patch
2025-09-09 13:13:24 +02:00
Eric Allam 892adfe81c fix(mcp): don't require a trigger.config.ts file to be present when installing rules (#2453) 2025-08-29 11:20:45 +01:00
Eric Allam 6f6ca01584 feat(mcp): add wait_for_run_to_complete tool so agents don't spam the get_run_details call after triggering (#2445)
* feat(mcp): add wait_for_run_to_complete tool so agents don't spam the get_run_details call after triggering

This also fixes the search docs MCP tool

* Install mcp using the latest tag, not the specific version
2025-08-28 10:13:46 +01:00