Files
triggerdotdev--trigger.dev/references
Eric Allam 469b039090 feat: OTEL metrics pipeline for task workers (#3061)
- Adds an end-to-end OTEL metrics pipeline: task workers collect and
export metrics via OpenTelemetry, the webapp ingests them into
ClickHouse, and they're queryable through the existing dashboard query
engine
- Workers emit process CPU/memory metrics (via
`@opentelemetry/host-metrics`) and Node.js runtime metrics (event loop
utilization, event loop delay, heap usage)
- Users can create custom metrics in their tasks via
`otel.metrics.getMeter()` from `@trigger.dev/sdk`
- Metrics are automatically tagged with run context (run ID, task slug,
machine, worker version) so they can be sliced per-run, per-task, or
per-machine
- The TSQL query engine gains metrics table support with typed attribute
columns, `prettyFormat()` for human-readable values, and per-schema time
bucket thresholds
- Includes reference tasks
(`references/hello-world/src/trigger/metrics.ts`) demonstrating
CPU-intensive, memory-ramp, bursty workload, and custom metrics patterns

## What changed

### Metrics collection (packages/core, packages/cli-v3)
- **Metrics export pipeline** — `TracingSDK` now sets up a
`MeterProvider` with a `PeriodicExportingMetricReader` that chains
through `TaskContextMetricExporter` (adds run context attributes) and
`BufferingMetricExporter` (batches exports to reduce overhead)
- **Host metrics** — Enabled `@opentelemetry/host-metrics` for process
CPU, memory, and system-level metrics
- **Node.js runtime metrics** — New `nodejsRuntimeMetrics.ts` module
using `performance.eventLoopUtilization()`, `monitorEventLoopDelay()`,
and `process.memoryUsage()` to emit 6 observable gauges
- File system and diskio metrics
- **Custom metrics** — Exposed `otel.metrics` from `@trigger.dev/sdk` so
users can create counters, histograms, and gauges in their tasks
- **Machine ID** — Stable per-worker machine identifier for grouping
metrics
- **Dev worker** — Drops `system.*` metrics to reduce noise, keeps
sending metrics between runs in warm workers

### Metrics ingestion (apps/webapp)
- **OTEL endpoint** — `otel.v1.metrics.ts` accepts OTEL metric export
requests (JSON and protobuf), converts to ClickHouse rows
- **ClickHouse schema** — `017_create_metrics_v1.sql` with 10-second
aggregation buckets, JSON attributes column, 60-day TTLs

### Query engine (internal-packages/tsql, apps/webapp)
- **Metrics query schema** — Typed columns for metric attributes
(`task_identifier`, `run_id`, `machine_name`, `worker_version`, etc.)
extracted from the JSON attributes column
- **`prettyFormat()`** — TSQL function that annotates columns with
format hints (`bytes`, `percent`, `durationSeconds`) for frontend
rendering without changing the underlying data
- **Per-schema time buckets** — Different tables can define their own
time bucket thresholds (metrics uses tighter intervals than runs)
- **AI query integration** — The AI query service knows about the
metrics table and can generate metric queries
- **Chart improvements** — Better formatting for byte values,
percentages, and durations in charts and tables

### Reference project
- **`references/hello-world/src/trigger/metrics.ts`** — 6 example tasks:
`cpu-intensive`, `memory-ramp`, `bursty-workload`, `sustained-workload`,
`concurrent-load`, `custom-metrics`

## Test plan

- [ ] Build all packages and webapp
- [ ] Start dev worker with hello-world reference project
- [ ] Run `cpu-intensive`, `memory-ramp`, and `custom-metrics` tasks
- [ ] Verify metrics in ClickHouse: `SELECT DISTINCT metric_name FROM
metrics_v1`
- [ ] Query via dashboard AI: "show me CPU utilization over time"
- [ ] Verify `prettyFormat` renders correctly in chart tooltips and
table cells
- [ ] Confirm dev worker drops `system.*` metrics but keeps `process.*`
and `nodejs.*`
2026-02-20 13:16:34 +00:00
..
2025-07-24 15:09:50 +01:00
2025-03-05 14:40:14 +00:00

Trigger.dev References

Contains code that tests or uses the @trigger.dev/* packages in some way, either by using them to test out a framework adapter, an integration, or parts of the main SDK.

All the dependencies to the @trigger.dev/* packages will be both referenced in the package.json dependencies as workspace:*, as well as using a direct path from the tsconfig.json file like so:

{
  "extends": "@trigger.dev/tsconfig/node18.json",
  "include": ["./src/**/*.ts"],
  "compilerOptions": {
    "baseUrl": ".",
    "lib": ["DOM", "DOM.Iterable"],
    "paths": {
      "@/*": ["./src/*"],
      "@trigger.dev/sdk": ["../../packages/trigger-sdk/src/index"],
      "@trigger.dev/sdk/*": ["../../packages/trigger-sdk/src/*"],
      "@trigger.dev/express": ["../../packages/express/src/index"],
      "@trigger.dev/express/*": ["../../packages/express/src/*"],
      "@trigger.dev/core": ["../../packages/core/src/index"],
      "@trigger.dev/core/*": ["../../packages/core/src/*"],
      "@trigger.dev/integration-kit": ["../../packages/integration-kit/src/index"],
      "@trigger.dev/integration-kit/*": ["../../packages/integration-kit/src/*"],
      "@trigger.dev/github": ["../../integrations/github/src/index"],
      "@trigger.dev/github/*": ["../../integrations/github/src/*"],
      "@trigger.dev/slack": ["../../integrations/slack/src/index"],
      "@trigger.dev/slack/*": ["../../integrations/slack/src/*"],
      "@trigger.dev/openai": ["../../integrations/openai/src/index"],
      "@trigger.dev/openai/*": ["../../integrations/openai/src/*"],
      "@trigger.dev/resend": ["../../integrations/resend/src/index"],
      "@trigger.dev/resend/*": ["../../integrations/resend/src/*"],
      "@trigger.dev/typeform": ["../../integrations/typeform/src/index"],
      "@trigger.dev/typeform/*": ["../../integrations/typeform/src/*"],
      "@trigger.dev/plain": ["../../integrations/plain/src/index"],
      "@trigger.dev/plain/*": ["../../integrations/plain/src/*"],
      "@trigger.dev/supabase": ["../../integrations/supabase/src/index"],
      "@trigger.dev/supabase/*": ["../../integrations/supabase/src/*"],
      "@trigger.dev/stripe": ["../../integrations/stripe/src/index"],
      "@trigger.dev/stripe/*": ["../../integrations/stripe/src/*"],
      "@trigger.dev/sendgrid": ["../../integrations/sendgrid/src/index"],
      "@trigger.dev/sendgrid/*": ["../../integrations/sendgrid/src/*"],
      "@trigger.dev/airtable": ["../../integrations/airtable/src/index"],
      "@trigger.dev/airtable/*": ["../../integrations/airtable/src/*"]
    }
  }
}

Creating a New Reference Project

This guide assumes that you have followed the Contributing.md instructions to set up a local trigger.dev instance. If not, please complete the setup before continuing.

Step-by-Step Instructions

  1. Run an HTTP tunnel: You will need to run an HTTP tunnel to expose your local webapp, it is required for some API calls during building the image to deploy on your local instance. This is optional if you do not plan to test deployment on your local instance.
  • Download the ngrok CLI. This can be done by following the instructions on ngrok's website.
  • Create an account on ngrok to obtain the authtoken and add it to the CLI.
ngrok config add-authtoken <your-auth-token>

Replace the with the token you obtain from ngrok.

  • Run the tunnel.
ngrok http <your-app-port>

Replace the with the webapp port, default is 3030.

  1. Add your tunnel URL to the env: After running the ngrok tunnel, you will see URL in your terminal, it will look something like https://<your-tunnel-address>.ngrok-free.app. Replace the APP_ORIGIN variable with this URL in your .env file in the root of the trigger.dev project.

  2. Run the webapp on localhost:

pnpm run dev --filter webapp --filter coordinator --filter docker-provider
  1. Build the CLI in a new terminal window:
# Build the CLI
pnpm run build --filter trigger.dev

# Make it accessible to `pnpm exec`
pnpm i
  1. Set up a new project in the webapp:
  • Open the webapp running on localhost:3030.
  • Create a new project in the webapp UI.
  • Go to the Project Settings page and copy the project reference id from there.
  1. Copy the hello-world project as a template:
cp -r references/hello-world references/<new-project>

Replace <new-project> with your desired project name.

  1. Update project details:
  • Open <new-project>/package.json and change the name field. (Tip: Use the same name as in the webapp to avoid confusion.)

  • Open <new-project>/trigger.config.ts and update the project field with the project reference you copied from the webapp.

  • Run pnpm i in your <new-project> directory to sync the dependencies.

  1. Authorize the CLI for your project:
pnpm exec trigger login -a http://localhost:3030 --profile local
  1. Run the new project: You can now run your project using the CLI with the following command:
pnpm exec trigger dev --profile local

You can also deploy them against your local instance with the following command:

pnpm exec trigger deploy --self-hosted --load-image --profile local