Eric Allam
469b039090
- Adds an end-to-end OTEL metrics pipeline: task workers collect and export metrics via OpenTelemetry, the webapp ingests them into ClickHouse, and they're queryable through the existing dashboard query engine - Workers emit process CPU/memory metrics (via `@opentelemetry/host-metrics`) and Node.js runtime metrics (event loop utilization, event loop delay, heap usage) - Users can create custom metrics in their tasks via `otel.metrics.getMeter()` from `@trigger.dev/sdk` - Metrics are automatically tagged with run context (run ID, task slug, machine, worker version) so they can be sliced per-run, per-task, or per-machine - The TSQL query engine gains metrics table support with typed attribute columns, `prettyFormat()` for human-readable values, and per-schema time bucket thresholds - Includes reference tasks (`references/hello-world/src/trigger/metrics.ts`) demonstrating CPU-intensive, memory-ramp, bursty workload, and custom metrics patterns ## What changed ### Metrics collection (packages/core, packages/cli-v3) - **Metrics export pipeline** — `TracingSDK` now sets up a `MeterProvider` with a `PeriodicExportingMetricReader` that chains through `TaskContextMetricExporter` (adds run context attributes) and `BufferingMetricExporter` (batches exports to reduce overhead) - **Host metrics** — Enabled `@opentelemetry/host-metrics` for process CPU, memory, and system-level metrics - **Node.js runtime metrics** — New `nodejsRuntimeMetrics.ts` module using `performance.eventLoopUtilization()`, `monitorEventLoopDelay()`, and `process.memoryUsage()` to emit 6 observable gauges - File system and diskio metrics - **Custom metrics** — Exposed `otel.metrics` from `@trigger.dev/sdk` so users can create counters, histograms, and gauges in their tasks - **Machine ID** — Stable per-worker machine identifier for grouping metrics - **Dev worker** — Drops `system.*` metrics to reduce noise, keeps sending metrics between runs in warm workers ### Metrics ingestion (apps/webapp) - **OTEL endpoint** — `otel.v1.metrics.ts` accepts OTEL metric export requests (JSON and protobuf), converts to ClickHouse rows - **ClickHouse schema** — `017_create_metrics_v1.sql` with 10-second aggregation buckets, JSON attributes column, 60-day TTLs ### Query engine (internal-packages/tsql, apps/webapp) - **Metrics query schema** — Typed columns for metric attributes (`task_identifier`, `run_id`, `machine_name`, `worker_version`, etc.) extracted from the JSON attributes column - **`prettyFormat()`** — TSQL function that annotates columns with format hints (`bytes`, `percent`, `durationSeconds`) for frontend rendering without changing the underlying data - **Per-schema time buckets** — Different tables can define their own time bucket thresholds (metrics uses tighter intervals than runs) - **AI query integration** — The AI query service knows about the metrics table and can generate metric queries - **Chart improvements** — Better formatting for byte values, percentages, and durations in charts and tables ### Reference project - **`references/hello-world/src/trigger/metrics.ts`** — 6 example tasks: `cpu-intensive`, `memory-ramp`, `bursty-workload`, `sustained-workload`, `concurrent-load`, `custom-metrics` ## Test plan - [ ] Build all packages and webapp - [ ] Start dev worker with hello-world reference project - [ ] Run `cpu-intensive`, `memory-ramp`, and `custom-metrics` tasks - [ ] Verify metrics in ClickHouse: `SELECT DISTINCT metric_name FROM metrics_v1` - [ ] Query via dashboard AI: "show me CPU utilization over time" - [ ] Verify `prettyFormat` renders correctly in chart tooltips and table cells - [ ] Confirm dev worker drops `system.*` metrics but keeps `process.*` and `nodejs.*`
OTLP Importer
Getting started
Install dependencies:
brew install protobuf
apt install -y protobuf-compiler
Alternatively, follow the manual install instructions for the protobuf compiler.
On Windows, download the correct binary from the latest release and extract the protoc binary to this directory, or add it to your PATH.
Submodules
Submodule is always pointing to certain revision number. So updating the submodule repo will not have impact on your code. Knowing this if you want to change the submodule to point to a different version (when for example proto has changed) here is how to do it:
Updating submodule to point to certain revision number
-
Make sure you are in the same folder as this instruction
-
Update your submodules by running this command
git submodule sync --recursive git submodule update --init --recursive -
Find the SHA which you want to update to and copy it (the long one) the latest sha when this guide was written is
c451441d7b73f702d1647574c730daf7786f188c -
Enter a submodule directory from this directory
cd protos -
Updates files in the submodule tree to given commit:
git checkout -q <sha> -
Return to the main directory:
cd ../ -
Please run
git statusyou should see something likeHead detached at. This is correct, go to next step -
Now thing which is very important. You have to commit this to apply these changes
git commit -am "chore: updating protos submodule for @trigger.dev/otlp-importer" -
If you look now at git log you will notice that the folder
protoshas been changed and it will show what was the previous sha and what is current one.