docs: otel metrics (#3096)
This commit is contained in:
Eric Allam
@@ -154,21 +154,30 @@ Some ones we recommend:
|
||||
|
||||
### Telemetry Exporters
|
||||
|
||||
You can also configure custom telemetry exporters to send your traces and logs to other external services. For example, you can send your logs to [Axiom](https://axiom.co/docs/guides/opentelemetry-nodejs#exporter-instrumentation-ts). First, add the opentelemetry exporter packages to your package.json file:
|
||||
You can also configure custom telemetry exporters to send your traces, logs, and metrics to other external services. For example, you can send your logs to [Axiom](https://axiom.co/docs/guides/opentelemetry-nodejs#exporter-instrumentation-ts). First, add the opentelemetry exporter packages to your package.json file:
|
||||
|
||||
```json package.json
|
||||
"dependencies": {
|
||||
"@opentelemetry/exporter-logs-otlp-http": "0.52.1",
|
||||
"@opentelemetry/exporter-trace-otlp-http": "0.52.1"
|
||||
"@opentelemetry/exporter-trace-otlp-http": "0.52.1",
|
||||
"@opentelemetry/exporter-metrics-otlp-proto": "0.52.1"
|
||||
}
|
||||
```
|
||||
|
||||
<Note>
|
||||
Axiom's `/v1/metrics` endpoint only supports protobuf (`application/x-protobuf`), not JSON. Use
|
||||
`@opentelemetry/exporter-metrics-otlp-proto` instead of
|
||||
`@opentelemetry/exporter-metrics-otlp-http` for metrics. Traces and logs work fine with the
|
||||
`-http` (JSON) exporters.
|
||||
</Note>
|
||||
|
||||
Then, configure the exporters in your `trigger.config.ts` file:
|
||||
|
||||
```ts trigger.config.ts
|
||||
import { defineConfig } from "@trigger.dev/sdk";
|
||||
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
|
||||
import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";
|
||||
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-proto";
|
||||
|
||||
// Initialize OTLP trace exporter with the endpoint URL and headers;
|
||||
export default defineConfig({
|
||||
@@ -196,18 +205,28 @@ export default defineConfig({
|
||||
},
|
||||
}),
|
||||
],
|
||||
metricExporters: [
|
||||
new OTLPMetricExporter({
|
||||
url: "https://api.axiom.co/v1/metrics",
|
||||
headers: {
|
||||
Authorization: `Bearer ${process.env.AXIOM_API_TOKEN}`,
|
||||
"x-axiom-metrics-dataset": process.env.AXIOM_METRICS_DATASET,
|
||||
},
|
||||
}),
|
||||
],
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
Make sure to set the `AXIOM_API_TOKEN` and `AXIOM_DATASET` environment variables in your project.
|
||||
Make sure to set the `AXIOM_API_TOKEN`, `AXIOM_DATASET`, and `AXIOM_METRICS_DATASET` environment variables in your project. Axiom requires a separate, dedicated dataset for metrics — you cannot reuse the same dataset for traces/logs and metrics.
|
||||
|
||||
It's important to note that you cannot configure exporters using `OTEL_*` environment variables, as they would conflict with our internal telemetry. Instead you should configure the exporters via passing in arguments to the `OTLPTraceExporter` and `OTLPLogExporter` constructors. For example, here is how you can configure exporting to Honeycomb:
|
||||
It's important to note that you cannot configure exporters using `OTEL_*` environment variables, as they would conflict with our internal telemetry. Instead you should configure the exporters via passing in arguments to the `OTLPTraceExporter`, `OTLPLogExporter`, and `OTLPMetricExporter` constructors. For example, here is how you can configure exporting to Honeycomb:
|
||||
|
||||
```ts trigger.config.ts
|
||||
import { defineConfig } from "@trigger.dev/sdk";
|
||||
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
|
||||
import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";
|
||||
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
|
||||
|
||||
// Initialize OTLP trace exporter with the endpoint URL and headers;
|
||||
export default defineConfig({
|
||||
@@ -235,6 +254,15 @@ export default defineConfig({
|
||||
},
|
||||
}),
|
||||
],
|
||||
metricExporters: [
|
||||
new OTLPMetricExporter({
|
||||
url: "https://api.honeycomb.io/v1/metrics",
|
||||
headers: {
|
||||
"x-honeycomb-team": process.env.HONEYCOMB_API_KEY,
|
||||
"x-honeycomb-dataset": process.env.HONEYCOMB_DATASET,
|
||||
},
|
||||
}),
|
||||
],
|
||||
},
|
||||
});
|
||||
```
|
||||
@@ -465,8 +493,9 @@ export default defineConfig({
|
||||
```
|
||||
|
||||
<Note>
|
||||
Any packages that install or build a native binary or use WebAssembly (WASM) should be added to external, as they
|
||||
cannot be bundled. For example, `re2`, `sharp`, `sqlite3`, and WASM packages should be added to external.
|
||||
Any packages that install or build a native binary or use WebAssembly (WASM) should be added to
|
||||
external, as they cannot be bundled. For example, `re2`, `sharp`, `sqlite3`, and WASM packages
|
||||
should be added to external.
|
||||
</Note>
|
||||
|
||||
### JSX
|
||||
|
||||
@@ -9,6 +9,12 @@ In the Trigger.dev dashboard we have built-in dashboards and you can create your
|
||||
|
||||
Metrics dashboards are powered by [TRQL queries](/insights/query) with widgets that can be displayed as charts, tables, or single values. They automatically refresh to show the latest data.
|
||||
|
||||
### Available metrics data
|
||||
|
||||
Trigger.dev automatically collects process metrics (CPU, memory) and Node.js runtime metrics (event loop, heap) for all deployed tasks -- no configuration needed. Requires SDK version **4.4.1 or later**. You can also create custom metrics using the `otel.metrics` API from the SDK.
|
||||
|
||||
All of this data is available in the `metrics` table for use in dashboard widgets. See [Logging, tracing & metrics](/logging#metrics) for the full list of automatic metrics and how to create custom ones, or the [Query page](/insights/query#metrics-table-columns) for the `metrics` table schema.
|
||||
|
||||
|
||||
|
||||
### Visualization types
|
||||
|
||||
+40
-4
@@ -6,7 +6,43 @@ description: "Query allows you to write custom queries against your data using T
|
||||
### Available tables
|
||||
|
||||
- `runs`: contains all task run data including status, timing, costs, and metadata
|
||||
- `metrics`: contains metrics data for your runs including CPU, memory, and your custom metrics.
|
||||
- `metrics`: contains metrics data for your runs including CPU, memory, and your custom metrics
|
||||
|
||||
### `metrics` table columns
|
||||
|
||||
| Column | Type | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| `metric_name` | string | Metric identifier (e.g., `process.cpu.utilization`) |
|
||||
| `metric_type` | string | `gauge`, `sum`, or `histogram` |
|
||||
| `value` | number | The observed value |
|
||||
| `bucket_start` | datetime | 10-second aggregation bucket start time |
|
||||
| `run_id` | string | Associated run ID |
|
||||
| `task_identifier` | string | Task slug |
|
||||
| `attempt_number` | number | Attempt number |
|
||||
| `machine_id` | string | Machine that produced the metric |
|
||||
| `machine_name` | string | Machine preset (e.g., `small-1x`) |
|
||||
| `worker_version` | string | Worker version |
|
||||
| `environment_type` | string | `PRODUCTION`, `STAGING`, `DEVELOPMENT`, `PREVIEW` |
|
||||
| `attributes` | json | Raw JSON attributes for custom data |
|
||||
|
||||
See [Logging, tracing & metrics](/logging#automatic-system-and-runtime-metrics) for the full list of automatically collected metrics and how to create custom metrics.
|
||||
|
||||
### `prettyFormat()`
|
||||
|
||||
Use `prettyFormat()` to format metric values for display:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
timeBucket(),
|
||||
prettyFormat(avg(value), 'bytes') AS avg_memory
|
||||
FROM metrics
|
||||
WHERE metric_name = 'process.memory.usage'
|
||||
GROUP BY timeBucket
|
||||
ORDER BY timeBucket
|
||||
LIMIT 1000
|
||||
```
|
||||
|
||||
Available format types: `bytes`, `percent`, `duration`, `durationSeconds`, `quantity`, `costInDollars`.
|
||||
|
||||
## Using the Query dashboard
|
||||
|
||||
@@ -191,7 +227,7 @@ SELECT
|
||||
task_identifier,
|
||||
avg(value) AS avg_memory
|
||||
FROM metrics
|
||||
WHERE metric_name = 'system.memory.usage'
|
||||
WHERE metric_name = 'process.memory.usage'
|
||||
GROUP BY task_identifier
|
||||
ORDER BY avg_memory DESC
|
||||
LIMIT 20
|
||||
@@ -500,14 +536,14 @@ LIMIT 1000
|
||||
|
||||
### Memory usage by task (past 7d)
|
||||
|
||||
Average memory usage per task identifier over the last 7 days.
|
||||
Average process memory usage per task identifier over the last 7 days.
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
task_identifier,
|
||||
avg(value) AS avg_memory
|
||||
FROM metrics
|
||||
WHERE metric_name = 'system.memory.usage'
|
||||
WHERE metric_name = 'process.memory.usage'
|
||||
GROUP BY task_identifier
|
||||
ORDER BY avg_memory DESC
|
||||
LIMIT 20
|
||||
|
||||
+118
-2
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Logging and tracing"
|
||||
description: "How to use the built-in logging and tracing system."
|
||||
title: "Logging, tracing & metrics"
|
||||
description: "How to use the built-in logging, tracing, and metrics system."
|
||||
---
|
||||
|
||||
|
||||
@@ -77,3 +77,119 @@ export const customTrace = task({
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Metrics
|
||||
|
||||
Trigger.dev collects system and runtime metrics automatically for deployed tasks, and provides an API for recording custom metrics using OpenTelemetry.
|
||||
|
||||
You can view metrics in the [Metrics dashboards](/insights/metrics), query them with [TRQL](/insights/query), and export them to external services via [telemetry exporters](/config/config-file#telemetry-exporters).
|
||||
|
||||
### Custom metrics API
|
||||
|
||||
Import `otel` from `@trigger.dev/sdk` and use the standard OpenTelemetry Metrics API to create custom instruments.
|
||||
|
||||
Create instruments **at module level** (outside the task `run` function) so they are reused across runs:
|
||||
|
||||
```ts /trigger/metrics.ts
|
||||
import { task, logger, otel } from "@trigger.dev/sdk";
|
||||
|
||||
// Create a meter — instruments are created once at module level
|
||||
const meter = otel.metrics.getMeter("my-app");
|
||||
|
||||
const itemsProcessed = meter.createCounter("items.processed", {
|
||||
description: "Total number of items processed",
|
||||
unit: "items",
|
||||
});
|
||||
|
||||
const itemDuration = meter.createHistogram("item.duration", {
|
||||
description: "Time spent processing each item",
|
||||
unit: "ms",
|
||||
});
|
||||
|
||||
const queueDepth = meter.createUpDownCounter("queue.depth", {
|
||||
description: "Current queue depth",
|
||||
unit: "items",
|
||||
});
|
||||
|
||||
export const processQueue = task({
|
||||
id: "process-queue",
|
||||
run: async (payload: { items: string[] }) => {
|
||||
queueDepth.add(payload.items.length);
|
||||
|
||||
for (const item of payload.items) {
|
||||
const start = performance.now();
|
||||
|
||||
// ... process item ...
|
||||
|
||||
const elapsed = performance.now() - start;
|
||||
|
||||
itemsProcessed.add(1, { "item.type": "order" });
|
||||
itemDuration.record(elapsed, { "item.type": "order" });
|
||||
queueDepth.add(-1);
|
||||
}
|
||||
|
||||
logger.info("Queue processed", { count: payload.items.length });
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
#### Available instrument types
|
||||
|
||||
| Instrument | Method | Use case |
|
||||
| :--- | :--- | :--- |
|
||||
| Counter | `meter.createCounter()` | Monotonically increasing values (items processed, requests sent) |
|
||||
| Histogram | `meter.createHistogram()` | Distributions of values (durations, sizes) |
|
||||
| UpDownCounter | `meter.createUpDownCounter()` | Values that go up and down (queue depth, active connections) |
|
||||
|
||||
All instruments accept optional attributes when recording values. Attributes let you break down metrics by dimension (e.g., by item type, status, or region).
|
||||
|
||||
### Automatic system and runtime metrics
|
||||
|
||||
Trigger.dev automatically collects the following metrics for deployed tasks. No configuration is needed. Requires SDK version **4.4.1 or later**.
|
||||
|
||||
| Metric name | Type | Unit | Description |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `process.cpu.utilization` | gauge | ratio | Process CPU usage (0-1) |
|
||||
| `process.cpu.time` | counter | seconds | CPU time consumed |
|
||||
| `process.memory.usage` | gauge | bytes | Process memory usage |
|
||||
| `nodejs.event_loop.utilization` | gauge | ratio | Event loop utilization (0-1) |
|
||||
| `nodejs.event_loop.delay.p95` | gauge | seconds | Event loop delay p95 |
|
||||
| `nodejs.event_loop.delay.max` | gauge | seconds | Event loop delay max |
|
||||
| `nodejs.heap.used` | gauge | bytes | V8 heap used |
|
||||
| `nodejs.heap.total` | gauge | bytes | V8 heap total |
|
||||
|
||||
<Note>
|
||||
In dev mode (`trigger dev`), only `process.*` and custom metrics are available.
|
||||
</Note>
|
||||
|
||||
### Context attributes
|
||||
|
||||
All metrics (both automatic and custom) are tagged with run context so you can filter and group them:
|
||||
|
||||
- `run_id` — the run that produced the metric
|
||||
- `task_identifier` — the task slug
|
||||
- `attempt_number` — the attempt number
|
||||
- `machine_name` — the machine preset (e.g., `small-1x`)
|
||||
- `worker_version` — the deployed worker version
|
||||
- `environment_type` — `PRODUCTION`, `STAGING`, `DEVELOPMENT`, or `PREVIEW`
|
||||
|
||||
### Querying metrics
|
||||
|
||||
Use [TRQL](/insights/query) to query metrics data. For example, to see average CPU utilization over time:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
timeBucket(),
|
||||
avg(value) AS avg_cpu
|
||||
FROM metrics
|
||||
WHERE metric_name = 'process.cpu.utilization'
|
||||
GROUP BY timeBucket
|
||||
ORDER BY timeBucket
|
||||
LIMIT 1000
|
||||
```
|
||||
|
||||
See the [Query page](/insights/query#metrics-table-columns) for the full `metrics` table schema.
|
||||
|
||||
### Exporting metrics
|
||||
|
||||
You can send metrics to external observability services (Axiom, Honeycomb, Datadog, etc.) by configuring [telemetry exporters](/config/config-file#telemetry-exporters) in your `trigger.config.ts`.
|
||||
|
||||
Reference in New Issue
Block a user