github-featured/triggerdotdev--trigger.dev
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity
Files
8fded28fcd6fd327c6600442cb2b7e1cd7ce99bd
triggerdotdev--trigger.dev/docs/self-hosting/env/webapp.mdx
T
Katia Bulatova 4569657923 feat(webapp): dashboard agent — chat, reports, investigate (#4418) 
## What & why

This is the system behind the Dashboard Agent — an assistant that
answers questions about a project's runs, errors, queues, deploys and
health, and can investigate failures end to end.

The agent runs as a chat.agent task in its own Trigger project. It has
no access to the main database or ClickHouse; all platform data is read
through the public API using a delegated, read-only user token.

Everything here is behind `canAccessDashboardAgent` and inert with the
flag off. The UI that mounts the panel lands in #4529.

## Stack

`#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook
gallery. The scenario/contract reference for the whole stack is
`internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch
branch): it states, per feature, what makes each thing happen and where
that is decided.

## What's inside

**Agent runtime and tools** — `internal-packages/dashboard-agent`:
prompt, tool set (API reads, TRQL query, docs, navigation,
evidence/investigations, repo source), conversation compaction, a
prompt-prefix token budget pinned by snapshot test, and sampled
LLM-judged turn evals. The package cannot import webapp server code,
which is what makes the "no DB access" claim structural rather than a
convention.

**Contracts** — `internal-packages/dashboard-agent-contracts`:
`trigger://` URIs, intents, and the block envelope every rendered card
travels in.

**Conversation store** — `internal-packages/dashboard-agent-db`: drizzle
over postgres-js in its own `trigger_dashboard_agent` Postgres schema,
plus one additive migration.

**Auth boundary** — the user-actor token gains an optional environment
claim; one guard (`userActorEnvironment.server.ts`) enforces it so
routes don't each re-derive the rule. Token minting, cap ceiling, and
the RBAC fallback path for self-hosted.

**Transport** — webapp resource routes that mint the token and proxy
each turn, and SDK-side mid-turn reconnect.

**Public API the agent reads through** — orgs, projects, environments,
runs, queue metrics, workers, a run's commit metadata, repo snapshot,
reports, and `POST /api/v1/query`.

**Reports** — the health report's layout is declared once and shared by
the card, the markdown surface and the JSON/MCP surface, so the same
report reads the same in the dashboard, the terminal and an editor.

**Block renderers** — the report and investigation cards the flows above
already emit (`app/components/dashboard-agent/`). The panel that hosts
them, and the rest of the chat UI, is #4529.

**Query safety and CSP** — see below.

## Key decisions

- **The agent is a separate Trigger project, not webapp code.** It reads
platform data over the public API with a delegated user-actor token
whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no
webapp imports.
- **The PAT-only auth helper now refuses user-actor tokens.** This is an
intentional behavioral change: its callers consume only a bare userId
and do not enforce delegated-token capabilities. Actor-aware routes
continue through the scoped route builders instead.
- **RBAC fallback builds a delegated token's ability from its own cap**,
never the blanket ability a PAT gets (read-only when the token declares
none). Without this, the agent's read-only cap would buy a write JWT on
self-hosted.
- **Org creation checks RBAC only for user-actor tokens, and only after
the env gate**, so an install with `ORG_CREATION_API_ENABLED` off
returns 404 rather than 403, and an ordinary PAT never consults an
ability the route has no org to scope. Both orderings are pinned by
test.
- **The query path is read-only in depth.** TRQL rejects write
statements at the grammar level (they don't parse, rather than being
filtered), ClickHouse runs with `readonly=1`, and the org/project/env
filters are injected server-side from the credential — the request body
cannot widen scope. An unparseable query denies instead of falling
through to the permissive resource.
- **Document-wide img-src CSP.** Remote images are an
outbound-request/exfiltration surface, so the policy permits only
own-origin/data/blob, the required SSO avatar hosts, and the favicon
endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST;
wildcard hosts and bare schemes are intentionally not allowed.
- **The chat transport reconnects on a mid-turn EOF**
(`@trigger.dev/sdk`). A body that ends without a turn-complete is
terminal only when the server says `X-Session-Settled: true`; otherwise
the transport resubscribes from `lastEventId` with bounded backoff, and
any record re-earns the budget. Previously a closed long-poll window or
a proxy restart left the reply stuck as if still generating.
- **Conversations live in their own datastore**, schema-scoped and
foreign-key-free (it references `organizationId`/`userId` by id, because
in cloud it is a different database). It is a display read-model for the
History tab and transport resume; `chat.agent`'s object-store snapshot
remains the model's source of truth.
- **Deterministic first.** Reports and health checks contain no LLM —
they are computed from the same data the dashboard shows, and the model
only narrates and links them. That is what makes a number in an answer
auditable.

## Testing

- 63 new test files, run with `pnpm run test --filter webapp` and
per-package vitest. Heaviest coverage on the auth boundary
(`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`,
`contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the
report layout, and the SDK reconnect.
- The agent package has a separate eval lane (`pnpm run test:evals`,
`vitest.eval.config.ts`) that hits the real model, so it never runs in
`pnpm test`.
- Live-tested against a local stack scenario by scenario; the GUIDEBOOK
lists the condition each behaviour is expected under, which is what
those runs were checked against.

## Changelog

`.server-changes/dashboard-agent.md`, plus changesets for
`@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat
reconnect) and the CLI's `mint-token` help text.
2026-08-11 18:56:14 +02:00

290 lines
40 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: "Webapp"
description: "Environment variables for the webapp container."
sidebarTitle: "Webapp"
mode: "wide"
---
| Name | Required | Default | Description |
| :----------------------------------------------- | :------- | :-------------------- | :----------------------------------------------------------------------------------------------------------------- |
| **Secrets** | | | |
| `SESSION_SECRET` | Yes | — | Session encryption secret. Run: `openssl rand -hex 16` |
| `MAGIC_LINK_SECRET` | Yes | — | Magic link encryption secret. Run: `openssl rand -hex 16` |
| `ENCRYPTION_KEY` | Yes | — | Secret store encryption key. Run: `openssl rand -hex 16` |
| `MANAGED_WORKER_SECRET` | Yes | — | Managed worker secret. Must be set and match supervisor. Run: `openssl rand -hex 32` |
| `ALLOW_INSECURE_DEFAULT_SECRETS` | No | false | Boot even if a secret is still a known-insecure published default. Temporary escape hatch for values you can't safely rotate yet (see [Secrets](/self-hosting/kubernetes#secrets)). |
| **Domains & ports** | | | |
| `REMIX_APP_PORT` | No | 3030 | Remix app port. |
| `APP_ORIGIN` | Yes | http://localhost:3030 | App origin URL. |
| `LOGIN_ORIGIN` | Yes | http://localhost:3030 | Login origin URL. Most likely the same as `APP_ORIGIN`. |
| `API_ORIGIN` | No | `APP_ORIGIN` | API origin URL. |
| `STREAM_ORIGIN` | No | `APP_ORIGIN` | Realtime stream origin URL. |
| `ELECTRIC_ORIGIN` | No | http://localhost:3060 | Electric origin URL. |
| **Postgres** | | | |
| `DATABASE_URL` | Yes | — | PostgreSQL connection string. |
| `DIRECT_URL` | Yes | — | Direct DB connection string used for migrations etc. |
| `DATABASE_CONNECTION_LIMIT` | No | 10 | Max DB connections. |
| `DATABASE_POOL_TIMEOUT` | No | 60 | DB pool timeout (s). |
| `DATABASE_CONNECTION_TIMEOUT` | No | 20 | DB connect timeout (s). |
| `DATABASE_WRITER_POOL_TIMEOUT` | No | `DATABASE_POOL_TIMEOUT` | Writer pool timeout (s); overrides the shared default for the writer only. |
| `DATABASE_WRITER_CONNECTION_TIMEOUT` | No | `DATABASE_CONNECTION_TIMEOUT` | Writer connect timeout (s); overrides the shared default for the writer only. |
| `DATABASE_READ_REPLICA_URL` | No | `DATABASE_URL` | Read-replica DB string. |
| `DATABASE_READ_REPLICA_POOL_TIMEOUT` | No | `DATABASE_POOL_TIMEOUT` | Read-replica pool timeout (s); overrides the shared default for the replica only. |
| `DATABASE_READ_REPLICA_CONNECTION_TIMEOUT` | No | `DATABASE_CONNECTION_TIMEOUT` | Read-replica connect timeout (s); overrides the shared default for the replica only. |
| **Redis** | | | |
| `REDIS_HOST` | Yes | — | Redis host. |
| `REDIS_PORT` | Yes | — | Redis port. |
| `REDIS_READER_HOST` | No | `REDIS_HOST` | Redis reader host. |
| `REDIS_READER_PORT` | No | `REDIS_PORT` | Redis reader port. |
| `REDIS_USERNAME` | No | — | Redis username. |
| `REDIS_PASSWORD` | No | — | Redis password. |
| `REDIS_TLS_DISABLED` | No | — | Disable Redis TLS. |
| **Auth** | | | |
| `WHITELISTED_EMAILS` | No | — | Whitelisted emails regex. |
| `LOGIN_RATE_LIMITS_ENABLED` | No | true | Enable rate limiting on magic-link login. |
| `AUTH_GITHUB_CLIENT_ID` | No | — | GitHub client ID. |
| `AUTH_GITHUB_CLIENT_SECRET` | No | — | GitHub client secret. |
| `CSP_IMG_SRC_ALLOWLIST` | No | — | Extra hosts the dashboard may load images from, e.g. an SSO or avatar host serving profile images. A comma-separated list of exact origins (`https://sso.example.com`); a wildcard host such as `https://*.example.com` is refused. |
| **Email** | | | |
| `EMAIL_TRANSPORT` | No | — | Email transport type. One of `resend`, `smtp`, `aws-ses`. |
| `FROM_EMAIL` | No | — | From email address. |
| `REPLY_TO_EMAIL` | No | — | Reply-to email address. |
| `RESEND_API_KEY` | No | — | Resend API key. |
| `SMTP_HOST` | No | — | SMTP host. |
| `SMTP_PORT` | No | — | SMTP port. |
| `SMTP_SECURE` | No | — | SMTP secure flag. |
| `SMTP_USER` | No | — | SMTP user. |
| `SMTP_PASSWORD` | No | — | SMTP password. |
| `AWS_REGION` | No | — | AWS region for SES. |
| `AWS_ACCESS_KEY_ID` | No | — | AWS access key ID for SES. |
| `AWS_SECRET_ACCESS_KEY` | No | — | AWS secret access key for SES. |
| **Worker** | | | |
| `GRACEFUL_SHUTDOWN_TIMEOUT` | No | 60000 (1m) | Graceful shutdown timeout (ms). Affects shutdown time. |
| **Concurrency limits** | | | |
| `DEFAULT_ENV_EXECUTION_CONCURRENCY_LIMIT` | No | 100 | Default env execution concurrency. |
| `DEFAULT_ORG_EXECUTION_CONCURRENCY_LIMIT` | No | 300 | Default org execution concurrency, needs to be 3x env concurrency. |
| `DEFAULT_ENV_EXECUTION_CONCURRENCY_BURST_FACTOR` | No | 1.0 | Burst factor for env concurrency. |
| `DEFAULT_DEV_ENV_EXECUTION_ATTEMPTS` | No | 1 | Default max attempts for dev environment runs. |
| **Dev** | | | |
| `DEV_MAX_CONCURRENT_RUNS` | No | 25 | Sets the max concurrency for dev runs via the CLI. |
| `DEV_OTEL_EXPORTER_OTLP_ENDPOINT` | No | `APP_ORIGIN/otel` | OTel endpoint for dev runs. |
| **Rate limiting** | | | |
| `API_RATE_LIMIT_REFILL_INTERVAL` | No | 10s | API rate limit refill interval. |
| `API_RATE_LIMIT_MAX` | No | 750 | API rate limit max. |
| `API_RATE_LIMIT_REFILL_RATE` | No | 250 | API rate limit refill rate. |
| `API_RATE_LIMIT_REQUEST_LOGS_ENABLED` | No | 0 | API rate limit request logs. |
| `API_RATE_LIMIT_REJECTION_LOGS_ENABLED` | No | 1 | API rate limit rejection logs. |
| `API_RATE_LIMIT_LIMITER_LOGS_ENABLED` | No | 0 | API rate limit limiter logs. |
| `API_RATE_LIMIT_JWT_WINDOW` | No | 1m | API rate limit JWT window. |
| `API_RATE_LIMIT_JWT_TOKENS` | No | 60 | API rate limit JWT tokens. |
| `DEPLOYMENT_RATE_LIMIT_REFILL_INTERVAL` | No | 10s | Deployment endpoints rate limit refill interval. |
| `DEPLOYMENT_RATE_LIMIT_MAX` | No | 1500 | Deployment endpoints rate limit max. |
| `DEPLOYMENT_RATE_LIMIT_REFILL_RATE` | No | 500 | Deployment endpoints rate limit refill rate. |
| `DEPLOYMENT_RATE_LIMIT_REQUEST_LOGS_ENABLED` | No | 0 | Deployment endpoints rate limit request logs. |
| `DEPLOYMENT_RATE_LIMIT_REJECTION_LOGS_ENABLED` | No | 1 | Deployment endpoints rate limit rejection logs. |
| `DEPLOYMENT_RATE_LIMIT_LIMITER_LOGS_ENABLED` | No | 0 | Deployment endpoints rate limit limiter logs. |
| **Deploy & Registry** | | | |
| `DEPLOY_REGISTRY_HOST` | Yes | — | Deploy registry host. |
| `DEPLOY_REGISTRY_USERNAME` | No | — | Deploy registry username. |
| `DEPLOY_REGISTRY_PASSWORD` | No | — | Deploy registry password. |
| `DEPLOY_REGISTRY_NAMESPACE` | No | trigger | Deploy registry namespace. |
| `DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY` | No | — | Raw IAM policy JSON applied via SetRepositoryPolicy to every ECR repo created by the webapp. Use to grant cross-account pull access to EKS workers when the ECR account is separate from the cluster account. |
| `DEPLOY_IMAGE_PLATFORM` | No | linux/amd64 | Deploy image platform, same values as docker `--platform` flag. |
| `DEPLOY_TIMEOUT_MS` | No | 480000 (8m) | Deploy timeout (ms). |
| `DEPLOY_QUEUE_TIMEOUT_MS` | No | 900000 (15m) | Deploy queue timeout (ms). |
| **Object store (S3)** | | | |
| `OBJECT_STORE_BASE_URL` | No | — | Object store base URL (default provider). |
| `OBJECT_STORE_BUCKET` | No | — | Object store bucket name (default provider). |
| `OBJECT_STORE_ACCESS_KEY_ID` | No | — | Object store access key (default provider). |
| `OBJECT_STORE_SECRET_ACCESS_KEY` | No | — | Object store secret key (default provider). |
| `OBJECT_STORE_REGION` | No | — | Object store region (default provider). |
| `OBJECT_STORE_SERVICE` | No | s3 | Object store service (default provider). |
| `OBJECT_STORE_DEFAULT_PROTOCOL` | No | — | Protocol for new uploads (e.g. `s3`, `r2`). Enables protocol-prefixed storage. See migration guide below. |
| `OBJECT_STORE_{PROTOCOL}_BASE_URL` | No | — | Named provider base URL (replace `{PROTOCOL}`, e.g. `OBJECT_STORE_S3_BASE_URL`). |
| `OBJECT_STORE_{PROTOCOL}_ACCESS_KEY_ID` | No | — | Named provider access key. |
| `OBJECT_STORE_{PROTOCOL}_SECRET_ACCESS_KEY` | No | — | Named provider secret key. |
| `OBJECT_STORE_{PROTOCOL}_REGION` | No | — | Named provider region. |
| `OBJECT_STORE_{PROTOCOL}_SERVICE` | No | — | Named provider service. |
| `ARTIFACTS_OBJECT_STORE_BUCKET` | No | — | Optional separate bucket for artifacts. If not set, uses main object store. |
| `ARTIFACTS_OBJECT_STORE_BASE_URL` | No | — | Optional artifacts store base URL. |
| `ARTIFACTS_OBJECT_STORE_ACCESS_KEY_ID` | No | — | Optional artifacts store access key. |
| `ARTIFACTS_OBJECT_STORE_SECRET_ACCESS_KEY` | No | — | Optional artifacts store secret key. |
| `ARTIFACTS_OBJECT_STORE_REGION` | No | — | Optional artifacts store region. |
| **Alerts** | | | |
| `ORG_SLACK_INTEGRATION_CLIENT_ID` | No | — | Slack client ID. Required for Slack alerts. |
| `ORG_SLACK_INTEGRATION_CLIENT_SECRET` | No | — | Slack client secret. Required for Slack alerts. |
| `ALERT_EMAIL_TRANSPORT` | No | — | Alert email transport. |
| `ALERT_FROM_EMAIL` | No | — | Alert from email. |
| `ALERT_REPLY_TO_EMAIL` | No | — | Alert reply-to email. |
| `ALERT_RESEND_API_KEY` | No | — | Alert Resend API key. |
| `ALERT_SMTP_HOST` | No | — | Alert SMTP host. |
| `ALERT_SMTP_PORT` | No | — | Alert SMTP port. |
| `ALERT_SMTP_SECURE` | No | — | Alert SMTP secure. |
| `ALERT_SMTP_USER` | No | — | Alert SMTP user. |
| `ALERT_SMTP_PASSWORD` | No | — | Alert SMTP password. |
| **Limits** | | | |
| `TASK_PAYLOAD_OFFLOAD_THRESHOLD` | No | 524288 (512KB) | Max task payload size before offloading to S3. |
| `TASK_PAYLOAD_MAXIMUM_SIZE` | No | 3145728 (3MB) | Max task payload size. |
| `BATCH_TASK_PAYLOAD_MAXIMUM_SIZE` | No | 1000000 (1MB) | Max batch payload size. |
| `BATCH_CONCURRENCY_LIMIT_DEFAULT` | No | 5 | Default concurrency for batch processing. |
| `BATCH_RATE_LIMIT_REFILL_RATE` | No | 100 | Batch rate limit refill rate. |
| `BATCH_RATE_LIMIT_MAX` | No | 1200 | Batch rate limit max. |
| `BATCH_RATE_LIMIT_REFILL_INTERVAL` | No | 10s | Batch rate limit refill interval. |
| `TASK_RUN_METADATA_MAXIMUM_SIZE` | No | 262144 (256KB) | Max metadata size. |
| `MAX_BATCH_V2_TRIGGER_ITEMS` | No | 500 | Max batch size (legacy v2 API). |
| `STREAMING_BATCH_MAX_ITEMS` | No | 1000 | Max items in streaming batch (v3 API, requires SDK 4.3.1+). |
| `STREAMING_BATCH_ITEM_MAXIMUM_SIZE` | No | 3145728 (3MB) | Max size per item in streaming batch. |
| `STREAMING_BATCH_INGEST_CONCURRENCY` | No | 10 | Items ingested concurrently per streaming batch request. Peak memory ≈ this × item size. Set to 1 for sequential. |
| **OTel limits** | | | |
| `TRIGGER_OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` | No | 1024 | OTel span attribute count limit. |
| `TRIGGER_OTEL_LOG_ATTRIBUTE_COUNT_LIMIT` | No | 1024 | OTel log attribute count limit. |
| `TRIGGER_OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` | No | 131072 | OTel span attribute value length limit. |
| `TRIGGER_OTEL_LOG_ATTRIBUTE_VALUE_LENGTH_LIMIT` | No | 131072 | OTel log attribute value length limit. |
| `TRIGGER_OTEL_SPAN_EVENT_COUNT_LIMIT` | No | 10 | OTel span event count limit. |
| `TRIGGER_OTEL_LINK_COUNT_LIMIT` | No | 2 | OTel link count limit. |
| `TRIGGER_OTEL_ATTRIBUTE_PER_LINK_COUNT_LIMIT` | No | 10 | OTel attribute per link count limit. |
| `TRIGGER_OTEL_ATTRIBUTE_PER_EVENT_COUNT_LIMIT` | No | 10 | OTel attribute per event count limit. |
| `SERVER_OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` | No | 8192 | OTel span attribute value length limit. |
| **Task events** | | | |
| `EVENT_REPOSITORY_DEFAULT_STORE` | No | postgres | Where to store task events. Set to `clickhouse_v2` to store in ClickHouse (recommended for production). |
| `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` | No | 0 | Skip all PostgreSQL task-event writes (set to `1`). Only enable when `EVENT_REPOSITORY_DEFAULT_STORE` is `clickhouse_v2`, otherwise task events are lost. |
| **Realtime** | | | |
| `REALTIME_STREAM_VERSION` | No | v1 | Stream version exposed to tasks via the `TRIGGER_REALTIME_STREAM_VERSION` variable. Distinct from `REALTIME_STREAMS_DEFAULT_VERSION`. One of `v1`, `v2`. |
| `REALTIME_STREAM_MAX_LENGTH` | No | 1000 | Realtime stream max length. |
| `REALTIME_STREAM_TTL` | No | 86400 (1d) | Realtime stream TTL (s). |
| `REALTIME_STREAMS_DEFAULT_VERSION` | No | v1 | Server-side default the webapp uses when a stream request doesn't pin a version (modern SDKs request `v2`). The Docker and Helm self-hosting defaults set this to `v2`. One of `v1`, `v2`. |
| `REALTIME_STREAMS_S2_BASIN` | No | — | S2 basin that holds v2 realtime streams. Required for `v2`. Must be at least 8 characters. |
| `REALTIME_STREAMS_S2_ENDPOINT` | No | — | Custom S2 API endpoint, including the `/v1` suffix (e.g. `http://s2/v1` for the bundled s2-lite). Omit to use hosted S2 at s2.dev. |
| `REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS` | No | false | Skip minting per-stream access tokens. Set to `true` for s2-lite, which needs no authentication. |
| `REALTIME_STREAMS_S2_ACCESS_TOKEN` | No | — | S2 access token. Required for hosted S2 unless `REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS` is `true`. |
| **Bootstrap** | | | |
| `TRIGGER_BOOTSTRAP_ENABLED` | No | 0 | Trigger bootstrap enabled. |
| `TRIGGER_BOOTSTRAP_WORKER_GROUP_NAME` | No | — | Trigger bootstrap worker group name. |
| `TRIGGER_BOOTSTRAP_WORKER_TOKEN_PATH` | No | — | Trigger bootstrap worker token path. |
| **Run engine** | | | |
| `RUN_ENGINE_WORKER_COUNT` | No | 4 | Run engine worker count. |
| `RUN_ENGINE_TASKS_PER_WORKER` | No | 10 | Run engine tasks per worker. |
| `RUN_ENGINE_WORKER_CONCURRENCY_LIMIT` | No | 10 | Run engine worker concurrency limit. |
| `RUN_ENGINE_WORKER_POLL_INTERVAL` | No | 100 | Run engine worker poll interval (ms). |
| `RUN_ENGINE_WORKER_IMMEDIATE_POLL_INTERVAL` | No | 100 | Run engine worker immediate poll interval (ms). |
| `RUN_ENGINE_WORKER_SHUTDOWN_TIMEOUT_MS` | No | 60000 (1m) | Run engine worker shutdown timeout (ms). |
| `RUN_ENGINE_RATE_LIMIT_REFILL_INTERVAL` | No | 10s | Run engine rate limit refill interval. |
| `RUN_ENGINE_RATE_LIMIT_MAX` | No | 1200 | Run engine rate limit max. |
| `RUN_ENGINE_RATE_LIMIT_REFILL_RATE` | No | 400 | Run engine rate limit refill rate. |
| `RUN_ENGINE_RATE_LIMIT_REQUEST_LOGS_ENABLED` | No | 0 | Run engine rate limit request logs. |
| `RUN_ENGINE_RATE_LIMIT_REJECTION_LOGS_ENABLED` | No | 1 | Run engine rate limit rejection logs. |
| `RUN_ENGINE_RATE_LIMIT_LIMITER_LOGS_ENABLED` | No | 0 | Run engine rate limit limiter logs. |
| `RUN_ENGINE_DEFAULT_MAX_TTL` | No | — | Maximum TTL for all runs (e.g. "14d"). Runs without a TTL use this as default; runs with a larger TTL are clamped. |
| `MAXIMUM_DEV_QUEUE_SIZE` | No | — | Maximum queued runs per queue in development environments. |
| `MAXIMUM_DEPLOYED_QUEUE_SIZE` | No | — | Maximum queued runs per queue in deployed (staging/prod) environments. |
| **Misc** | | | |
| `TRIGGER_TELEMETRY_DISABLED` | No | — | Disable telemetry. |
| `NODE_MAX_OLD_SPACE_SIZE` | No | 8192 | Maximum memory allocation for Node.js heap in MiB (e.g. "4096" for 4GB). |
| `OPENAI_API_KEY` | No | — | OpenAI API key. |
| `MACHINE_PRESETS_OVERRIDE_PATH` | No | — | Path to machine presets override file. See [machine overrides](/self-hosting/overview#machine-overrides). |
| `APP_ENV` | No | `NODE_ENV` | App environment. Used for things like the title tag. |
| `ADMIN_EMAILS` | No | — | Regex of user emails to automatically promote to admin on signup. Does not apply to existing users. |
| `EVENT_LOOP_MONITOR_ENABLED` | No | 1 | Node.js event loop lag monitor. |
## Multi-Provider Object Storage
The object storage system supports multiple S3-compatible providers (R2, S3, GCS, MinIO, etc.) using protocol prefixes. This enables migrating between providers without breaking existing runs.
### How It Works
When data exceeds the configured threshold (`TASK_PAYLOAD_OFFLOAD_THRESHOLD`), it's uploaded to object storage. The storage location is saved in the database with an optional protocol prefix:
- **With protocol**: `s3://run_abc/payload.json` or `r2://batch_123/item_0/payload.json`
- **Without protocol** (legacy): `batch_123/item_0/payload.json` (uses default provider)
### Configuration
#### Default Provider (Backward Compatible)
The default provider is used for data without a protocol prefix:
```bash
# Default provider (backward compatible - no protocol prefix)
OBJECT_STORE_BASE_URL=https://r2.example.com
OBJECT_STORE_ACCESS_KEY_ID=...
OBJECT_STORE_SECRET_ACCESS_KEY=...
OBJECT_STORE_REGION=auto
OBJECT_STORE_SERVICE=s3
```
#### Named Providers
Named providers are accessed via protocol-prefixed URIs. Configure them using `OBJECT_STORE_{PROTOCOL}_*` variables:
```bash
# S3 provider (accessed via s3:// prefix)
OBJECT_STORE_S3_BASE_URL=https://s3.amazonaws.com
OBJECT_STORE_S3_ACCESS_KEY_ID=...
OBJECT_STORE_S3_SECRET_ACCESS_KEY=...
OBJECT_STORE_S3_REGION=us-east-1
OBJECT_STORE_S3_SERVICE=s3
# R2 provider (accessed via r2:// prefix)
OBJECT_STORE_R2_BASE_URL=https://...r2.cloudflarestorage.com
OBJECT_STORE_R2_ACCESS_KEY_ID=...
OBJECT_STORE_R2_SECRET_ACCESS_KEY=...
OBJECT_STORE_R2_REGION=auto
OBJECT_STORE_R2_SERVICE=s3
```
#### Default Protocol for New Uploads
Set `OBJECT_STORE_DEFAULT_PROTOCOL` to specify which provider to use for new uploads:
```bash
# Use S3 for new uploads (old data without prefix still uses default provider)
OBJECT_STORE_DEFAULT_PROTOCOL=s3
```
### Migration Guide
To migrate from R2 to S3 without breaking existing runs:
<Steps>
<Step title="Configure S3 provider">
Add S3 credentials as a named provider:
```bash
OBJECT_STORE_S3_BASE_URL=https://s3.amazonaws.com
OBJECT_STORE_S3_ACCESS_KEY_ID=...
OBJECT_STORE_S3_SECRET_ACCESS_KEY=...
OBJECT_STORE_S3_REGION=us-east-1
```
Keep your existing `OBJECT_STORE_*` variables (R2) as the default provider.
</Step>
<Step title="Test the configuration">
Restart the webapp and verify both providers work:
- Old runs (no prefix) should still access R2
- New runs with `s3://` prefix should use S3
</Step>
<Step title="Switch to S3 for new uploads">
Set the default protocol to use S3 for new uploads:
```bash
OBJECT_STORE_DEFAULT_PROTOCOL=s3
```
After this change:
- New data uses `s3://` prefix and goes to S3
- Old data (no prefix) still uses R2
- Data with explicit protocol uses the corresponding provider
</Step>
<Step title="Optionally decommission R2">
Once all active runs using R2 data have completed (check your data retention policies), you can remove the R2 credentials. Keep `OBJECT_STORE_DEFAULT_PROTOCOL=s3` to ensure new data continues using S3.
</Step>
</Steps>
Reference in New Issue View Git Blame Copy Permalink