github-featured/triggerdotdev--trigger.dev
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity
Files
bfa3341d6bb4e192f7097afe01f2bbb58dd2aa8d
triggerdotdev--trigger.dev/docs/self-hosting/env/webapp.mdx
T
ThullyoCunha f1736595cd
🚀 Publish Trigger.dev Docker / units (push) Failing after 13m3s
Details
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 13m3s
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
feat(webapp): apply default repository policy on ECR repo creation (#3467) 
## Summary

Self-hosters that operate the webapp's ECR account separately from the
account running the EKS workers (e.g., a shared platform account that
hosts the registry plus per-team accounts that host clusters) currently
hit a 403 Forbidden the first time **any** project is deployed:

```
Failed to pull image "<acct-A>.dkr.ecr.<region>.amazonaws.com/<namespace>/proj_…:…":
unexpected status from HEAD request to .../v2/.../manifests/sha256:…: 403 Forbidden
```

`ensureEcrRepositoryExists` in
`apps/webapp/app/v3/getDeploymentImageRef.server.ts` calls
`CreateRepository` and `PutLifecyclePolicy`, but never
`SetRepositoryPolicy` — so the new repo inherits the AWS default (only
the registry-owner account can read/pull). Workers in the cluster
account get 403 every single deploy. The only workarounds today are
running a one-off post-create script or pre-creating every repo by hand.

## Proposed change

Add an optional env var:

```
DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY  (V4 mirror: V4_DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY)
```

Raw IAM policy JSON. When set, the webapp calls `SetRepositoryPolicy`
immediately after `CreateRepository` so every new repo carries that
policy from creation. Operators control the principal/actions; we don't
bake in any opinions about cross-account boundaries.

Example value (for the typical self-host case — grant pull to the
cluster account):

```json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowClusterAccountPull",
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::<cluster-account-id>:root"},
    "Action": [
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "ecr:BatchCheckLayerAvailability"
    ]
  }]
}
```

## Why env var (not a chart-level field)

- Mirrors the shape of the sibling vars (`DEPLOY_REGISTRY_ECR_TAGS`,
`DEPLOY_REGISTRY_ECR_ASSUME_ROLE_ARN`, etc.) which are already
operator-supplied via `webapp.extraEnvVars` in self-host setups.
- Cloud is unaffected — the env var is optional, unset by default;
existing behavior unchanged.
- Existing repos are unaffected — only newly-created repos get the
policy.
- `RepositoryCreationTemplate` from the AWS provider isn't an
alternative here: it only applies to repos created via
pull-through-cache or replication, not to `ecr:CreateRepository` API
calls.

## Implementation

- `apps/webapp/app/env.server.ts` — declare
`DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY` and the V4 fallback.
- `apps/webapp/app/v3/registryConfig.server.ts` — propagate
`ecrDefaultRepositoryPolicy` to `RegistryConfig`.
- `apps/webapp/app/v3/getDeploymentImageRef.server.ts` —
`createEcrRepository` accepts the policy; if set, calls
`SetRepositoryPolicy` after `PutLifecyclePolicy`.
- `docs/self-hosting/env/webapp.mdx` — documentation row added under
**Deploy & Registry**.

## Verification

Verified end-to-end against a self-hosted Trigger.dev on EKS where the
ECR account is separate from the cluster account:

- **Without the env var** (current `main`): the new project's first run
pod stays in `ImagePullBackOff` with `403 Forbidden`.
- **With the env var set** to a JSON granting
`ecr:BatchGetImage`/`GetDownloadUrlForLayer`/`BatchCheckLayerAvailability`
to the cluster account: a fresh `trigger.dev deploy --env prod` followed
by a `hello-world` run completes in ~5s end-to-end on the first try.

Manually also confirmed that existing repos are untouched (the call only
fires inside `createEcrRepository`, which only runs when
`DescribeRepositories` returned `RepositoryNotFoundException`).

## Out of scope

- Chart values surface for this — operators already pass the existing
ECR vars via `webapp.extraEnvVars`, so this follows the same pattern.
Happy to add a first-class chart field in a follow-up if that's the
preferred direction.
- IAM-policy validation in the webapp — we forward the JSON verbatim to
AWS and surface AWS's error messages on misuse, matching how
`DEPLOY_REGISTRY_ECR_TAGS` is handled today.

This is a draft pending CI / CodeRabbit pass — happy to iterate on
direction (e.g., split into per-action env vars, or extend the chart
values schema) if any of the above choices feels off.

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
2026-04-29 15:17:23 +01:00

257 lines
33 KiB
Plaintext
Raw Blame History

---
title: "Webapp"
description: "Environment variables for the webapp container."
sidebarTitle: "Webapp"
mode: "wide"
---
| Name | Required | Default | Description |
| :----------------------------------------------- | :------- | :-------------------- | :----------------------------------------------------------------------------------------------------------------- |
| **Secrets** | | | |
| `SESSION_SECRET` | Yes | — | Session encryption secret. Run: `openssl rand -hex 16` |
| `MAGIC_LINK_SECRET` | Yes | — | Magic link encryption secret. Run: `openssl rand -hex 16` |
| `ENCRYPTION_KEY` | Yes | — | Secret store encryption key. Run: `openssl rand -hex 16` |
| `MANAGED_WORKER_SECRET` | No | managed-secret | Managed worker secret. Should be changed and match supervisor. |
| **Domains & ports** | | | |
| `REMIX_APP_PORT` | No | 3030 | Remix app port. |
| `APP_ORIGIN` | Yes | http://localhost:3030 | App origin URL. |
| `LOGIN_ORIGIN` | Yes | http://localhost:3030 | Login origin URL. Most likely the same as `APP_ORIGIN`. |
| `API_ORIGIN` | No | `APP_ORIGIN` | API origin URL. |
| `STREAM_ORIGIN` | No | `APP_ORIGIN` | Realtime stream origin URL. |
| `ELECTRIC_ORIGIN` | No | http://localhost:3060 | Electric origin URL. |
| **Postgres** | | | |
| `DATABASE_URL` | Yes | — | PostgreSQL connection string. |
| `DIRECT_URL` | Yes | — | Direct DB connection string used for migrations etc. |
| `DATABASE_CONNECTION_LIMIT` | No | 10 | Max DB connections. |
| `DATABASE_POOL_TIMEOUT` | No | 60 | DB pool timeout (s). |
| `DATABASE_CONNECTION_TIMEOUT` | No | 20 | DB connect timeout (s). |
| `DATABASE_READ_REPLICA_URL` | No | `DATABASE_URL` | Read-replica DB string. |
| **Redis** | | | |
| `REDIS_HOST` | Yes | — | Redis host. |
| `REDIS_PORT` | Yes | — | Redis port. |
| `REDIS_READER_HOST` | No | `REDIS_HOST` | Redis reader host. |
| `REDIS_READER_PORT` | No | `REDIS_PORT` | Redis reader port. |
| `REDIS_USERNAME` | No | — | Redis username. |
| `REDIS_PASSWORD` | No | — | Redis password. |
| `REDIS_TLS_DISABLED` | No | — | Disable Redis TLS. |
| **Auth** | | | |
| `WHITELISTED_EMAILS` | No | — | Whitelisted emails regex. |
| `AUTH_GITHUB_CLIENT_ID` | No | — | GitHub client ID. |
| `AUTH_GITHUB_CLIENT_SECRET` | No | — | GitHub client secret. |
| **Email** | | | |
| `EMAIL_TRANSPORT` | No | — | Email transport type. One of `resend`, `smtp`, `aws-ses`. |
| `FROM_EMAIL` | No | — | From email address. |
| `REPLY_TO_EMAIL` | No | — | Reply-to email address. |
| `RESEND_API_KEY` | No | — | Resend API key. |
| `SMTP_HOST` | No | — | SMTP host. |
| `SMTP_PORT` | No | — | SMTP port. |
| `SMTP_SECURE` | No | — | SMTP secure flag. |
| `SMTP_USER` | No | — | SMTP user. |
| `SMTP_PASSWORD` | No | — | SMTP password. |
| `AWS_REGION` | No | — | AWS region for SES. |
| `AWS_ACCESS_KEY_ID` | No | — | AWS access key ID for SES. |
| `AWS_SECRET_ACCESS_KEY` | No | — | AWS secret access key for SES. |
| **Graphile & Redis worker** | | | |
| `WORKER_CONCURRENCY` | No | 10 | Redis worker concurrency. |
| `WORKER_POLL_INTERVAL` | No | 1000 | Redis worker poll interval (ms). |
| `WORKER_SCHEMA` | No | graphile_worker | Graphile worker schema. |
| `GRACEFUL_SHUTDOWN_TIMEOUT` | No | 60000 (1m) | Graphile graceful shutdown timeout (ms). Affects shutdown time. |
| **Concurrency limits** | | | |
| `DEFAULT_ENV_EXECUTION_CONCURRENCY_LIMIT` | No | 100 | Default env execution concurrency. |
| `DEFAULT_ORG_EXECUTION_CONCURRENCY_LIMIT` | No | 300 | Default org execution concurrency, needs to be 3x env concurrency. |
| **Dev** | | | |
| `DEV_MAX_CONCURRENT_RUNS` | No | 25 | Sets the max concurrency for dev runs via the CLI. |
| `DEV_OTEL_EXPORTER_OTLP_ENDPOINT` | No | `APP_ORIGIN/otel` | OTel endpoint for dev runs. |
| **Rate limiting** | | | |
| `API_RATE_LIMIT_REFILL_INTERVAL` | No | 10s | API rate limit refill interval. |
| `API_RATE_LIMIT_MAX` | No | 750 | API rate limit max. |
| `API_RATE_LIMIT_REFILL_RATE` | No | 250 | API rate limit refill rate. |
| `API_RATE_LIMIT_REQUEST_LOGS_ENABLED` | No | 0 | API rate limit request logs. |
| `API_RATE_LIMIT_REJECTION_LOGS_ENABLED` | No | 1 | API rate limit rejection logs. |
| `API_RATE_LIMIT_LIMITER_LOGS_ENABLED` | No | 0 | API rate limit limiter logs. |
| `API_RATE_LIMIT_JWT_WINDOW` | No | 1m | API rate limit JWT window. |
| `API_RATE_LIMIT_JWT_TOKENS` | No | 60 | API rate limit JWT tokens. |
| **Deploy & Registry** | | | |
| `DEPLOY_REGISTRY_HOST` | Yes | — | Deploy registry host. |
| `DEPLOY_REGISTRY_USERNAME` | No | — | Deploy registry username. |
| `DEPLOY_REGISTRY_PASSWORD` | No | — | Deploy registry password. |
| `DEPLOY_REGISTRY_NAMESPACE` | No | trigger | Deploy registry namespace. |
| `DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY` | No | — | Raw IAM policy JSON applied via SetRepositoryPolicy to every ECR repo created by the webapp. Use to grant cross-account pull access to EKS workers when the ECR account is separate from the cluster account. |
| `DEPLOY_IMAGE_PLATFORM` | No | linux/amd64 | Deploy image platform, same values as docker `--platform` flag. |
| `DEPLOY_TIMEOUT_MS` | No | 480000 (8m) | Deploy timeout (ms). |
| **Object store (S3)** | | | |
| `OBJECT_STORE_BASE_URL` | No | — | Object store base URL (default provider). |
| `OBJECT_STORE_ACCESS_KEY_ID` | No | — | Object store access key (default provider). |
| `OBJECT_STORE_SECRET_ACCESS_KEY` | No | — | Object store secret key (default provider). |
| `OBJECT_STORE_REGION` | No | — | Object store region (default provider). |
| `OBJECT_STORE_SERVICE` | No | s3 | Object store service (default provider). |
| `OBJECT_STORE_DEFAULT_PROTOCOL` | No | — | Protocol to use for new uploads (e.g., `s3`, `r2`). Enables protocol-prefixed storage. See migration guide below. |
| `OBJECT_STORE_{PROTOCOL}_BASE_URL` | No | — | Named provider base URL (replace `{PROTOCOL}` with protocol name, e.g., `OBJECT_STORE_S3_BASE_URL`). |
| `OBJECT_STORE_{PROTOCOL}_ACCESS_KEY_ID` | No | — | Named provider access key. |
| `OBJECT_STORE_{PROTOCOL}_SECRET_ACCESS_KEY` | No | — | Named provider secret key. |
| `OBJECT_STORE_{PROTOCOL}_REGION` | No | — | Named provider region. |
| `OBJECT_STORE_{PROTOCOL}_SERVICE` | No | — | Named provider service. |
| **Alerts** | | | |
| `ORG_SLACK_INTEGRATION_CLIENT_ID` | No | — | Slack client ID. Required for Slack alerts. |
| `ORG_SLACK_INTEGRATION_CLIENT_SECRET` | No | — | Slack client secret. Required for Slack alerts. |
| `ALERT_EMAIL_TRANSPORT` | No | — | Alert email transport. |
| `ALERT_FROM_EMAIL` | No | — | Alert from email. |
| `ALERT_REPLY_TO_EMAIL` | No | — | Alert reply-to email. |
| `ALERT_RESEND_API_KEY` | No | — | Alert Resend API key. |
| `ALERT_SMTP_HOST` | No | — | Alert SMTP host. |
| `ALERT_SMTP_PORT` | No | — | Alert SMTP port. |
| `ALERT_SMTP_SECURE` | No | — | Alert SMTP secure. |
| `ALERT_SMTP_USER` | No | — | Alert SMTP user. |
| `ALERT_SMTP_PASSWORD` | No | — | Alert SMTP password. |
| **Limits** | | | |
| `TASK_PAYLOAD_OFFLOAD_THRESHOLD` | No | 524288 (512KB) | Max task payload size before offloading to S3. |
| `TASK_PAYLOAD_MAXIMUM_SIZE` | No | 3145728 (3MB) | Max task payload size. |
| `BATCH_TASK_PAYLOAD_MAXIMUM_SIZE` | No | 1000000 (1MB) | Max batch payload size. |
| `TASK_RUN_METADATA_MAXIMUM_SIZE` | No | 262144 (256KB) | Max metadata size. |
| `MAX_BATCH_V2_TRIGGER_ITEMS` | No | 500 | Max batch size (legacy v2 API). |
| `STREAMING_BATCH_MAX_ITEMS` | No | 1000 | Max items in streaming batch (v3 API, requires SDK 4.3.1+). |
| `STREAMING_BATCH_ITEM_MAXIMUM_SIZE` | No | 3145728 (3MB) | Max size per item in streaming batch. |
| `MAXIMUM_DEV_QUEUE_SIZE` | No | — | Max dev queue size. |
| `MAXIMUM_DEPLOYED_QUEUE_SIZE` | No | — | Max deployed queue size. |
| **OTel limits** | | | |
| `TRIGGER_OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` | No | 1024 | OTel span attribute count limit. |
| `TRIGGER_OTEL_LOG_ATTRIBUTE_COUNT_LIMIT` | No | 1024 | OTel log attribute count limit. |
| `TRIGGER_OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` | No | 131072 | OTel span attribute value length limit. |
| `TRIGGER_OTEL_LOG_ATTRIBUTE_VALUE_LENGTH_LIMIT` | No | 131072 | OTel log attribute value length limit. |
| `TRIGGER_OTEL_SPAN_EVENT_COUNT_LIMIT` | No | 10 | OTel span event count limit. |
| `TRIGGER_OTEL_LINK_COUNT_LIMIT` | No | 2 | OTel link count limit. |
| `TRIGGER_OTEL_ATTRIBUTE_PER_LINK_COUNT_LIMIT` | No | 10 | OTel attribute per link count limit. |
| `TRIGGER_OTEL_ATTRIBUTE_PER_EVENT_COUNT_LIMIT` | No | 10 | OTel attribute per event count limit. |
| `SERVER_OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` | No | 8192 | OTel span attribute value length limit. |
| **Realtime** | | | |
| `REALTIME_STREAM_MAX_LENGTH` | No | 1000 | Realtime stream max length. |
| `REALTIME_STREAM_TTL` | No | 86400 (1d) | Realtime stream TTL (s). |
| **Bootstrap** | | | |
| `TRIGGER_BOOTSTRAP_ENABLED` | No | 0 | Trigger bootstrap enabled. |
| `TRIGGER_BOOTSTRAP_WORKER_GROUP_NAME` | No | — | Trigger bootstrap worker group name. |
| `TRIGGER_BOOTSTRAP_WORKER_TOKEN_PATH` | No | — | Trigger bootstrap worker token path. |
| **Run engine** | | | |
| `RUN_ENGINE_WORKER_COUNT` | No | 4 | Run engine worker count. |
| `RUN_ENGINE_TASKS_PER_WORKER` | No | 10 | Run engine tasks per worker. |
| `RUN_ENGINE_WORKER_CONCURRENCY_LIMIT` | No | 10 | Run engine worker concurrency limit. |
| `RUN_ENGINE_WORKER_POLL_INTERVAL` | No | 100 | Run engine worker poll interval (ms). |
| `RUN_ENGINE_WORKER_IMMEDIATE_POLL_INTERVAL` | No | 100 | Run engine worker immediate poll interval (ms). |
| `RUN_ENGINE_WORKER_SHUTDOWN_TIMEOUT_MS` | No | 60000 (1m) | Run engine worker shutdown timeout (ms). |
| `RUN_ENGINE_RATE_LIMIT_REFILL_INTERVAL` | No | 10s | Run engine rate limit refill interval. |
| `RUN_ENGINE_RATE_LIMIT_MAX` | No | 1200 | Run engine rate limit max. |
| `RUN_ENGINE_RATE_LIMIT_REFILL_RATE` | No | 400 | Run engine rate limit refill rate. |
| `RUN_ENGINE_RATE_LIMIT_REQUEST_LOGS_ENABLED` | No | 0 | Run engine rate limit request logs. |
| `RUN_ENGINE_RATE_LIMIT_REJECTION_LOGS_ENABLED` | No | 1 | Run engine rate limit rejection logs. |
| `RUN_ENGINE_RATE_LIMIT_LIMITER_LOGS_ENABLED` | No | 0 | Run engine rate limit limiter logs. |
| `RUN_ENGINE_DEFAULT_MAX_TTL` | No | — | Maximum TTL for all runs (e.g. "14d"). Runs without a TTL use this as default; runs with a larger TTL are clamped. |
| `MAXIMUM_DEV_QUEUE_SIZE` | No | — | Maximum queued runs per queue in development environments. |
| `MAXIMUM_DEPLOYED_QUEUE_SIZE` | No | — | Maximum queued runs per queue in deployed (staging/prod) environments. |
| **Misc** | | | |
| `TRIGGER_TELEMETRY_DISABLED` | No | — | Disable telemetry. |
| `NODE_MAX_OLD_SPACE_SIZE` | No | 8192 | Maximum memory allocation for Node.js heap in MiB (e.g. "4096" for 4GB). |
| `OPENAI_API_KEY` | No | — | OpenAI API key. |
| `MACHINE_PRESETS_OVERRIDE_PATH` | No | — | Path to machine presets override file. See [machine overrides](/self-hosting/overview#machine-overrides). |
| `APP_ENV` | No | `NODE_ENV` | App environment. Used for things like the title tag. |
| `ADMIN_EMAILS` | No | — | Regex of user emails to automatically promote to admin on signup. Does not apply to existing users. |
| `EVENT_LOOP_MONITOR_ENABLED` | No | 1 | Node.js event loop lag monitor. |
## Multi-Provider Object Storage
The object storage system supports multiple S3-compatible providers (R2, S3, GCS, MinIO, etc.) using protocol prefixes. This enables migrating between providers without breaking existing runs.
### How It Works
When data exceeds the configured threshold (`TASK_PAYLOAD_OFFLOAD_THRESHOLD`), it's uploaded to object storage. The storage location is saved in the database with an optional protocol prefix:
- **With protocol**: `s3://run_abc/payload.json` or `r2://batch_123/item_0/payload.json`
- **Without protocol** (legacy): `batch_123/item_0/payload.json` (uses default provider)
### Configuration
#### Default Provider (Backward Compatible)
The default provider is used for data without a protocol prefix:
```bash
# Default provider (backward compatible - no protocol prefix)
OBJECT_STORE_BASE_URL=https://r2.example.com
OBJECT_STORE_ACCESS_KEY_ID=...
OBJECT_STORE_SECRET_ACCESS_KEY=...
OBJECT_STORE_REGION=auto
OBJECT_STORE_SERVICE=s3
```
#### Named Providers
Named providers are accessed via protocol-prefixed URIs. Configure them using `OBJECT_STORE_{PROTOCOL}_*` variables:
```bash
# S3 provider (accessed via s3:// prefix)
OBJECT_STORE_S3_BASE_URL=https://s3.amazonaws.com
OBJECT_STORE_S3_ACCESS_KEY_ID=...
OBJECT_STORE_S3_SECRET_ACCESS_KEY=...
OBJECT_STORE_S3_REGION=us-east-1
OBJECT_STORE_S3_SERVICE=s3
# R2 provider (accessed via r2:// prefix)
OBJECT_STORE_R2_BASE_URL=https://...r2.cloudflarestorage.com
OBJECT_STORE_R2_ACCESS_KEY_ID=...
OBJECT_STORE_R2_SECRET_ACCESS_KEY=...
OBJECT_STORE_R2_REGION=auto
OBJECT_STORE_R2_SERVICE=s3
```
#### Default Protocol for New Uploads
Set `OBJECT_STORE_DEFAULT_PROTOCOL` to specify which provider to use for new uploads:
```bash
# Use S3 for new uploads (old data without prefix still uses default provider)
OBJECT_STORE_DEFAULT_PROTOCOL=s3
```
### Migration Guide
To migrate from R2 to S3 without breaking existing runs:
<Steps>
<Step title="Configure S3 provider">
Add S3 credentials as a named provider:
```bash
OBJECT_STORE_S3_BASE_URL=https://s3.amazonaws.com
OBJECT_STORE_S3_ACCESS_KEY_ID=...
OBJECT_STORE_S3_SECRET_ACCESS_KEY=...
OBJECT_STORE_S3_REGION=us-east-1
```
Keep your existing `OBJECT_STORE_*` variables (R2) as the default provider.
</Step>
<Step title="Test the configuration">
Restart the webapp and verify both providers work:
- Old runs (no prefix) should still access R2
- New runs with `s3://` prefix should use S3
</Step>
<Step title="Switch to S3 for new uploads">
Set the default protocol to use S3 for new uploads:
```bash
OBJECT_STORE_DEFAULT_PROTOCOL=s3
```
After this change:
- New data uses `s3://` prefix and goes to S3
- Old data (no prefix) still uses R2
- Data with explicit protocol uses the corresponding provider
</Step>
<Step title="Optionally decommission R2">
Once all active runs using R2 data have completed (check your data retention policies), you can remove the R2 credentials. Keep `OBJECT_STORE_DEFAULT_PROTOCOL=s3` to ensure new data continues using S3.
</Step>
</Steps>
Reference in New Issue View Git Blame Copy Permalink