Matt Aitken
f48c89752c
## Problem
The item-streaming endpoint of the two-phase batch API (`POST
/api/v3/batches/:batchId/items`) processed streamed items strictly
sequentially. For a batch of many large payloads, each offloaded to
object storage inline, this serialized N object-store round-trips inside
a single request and could exceed Node's default `server.requestTimeout`
(300s). The webapp then returned `408`, which the SDK reads as `408
terminated` and retries up to 5 times, turning a slow ingest into a
failure that takes tens of minutes to surface.
## Fix
Ingest now runs through `p-map` over the NDJSON async iterable with
bounded concurrency (`STREAMING_BATCH_INGEST_CONCURRENCY`, default 10):
- `p-map` pulls lazily from the stream, so at most `concurrency` items
are read and in-flight at once. Peak memory stays bounded to roughly
`concurrency × STREAMING_BATCH_ITEM_MAXIMUM_SIZE` and request-body
backpressure is preserved.
- Set the env to `1` for fully sequential ingestion (escape hatch).
## Why this is safe (ordering and idempotency unchanged)
- Ordering derives from each item's index (enqueue `timestamp =
batch.createdAt + index`), not enqueue order.
- Dedup is atomic per index in `enqueueBatchItem`.
- The NDJSON parser now stamps oversized-item markers with their emit
position, removing the consumer's sequential `lastIndex` assumption (the
only order-dependent bit).
- The count-check and conditional-seal path is untouched.
## Scope
This speeds up every batch ingested through the streaming endpoint, not
just large-payload batches. Each item does a per-item Redis enqueue
regardless of size, and those now overlap. Large payloads benefit most
because they add an object-store offload round-trip on top of the
enqueue.
## Verification
Added an integration test (`streamBatchItems.test.ts`) that drives the
real service against Postgres + Redis + RunEngine and times a 150-item
batch at increasing concurrency. Object-store offload is modelled as a
fixed per-item latency (local round-trips are too small to compare
meaningfully):
```
runCount=150
large payloads (10ms/item offload):
concurrency=1 1739ms
concurrency=10 192ms (9.1x faster)
concurrency=50 57ms (30.7x faster)
small payloads (Redis enqueue only, no offload):
concurrency=1 90ms
concurrency=10 24ms (3.7x faster)
```
The test asserts correctness at every concurrency (all items accepted,
sealed, enqueued exactly once), that parallel ingest beats the
sequential floor, and that the small-payload case is strictly faster
than sequential, so the win is not specific to large payloads.
Also exercised end-to-end over real HTTP against a local server: a
20-item batch (12MB body) ingests and seals, a re-stream of the sealed
batch returns `sealed: true` with zero re-accepted items (idempotent
retry), and an oversized item still seals at its correct index.
Existing coverage stays green: concurrent ingest of a 100-item batch,
in-flight processing never exceeding the configured concurrency,
concurrent dedup on streaming retry, and emit-position marker indexing.
## Follow-ups (not in this PR)
- SDK pre-offload of large item payloads (send `application/store` refs
instead of raw blobs) to remove object-store work from the request hot
path and shrink the request body.
- Optional `server.requestTimeout` bump as a safety net.
## CI fix
Added `.github/workflows/codeql.yml` to replace GitHub's automatic
("dynamic") CodeQL scanning. The dynamic setup was failing to upload
SARIF results because the auto-generated `GITHUB_TOKEN` lacked the
`security-events: write` permission. The explicit workflow grants that
permission at the job level and pins all actions to commit SHAs,
consistent with the repo's security conventions.
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
- Integration test (`streamBatchItems.test.ts`) validates correctness
and performance at concurrency 1, 10, and 50 for both large and small
payloads.
- End-to-end verified over real HTTP: 20-item/12MB batch ingests and
seals, idempotent retry returns `sealed: true`, oversized item seals at
correct index.
---
## Changelog
Streaming batch ingest now processes items with bounded concurrency
instead of one at a time, so batches of many large payloads ingest far
faster and no longer time out. Concurrency is configurable via
`STREAMING_BATCH_INGEST_CONCURRENCY` (default 10); set it to 1 for fully
sequential ingestion.
---
## Screenshots
_[Screenshots]_
💯
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
277 lines
37 KiB
Plaintext
277 lines
37 KiB
Plaintext
---
|
||
title: "Webapp"
|
||
description: "Environment variables for the webapp container."
|
||
sidebarTitle: "Webapp"
|
||
mode: "wide"
|
||
---
|
||
|
||
| Name | Required | Default | Description |
|
||
| :----------------------------------------------- | :------- | :-------------------- | :----------------------------------------------------------------------------------------------------------------- |
|
||
| **Secrets** | | | |
|
||
| `SESSION_SECRET` | Yes | — | Session encryption secret. Run: `openssl rand -hex 16` |
|
||
| `MAGIC_LINK_SECRET` | Yes | — | Magic link encryption secret. Run: `openssl rand -hex 16` |
|
||
| `ENCRYPTION_KEY` | Yes | — | Secret store encryption key. Run: `openssl rand -hex 16` |
|
||
| `MANAGED_WORKER_SECRET` | No | managed-secret | Managed worker secret. Should be changed and match supervisor. |
|
||
| **Domains & ports** | | | |
|
||
| `REMIX_APP_PORT` | No | 3030 | Remix app port. |
|
||
| `APP_ORIGIN` | Yes | http://localhost:3030 | App origin URL. |
|
||
| `LOGIN_ORIGIN` | Yes | http://localhost:3030 | Login origin URL. Most likely the same as `APP_ORIGIN`. |
|
||
| `API_ORIGIN` | No | `APP_ORIGIN` | API origin URL. |
|
||
| `STREAM_ORIGIN` | No | `APP_ORIGIN` | Realtime stream origin URL. |
|
||
| `ELECTRIC_ORIGIN` | No | http://localhost:3060 | Electric origin URL. |
|
||
| **Postgres** | | | |
|
||
| `DATABASE_URL` | Yes | — | PostgreSQL connection string. |
|
||
| `DIRECT_URL` | Yes | — | Direct DB connection string used for migrations etc. |
|
||
| `DATABASE_CONNECTION_LIMIT` | No | 10 | Max DB connections. |
|
||
| `DATABASE_POOL_TIMEOUT` | No | 60 | DB pool timeout (s). |
|
||
| `DATABASE_CONNECTION_TIMEOUT` | No | 20 | DB connect timeout (s). |
|
||
| `DATABASE_READ_REPLICA_URL` | No | `DATABASE_URL` | Read-replica DB string. |
|
||
| **Redis** | | | |
|
||
| `REDIS_HOST` | Yes | — | Redis host. |
|
||
| `REDIS_PORT` | Yes | — | Redis port. |
|
||
| `REDIS_READER_HOST` | No | `REDIS_HOST` | Redis reader host. |
|
||
| `REDIS_READER_PORT` | No | `REDIS_PORT` | Redis reader port. |
|
||
| `REDIS_USERNAME` | No | — | Redis username. |
|
||
| `REDIS_PASSWORD` | No | — | Redis password. |
|
||
| `REDIS_TLS_DISABLED` | No | — | Disable Redis TLS. |
|
||
| **Auth** | | | |
|
||
| `WHITELISTED_EMAILS` | No | — | Whitelisted emails regex. |
|
||
| `LOGIN_RATE_LIMITS_ENABLED` | No | true | Enable rate limiting on magic-link login. |
|
||
| `AUTH_GITHUB_CLIENT_ID` | No | — | GitHub client ID. |
|
||
| `AUTH_GITHUB_CLIENT_SECRET` | No | — | GitHub client secret. |
|
||
| **Email** | | | |
|
||
| `EMAIL_TRANSPORT` | No | — | Email transport type. One of `resend`, `smtp`, `aws-ses`. |
|
||
| `FROM_EMAIL` | No | — | From email address. |
|
||
| `REPLY_TO_EMAIL` | No | — | Reply-to email address. |
|
||
| `RESEND_API_KEY` | No | — | Resend API key. |
|
||
| `SMTP_HOST` | No | — | SMTP host. |
|
||
| `SMTP_PORT` | No | — | SMTP port. |
|
||
| `SMTP_SECURE` | No | — | SMTP secure flag. |
|
||
| `SMTP_USER` | No | — | SMTP user. |
|
||
| `SMTP_PASSWORD` | No | — | SMTP password. |
|
||
| `AWS_REGION` | No | — | AWS region for SES. |
|
||
| `AWS_ACCESS_KEY_ID` | No | — | AWS access key ID for SES. |
|
||
| `AWS_SECRET_ACCESS_KEY` | No | — | AWS secret access key for SES. |
|
||
| **Graphile & Redis worker** | | | |
|
||
| `WORKER_CONCURRENCY` | No | 10 | Redis worker concurrency. |
|
||
| `WORKER_POLL_INTERVAL` | No | 1000 | Redis worker poll interval (ms). |
|
||
| `WORKER_SCHEMA` | No | graphile_worker | Graphile worker schema. |
|
||
| `GRACEFUL_SHUTDOWN_TIMEOUT` | No | 60000 (1m) | Graphile graceful shutdown timeout (ms). Affects shutdown time. |
|
||
| **Concurrency limits** | | | |
|
||
| `DEFAULT_ENV_EXECUTION_CONCURRENCY_LIMIT` | No | 100 | Default env execution concurrency. |
|
||
| `DEFAULT_ORG_EXECUTION_CONCURRENCY_LIMIT` | No | 300 | Default org execution concurrency, needs to be 3x env concurrency. |
|
||
| `DEFAULT_ENV_EXECUTION_CONCURRENCY_BURST_FACTOR` | No | 1.0 | Burst factor for env concurrency. |
|
||
| `DEFAULT_DEV_ENV_EXECUTION_ATTEMPTS` | No | 1 | Default max attempts for dev environment runs. |
|
||
| **Dev** | | | |
|
||
| `DEV_MAX_CONCURRENT_RUNS` | No | 25 | Sets the max concurrency for dev runs via the CLI. |
|
||
| `DEV_OTEL_EXPORTER_OTLP_ENDPOINT` | No | `APP_ORIGIN/otel` | OTel endpoint for dev runs. |
|
||
| **Rate limiting** | | | |
|
||
| `API_RATE_LIMIT_REFILL_INTERVAL` | No | 10s | API rate limit refill interval. |
|
||
| `API_RATE_LIMIT_MAX` | No | 750 | API rate limit max. |
|
||
| `API_RATE_LIMIT_REFILL_RATE` | No | 250 | API rate limit refill rate. |
|
||
| `API_RATE_LIMIT_REQUEST_LOGS_ENABLED` | No | 0 | API rate limit request logs. |
|
||
| `API_RATE_LIMIT_REJECTION_LOGS_ENABLED` | No | 1 | API rate limit rejection logs. |
|
||
| `API_RATE_LIMIT_LIMITER_LOGS_ENABLED` | No | 0 | API rate limit limiter logs. |
|
||
| `API_RATE_LIMIT_JWT_WINDOW` | No | 1m | API rate limit JWT window. |
|
||
| `API_RATE_LIMIT_JWT_TOKENS` | No | 60 | API rate limit JWT tokens. |
|
||
| **Deploy & Registry** | | | |
|
||
| `DEPLOY_REGISTRY_HOST` | Yes | — | Deploy registry host. |
|
||
| `DEPLOY_REGISTRY_USERNAME` | No | — | Deploy registry username. |
|
||
| `DEPLOY_REGISTRY_PASSWORD` | No | — | Deploy registry password. |
|
||
| `DEPLOY_REGISTRY_NAMESPACE` | No | trigger | Deploy registry namespace. |
|
||
| `DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY` | No | — | Raw IAM policy JSON applied via SetRepositoryPolicy to every ECR repo created by the webapp. Use to grant cross-account pull access to EKS workers when the ECR account is separate from the cluster account. |
|
||
| `DEPLOY_IMAGE_PLATFORM` | No | linux/amd64 | Deploy image platform, same values as docker `--platform` flag. |
|
||
| `DEPLOY_TIMEOUT_MS` | No | 480000 (8m) | Deploy timeout (ms). |
|
||
| `DEPLOY_QUEUE_TIMEOUT_MS` | No | 900000 (15m) | Deploy queue timeout (ms). |
|
||
| **Object store (S3)** | | | |
|
||
| `OBJECT_STORE_BASE_URL` | No | — | Object store base URL (default provider). |
|
||
| `OBJECT_STORE_BUCKET` | No | — | Object store bucket name (default provider). |
|
||
| `OBJECT_STORE_ACCESS_KEY_ID` | No | — | Object store access key (default provider). |
|
||
| `OBJECT_STORE_SECRET_ACCESS_KEY` | No | — | Object store secret key (default provider). |
|
||
| `OBJECT_STORE_REGION` | No | — | Object store region (default provider). |
|
||
| `OBJECT_STORE_SERVICE` | No | s3 | Object store service (default provider). |
|
||
| `OBJECT_STORE_DEFAULT_PROTOCOL` | No | — | Protocol for new uploads (e.g. `s3`, `r2`). Enables protocol-prefixed storage. See migration guide below. |
|
||
| `OBJECT_STORE_{PROTOCOL}_BASE_URL` | No | — | Named provider base URL (replace `{PROTOCOL}`, e.g. `OBJECT_STORE_S3_BASE_URL`). |
|
||
| `OBJECT_STORE_{PROTOCOL}_ACCESS_KEY_ID` | No | — | Named provider access key. |
|
||
| `OBJECT_STORE_{PROTOCOL}_SECRET_ACCESS_KEY` | No | — | Named provider secret key. |
|
||
| `OBJECT_STORE_{PROTOCOL}_REGION` | No | — | Named provider region. |
|
||
| `OBJECT_STORE_{PROTOCOL}_SERVICE` | No | — | Named provider service. |
|
||
| `ARTIFACTS_OBJECT_STORE_BUCKET` | No | — | Optional separate bucket for artifacts. If not set, uses main object store. |
|
||
| `ARTIFACTS_OBJECT_STORE_BASE_URL` | No | — | Optional artifacts store base URL. |
|
||
| `ARTIFACTS_OBJECT_STORE_ACCESS_KEY_ID` | No | — | Optional artifacts store access key. |
|
||
| `ARTIFACTS_OBJECT_STORE_SECRET_ACCESS_KEY` | No | — | Optional artifacts store secret key. |
|
||
| `ARTIFACTS_OBJECT_STORE_REGION` | No | — | Optional artifacts store region. |
|
||
| **Alerts** | | | |
|
||
| `ORG_SLACK_INTEGRATION_CLIENT_ID` | No | — | Slack client ID. Required for Slack alerts. |
|
||
| `ORG_SLACK_INTEGRATION_CLIENT_SECRET` | No | — | Slack client secret. Required for Slack alerts. |
|
||
| `ALERT_EMAIL_TRANSPORT` | No | — | Alert email transport. |
|
||
| `ALERT_FROM_EMAIL` | No | — | Alert from email. |
|
||
| `ALERT_REPLY_TO_EMAIL` | No | — | Alert reply-to email. |
|
||
| `ALERT_RESEND_API_KEY` | No | — | Alert Resend API key. |
|
||
| `ALERT_SMTP_HOST` | No | — | Alert SMTP host. |
|
||
| `ALERT_SMTP_PORT` | No | — | Alert SMTP port. |
|
||
| `ALERT_SMTP_SECURE` | No | — | Alert SMTP secure. |
|
||
| `ALERT_SMTP_USER` | No | — | Alert SMTP user. |
|
||
| `ALERT_SMTP_PASSWORD` | No | — | Alert SMTP password. |
|
||
| **Limits** | | | |
|
||
| `TASK_PAYLOAD_OFFLOAD_THRESHOLD` | No | 524288 (512KB) | Max task payload size before offloading to S3. |
|
||
| `TASK_PAYLOAD_MAXIMUM_SIZE` | No | 3145728 (3MB) | Max task payload size. |
|
||
| `BATCH_TASK_PAYLOAD_MAXIMUM_SIZE` | No | 1000000 (1MB) | Max batch payload size. |
|
||
| `BATCH_CONCURRENCY_LIMIT_DEFAULT` | No | 5 | Default concurrency for batch processing. |
|
||
| `BATCH_RATE_LIMIT_REFILL_RATE` | No | 100 | Batch rate limit refill rate. |
|
||
| `BATCH_RATE_LIMIT_MAX` | No | 1200 | Batch rate limit max. |
|
||
| `BATCH_RATE_LIMIT_REFILL_INTERVAL` | No | 10s | Batch rate limit refill interval. |
|
||
| `TASK_RUN_METADATA_MAXIMUM_SIZE` | No | 262144 (256KB) | Max metadata size. |
|
||
| `MAX_BATCH_V2_TRIGGER_ITEMS` | No | 500 | Max batch size (legacy v2 API). |
|
||
| `STREAMING_BATCH_MAX_ITEMS` | No | 1000 | Max items in streaming batch (v3 API, requires SDK 4.3.1+). |
|
||
| `STREAMING_BATCH_ITEM_MAXIMUM_SIZE` | No | 3145728 (3MB) | Max size per item in streaming batch. |
|
||
| `STREAMING_BATCH_INGEST_CONCURRENCY` | No | 10 | Items ingested concurrently per streaming batch request. Peak memory ≈ this × item size. Set to 1 for sequential. |
|
||
| **OTel limits** | | | |
|
||
| `TRIGGER_OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` | No | 1024 | OTel span attribute count limit. |
|
||
| `TRIGGER_OTEL_LOG_ATTRIBUTE_COUNT_LIMIT` | No | 1024 | OTel log attribute count limit. |
|
||
| `TRIGGER_OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` | No | 131072 | OTel span attribute value length limit. |
|
||
| `TRIGGER_OTEL_LOG_ATTRIBUTE_VALUE_LENGTH_LIMIT` | No | 131072 | OTel log attribute value length limit. |
|
||
| `TRIGGER_OTEL_SPAN_EVENT_COUNT_LIMIT` | No | 10 | OTel span event count limit. |
|
||
| `TRIGGER_OTEL_LINK_COUNT_LIMIT` | No | 2 | OTel link count limit. |
|
||
| `TRIGGER_OTEL_ATTRIBUTE_PER_LINK_COUNT_LIMIT` | No | 10 | OTel attribute per link count limit. |
|
||
| `TRIGGER_OTEL_ATTRIBUTE_PER_EVENT_COUNT_LIMIT` | No | 10 | OTel attribute per event count limit. |
|
||
| `SERVER_OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` | No | 8192 | OTel span attribute value length limit. |
|
||
| **Task events** | | | |
|
||
| `EVENT_REPOSITORY_DEFAULT_STORE` | No | postgres | Where to store task events. Set to `clickhouse_v2` to store in ClickHouse (recommended for production). |
|
||
| **Realtime** | | | |
|
||
| `REALTIME_STREAM_VERSION` | No | v1 | Realtime stream protocol version. One of `v1`, `v2`. |
|
||
| `REALTIME_STREAM_MAX_LENGTH` | No | 1000 | Realtime stream max length. |
|
||
| `REALTIME_STREAM_TTL` | No | 86400 (1d) | Realtime stream TTL (s). |
|
||
| **Bootstrap** | | | |
|
||
| `TRIGGER_BOOTSTRAP_ENABLED` | No | 0 | Trigger bootstrap enabled. |
|
||
| `TRIGGER_BOOTSTRAP_WORKER_GROUP_NAME` | No | — | Trigger bootstrap worker group name. |
|
||
| `TRIGGER_BOOTSTRAP_WORKER_TOKEN_PATH` | No | — | Trigger bootstrap worker token path. |
|
||
| **Run engine** | | | |
|
||
| `RUN_ENGINE_WORKER_COUNT` | No | 4 | Run engine worker count. |
|
||
| `RUN_ENGINE_TASKS_PER_WORKER` | No | 10 | Run engine tasks per worker. |
|
||
| `RUN_ENGINE_WORKER_CONCURRENCY_LIMIT` | No | 10 | Run engine worker concurrency limit. |
|
||
| `RUN_ENGINE_WORKER_POLL_INTERVAL` | No | 100 | Run engine worker poll interval (ms). |
|
||
| `RUN_ENGINE_WORKER_IMMEDIATE_POLL_INTERVAL` | No | 100 | Run engine worker immediate poll interval (ms). |
|
||
| `RUN_ENGINE_WORKER_SHUTDOWN_TIMEOUT_MS` | No | 60000 (1m) | Run engine worker shutdown timeout (ms). |
|
||
| `RUN_ENGINE_RATE_LIMIT_REFILL_INTERVAL` | No | 10s | Run engine rate limit refill interval. |
|
||
| `RUN_ENGINE_RATE_LIMIT_MAX` | No | 1200 | Run engine rate limit max. |
|
||
| `RUN_ENGINE_RATE_LIMIT_REFILL_RATE` | No | 400 | Run engine rate limit refill rate. |
|
||
| `RUN_ENGINE_RATE_LIMIT_REQUEST_LOGS_ENABLED` | No | 0 | Run engine rate limit request logs. |
|
||
| `RUN_ENGINE_RATE_LIMIT_REJECTION_LOGS_ENABLED` | No | 1 | Run engine rate limit rejection logs. |
|
||
| `RUN_ENGINE_RATE_LIMIT_LIMITER_LOGS_ENABLED` | No | 0 | Run engine rate limit limiter logs. |
|
||
| `RUN_ENGINE_DEFAULT_MAX_TTL` | No | — | Maximum TTL for all runs (e.g. "14d"). Runs without a TTL use this as default; runs with a larger TTL are clamped. |
|
||
| `MAXIMUM_DEV_QUEUE_SIZE` | No | — | Maximum queued runs per queue in development environments. |
|
||
| `MAXIMUM_DEPLOYED_QUEUE_SIZE` | No | — | Maximum queued runs per queue in deployed (staging/prod) environments. |
|
||
| **Misc** | | | |
|
||
| `PROVIDER_SECRET` | No | provider-secret | Secret for provider auth. **Must be set to a secure value in self-hosted/production**; the default is insecure. |
|
||
| `COORDINATOR_SECRET` | No | coordinator-secret | Secret for coordinator auth. **Must be set to a secure value in self-hosted/production**; the default is insecure. |
|
||
| `TRIGGER_TELEMETRY_DISABLED` | No | — | Disable telemetry. |
|
||
| `NODE_MAX_OLD_SPACE_SIZE` | No | 8192 | Maximum memory allocation for Node.js heap in MiB (e.g. "4096" for 4GB). |
|
||
| `OPENAI_API_KEY` | No | — | OpenAI API key. |
|
||
| `MACHINE_PRESETS_OVERRIDE_PATH` | No | — | Path to machine presets override file. See [machine overrides](/self-hosting/overview#machine-overrides). |
|
||
| `APP_ENV` | No | `NODE_ENV` | App environment. Used for things like the title tag. |
|
||
| `ADMIN_EMAILS` | No | — | Regex of user emails to automatically promote to admin on signup. Does not apply to existing users. |
|
||
| `EVENT_LOOP_MONITOR_ENABLED` | No | 1 | Node.js event loop lag monitor. |
|
||
|
||
## Multi-Provider Object Storage
|
||
|
||
The object storage system supports multiple S3-compatible providers (R2, S3, GCS, MinIO, etc.) using protocol prefixes. This enables migrating between providers without breaking existing runs.
|
||
|
||
### How It Works
|
||
|
||
When data exceeds the configured threshold (`TASK_PAYLOAD_OFFLOAD_THRESHOLD`), it's uploaded to object storage. The storage location is saved in the database with an optional protocol prefix:
|
||
|
||
- **With protocol**: `s3://run_abc/payload.json` or `r2://batch_123/item_0/payload.json`
|
||
- **Without protocol** (legacy): `batch_123/item_0/payload.json` (uses default provider)
|
||
|
||
### Configuration
|
||
|
||
#### Default Provider (Backward Compatible)
|
||
|
||
The default provider is used for data without a protocol prefix:
|
||
|
||
```bash
|
||
# Default provider (backward compatible - no protocol prefix)
|
||
OBJECT_STORE_BASE_URL=https://r2.example.com
|
||
OBJECT_STORE_ACCESS_KEY_ID=...
|
||
OBJECT_STORE_SECRET_ACCESS_KEY=...
|
||
OBJECT_STORE_REGION=auto
|
||
OBJECT_STORE_SERVICE=s3
|
||
```
|
||
|
||
#### Named Providers
|
||
|
||
Named providers are accessed via protocol-prefixed URIs. Configure them using `OBJECT_STORE_{PROTOCOL}_*` variables:
|
||
|
||
```bash
|
||
# S3 provider (accessed via s3:// prefix)
|
||
OBJECT_STORE_S3_BASE_URL=https://s3.amazonaws.com
|
||
OBJECT_STORE_S3_ACCESS_KEY_ID=...
|
||
OBJECT_STORE_S3_SECRET_ACCESS_KEY=...
|
||
OBJECT_STORE_S3_REGION=us-east-1
|
||
OBJECT_STORE_S3_SERVICE=s3
|
||
|
||
# R2 provider (accessed via r2:// prefix)
|
||
OBJECT_STORE_R2_BASE_URL=https://...r2.cloudflarestorage.com
|
||
OBJECT_STORE_R2_ACCESS_KEY_ID=...
|
||
OBJECT_STORE_R2_SECRET_ACCESS_KEY=...
|
||
OBJECT_STORE_R2_REGION=auto
|
||
OBJECT_STORE_R2_SERVICE=s3
|
||
```
|
||
|
||
#### Default Protocol for New Uploads
|
||
|
||
Set `OBJECT_STORE_DEFAULT_PROTOCOL` to specify which provider to use for new uploads:
|
||
|
||
```bash
|
||
# Use S3 for new uploads (old data without prefix still uses default provider)
|
||
OBJECT_STORE_DEFAULT_PROTOCOL=s3
|
||
```
|
||
|
||
### Migration Guide
|
||
|
||
To migrate from R2 to S3 without breaking existing runs:
|
||
|
||
<Steps>
|
||
<Step title="Configure S3 provider">
|
||
Add S3 credentials as a named provider:
|
||
|
||
```bash
|
||
OBJECT_STORE_S3_BASE_URL=https://s3.amazonaws.com
|
||
OBJECT_STORE_S3_ACCESS_KEY_ID=...
|
||
OBJECT_STORE_S3_SECRET_ACCESS_KEY=...
|
||
OBJECT_STORE_S3_REGION=us-east-1
|
||
```
|
||
|
||
Keep your existing `OBJECT_STORE_*` variables (R2) as the default provider.
|
||
|
||
</Step>
|
||
|
||
<Step title="Test the configuration">
|
||
Restart the webapp and verify both providers work:
|
||
|
||
- Old runs (no prefix) should still access R2
|
||
- New runs with `s3://` prefix should use S3
|
||
|
||
</Step>
|
||
|
||
<Step title="Switch to S3 for new uploads">
|
||
Set the default protocol to use S3 for new uploads:
|
||
|
||
```bash
|
||
OBJECT_STORE_DEFAULT_PROTOCOL=s3
|
||
```
|
||
|
||
After this change:
|
||
|
||
- New data uses `s3://` prefix and goes to S3
|
||
- Old data (no prefix) still uses R2
|
||
- Data with explicit protocol uses the corresponding provider
|
||
|
||
</Step>
|
||
|
||
<Step title="Optionally decommission R2">
|
||
Once all active runs using R2 data have completed (check your data retention policies), you can remove the R2 credentials. Keep `OBJECT_STORE_DEFAULT_PROTOCOL=s3` to ensure new data continues using S3.
|
||
</Step>
|
||
</Steps>
|