## Summary
Depends on #4136.
Docker Compose self-hosting now uses the maintained `latest` image tag
by default instead of the frozen prerelease tag. The version-locking
docs keep pointing production users at explicit versioned tags when they
want pinned upgrades.
## Summary
Stable v4 Docker image builds now also publish `v4` and `latest` tags,
giving Docker-based self-hosters a maintained floating tag to use after
a stable release. The Kubernetes guide now uses the current Helm chart
line and current pinned examples, so new installs and upgrades resolve
to the 4.5 chart line instead of the 4.0 line.
## Design
The publish workflows add the floating tags only for stable `v4.x.x`
image tags. Prerelease and `main` builds keep their existing tags.
A batch of technical-SEO fixes across the docs, all reader-facing
(titles, links, redirects):
- Canonicalize the duplicate CLI command pages: the bare `/cli-dev` and
`/cli-deploy` paths now permanently redirect to their `-commands`
equivalents, and a duplicate navigation entry is removed.
- Give the three pages that all rendered as "Overview" distinct titles
(Building with AI, self-hosting overview, Management API overview), with
sidebar labels unchanged.
- Replace the generic "Learn more" links in the introduction's
build-extension list with descriptive anchor text.
- Switch two http links to https in the Supabase guides, point a
troubleshooting page's help link to Discord, and add missing meta
descriptions to three help and troubleshooting pages.
## Problem
The item-streaming endpoint of the two-phase batch API (`POST
/api/v3/batches/:batchId/items`) processed streamed items strictly
sequentially. For a batch of many large payloads, each offloaded to
object storage inline, this serialized N object-store round-trips inside
a single request and could exceed Node's default `server.requestTimeout`
(300s). The webapp then returned `408`, which the SDK reads as `408
terminated` and retries up to 5 times, turning a slow ingest into a
failure that takes tens of minutes to surface.
## Fix
Ingest now runs through `p-map` over the NDJSON async iterable with
bounded concurrency (`STREAMING_BATCH_INGEST_CONCURRENCY`, default 10):
- `p-map` pulls lazily from the stream, so at most `concurrency` items
are read and in-flight at once. Peak memory stays bounded to roughly
`concurrency × STREAMING_BATCH_ITEM_MAXIMUM_SIZE` and request-body
backpressure is preserved.
- Set the env to `1` for fully sequential ingestion (escape hatch).
## Why this is safe (ordering and idempotency unchanged)
- Ordering derives from each item's index (enqueue `timestamp =
batch.createdAt + index`), not enqueue order.
- Dedup is atomic per index in `enqueueBatchItem`.
- The NDJSON parser now stamps oversized-item markers with their emit
position, removing the consumer's sequential `lastIndex` assumption (the
only order-dependent bit).
- The count-check and conditional-seal path is untouched.
## Scope
This speeds up every batch ingested through the streaming endpoint, not
just large-payload batches. Each item does a per-item Redis enqueue
regardless of size, and those now overlap. Large payloads benefit most
because they add an object-store offload round-trip on top of the
enqueue.
## Verification
Added an integration test (`streamBatchItems.test.ts`) that drives the
real service against Postgres + Redis + RunEngine and times a 150-item
batch at increasing concurrency. Object-store offload is modelled as a
fixed per-item latency (local round-trips are too small to compare
meaningfully):
```
runCount=150
large payloads (10ms/item offload):
concurrency=1 1739ms
concurrency=10 192ms (9.1x faster)
concurrency=50 57ms (30.7x faster)
small payloads (Redis enqueue only, no offload):
concurrency=1 90ms
concurrency=10 24ms (3.7x faster)
```
The test asserts correctness at every concurrency (all items accepted,
sealed, enqueued exactly once), that parallel ingest beats the
sequential floor, and that the small-payload case is strictly faster
than sequential, so the win is not specific to large payloads.
Also exercised end-to-end over real HTTP against a local server: a
20-item batch (12MB body) ingests and seals, a re-stream of the sealed
batch returns `sealed: true` with zero re-accepted items (idempotent
retry), and an oversized item still seals at its correct index.
Existing coverage stays green: concurrent ingest of a 100-item batch,
in-flight processing never exceeding the configured concurrency,
concurrent dedup on streaming retry, and emit-position marker indexing.
## Follow-ups (not in this PR)
- SDK pre-offload of large item payloads (send `application/store` refs
instead of raw blobs) to remove object-store work from the request hot
path and shrink the request body.
- Optional `server.requestTimeout` bump as a safety net.
## CI fix
Added `.github/workflows/codeql.yml` to replace GitHub's automatic
("dynamic") CodeQL scanning. The dynamic setup was failing to upload
SARIF results because the auto-generated `GITHUB_TOKEN` lacked the
`security-events: write` permission. The explicit workflow grants that
permission at the job level and pins all actions to commit SHAs,
consistent with the repo's security conventions.
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
- Integration test (`streamBatchItems.test.ts`) validates correctness
and performance at concurrency 1, 10, and 50 for both large and small
payloads.
- End-to-end verified over real HTTP: 20-item/12MB batch ingests and
seals, idempotent retry returns `sealed: true`, oversized item seals at
correct index.
---
## Changelog
Streaming batch ingest now processes items with bounded concurrency
instead of one at a time, so batches of many large payloads ingest far
faster and no longer time out. Concurrency is configurable via
`STREAMING_BATCH_INGEST_CONCURRENCY` (default 10); set it to 1 for fully
sequential ingestion.
---
## Screenshots
_[Screenshots]_
💯
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
- Recommend deploying NodeLocal DNS and lowering `ndots` to `1` in the
Kubernetes self-hosting guide.
- Recommend storing task events in ClickHouse
(`EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2`) in both the Docker and
Kubernetes guides, plus a new row in the webapp env var reference.
## Summary
Self-hosters that operate the webapp's ECR account separately from the
account running the EKS workers (e.g., a shared platform account that
hosts the registry plus per-team accounts that host clusters) currently
hit a 403 Forbidden the first time **any** project is deployed:
```
Failed to pull image "<acct-A>.dkr.ecr.<region>.amazonaws.com/<namespace>/proj_…:…":
unexpected status from HEAD request to .../v2/.../manifests/sha256:…: 403 Forbidden
```
`ensureEcrRepositoryExists` in
`apps/webapp/app/v3/getDeploymentImageRef.server.ts` calls
`CreateRepository` and `PutLifecyclePolicy`, but never
`SetRepositoryPolicy` — so the new repo inherits the AWS default (only
the registry-owner account can read/pull). Workers in the cluster
account get 403 every single deploy. The only workarounds today are
running a one-off post-create script or pre-creating every repo by hand.
## Proposed change
Add an optional env var:
```
DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY (V4 mirror: V4_DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY)
```
Raw IAM policy JSON. When set, the webapp calls `SetRepositoryPolicy`
immediately after `CreateRepository` so every new repo carries that
policy from creation. Operators control the principal/actions; we don't
bake in any opinions about cross-account boundaries.
Example value (for the typical self-host case — grant pull to the
cluster account):
```json
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "AllowClusterAccountPull",
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::<cluster-account-id>:root"},
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
]
}]
}
```
## Why env var (not a chart-level field)
- Mirrors the shape of the sibling vars (`DEPLOY_REGISTRY_ECR_TAGS`,
`DEPLOY_REGISTRY_ECR_ASSUME_ROLE_ARN`, etc.) which are already
operator-supplied via `webapp.extraEnvVars` in self-host setups.
- Cloud is unaffected — the env var is optional, unset by default;
existing behavior unchanged.
- Existing repos are unaffected — only newly-created repos get the
policy.
- `RepositoryCreationTemplate` from the AWS provider isn't an
alternative here: it only applies to repos created via
pull-through-cache or replication, not to `ecr:CreateRepository` API
calls.
## Implementation
- `apps/webapp/app/env.server.ts` — declare
`DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY` and the V4 fallback.
- `apps/webapp/app/v3/registryConfig.server.ts` — propagate
`ecrDefaultRepositoryPolicy` to `RegistryConfig`.
- `apps/webapp/app/v3/getDeploymentImageRef.server.ts` —
`createEcrRepository` accepts the policy; if set, calls
`SetRepositoryPolicy` after `PutLifecyclePolicy`.
- `docs/self-hosting/env/webapp.mdx` — documentation row added under
**Deploy & Registry**.
## Verification
Verified end-to-end against a self-hosted Trigger.dev on EKS where the
ECR account is separate from the cluster account:
- **Without the env var** (current `main`): the new project's first run
pod stays in `ImagePullBackOff` with `403 Forbidden`.
- **With the env var set** to a JSON granting
`ecr:BatchGetImage`/`GetDownloadUrlForLayer`/`BatchCheckLayerAvailability`
to the cluster account: a fresh `trigger.dev deploy --env prod` followed
by a `hello-world` run completes in ~5s end-to-end on the first try.
Manually also confirmed that existing repos are untouched (the call only
fires inside `createEcrRepository`, which only runs when
`DescribeRepositories` returned `RepositoryNotFoundException`).
## Out of scope
- Chart values surface for this — operators already pass the existing
ECR vars via `webapp.extraEnvVars`, so this follows the same pattern.
Happy to add a first-class chart field in a follow-up if that's the
preferred direction.
- IAM-policy validation in the webapp — we forward the JSON verbatim to
AWS and surface AWS's error messages on misuse, matching how
`DEPLOY_REGISTRY_ECR_TAGS` is handled today.
This is a draft pending CI / CodeRabbit pass — happy to iterate on
direction (e.g., split into per-action env vars, or extend the chart
values schema) if any of the above choices feels off.
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
This allows seamless migration to different object storage.
Existing runs that have offloaded payloads/outputs will continue to use
the default object store (configured using `OBJECT_STORE_*` env vars).
You can add additional stores by setting new env vars:
- `OBJECT_STORE_DEFAULT_PROTOCOL` this determines where new run large
payloads will get stored.
- If you set that you need to set new env vars for that protocol.
Example:
```
OBJECT_STORE_DEFAULT_PROTOCOL=“s3"
OBJECT_STORE_S3_BASE_URL=https://s3.us-east-1.amazonaws.com
OBJECT_STORE_S3_ACCESS_KEY_ID=<val>
OBJECT_STORE_S3_SECRET_ACCESS_KEY=<val>
OBJECT_STORE_S3_REGION=us-east-1
OBJECT_STORE_S3_SERVICE=s3
```
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Adds documentation for creating additional worker groups via the admin
API endpoint, including how to make users admin (new vs existing users),
and clarifies that ADMIN_EMAILS only applies on signup.
* Adds new features table to top of v4 upgrade guide
* Adds wait idempotency to wait-until, wait-for, and wait-for-token pages
* Adds new priority docs page and updates the v4 upgrade guide
* Adds new task lifecycle hooks
* Removes the message about requiring tasks to be exported
* Adds new global lifecycle hooks section
* Moves sections from upgrade guide into the table
* Adds hidden task page
* Improves the global lifecycle hooks section
* Updates middleware and locals section
* Adds new useWaitToken page to the react hooks section
* Adds a new ai.tool section
* Moves Docker (legacy) page into self-hosting section
* Removes known issues from v4 upgrade guide
* Replace “toolTask” with “ai.tool” in the Streams page example
* Renames guide to “Migrating from v3” and adds redirect
* Remove references to v4
* Removes changelog from migration guide
* The installation guide now references `@latest update`
* Changes all references from `/sdk/v3` to `/sdk`
* Updates @v4-beta to @latest
* Fixed broken link
* Fixes broken link
* Adds an upgrade to v4 using AI section
* Fixes 2 broken links
* Adds an entry for targetting preview branches
* Updates the run statuses
* Adds boolean helpers section to the runs and realtime pages
* Updates the concurrency page
* Updates the test page to include the new options
* Adds SDK and curl options for the preview branch targeting
* Updates new bulk actions page
* Remove the releasing concurrency section
* Got rid of some more @v4-beta mentions
* Improved rate limit docs
* Improved migrating docs
* Removed commented sections of the docs
* useWaitToken hook
* Fixed the description
* Fix for missing test image
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: Dan <8297864+D-K-P@users.noreply.github.com>
* fix both internal and external registry host
* add next steps to self-hosting overview
* separate into webapp and registry ingress
* fix external registry config check
* fix localhost registry warning
* bump chart version
* shallow clone
* fix image tag locking example
* electric should use db url env var
* add htpasswd note
* use local driver with log rotation by default
* make worker urls more easily configurable
* configure dev otel endpoint via .env
* increase min recommended worker specs
* move worker url section
Chris Arderne
