## Summary
1 improvement, 1 bug fix.
## Improvements
- Fail attempts on uncaught exceptions instead of hanging to
`MAX_DURATION_EXCEEDED`. A Node `EventEmitter` (e.g. `node-redis`)
emitting `"error"` with no `.on("error", ...)` listener escalates to
`uncaughtException`, which the worker previously reported but did not
act on — runs drifted to maxDuration with empty attempts. They now fail
fast with the original error and status `FAILED`, and respect the task's
normal retry policy. You should still attach `.on("error", ...)`
listeners to long-lived clients to handle errors gracefully.
([#3529](https://github.com/triggerdotdev/trigger.dev/pull/3529))
## Bug fixes
- Fix dev workers spinning at 100% CPU after the parent CLI disconnects.
Orphaned `trigger-dev-run-worker` (and indexer) processes were caught in
an `uncaughtException` feedback loop: a periodic IPC send via
`process.send` would throw `ERR_IPC_CHANNEL_CLOSED` once the parent
closed the channel, which re-entered the same handler that itself called
`process.send`, scheduled via `setImmediate` and amplified by
source-map-support's `prepareStackTrace`. Fixed by (1) silently dropping
packets in `ZodIpcConnection` when the channel is disconnected, (2)
adding a `process.on("disconnect", ...)` handler in dev workers so they
exit cleanly when the CLI closes the IPC channel, and (3) wrapping all
`uncaughtException`-path `process.send` calls in a `safeSend` guard that
checks `process.connected` and swallows synchronous throws.
([#3491](https://github.com/triggerdotdev/trigger.dev/pull/3491))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## trigger.dev@4.4.6
### Patch Changes
- Fix dev workers spinning at 100% CPU after the parent CLI disconnects.
Orphaned `trigger-dev-run-worker` (and indexer) processes were caught in
an `uncaughtException` feedback loop: a periodic IPC send via
`process.send` would throw `ERR_IPC_CHANNEL_CLOSED` once the parent
closed the channel, which re-entered the same handler that itself called
`process.send`, scheduled via `setImmediate` and amplified by
source-map-support's `prepareStackTrace`. Fixed by (1) silently dropping
packets in `ZodIpcConnection` when the channel is disconnected, (2)
adding a `process.on("disconnect", ...)` handler in dev workers so they
exit cleanly when the CLI closes the IPC channel, and (3) wrapping all
`uncaughtException`-path `process.send` calls in a `safeSend` guard that
checks `process.connected` and swallows synchronous throws.
([#3491](https://github.com/triggerdotdev/trigger.dev/pull/3491))
- Fail attempts on uncaught exceptions instead of hanging to
`MAX_DURATION_EXCEEDED`. A Node `EventEmitter` (e.g. `node-redis`)
emitting `"error"` with no `.on("error", ...)` listener escalates to
`uncaughtException`, which the worker previously reported but did not
act on — runs drifted to maxDuration with empty attempts. They now fail
fast with the original error and status `FAILED`, and respect the task's
normal retry policy. You should still attach `.on("error", ...)`
listeners to long-lived clients to handle errors gracefully.
([#3529](https://github.com/triggerdotdev/trigger.dev/pull/3529))
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
- `@trigger.dev/build@4.4.6`
- `@trigger.dev/schema-to-json@4.4.6`
## @trigger.dev/core@4.4.6
### Patch Changes
- Fix dev workers spinning at 100% CPU after the parent CLI disconnects.
Orphaned `trigger-dev-run-worker` (and indexer) processes were caught in
an `uncaughtException` feedback loop: a periodic IPC send via
`process.send` would throw `ERR_IPC_CHANNEL_CLOSED` once the parent
closed the channel, which re-entered the same handler that itself called
`process.send`, scheduled via `setImmediate` and amplified by
source-map-support's `prepareStackTrace`. Fixed by (1) silently dropping
packets in `ZodIpcConnection` when the channel is disconnected, (2)
adding a `process.on("disconnect", ...)` handler in dev workers so they
exit cleanly when the CLI closes the IPC channel, and (3) wrapping all
`uncaughtException`-path `process.send` calls in a `safeSend` guard that
checks `process.connected` and swallows synchronous throws.
([#3491](https://github.com/triggerdotdev/trigger.dev/pull/3491))
- Fail attempts on uncaught exceptions instead of hanging to
`MAX_DURATION_EXCEEDED`. A Node `EventEmitter` (e.g. `node-redis`)
emitting `"error"` with no `.on("error", ...)` listener escalates to
`uncaughtException`, which the worker previously reported but did not
act on — runs drifted to maxDuration with empty attempts. They now fail
fast with the original error and status `FAILED`, and respect the task's
normal retry policy. You should still attach `.on("error", ...)`
listeners to long-lived clients to handle errors gracefully.
([#3529](https://github.com/triggerdotdev/trigger.dev/pull/3529))
## @trigger.dev/python@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
- `@trigger.dev/build@4.4.6`
- `@trigger.dev/sdk@4.4.6`
## @trigger.dev/react-hooks@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/redis-worker@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/rsc@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/schema-to-json@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/sdk@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Fixes#3520. The bundled bitnami clickhouse subchart was pinned at
`9.3.7` (clickhouse `25.6.1-debian-12-r0`), which hits a memory-tracker
accounting bug under sustained ingest - the global counter overflows to
~7 EiB and every query gets rejected by OvercommitTracker until the pod
is restarted. Self-hosters running 4.0.5 through 4.4.5 are exposed
regardless of chart version since the subchart pin hadn't moved.
Bumping to `9.4.4` (clickhouse `25.7.5-debian-12-r0`) pulls in the
25.7.x memory-tracker fixes. This is also the latest publicly packaged
release at `oci://registry-1.docker.io/bitnamicharts` - that registry
has been frozen since 2025-08-28 (Bitnami catalog changes), but the
chart source remains under Apache 2 on `bitnami/charts`. The image
continues to resolve via `bitnamilegacy/clickhouse` per the existing
`values.yaml` override, since `bitnami/clickhouse` itself moved to
paid-only.
Verified locally: `helm dependency update` + `helm lint` + `helm
template` + kubeconform across all 57 rendered manifests. Rendered
statefulset image is
`docker.io/bitnamilegacy/clickhouse:25.7.5-debian-12-r0`.
Reported by external contributor. The supervisor template hardcoded a
short DNS name for `OTEL_EXPORTER_OTLP_ENDPOINT`, which the supervisor
then propagates verbatim into runner pods
(`apps/supervisor/src/workloadManager/kubernetes.ts:196`). When runners
are spawned in a different namespace via
`supervisor.config.kubernetes.namespace`, the short name doesn't resolve
and span/log export silently fails - runs complete fine but the
dashboard shows nothing.
Same FQDN pattern the chart already uses for
`TRIGGER_WORKLOAD_API_DOMAIN` (line 203). Verified with `helm template
trigger . --namespace my-ns` - renders
`http://trigger-webapp.my-ns.svc.cluster.local:3030/otel`.
Cheers Niels
Follow-up to v4.4.5 release. The `bump-chart-version` job on the release
PR was cancelled before it could run, so Chart.yaml was merged still
pointing at 4.4.4. The helm release job ([failed
run](https://github.com/triggerdotdev/trigger.dev/actions/runs/25218553990/job/73947054128))
caught it via its version-match guard.
Once this merges I'll re-run the helm release workflow manually.
Today the helm prerelease workflow only fires on PRs that touch
`hosting/k8s/helm/**`. Two consequences we ran into:
1. The `changeset-release/main` PR's prerelease comment goes stale once
the release branch gets force-pushed without a helm-touching commit (the
bot's `Chart.yaml` bump alone doesn't seem to refire the trigger
reliably).
2. The release PR's chart references an `appVersion` (e.g. `v4.4.5`)
whose Docker images don't exist until *after* merge + tag. So that
prerelease chart can't actually be installed end-to-end.
Renames the workflow to `helm-prerelease.yml` and adds two new triggers:
- **`push: main`** with `paths: hosting/k8s/helm/**` -> rolling
prereleases versioned `<base>-main.<sha>`. `appVersion` stays at
whatever `Chart.yaml` has (i.e. last released), so installs pull real
images. Tests that chart structure is deployable, even if the app code
is one release behind.
- **`workflow_dispatch`** with optional `app_version` input -> manually
trigger a prerelease and optionally override `appVersion` (e.g. pin to
`main` or a specific tag). Useful for testing chart + app-version
combinations on demand.
PR behavior unchanged: same `<base>-pr<N>.<sha>` versioning, same
posted/updated comment.
Why not also bypass paths for `changeset-release/main`? The release PR's
chart references not-yet-built `v4.4.5` images, so those prereleases
aren't actually installable. The rolling main prerelease covers the
testable case better.
Why not SHA-pin `appVersion` to a built image like `main-<sha>`? Bigger
change - the docker publish workflows currently only push `:main` (no
SHA-suffixed tag). Worth doing later if we want first-class "install one
chart, get exactly that commit's app code" testing, but out of scope
here.
Diff is mostly a rename. Substantive changes:
- new `push` and `workflow_dispatch` triggers
- `prerelease` job `if:` extended for the new event types
- version logic branches per event
- new "Override appVersion" step (workflow_dispatch only)
- new "Write run summary" step so non-PR runs surface the install
instructions
- PR comment steps gated on `github.event_name == 'pull_request'`
- concurrency group falls back to `github.ref` for non-PR runs
## Summary
When the official Helm chart is deployed with an external ClickHouse and
`clickhouse.external.existingSecret` set — the documented path for not
committing secrets to `values.yaml` — the webapp pod crash-loops on
startup:
```
goose run: parse "http://default:${CLICKHOUSE_PASSWORD}@<host>:8123?secure=false": net/url: invalid userinfo
```
Context in vouch request #3443. Re-opening in draft status per bot
policy (previous attempt was #3445, closed by automation because it
wasn't draft; no changes to the patch).
## Root cause
Two pieces interact:
1. `hosting/k8s/helm/templates/_helpers.tpl` renders `CLICKHOUSE_URL`
(and `RUN_REPLICATION_CLICKHOUSE_URL`) with a shell-style literal
`${CLICKHOUSE_PASSWORD}` expecting bash expansion at container start.
2. `docker/scripts/entrypoint.sh` does `export
GOOSE_DBSTRING="$CLICKHOUSE_URL"` — single-pass POSIX sh substitution,
so the inner `${...}` survives as literal text and goose rejects it.
Reproduces against the latest published chart
(`oci://ghcr.io/triggerdotdev/charts/trigger:4.0.5`) and `main`.
## Fix
Switch the two helpers (external + `existingSecret` branch) from
shell-style `${CLICKHOUSE_PASSWORD}` to Kubernetes'
`$(CLICKHOUSE_PASSWORD)`. Kubelet substitutes `$(VAR)` at pod-creation
time from earlier env entries, and the chart already declares
`CLICKHOUSE_PASSWORD` from the Secret immediately before
`CLICKHOUSE_URL`, so the URL reaches the entrypoint with the real
password already inlined. No entrypoint change, no image change. The
plain-password branch (no `existingSecret`) is unchanged.
Operator caveat added as template comments: `CLICKHOUSE_PASSWORD` must
be URL-userinfo-safe since kubelet substitutes verbatim without
percent-encoding. Hex-encoded passwords (e.g. `openssl rand -hex 32`)
are safe by construction.
## Verification
- `helm template` against `external.existingSecret` now renders `value:
"http://default:$(CLICKHOUSE_PASSWORD)@<host>:8123?secure=false"` (was
`${CLICKHOUSE_PASSWORD}`).
- `helm template` against the plain-password branch is byte-identical to
before.
- Deployed end-to-end on a staging EKS cluster (Meistrari platform):
webapp container reaches `goose: successfully migrated database to
version: 6`, Node.js ClickHouse client connects at runtime.
## Alternatives considered
- **Change `entrypoint.sh`** to `eval` / `envsubst` the URL — larger
surface, touches every deployment mode (Docker Compose + k8s) and every
container image.
- **Mirror the Postgres pattern** (chart reads the full URL via
`valueFrom.secretKeyRef`, as in `trigger-v4.postgres.useSecretUrl`) —
cleaner long-term but requires a new `values.yaml` field and a migration
path for existing users. Happy to follow up with that as a separate PR
if the minimal fix here isn't the preferred direction.
## Changeset
None added — the Helm chart isn't versioned through `@changesets/cli`
(docs/chart-only PRs historically merge without a changeset, e.g.
#2671). Happy to add one if the policy changed.
Closes#3443.
Mirrors the existing `supervisor.serviceAccount` pattern onto webapp so
operators can annotate the SA (IRSA `eks.amazonaws.com/role-arn`,
Workload Identity, etc.) or bring their own SA. Without this,
`webapp.serviceAccount.annotations` isn't exposed and operators have to
patch the SA out-of-band.
```yaml
webapp:
serviceAccount:
create: true
name: ""
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/trigger-webapp
```
Three pieces, same as supervisor:
- `webapp.serviceAccount.create` toggle on the SA block
- `webapp.serviceAccount.annotations` + `name` values
- `trigger-v4.webappServiceAccountName` helper, used by the SA, the
token-syncer RoleBinding subject, and the Deployment's
`serviceAccountName`
Role + RoleBinding are left unguarded (matching supervisor's shape where
`rbac.create` is a separate toggle from `serviceAccount.create`) -
BYO-SA users take on the responsibility of ensuring the SA they supply
has the permissions the RoleBinding grants.
Verified with `helm template` against default values, an IRSA annotation
override, and `create: false` with a custom name.
This allows seamless migration to different object storage.
Existing runs that have offloaded payloads/outputs will continue to use
the default object store (configured using `OBJECT_STORE_*` env vars).
You can add additional stores by setting new env vars:
- `OBJECT_STORE_DEFAULT_PROTOCOL` this determines where new run large
payloads will get stored.
- If you set that you need to set new env vars for that protocol.
Example:
```
OBJECT_STORE_DEFAULT_PROTOCOL=“s3"
OBJECT_STORE_S3_BASE_URL=https://s3.us-east-1.amazonaws.com
OBJECT_STORE_S3_ACCESS_KEY_ID=<val>
OBJECT_STORE_S3_SECRET_ACCESS_KEY=<val>
OBJECT_STORE_S3_REGION=us-east-1
OBJECT_STORE_S3_SERVICE=s3
```
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
When users set `secrets.enabled=false` to use an external secret via
`secrets.existingSecret`, the environment variables `SESSION_SECRET`,
`MAGIC_LINK_SECRET`, `ENCRYPTION_KEY`, and `MANAGED_WORKER_SECRET` were
not being populated from the secret.
Fixes#2859
Also adds automatic helm prereleases for PRs
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <nicktrn@users.noreply.github.com>
- Move S3 credentials from secrets.objectStore to s3.auth
- Update external PostgreSQL config to use databaseUrl/directUrl approach
- Add existingSecret support for PostgreSQL with secretKeys
- Add TLS configuration for external Redis
- Add existingSecret support for Redis, ClickHouse and S3
- Add complete external S3 configuration example
- Improve secure credential management documentation
These changes align the production example file with the current values.yaml
structure, making it easier to configure external services with better
support for secret management.
* fix(charts): use bitnamilegacy repo for MinIO Object Browser
The Bitnami `minio-object-browser` image has been removed from Docker
Hub. This patch updates the Trigger Helm chart to reference the
`bitnamilegacy/minio-object-browser` repository under the
`s3.console.image` section, ensuring that the legacy MinIO browser pod
can still be deployed without image pull errors.
All other MinIO components continue using the `bitnamilegacy/minio`
image for consistency across Bitnami Legacy dependencies.
* chore(helm): bump Helm chart version to 4.0.4
* chore(docker): use bitnami legacy repo
* chore(helm): use bitnami legacy repo
* Make Helm webapp chart images configurable
Adds configurability for init and token syncer container images through
new values in the Helm chart configuration
* chore(helm): refactor utility image config
* chore(helm): bump chart version to 4.0.3
---------
Co-authored-by: LeoKaynan <leokaynan@hotmail.com>
* fix both internal and external registry host
* add next steps to self-hosting overview
* separate into webapp and registry ingress
* fix external registry config check
* fix localhost registry warning
* bump chart version
* shallow clone
* fix image tag locking example
* electric should use db url env var
* add htpasswd note
* use local driver with log rotation by default
* make worker urls more easily configurable
* configure dev otel endpoint via .env
* increase min recommended worker specs
* move worker url section
github-actions[bot]