cb9aefd49b
## Summary Self-hosted deployments now run ClickHouse from the official [`clickhouse/clickhouse-server`](https://hub.docker.com/r/clickhouse/clickhouse-server) image instead of `bitnamilegacy/clickhouse`. Bitnami's free image catalog is EOL and the frozen legacy archive tops out at ClickHouse 25.7.5, below the 25.8 minimum the platform requires since v4.5.0, which broke every ClickHouse insert on chart-bundled deployments. Both stacks now default to 26.2, the same version the platform is developed and tested against. Existing deployments keep their ClickHouse data with no manual migration. Fixes #4197. ## Details **Docker Compose**: the `clickhouse` service uses the official image with its native env vars, plus the recommended `nofile` ulimits. It reuses the same named volume as before: a `data-paths.xml` config override points ClickHouse at the `data/` subdirectory of the volume, which is exactly the layout the Bitnami image used, so old volumes work in place (including SQL-created users) and fresh installs get the identical layout. The service follows the required-secrets model: `CLICKHOUSE_PASSWORD` must be set, matching the other services. **Helm chart**: the Bitnami ClickHouse subchart is replaced by a chart-owned single-node StatefulSet and Service running the official image (non-root, HTTP `/ping` probes, config overrides mounted into `config.d`, and the same `data-paths.xml` layout compatibility). On upgrade, the chart automatically adopts the data PVC left behind by the old subchart (`data-<release>-clickhouse-shard0-0`) via `lookup`, and `fsGroup` relabeling handles the uid change on first mount. Both the ClickHouse server and the webapp read the password from the same chart-managed datastore secret (auto-generated and retained across upgrades), so the server credential and the app's connection URL always match. Existing `clickhouse.*` values keep working: `auth` (including `existingSecret`/`existingSecretKey`), `persistence` (including `global.storageClass`), `resources`, `secure`, `external.*`, `configdFiles`, and now `nodeSelector`/`tolerations`/`affinity`. Bitnami-only keys (`shards`, `replicaCount`, `keeper`, `resourcesPreset`) are gone; default `resources` requests/limits match what the old preset applied. The docs state the 25.8 minimum for bring-your-own ClickHouse. ## Upgrade caveats An adversarial review of the upgrade path found a few cohorts that need awareness (all documented): - **GitOps tools that render with `helm template`** (no cluster access): PVC auto-detection can't run, so `clickhouse.persistence.existingClaim` must be set to the old PVC name or ClickHouse starts on a fresh empty volume. Documented in the values file and the Kubernetes self-hosting docs. Tools that run real helm installs (e.g. Flux) adopt automatically. - **A pinned `CLICKHOUSE_IMAGE_TAG`** pointing at a Bitnami tag must be updated to an official image tag; documented in the Docker self-hosting docs. - **Storage without `fsGroup` support** (NFS, hostPath): set `clickhouse.volumePermissions.enabled: true` for a one-time ownership-fixing init container. - **Rollback is not automatic**: once the official image has run, file ownership changes and the Bitnami image can no longer read the volume without a manual chown, and ClickHouse does not support downgrades across the version gap. ## Verification - Full upgrade simulation for Compose, twice (before and after rebasing onto the required-secrets release): booted the ClickHouse service from the old compose file on `main` (Bitnami), wrote thousands of rows, then brought the same project up with this branch's compose file. The official 26.2 server came up healthy on the same volume with all rows intact, SQL-created users working, and writes succeeding. - Adoption scenarios tested against real containers: old volume + root entrypoint (Compose), old volume owned by the Bitnami uid + non-root 101 with fsGroup-style group permissions (Kubernetes), and fresh volumes for both. - `helm lint`, `helm template` (default values, `existingClaim` set, external ClickHouse, volumePermissions/scheduling toggles, and the production example) and kubeconform all pass, mirroring the release CI steps. The rendered webapp Deployment and ClickHouse StatefulSet resolve to the same datastore secret key. - Inserts using `input_format_json_infer_array_of_dynamic_from_array_of_different_types` (the setting that fails on 25.7.5) succeed on the upgraded volume. ## Upgrade preflight and docs A production upgrade report on this branch surfaced two hazards that predate this PR — both landed in chart 4.5.6 (#4316) — so they are fixed here rather than left for the next person to hit. **`secrets.existingSecret` gained two required keys.** The webapp started reading `PROVIDER_SECRET` and `COORDINATOR_SECRET`, and when `existingSecret` is set the chart generates nothing, so a missing key only surfaced as a `CreateContainerConfigError` partway through the webapp rollout. The pre-install/pre-upgrade validation now looks the Secret up and fails with the complete list of missing keys, leaving the running release untouched. It is skipped under `helm template` and client-side dry-run, where `lookup` cannot read the cluster. **Bundled datastore credentials moved into the chart-managed Secret** (`<release>-clickhouse`/`admin-password` → `trigger-datastore`/`clickhouse-admin-password`). The chart wires both ends itself, but consumers outside it — maintenance CronJobs, Grafana datasources, secret syncs — have to be repointed. A new `## Upgrading` section in the Kubernetes docs carries the old→new mapping, the two new keys, and a pointer to the ClickHouse image notes. The existingSecret key list in the docs also named `OBJECT_STORE_ACCESS_KEY_ID`/`OBJECT_STORE_SECRET_ACCESS_KEY`, which are env var names rather than keys the chart reads; corrected to the real key names and the condition under which they apply. Verified on a throwaway kind cluster with `--dry-run=server`: a pre-4.5.6 Secret fails with both key names listed, the documented `kubectl patch` clears it, and default values, `existingClaim`, external ClickHouse, volumePermissions/scheduling and the production example all still render. A real `helm install` followed by an upgrade against an incomplete Secret aborts with the release still at revision 1 and `deployed`. `helm lint`, the CI render and kubeconform (59 resources, 0 invalid) pass. --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
174 lines
7.9 KiB
Bash
174 lines
7.9 KiB
Bash
# Trigger.dev self-hosting environment variables
|
|
# - These are the default values for the self-hosting stack
|
|
# - You should change them to suit your needs, especially the secrets
|
|
# - See the docs for more information: https://trigger.dev/docs/self-hosting/overview
|
|
|
|
# Secrets — REQUIRED, no defaults. The stack will not boot until each is set to a unique value.
|
|
# Generate each with: openssl rand -hex 16
|
|
SESSION_SECRET=
|
|
MAGIC_LINK_SECRET=
|
|
ENCRYPTION_KEY=
|
|
# These authenticate the internal control-plane connections. Generate each with: openssl rand -hex 16
|
|
# COORDINATOR_SECRET must match the coordinator's PLATFORM_SECRET; MANAGED_WORKER_SECRET the supervisor's.
|
|
PROVIDER_SECRET=
|
|
COORDINATOR_SECRET=
|
|
MANAGED_WORKER_SECRET=
|
|
|
|
# Worker token
|
|
# - This is the token for the worker to connect to the webapp
|
|
# - When running the combined stack, this is set automatically during bootstrap
|
|
# - For the split setup, you will have to set this manually. The token is available in the webapp logs but will only be shown once.
|
|
# - See the docs for more information: https://trigger.dev/docs/self-hosting/docker
|
|
# TRIGGER_WORKER_TOKEN=
|
|
|
|
# Worker URLs
|
|
# - In split setups, uncomment and set to the public URL of your webapp
|
|
# TRIGGER_API_URL=https://trigger.example.com
|
|
# OTEL_EXPORTER_OTLP_ENDPOINT=https://trigger.example.com/otel
|
|
|
|
# Postgres
|
|
# - Password is REQUIRED, no default. Run ./generate-secrets.sh to fill it (or openssl rand -hex 16).
|
|
# - DATABASE_URL / DIRECT_URL are derived from POSTGRES_PASSWORD automatically - only set them
|
|
# below to point at an external Postgres (POSTGRES_PASSWORD is then unused).
|
|
# POSTGRES_USER=postgres
|
|
POSTGRES_PASSWORD=
|
|
# POSTGRES_DB=postgres
|
|
# DATABASE_URL=postgresql://user:password@host:5432/main?schema=public&sslmode=disable
|
|
# DIRECT_URL=postgresql://user:password@host:5432/main?schema=public&sslmode=disable
|
|
|
|
# Trigger image tag
|
|
# - This is the version of the webapp and worker images to use, they should be locked to a specific version in production
|
|
# - For example: TRIGGER_IMAGE_TAG=v4.5.0
|
|
TRIGGER_IMAGE_TAG=latest
|
|
|
|
# Webapp
|
|
# - These should generally be set to the same value
|
|
# - In production, these should be set to the public URL of your webapp, e.g. https://trigger.example.com
|
|
APP_ORIGIN=http://localhost:8030
|
|
LOGIN_ORIGIN=http://localhost:8030
|
|
API_ORIGIN=http://localhost:8030
|
|
DEV_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8030/otel
|
|
# You may need to set this when testing locally or when using the combined setup
|
|
# API_ORIGIN=http://webapp:3000
|
|
|
|
# Webapp - memory management
|
|
# - This sets the maximum memory allocation for Node.js heap in MiB (e.g. "4096" for 4GB)
|
|
# - It should be set according to your total webapp machine's memory or any container limits you have set
|
|
# - Setting this too high or low WILL cause crashes, inefficient memory utilization and high CPU usage
|
|
# - You should allow for some memory overhead, we suggest at least 20%, for example:
|
|
# - 2GB machine: NODE_MAX_OLD_SPACE_SIZE=1600
|
|
# - 4GB machine: NODE_MAX_OLD_SPACE_SIZE=3200
|
|
# - 6GB machine: NODE_MAX_OLD_SPACE_SIZE=4800
|
|
# - 8GB machine: NODE_MAX_OLD_SPACE_SIZE=6400
|
|
# NODE_MAX_OLD_SPACE_SIZE=8192
|
|
|
|
# ClickHouse
|
|
# - Password is REQUIRED, no default. Run ./generate-secrets.sh to fill it.
|
|
# - CLICKHOUSE_URL / RUN_REPLICATION_CLICKHOUSE_URL are derived from CLICKHOUSE_PASSWORD
|
|
# automatically - only set them below to point at an external ClickHouse.
|
|
CLICKHOUSE_USER=default
|
|
CLICKHOUSE_PASSWORD=
|
|
# CLICKHOUSE_URL=http://user:password@host:8123?secure=false
|
|
# RUN_REPLICATION_CLICKHOUSE_URL=http://user:password@host:8123
|
|
|
|
# Docker Registry
|
|
# - Password is REQUIRED, no default. Run ./generate-secrets.sh to fill it - it also writes
|
|
# the matching registry/auth.htpasswd (bcrypt) the bundled registry authenticates against.
|
|
# - See the docs for more information: https://trigger.dev/docs/self-hosting/docker#registry-setup
|
|
DOCKER_REGISTRY_URL=localhost:5000
|
|
DOCKER_REGISTRY_USERNAME=registry-user
|
|
DOCKER_REGISTRY_PASSWORD=
|
|
# When using an external registry you will have to change this
|
|
# On Docker Hub it should generally be the same as your username
|
|
DOCKER_REGISTRY_NAMESPACE=trigger
|
|
|
|
# Object store
|
|
# - You need to log into the Minio dashboard and create a bucket called "packets"
|
|
# - See the docs for more information: https://trigger.dev/docs/self-hosting/docker#object-storage
|
|
# Default provider (backward compatible - no protocol prefix)
|
|
# - Secret access key is REQUIRED, no default. Run ./generate-secrets.sh to fill it.
|
|
# - For the bundled MinIO, these ARE its root credentials (MINIO_ROOT_USER/PASSWORD derive from them).
|
|
OBJECT_STORE_ACCESS_KEY_ID=admin
|
|
OBJECT_STORE_SECRET_ACCESS_KEY=
|
|
# You will have to uncomment and configure this for production
|
|
# OBJECT_STORE_BASE_URL=http://localhost:9000
|
|
# OBJECT_STORE_REGION=auto
|
|
# OBJECT_STORE_SERVICE=s3
|
|
# OBJECT_STORE_DEFAULT_PROTOCOL=s3 # Optional: protocol to use for new uploads (e.g., "s3", "r2")
|
|
#
|
|
# Named providers (protocol-prefixed data) - optional for multi-provider support
|
|
# OBJECT_STORE_S3_BASE_URL=https://s3.amazonaws.com
|
|
# OBJECT_STORE_S3_ACCESS_KEY_ID=
|
|
# OBJECT_STORE_S3_SECRET_ACCESS_KEY=
|
|
# OBJECT_STORE_S3_REGION=us-east-1
|
|
# OBJECT_STORE_S3_SERVICE=s3
|
|
#
|
|
# OBJECT_STORE_R2_BASE_URL=https://{bucket}.{accountId}.r2.cloudflarestorage.com
|
|
# OBJECT_STORE_R2_ACCESS_KEY_ID=
|
|
# OBJECT_STORE_R2_SECRET_ACCESS_KEY=
|
|
# OBJECT_STORE_R2_REGION=auto
|
|
# OBJECT_STORE_R2_SERVICE=s3
|
|
# Minio dashboard at http://localhost:9001
|
|
# - The bundled Minio's root credentials default to OBJECT_STORE_ACCESS_KEY_ID / OBJECT_STORE_SECRET_ACCESS_KEY.
|
|
# - For production, set a separate root user here and create a non-root user scoped to the "packets" bucket for OBJECT_STORE_*.
|
|
# MINIO_ROOT_USER=admin
|
|
# MINIO_ROOT_PASSWORD=
|
|
|
|
# Realtime streams
|
|
# - Realtime streams power AI-agent token streaming and run streams
|
|
# - They default to v2, backed by the bundled s2-lite service (open-source S2, https://s2.dev)
|
|
# - To fall back to the Redis-backed v1 streams, set REALTIME_STREAMS_DEFAULT_VERSION=v1
|
|
# REALTIME_STREAMS_DEFAULT_VERSION=v2
|
|
# The bundled s2-lite creates whatever basin you set here (min 8 characters)
|
|
# REALTIME_STREAMS_S2_BASIN=trigger-realtime
|
|
# REALTIME_STREAMS_S2_ENDPOINT=http://s2/v1
|
|
# REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS=true
|
|
# To use a hosted S2 (https://s2.dev) instead of the bundled s2-lite:
|
|
# - point the endpoint at your basin, disable token-skipping, and set an access token
|
|
# REALTIME_STREAMS_S2_SKIP_ACCESS_TOKENS=false
|
|
# REALTIME_STREAMS_S2_ACCESS_TOKEN=
|
|
# Pin the s2-lite image in production (full image reference, digest recommended)
|
|
# S2_IMAGE=ghcr.io/s2-streamstore/s2:latest@sha256:d6ded5ca7dd619fa7c946f06e39a98f9c95c6883c8bb884e5eaa129f232c920c
|
|
|
|
# Other image tags
|
|
# - These are the versions of the other images to use
|
|
# - You should lock these to a specific version in production
|
|
# POSTGRES_IMAGE_TAG=14
|
|
# REDIS_IMAGE_TAG=7
|
|
# ELECTRIC_IMAGE_TAG=1.0.13
|
|
# CLICKHOUSE_IMAGE_TAG=26.2
|
|
# REGISTRY_IMAGE_TAG=2
|
|
# MINIO_IMAGE_TAG=latest
|
|
# DOCKER_PROXY_IMAGE_TAG=latest
|
|
# TRAEFIK_IMAGE_TAG=v3.4
|
|
|
|
# Publish IPs
|
|
# - These are the IPs to publish the services to
|
|
# - Setting to 127.0.0.1 makes the service only accessible locally
|
|
# - When deploying to production, you will have to change these, depending on your setup
|
|
# WEBAPP_PUBLISH_IP=0.0.0.0
|
|
# POSTGRES_PUBLISH_IP=127.0.0.1
|
|
# REDIS_PUBLISH_IP=127.0.0.1
|
|
# ELECTRIC_PUBLISH_IP=127.0.0.1
|
|
# CLICKHOUSE_PUBLISH_IP=127.0.0.1
|
|
# REGISTRY_PUBLISH_IP=127.0.0.1
|
|
# MINIO_PUBLISH_IP=127.0.0.1
|
|
|
|
# Restart policy
|
|
# - Applies to all services, adjust as needed
|
|
# RESTART_POLICY=unless-stopped
|
|
|
|
# Docker logging
|
|
# - See the official docs: https://docs.docker.com/engine/logging/configure/
|
|
# LOGGING_DRIVER=local
|
|
# LOGGING_MAX_SIZE=20m
|
|
# LOGGING_MAX_FILES=5
|
|
# LOGGING_COMPRESS=true
|
|
|
|
# Traefik
|
|
# - Reverse proxy settings only serve as an example and require further configuration
|
|
# - See the partial overrides in docker-compose.traefik.yml for more details
|
|
# TRAEFIK_ENTRYPOINT=websecure
|
|
# TRAEFIK_HTTP_PUBLISH_IP=0.0.0.0
|
|
# TRAEFIK_HTTPS_PUBLISH_IP=0.0.0.0
|
|
# TRAEFIK_DASHBOARD_PUBLISH_IP=127.0.0.1 |