changeset-release/main
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 nicktrn
|
d6457521cb |
fix(hosting): disable clickhouse system-log telemetry and apply profile settings via users.d (#4762)
Carries over the self-hosted ClickHouse fix from #4546 by @Leafgard, whose commits are preserved here, plus follow-up polish. Opened in-repo because the fork is org-owned, which GitHub's "Allow edits from maintainers" doesn't cover. fixes #4343 ## What was wrong Two independent problems in `hosting/docker/clickhouse/`: 1. **The `<profiles>` block never applied.** It sits in `override.xml`, mounted under `config.d` - but ClickHouse only reads profile settings from the users config tree. Verified on the pinned image: before this change `max_block_size` sat at its default `65409` with `changed=0`, so the advertised low-memory settings had never taken effect at all. 2. **Every ClickHouse system log table was enabled and unbounded.** On a sub-16GB machine their background merges outgrow the memory cap; ClickHouse's [low-RAM guide](https://clickhouse.com/docs/operations/tips) recommends disabling them. The dev stack already does this - `hosting/docker` never got it. ## What this does - `clickhouse/override.xml`: disables the high-frequency telemetry tables, and bounds the ones worth keeping with a config-level `<ttl>` - `query_log` and `part_log` at 7 days, `error_log` at 30. A config-level TTL survives log-table recreation, unlike `ALTER ... MODIFY TTL`. - New `clickhouse/users-override.xml`, mounted at `users.d/override.xml`: carries the profile settings so they actually apply, completes the sub-16GB set with `max_threads=1`, and zeroes the memory/query profilers, whose samples were the main source feeding `trace_log`. - `webapp/docker-compose.yml`: adds the `users.d` mount. ## Verification Ran `clickhouse/clickhouse-server:26.2` with these exact mounts, and `25.12` to cover the documented 25.8 floor: - All 9 profile settings report `changed=1`, and a custom `CLICKHOUSE_USER` inherits them. - `users.d` merges rather than replaces: the `default` user, its password, `access_management` and the `readonly` profile all survive, so the compose healthcheck still passes. - `remove="1"` is a clean no-op on keys absent from a given version - no empty section, no accidental table, no startup error - so pinning `CLICKHOUSE_IMAGE_TAG` to an older supported tag won't crash-loop. - TTLs land in the real DDL: `TTL event_date + toIntervalDay(7)` / `(30)`. - In-place upgrade on a populated volume: clean restart, data preserved, and ClickHouse lazily renames the pre-existing `query_log`/`error_log` to `query_log_0`/`error_log_0` as it applies the new retention. ## Notes for review - **`part_log` is kept (bounded) rather than disabled.** It appears in neither report behind this change and isn't on ClickHouse's sub-16GB list, but it's the merge history you'd need to diagnose a recurrence. Measured at ~0.18 KiB per part event under insert churn - about 10x cheaper than `text_log` over the same window - so a TTL bounds it rather than removing it. - **The profile settings go live for the first time here.** On larger machines that's a real, intended throughput change: `max_threads=1`, `max_download_threads=1`, parallel parsing and formatting off. - **Disabling a log table stops new writes but doesn't delete existing data.** Reclaiming disk on an existing deployment needs `DROP TABLE system.<name> SYNC`, including the `*_log_0` leftovers. ## Known gaps, deliberately not in this PR - The Helm chart carries the same ineffective `<profiles>` block in `values.yaml` and mounts nothing into `users.d`, so this fix isn't currently expressible there. - `background_schedule_pool_log` is enabled by default with no TTL and is disabled by neither stack. - The dev stack's disable list has drifted from this one. - The compose healthcheck still logs a query every 5 seconds. --------- Co-authored-by: Yann SEGET <yann.seget@actemium.ch> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Matt Aitken
|
cb9aefd49b |
fix(hosting): deploy ClickHouse from the official image instead of Bitnami (#4249)
## Summary Self-hosted deployments now run ClickHouse from the official [`clickhouse/clickhouse-server`](https://hub.docker.com/r/clickhouse/clickhouse-server) image instead of `bitnamilegacy/clickhouse`. Bitnami's free image catalog is EOL and the frozen legacy archive tops out at ClickHouse 25.7.5, below the 25.8 minimum the platform requires since v4.5.0, which broke every ClickHouse insert on chart-bundled deployments. Both stacks now default to 26.2, the same version the platform is developed and tested against. Existing deployments keep their ClickHouse data with no manual migration. Fixes #4197. ## Details **Docker Compose**: the `clickhouse` service uses the official image with its native env vars, plus the recommended `nofile` ulimits. It reuses the same named volume as before: a `data-paths.xml` config override points ClickHouse at the `data/` subdirectory of the volume, which is exactly the layout the Bitnami image used, so old volumes work in place (including SQL-created users) and fresh installs get the identical layout. The service follows the required-secrets model: `CLICKHOUSE_PASSWORD` must be set, matching the other services. **Helm chart**: the Bitnami ClickHouse subchart is replaced by a chart-owned single-node StatefulSet and Service running the official image (non-root, HTTP `/ping` probes, config overrides mounted into `config.d`, and the same `data-paths.xml` layout compatibility). On upgrade, the chart automatically adopts the data PVC left behind by the old subchart (`data-<release>-clickhouse-shard0-0`) via `lookup`, and `fsGroup` relabeling handles the uid change on first mount. Both the ClickHouse server and the webapp read the password from the same chart-managed datastore secret (auto-generated and retained across upgrades), so the server credential and the app's connection URL always match. Existing `clickhouse.*` values keep working: `auth` (including `existingSecret`/`existingSecretKey`), `persistence` (including `global.storageClass`), `resources`, `secure`, `external.*`, `configdFiles`, and now `nodeSelector`/`tolerations`/`affinity`. Bitnami-only keys (`shards`, `replicaCount`, `keeper`, `resourcesPreset`) are gone; default `resources` requests/limits match what the old preset applied. The docs state the 25.8 minimum for bring-your-own ClickHouse. ## Upgrade caveats An adversarial review of the upgrade path found a few cohorts that need awareness (all documented): - **GitOps tools that render with `helm template`** (no cluster access): PVC auto-detection can't run, so `clickhouse.persistence.existingClaim` must be set to the old PVC name or ClickHouse starts on a fresh empty volume. Documented in the values file and the Kubernetes self-hosting docs. Tools that run real helm installs (e.g. Flux) adopt automatically. - **A pinned `CLICKHOUSE_IMAGE_TAG`** pointing at a Bitnami tag must be updated to an official image tag; documented in the Docker self-hosting docs. - **Storage without `fsGroup` support** (NFS, hostPath): set `clickhouse.volumePermissions.enabled: true` for a one-time ownership-fixing init container. - **Rollback is not automatic**: once the official image has run, file ownership changes and the Bitnami image can no longer read the volume without a manual chown, and ClickHouse does not support downgrades across the version gap. ## Verification - Full upgrade simulation for Compose, twice (before and after rebasing onto the required-secrets release): booted the ClickHouse service from the old compose file on `main` (Bitnami), wrote thousands of rows, then brought the same project up with this branch's compose file. The official 26.2 server came up healthy on the same volume with all rows intact, SQL-created users working, and writes succeeding. - Adoption scenarios tested against real containers: old volume + root entrypoint (Compose), old volume owned by the Bitnami uid + non-root 101 with fsGroup-style group permissions (Kubernetes), and fresh volumes for both. - `helm lint`, `helm template` (default values, `existingClaim` set, external ClickHouse, volumePermissions/scheduling toggles, and the production example) and kubeconform all pass, mirroring the release CI steps. The rendered webapp Deployment and ClickHouse StatefulSet resolve to the same datastore secret key. - Inserts using `input_format_json_infer_array_of_dynamic_from_array_of_different_types` (the setting that fails on 25.7.5) succeed on the upgraded volume. ## Upgrade preflight and docs A production upgrade report on this branch surfaced two hazards that predate this PR — both landed in chart 4.5.6 (#4316) — so they are fixed here rather than left for the next person to hit. **`secrets.existingSecret` gained two required keys.** The webapp started reading `PROVIDER_SECRET` and `COORDINATOR_SECRET`, and when `existingSecret` is set the chart generates nothing, so a missing key only surfaced as a `CreateContainerConfigError` partway through the webapp rollout. The pre-install/pre-upgrade validation now looks the Secret up and fails with the complete list of missing keys, leaving the running release untouched. It is skipped under `helm template` and client-side dry-run, where `lookup` cannot read the cluster. **Bundled datastore credentials moved into the chart-managed Secret** (`<release>-clickhouse`/`admin-password` → `trigger-datastore`/`clickhouse-admin-password`). The chart wires both ends itself, but consumers outside it — maintenance CronJobs, Grafana datasources, secret syncs — have to be repointed. A new `## Upgrading` section in the Kubernetes docs carries the old→new mapping, the two new keys, and a pointer to the ClickHouse image notes. The existingSecret key list in the docs also named `OBJECT_STORE_ACCESS_KEY_ID`/`OBJECT_STORE_SECRET_ACCESS_KEY`, which are env var names rather than keys the chart reads; corrected to the real key names and the condition under which they apply. Verified on a throwaway kind cluster with `--dry-run=server`: a pre-4.5.6 Secret fails with both key names listed, the documented `kubectl patch` clears it, and default values, `existingClaim`, external ClickHouse, volumePermissions/scheduling and the production example all still render. A real `helm install` followed by an upgrade against an incomplete Secret aborts with the release still at revision 1 and `deployed`. `helm lint`, the CI render and kubeconform (59 resources, 0 invalid) pass. --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com> |
||
|
nicktrn
|
6535cf918b |
v4 helm chart and docs (#2195)
* v4 helm chart * ignore packaged chart archives * chart publish prep * add helm chart release workflow * enable draft gh release * stateful services should be statefulsets * expose internal otel config * podAnnotations for all services * postgresql.extraArgs * postgresql -> postgres * add clickhouse vars as extraEnv examples * support external secrets * improve clickhouse config improve clickhouse config * disable internal otel config by default * allow nil internal config * fix gh release permission * fix validation step for ci * secrets.enabled * simplify secret keys * bump chart version * fix contents permission * remove redundant print steps * remove helm release heading * optimise clickhouse config * support wait-for-it.sh * fix ch overrides * add low-resource ch overrides to compose setup * supervisor bootstrap config * disable otel trace logging * retain shared volume * enable clickhouse for compose setup * disable trace logging * bump chart * add extraManifests * telemetry config * add repo namespace config * scope cluster role to namespace to support multiple installs * external s3 config * values tweaks * helm docs * bump chart * set contents read permission Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * add object store root credentials warning * run init containers as uid 1000 * pass ch user creds with explicit flag * improve ingress anotation handling * remove duplicate colume claim block * trim htpasswd auth * lock curl images used for tests * add v4-beta package version warning * make schema and sslmode configurable * package tag info -> warning * improve secrets config example * update docs * bump version * bump webapp reqs to account for ch * make ch creds more easily configurable * publish ch http port * example point at latest beta chart * small docs tweaks --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> |
||
|
nicktrn
|
7a34c1102b |
Feat: v4 self-hosting (#2155)
* self-hosting stuff goes in /hosting * add v4 tags * add main compose file * draft overview * add webapp env vars * overview tweaks * add supervisor env vars * move old docker guide * new sidebar structure * update github actions docs * docker draft * use env vars for s3 creds * this might just work * split into multiple files * update guide * document machine overrides * split legacy docs into different section * some fixes * some tweaks * add login and init instructions * don't cursorignore .env.example |