Carries over the self-hosted ClickHouse fix from #4546 by @Leafgard,
whose commits are preserved here, plus follow-up polish. Opened in-repo
because the fork is org-owned, which GitHub's "Allow edits from
maintainers" doesn't cover.
fixes#4343
## What was wrong
Two independent problems in `hosting/docker/clickhouse/`:
1. **The `<profiles>` block never applied.** It sits in `override.xml`,
mounted under `config.d` - but ClickHouse only reads profile settings
from the users config tree. Verified on the pinned image: before this
change `max_block_size` sat at its default `65409` with `changed=0`, so
the advertised low-memory settings had never taken effect at all.
2. **Every ClickHouse system log table was enabled and unbounded.** On a
sub-16GB machine their background merges outgrow the memory cap;
ClickHouse's [low-RAM
guide](https://clickhouse.com/docs/operations/tips) recommends disabling
them. The dev stack already does this - `hosting/docker` never got it.
## What this does
- `clickhouse/override.xml`: disables the high-frequency telemetry
tables, and bounds the ones worth keeping with a config-level `<ttl>` -
`query_log` and `part_log` at 7 days, `error_log` at 30. A config-level
TTL survives log-table recreation, unlike `ALTER ... MODIFY TTL`.
- New `clickhouse/users-override.xml`, mounted at
`users.d/override.xml`: carries the profile settings so they actually
apply, completes the sub-16GB set with `max_threads=1`, and zeroes the
memory/query profilers, whose samples were the main source feeding
`trace_log`.
- `webapp/docker-compose.yml`: adds the `users.d` mount.
## Verification
Ran `clickhouse/clickhouse-server:26.2` with these exact mounts, and
`25.12` to cover the documented 25.8 floor:
- All 9 profile settings report `changed=1`, and a custom
`CLICKHOUSE_USER` inherits them.
- `users.d` merges rather than replaces: the `default` user, its
password, `access_management` and the `readonly` profile all survive, so
the compose healthcheck still passes.
- `remove="1"` is a clean no-op on keys absent from a given version - no
empty section, no accidental table, no startup error - so pinning
`CLICKHOUSE_IMAGE_TAG` to an older supported tag won't crash-loop.
- TTLs land in the real DDL: `TTL event_date + toIntervalDay(7)` /
`(30)`.
- In-place upgrade on a populated volume: clean restart, data preserved,
and ClickHouse lazily renames the pre-existing `query_log`/`error_log`
to `query_log_0`/`error_log_0` as it applies the new retention.
## Notes for review
- **`part_log` is kept (bounded) rather than disabled.** It appears in
neither report behind this change and isn't on ClickHouse's sub-16GB
list, but it's the merge history you'd need to diagnose a recurrence.
Measured at ~0.18 KiB per part event under insert churn - about 10x
cheaper than `text_log` over the same window - so a TTL bounds it rather
than removing it.
- **The profile settings go live for the first time here.** On larger
machines that's a real, intended throughput change: `max_threads=1`,
`max_download_threads=1`, parallel parsing and formatting off.
- **Disabling a log table stops new writes but doesn't delete existing
data.** Reclaiming disk on an existing deployment needs `DROP TABLE
system.<name> SYNC`, including the `*_log_0` leftovers.
## Known gaps, deliberately not in this PR
- The Helm chart carries the same ineffective `<profiles>` block in
`values.yaml` and mounts nothing into `users.d`, so this fix isn't
currently expressible there.
- `background_schedule_pool_log` is enabled by default with no TTL and
is disabled by neither stack.
- The dev stack's disable list has drifted from this one.
- The compose healthcheck still logs a query every 5 seconds.
---------
Co-authored-by: Yann SEGET <yann.seget@actemium.ch>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
nicktrn
d6457521cb