github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 2.2.9 e0442e3250
    比较

    [NA] [BE] [DOCS] feat: ClickHouse async-insert tuning knobs + TOO_MANY_PARTS recovery runbook (#7629)

    frostbyte_neo 发布于 2026-07-28 11:39:33 +00:00 | -1622 次提交 在此版本后已推送到 1.10.43/hotfix

    • [NA] [BE] [DOCS] feat: ClickHouse async-insert tuning knobs + TOO_MANY_PARTS recovery runbook

    Add operator knobs to reduce ClickHouse async-insert part fragmentation, plus a
    self-host runbook for TOO_MANY_PARTS ingestion failures and stuck merges.

    Backend:

    • ANALYTICS_DB_ASYNC_INSERT_BUSY_TIMEOUT_MIN_MS: present-only override, mirroring
      the existing MAX_MS (applied only when already present in the queryParameters
      custom_http_params chain).
    • ANALYTICS_DB_ASYNC_INSERT_MAX_DATA_SIZE: injected only when set; unset leaves the
      ClickHouse/server-side value untouched (upgrade-safe, never clobbers server tuning).
    • Remove dead AsyncInsertConfig (never wired in; misleading Javadoc).
    • config-test mirrors config.yml; docker-compose passes the new vars through.
    • Integration tests cover both fields against a real ClickHouse.

    Docs (self-host troubleshooting, Latest + v1):

    • Runbook to distinguish async-insert fragmentation from a stuck merge and recover
      (raise parts_to_throw_insert, DETACH corrupt parts, ZooKeeper queue-entry removal
      as last resort), with cluster-aware queries and data-safety warnings.
    • docs(self-host): address review — schema placeholders + accurate stuck-merge recovery
    • Use a <database_name> placeholder in the recovery commands (diagnostic queries,
      ALTER, DETACH, RESTART REPLICA, ZooKeeper path) instead of hardcoding the default
      opik schema, so custom-schema installs act on the right tables.
    • Correct the DETACH step: on a ReplicatedMergeTree the queued merge entry
      references its source parts by name, so DETACH only clears the merge when the part
      is intact on another replica (re-fetch). If it is corrupt on all replicas the entry
      persists and step 6 (remove the stuck queue entry) is the reliable recovery.

    Applied to both the Latest (docs-v2) and v1 (docs) troubleshooting pages.

    • fix(clickhouse): inject async_insert_max_data_size even when queryParameters is blank

    getQueryParametersOverrides returned early on a blank/empty queryParameters, which
    dropped inject-when-set settings. That silently ignored a configured
    async_insert_max_data_size when async insert is enabled at the ClickHouse
    server/profile level but the queryParameters chain is blank.

    Remove the blank short-circuit and rely on the existing "no overrides and nothing
    to inject" check, so the value is injected regardless of the chain being blank.
    Present-only overrides (the busy-timeout window) are unaffected — an empty chain
    carries nothing to override. Drops the now-unused StringUtils import and adds an
    integration test for the blank + max_data_size case (35 pass on ClickHouse 26.3.16).

    • refactor(clickhouse): unify async_insert overrides to inject-when-set

    Align the busy-timeout min/max env overrides to the same inject-when-set behavior
    as async_insert_max_data_size: a dedicated env is applied whenever set — overriding
    the value in the queryParameters chain, or added when the chain omits it (including
    a blank chain) — and leaves the chain/ClickHouse-server value untouched when unset.

    Previously the busy-timeout overrides were present-only: silently ignored when the
    setting was absent from a customized or stale ANALYTICS_DB_QUERY_PARAMETERS, so an
    operator could set the env and see no effect. Uniform behavior removes that footgun
    and is robust to overridden query strings.

    Behavior is unchanged for any deployment that leaves the envs unset or uses the
    shipped chain (which carries the busy-timeout settings). The only difference: a
    busy-timeout env now also applies when the setting is absent from a custom chain —
    a narrow, non-breaking refinement of the previously released MAX_MS override.

    Collapses the two-path override logic into one, flips the corresponding tests to
    assert injection, and unifies the config and runbook wording.

    • refactor(clickhouse): use StringUtils.isBlank in parseQueryParameters; test blank string

    Simplify the null-or-blank guard in parseQueryParameters to StringUtils.isBlank
    (null-, empty-, and whitespace-safe in one call) and cover a whitespace queryParameters
    end-to-end alongside the existing null case, confirming a blank chain plus an override
    still injects into custom_http_params on both client paths.

    • docs(self-host): parameterize the table in TOO_MANY_PARTS recovery steps

    The stuck-merge recovery hard-coded <database_name>.spans, so a traces incident
    would send the operator to raise thresholds, DETACH, restart, and edit the
    ZooKeeper path on the wrong table. Replace spans with a placeholder across
    the ALTER, DETACH, SYSTEM RESTART REPLICA and ZooKeeper-path commands, and note it
    is the table the diagnosis flagged (spans or traces). Applied to both doc versions.

    • refactor(clickhouse): rename configurableQueryParameters to configurableServerSettings

    The helper returns only ClickHouse server settings (destined for custom_http_params),
    not query/driver parameters, so the name was misleading. Rename for accuracy (it
    already matches the method's Javadoc). Private method, single caller; no behavior change.

    下载附件