github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 1.10.46-4594 33423825ec
    比较

    [OPIK-5050] [BE] fix: trace threads closing job performance and query correctness (#5726)

    frostbyte_neo 发布于 2026-03-20 17:07:57 +00:00

    • [OPIK-5050] [BE] fix: replace FINAL with LIMIT 1 BY in trace thread queries and add exponential backoff

    Replace FINAL with LIMIT 1 BY subqueries in TraceThreadDAO to avoid
    full table scans on the closing job hot path. Push status filter into
    subquery for FIND_PENDING_CLOSURE_THREADS_SQL to skip inactive rows
    early. Fix correctness issue in FIND_THREADS_BY_PROJECT_SQL where
    mutable column filters were applied before deduplication, potentially
    returning stale results. Add exponential backoff to
    TraceThreadsClosingJob to prevent hammering ClickHouse on consecutive
    failures.

    • [OPIK-5050] [BE] chore: address PR review feedback
    • Simplify LIMIT 1 BY (workspace_id, project_id, thread_id, id) to
      LIMIT 1 BY id in FIND_PENDING_CLOSURE_THREADS_SQL and
      OPEN_CLOSURE_THREADS_SQL for consistency with rest of codebase
    • Add SETTINGS log_comment to FIND_THREADS_BY_PROJECT_SQL,
      FIND_PENDING_CLOSURE_THREADS_SQL, and OPEN_CLOSURE_THREADS_SQL
      for query observability in ClickHouse
    • Fix log placeholder formatting to use single-quoted '{}' per
      project conventions

    • [OPIK-5050] [BE] chore: fix import ordering (spotless)

    • [OPIK-5050] [BE] fix: escape angle brackets in SQL to prevent StringTemplate interpolation

    The < and > SQL comparison operators in FIND_PENDING_CLOSURE_THREADS_SQL
    were being interpreted as StringTemplate delimiters after switching from
    raw string to getSTWithLogComment, silently corrupting the query.

    • [OPIK-5050] [BE] fix: revert log_comment on queries with SQL angle brackets

    FIND_PENDING_CLOSURE_THREADS_SQL and OPEN_CLOSURE_THREADS_SQL contain
    SQL < and > operators which StringTemplate interprets as template
    delimiters, silently corrupting the rendered query. Revert these two
    queries to use raw strings like the original code. Keep log_comment
    on FIND_THREADS_BY_PROJECT_SQL which only uses ST template expressions.

    • [OPIK-5050] [BE] perf: use time-bounded FINAL with minmax skip index for closing job query

    Replace LIMIT 1 BY approach with time-bounded FINAL + minmax skip index
    on last_updated_at. The closing job query now only scans recent granules
    instead of the entire trace_threads table, reducing granules read from
    369/369 to 12/369 (97% reduction) in benchmarks.

    • Add cached getMaxTimeoutMarkThreadAsInactive to compute lookback window
    • Bind cached_max_inactive_period parameter in DAO
    • Add use_skip_indexes_if_final=1 SETTINGS to enable skip index with FINAL
    • Add cache config for max_timeout (30min TTL)
    • [OPIK-5050] [BE] perf: add cold start lookback, GROUP BY optimization, increase default job interval
    • Add 7-day cold start lookback on first run after startup to catch threads
      that became stale during outages
    • Normal lookback floor: max(maxTimeout + 1h, 1 day) via minmax skip index
    • GROUP BY workspace_id, project_id, status with min(last_updated_at) in
      subquery to reduce rows before workspace_configurations JOIN
    • Increase default OPIK_CLOSE_TRACE_THREAD_JOB_INTERVAL from 3s to 15s
    • Add minmax skip index migration for last_updated_at (GRANULARITY 1)
    • [OPIK-5050] [BE] chore: add cache config docs, bump lock time to match interval
    • Add documentation comment for max_timeout_mark_thread_as_inactive cache
    • Increase closeTraceThreadJobLockTime from 4s to 14s to match the 15s
      job interval (prevents premature lock release with more accumulated work)

    • [OPIK-5050] [BE] chore: update MAX_BACKOFF_EXPONENT comment for 15s interval

    • [OPIK-5050] [BE] chore: improve backoff comment, use LEFT ANY JOIN for workspace config

    • Clarify MAX_BACKOFF_EXPONENT comment to describe doubling pattern
    • Use LEFT ANY JOIN for workspace_configurations (at most one row per
      workspace after FINAL dedup, communicates intent and is slightly more efficient)
    • [OPIK-5050] [BE] fix: move success handler to onComplete, add migration comment
    • Mono never emits onNext, so completedFirstRun/backoff reset was
      unreachable. Move success logic to onComplete (3rd subscribe arg).
    • Add --comment to migration file per convention.
    • [OPIK-5050] [BE] chore: move cold-start lookback and max backoff exponent to config

    Move COLD_START_LOOKBACK and MAX_BACKOFF_EXPONENT from hardcoded
    constants to TraceThreadConfig, backed by env vars
    OPIK_CLOSE_TRACE_THREAD_COLD_START_LOOKBACK (default 7d) and
    OPIK_CLOSE_TRACE_THREAD_MAX_BACKOFF_EXPONENT (default 5).

    Also fix log message in onComplete handler ("started" -> "completed").

    • [OPIK-5050] [BE] fix: rename migration 71 -> 73 to avoid conflict with main

    • [OPIK-5050] [BE] chore: remove benchmark script, expand config docs

    下载附件