github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 2.0.48-5449 991f5d3bf3
    比较

    [OPIK-6576] [SDK] feat: add load-test suite for traces, spans, and attachments (#6829)

    frostbyte_neo 发布于 2026-05-25 13:46:38 +00:00

    • [OPIK-6576] [SDK] feat: add load-test suite for traces, spans, and attachments

    Adds a pytest-driven load-test suite under tests_load/suite/python_sdk/
    covering the four ingestion shapes from OPIK-6576: high trace/span
    counts, heavy payloads, explicit and implicit attachments, and
    burst/spread/concurrent patterns. Tests exercise the SDK via the public
    surface only — @opik.track decorators and start_as_current_trace /
    start_as_current_span context managers — so they mirror real user code.

    Each scenario captures the submitted trace ids, calls opik.flush_tracker,
    then polls search_traces (with attachment-search workarounds for
    OPIK-6651) and the attachments list endpoint until every submitted id
    lands with required fields set. A regression-style assertion fails fast
    if any submitted id is missing post-flush, catching dropped messages
    (same shape as the OPIK-6444 unit regression, one level up).

    Per-phase timings and counts are written to tests_load/.last_run/
    <test_name>.json. The new .github/workflows/load_tests.yml runs the
    suite weekly via cron and on workflow_dispatch, with -n auto
    --dist=worksteal so independent scenarios run in parallel. A summary
    step aggregates the per-test JSON into a Markdown table appended to
    $GITHUB_STEP_SUMMARY so the metrics render directly on the workflow
    run page.

    Implements OPIK-6576: Load testing for spans & traces against
    open-source installation.

    • fix(sdk): release context project name owner on span context manager exit

    start_as_current_span calls _try_acquire_project_name via
    add_start_candidates on enter, which sets the context_storage project
    name with the span/trace id as owner. The finally block popped span/
    trace data but never called release_context_project_name_if_owner, so
    the owner leaked across context boundaries. After one
    start_as_current_span call, any later @opik.track invocation in the
    same thread silently inherited the leaked project name regardless of
    its own project_name argument.

    The decorator path in base_track_decorator.pop_end_candidates already
    releases by span/trace id on exit; this change makes the context-
    manager path symmetric. Two regression tests in
    test_span_context_manager cover the single-call and looped patterns.

    Discovered while building the OPIK-6576 load-test suite (test ordering
    across scenarios was non-deterministic because of this leak).

    • test(load): add race-stress concurrent scenario + pytest-timeout hang-guard

    The existing test_concurrent_writers_share_one_client at scale 1.0 would
    catch a missing-lock regression in BatchManager.flush_ready (OPIK-6444
    shape) only flakily, because the production CREATE_*_FLUSH_INTERVAL_SECONDS
    is 2.0 s and our 30-thread submission window is also ~2 s — only ~1 flush
    cycle fits inside the race window per run.

    Add test_concurrent_writers_race_stress, tuned specifically to surface
    lock regressions:

    • A race_stress_flush_interval context manager monkey-patches
      CREATE_TRACES      /CREATE_SPANS_MESSAGE_BATCHER_FLUSH_INTERVAL_SECONDS down
      from 2 s to 5 ms for the duration of the test, then restores them and
      resets the global Opik client so a fresh batcher picks up the patch.
    • 100 worker threads (vs 30 in the realistic test) and no think-time keep
      the submission window dense.
    • Volume: 100 * 500 = 50k traces at scale 1.0.

    Empirical catch rate against the OPIK-6444 lock removal at scale 0.05:
    ~100% — every run either fails with the clean "X trace(s) missing
    required fields (name, end_time)" assertion when partial loss leaks
    through, or hangs and is killed by the new 300 s test-level timeout.

    Hang-guard infrastructure:

    • New pytest-timeout dependency in tests_load/suite/python_sdk/requirements.txt
    • Global default timeout = 1200 s in pytest.ini (2x the longest legitimate
      scenario, test_spread_over_time at ~600 s)
    • Tighter @pytest.mark.timeout(300) on test_concurrent_writers_race_stress
    • timeout_method = thread so the hang is killed mid-test rather than
      blocking until process exit
    • test(load): add Dataset.insert() many-versions scenario

    Covers the failure mode documented in the Notion writeup "Dataset
    migration replay: silent data loss on the version chain":
    Dataset.insert() creates a new version on the backend, and the
    COPY_VERSION_ITEMS step (a ClickHouse INSERT...SELECT that snapshots
    the previous version's items into the new one) non-deterministically
    returns short on multi-replica deployments, truncating the new
    version. Subsequent versions cascade off the truncated baseline.

    The bug is purely server-side — single-thread sequential REST calls
    already trigger it on prod. Localhost docker-compose (single-replica
    ClickHouse) doesn't reproduce, so this test passes there as a green
    baseline; it'll surface the truncation when run against a multi-
    replica environment.

    Uses the high-level Dataset.insert() / Dataset.get_items() API (not
    the lower-level apply_dataset_item_changes the migrate CLI uses), per
    the requirement to exercise the public SDK surface.

    Volume at load_scale=1.0: 50 versions × 50 items × ~4 KB payload =
    2500 items, matching the production repro parameters from the
    Notion doc.

    • fix(load): use standard b64encode for attachment list path, matching SDK contract

    The attachments-list endpoint expects a base64-encoded base URL in the
    path query param. Our verify_attachments helper used
    urlsafe_b64encode, while the SDK's own attachment/client.py and
    the e2e verifiers in tests/e2e/verifiers.py use standard
    b64encode. The url-safe variant happened to round-trip correctly
    for the URLs we test against (no +// characters in them), but
    matching the established contract is the right thing to do.

    Caught by baz-reviewer on PR #6829.

    • chore(ci): unblock lint, eval-metrics, and ADK transfer-to-agent failures

    Three independent fixes carried in this PR because all three CI jobs
    were red on every recent SDK PR including this one. None of them are
    caused by the load-test suite this PR introduces.

    • Lint (pre-commit run --all-files): ruff format applied to
      sdks/python/tests/unit/api_objects/dataset/test_stream_dataset_items.py
      which landed in main without passing the formatter. Pure whitespace
      reformat (22 line diff, no semantic change).

    • Evaluation metrics (ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai'): ragas 0.4.x imports
      from a langchain_community submodule that was removed in
      langchain-community 0.3.30+ (migrated to the standalone
      langchain-google-vertexai package). Pin ragas to the 0.3.x line for
      Python 3.10+ to match the older constraint already in place for
      Python < 3.10. Follow-up: rework the test or upgrade ragas once it
      catches up with the langchain migration.

    • ADK (test_adk__transfer_to_agent__tracked_and_span_created
      failing intermittently): the test's root agent has
      instruction="Always transfer the user's message to the Translator sub-agent. Do not translate yourself.", but gemini-2.5-flash
      occasionally answers directly without invoking the sub-agent, which
      collapses the expected Translator span out of the trace. This is
      LLM non-determinism, not an SDK regression. Add
      pytest-rerunfailures to test_requirements.txt and mark the test
      @pytest.mark.flaky(reruns=2, reruns_delay=5) so it only fails when
      the SDK is genuinely broken. Inline comment explains the rationale.

    • revert(ci): drop ADK rerunfailures (ineffective); switch eval-metrics pin to root cause

    ADK: previous commit added pytest-rerunfailures + @pytest.mark.flaky on
    test_adk__transfer_to_agent__tracked_and_span_created on the theory that
    the failure was LLM non-determinism. CI proved otherwise — the test
    failed all 3 attempts (initial + 2 reruns) with the same deep-assert
    diff (no Translator span emitted), so the failure is deterministic and
    retries don't help. Revert the marker and the new dependency so we
    don't carry dead-weight infrastructure for no benefit. This test
    remains pre-existing failure on main and is left to be tackled in a
    focused follow-up.

    Eval metrics: the prior commit pinned ragas<0.4, but ragas 0.3.9 still
    imports from langchain_community.chat_models.vertexai import ChatVertexAI. That submodule was removed in langchain-community 0.4
    (replaced by the standalone langchain-google-vertexai package). Pinning
    ragas didn't help because the latest langchain-community gets installed
    either way. The root cause is langchain-community's removal, so pin
    langchain-community<0.4 directly and drop the ragas pin.

    下载附件