-
[NA] [SDK] ci(load): drop xdist concurrency from -n auto to -n 2 (#6857)
frostbyte_neo 发布于
2026-05-25 20:39:34 +00:00 The first three weekly Load Tests workflow runs (and several manual
workflow_dispatch retries) all failed in the same pattern: under
xdist's-n auto(= 4 worker processes on ubuntu-latest), the two
highest-volume scenarios — test_many_traces_one_span_each (100k traces
× 1 span = 200k observations) and test_many_spans_per_trace (5k × 50
spans) — were OS-killed mid-run with no Python traceback. xdist
reports them as "worker 'gw0' crashed" / "worker 'gw3' crashed",
which is what the OOM-killer hitting the worker process looks like
from xdist's side.Root cause is the runner topology, not any single test: 4 xdist
workers + a full docker-compose Opik stack (ClickHouse + MySQL + MinIO- backend + nginx) on the same 7 GB ubuntu-latest runner can't hold
all four high-volume scenarios in memory at once. Per-test fixes only
shifted which scenario got killed — adding think-time to
test_burst_single_loop (PR #6856) brought the burst wall-clock down
but the freed-up backend headroom just went to the next test in line,
which then ran longer and met the OOM-killer instead.
Halve concurrency:
-n auto->-n 2. Two scenarios run in parallel
at a time, the other two wait. Trade-off:- expected wall-clock goes from ~32 min to ~50 min (still well under
the 60 min job timeout); - per-runner memory and concurrent backend pressure both drop ~2x;
--dist=workstealkeeps balancing the very uneven per-test
durations (spread is window-locked at 600 s, others run 1-6 min).
If we ever want to bring concurrency back up, the proper fix is a
larger runner (e.g.ubuntu-latest-4-cores-16gb) or splitting the
suite into two workflow jobs that don't share a backend. Both are
follow-ups out of scope for this CI hotfix.README updated to reflect the new command and explain why.
下载附件
- backend + nginx) on the same 7 GB ubuntu-latest runner can't hold