-
[OPIK-6217] [SDK] perf: e2e test isolation + introduce pytest-xdist for parallel execution (#6499)
frostbyte_neo 发布于
2026-04-27 15:27:47 +00:00 - [OPIK-6217] [SDK] perf: e2e test isolation + introduce pytest-xdist for parallel execution
Run the Python SDK e2e suite under pytest-xdist with --dist=loadfile so
multiple test files execute in parallel against the docker-compose
backend. Locally this drops a representative slice from ~7 min serial
to ~2 min at -n 2 (~6x at -n 4), without changing what is tested.Isolation contract (so workers do not collide on shared backend state):
- Per-module project name from
generate_project_name("e2e", __name__)
(helper in tests/testlib/project_naming.py, re-exported from
tests.testlib). Test files declarePROJECT_NAME = ...at module top
and reference it directly in test bodies. configure_e2e_tests_env(autouse, module-scoped) reads the test
module's PROJECT_NAME and patches OPIK_PROJECT_NAME so the env var
the SDK writes to and the constant the tests verify against can never
drift; PROJECT_NAME is the single source of truth.- Parametrize values switched to booleans (
override_project_name) +
in-test computation, so xdist's collection-consistency check holds
without env-var indirection or static-literal exceptions. - Removed every hardcoded project / dataset / experiment / suite /
optimization name under tests/e2e/**. test_local_recording.pyopts out of OPIK_PROJECT_NAME with a
module-scoped autouse fixture (tests assert on the SDK's "Default
Project" fallback).
Tooling:
- Added pytest-xdist to tests/test_requirements.txt.
- python_sdk_e2e_tests.yml runs
-n 2 --dist=loadfile -p no:benchmark --tb=short --durations=20. Conservative 2-worker start; scale up
once stable. - Test-authoring contract documented in sdks/python/AGENTS.md.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(e2e): drop unused project_name locals after inlining
Leftovers from the PROJECT_NAME inlining in three tests; they failed
ruff F841 (assigned but never used).Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
-
Fix lint errors
-
ci(e2e): bump xdist workers to 3 and drop --tb=short
Local timing (full e2e suite, same scope as CI):
-n 2: 2:04
-n 3: 1:37 (this commit)
-n 4: 1:09
3 workers gives a healthy speedup over 2 without saturating the
docker-compose stack. Drop --tb=short — full tracebacks render fine
under xdist (failures are buffered per-test and printed sequentially
at the end, no interleaving).Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- ci(e2e): bump xdist workers from 3 to 4
Local timings: -n 3 → 1:37, -n 4 → 1:09. Past 4 gains flatten — the
suite becomes tail-bound on the slowest single test file and the
docker-compose backend on the same runner starts to thrash.Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(e2e): address review feedback on e2e isolation contract
- AGENTS.md: rewrite the "alternative projects" rule. The PR's actual
pattern is parametrize-on-boolean + compute the project in the test
body, not a static literal. The previous wording contradicted the
shipped code. - AGENTS.md: soften "no string literals" to "no bare hardcoded
literals" — strings derived from a unique-per-test fixture
(e.g. f"test_optimization_{dataset_name}") are fine. - project_naming.py: switch _random_chars to secrets.token_hex so
per-worker uniqueness is independent of any seeding in the test
environment (PYTHONHASHSEED, stray random.seed). Document why we
duplicate tests/conftest.py::random_chars (cycle-avoidance). - conftest.py::configure_e2e_tests_env: clarifying docstring on env-var
read timing — Opik(...) caches OPIK_PROJECT_NAME at init, the
patch works because each test gets a fresh client; @opik.track is
lazy via get_client_cached(), no import-time capture.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
下载附件