发布

RSS 订阅源

1.11.2-4867 0bd1684911

比较
[NA] [SDK] fix: harden opik export against server-side 429 rate limiting (#6087)

frostbyte_neo 发布于 2026-04-09 17:27:18 +00:00
- [NA] [SDK] fix: harden opik export against server-side 429 rate limiting
The backend recently added per-endpoint throttling (30 req/60s per
workspace) to trace/span read endpoints (OPIK-5187, ~March 25 2026).
The previous export code exhausted its 3 retries quickly and then
broke out of the pagination loop entirely, silently exporting only
half the traces.

Changes:
- Reduce MAX_WORKERS from 10 → 3 to avoid triggering rate limits
  with concurrent span fetches
- Replace the default SDK retry decorator on _fetch_traces_page and
  _fetch_spans with an export-specific one: 8 attempts, 30 s wait on
  429 (honouring Retry-After up to 120 s), 10–60 s exponential backoff
  for other transient errors
- Add a 1 s inter-page delay to avoid request bursts
- On page-fetch failure after all retries, skip the page and continue
  rather than breaking the loop — subsequent pages are still fetched,
  and the manifest stays in_progress so the next run fills the gaps
- Surface had_errors through export_single_project's return tuple (now
  4-tuple) and print an explicit "run again to fill gaps" warning
  instead of falsely reporting success
- Add 16 unit tests covering the new retry logic, page-skip behaviour,
  inter-page delay, and had_errors propagation
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] address baz-reviewer feedback on PR #6087
- Narrow page-fetch exception handler from bare except Exception to
  only ApiError + httpx transient errors; re-raise permanent errors
  (e.g. 400/401/403) so they are not silently skipped (high severity)
- Add MAX_CONSECUTIVE_PAGE_FAILURES = 5 cap: abort the loop if five
  pages in a row all fail, preventing an infinite skip loop
- Extract _print_project_export_status helper to deduplicate the
  had_errors/success/up-to-date print block shared by the two callers
- Fix export_single_project to return 0 as the project-exported
  flag when no traces were exported or skipped, instead of always 1
- Replace direct tests of private _export_wait_duration with tests
  that go through the public export_traces API and capture
  time.sleep calls, per project testing guidelines
- Rename all test methods to test_WHAT__CASE__EXPECTED convention
- Add two new tests: consecutive-failure cap aborts export; permanent
  400 error raises rather than being silently skipped
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] fix: use module-level import for retry_decorator per project convention
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] fix: parse HTTP-date Retry-After header in _export_wait_duration
Replace manual float() parsing with _parse_retry_after from http_client.py,
which handles both numeric-seconds and HTTP-date Retry-After values using
email.utils.parsedate_tz. Also wraps exc.headers dict in httpx.Headers for
compatibility. Adds a test covering HTTP-date format.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] fix: make inter-page sleep assertion robust against background threads
The test was checking mock_sleep.call_count == 2, but when run in the full
CI suite, background Opik message-processing threads (sleeping 0.1s) are
still alive and their time.sleep calls inflate the count. Fix by counting
only calls with _PAGE_FETCH_DELAY_SECONDS (1.0s), which background threads
never use.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] fix: filter background thread sleeps in partial-page rate-limit test
Count only _PAGE_FETCH_DELAY_SECONDS calls instead of all time.sleep
calls so background Opik threads don't cause false test failures.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] address review feedback: page-size option and softer backoff
- Raise default page_size from 100 to 500 (fewer round-trips, offsetting
  reduced concurrency) and expose as --page-size (1-1000) on both
  opik export project and opik export all — addresses petrotiurin's
  review comment
- Soften non-rate-limit exponential backoff from min=10s to min=2s
  (multiplier=2, giving ~2s/4s/8s/... rather than jumping to 10s) —
  addresses alexkuzmik's review comment
- Update docs and tests to reflect new defaults
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] scale non-429 backoff by MAX_WORKERS per alexkuzmik's review
With MAX_WORKERS=3 concurrent span-fetch workers, a single worker's
transient error should wait long enough for all concurrent requests to
drain. Multiply the base exponential wait by MAX_WORKERS (capped at 60s):
~6s, 12s, 24s, 48s, 60s... This keeps the formula self-consistent when
MAX_WORKERS changes, rather than hard-coding a magic constant.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] fix: jitter 429 waits and handle prompt history project-filter error
- project.py: parse ratelimit-reset / opik-...-ttl-millis headers as
  fallbacks to Retry-After, and add 0–5 s jitter to prevent concurrent
  workers from restarting simultaneously (thundering herd on rate limits)
- prompt.py: catch ValueError from get_prompt_history when the prompt is
  not associated with the default project; fall back to empty history so
  the current version is still exported instead of failing entirely
- test: update wait-duration assertions to accept header value + jitter range
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] fix: unify Rich console across export modules to prevent garbled progress output
All export modules now share a single Console instance from utils.py
instead of each creating their own. This lets Rich's Progress context
properly intercept and interleave console.print() calls from worker
threads, preventing messages from being spliced into progress bar lines.

Also replaces the opik SDK's plain StreamHandler with a RichHandler
(ERROR-level only) during CLI export, eliminating the "OPIK: Deprecation
warning..." text that was being written to stderr and corrupting the
progress bar display.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] fix: replace per-trace span search with bulk project span pagination
POST /spans/search has a 30 req/min per-workspace rate limit, making it
impractical for large projects (~3.5k traces/day = 3,500 API calls).

Switch to GET /spans?project_name=... (no trace_id filter), which paginates
all spans for a project at once (~280 calls for 280k spans) and is not
subject to the same strict rate limit. Spans carry trace_id, so we group
them client-side after collection.

Export is now three phases:
1. Paginate all traces → {trace_id: trace} dict
2. Paginate all project spans → {trace_id: [spans]} dict
3. Write trace files
Remove ThreadPoolExecutor / MAX_WORKERS (no longer needed), simplify the
non-429 backoff (was scaled by MAX_WORKERS to account for concurrent
worker pressure). Update tests accordingly.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] fix: clamp Retry-After to max instead of discarding, extract wait constants
When the server's Retry-After exceeds _EXPORT_MAX_RETRY_AFTER_SECONDS, clamp
to the max rather than ignoring it and falling back to the 30 s default. Also
extract the 30.0 and 5.0 literals as named constants for readability.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] fix RichHandler console binding bug; update export_single_project docstring
- Replace wrong-console RichHandlers on the opik logger instead of
  skipping setup when any RichHandler exists. Previously, if the SDK
  had already attached a RichHandler to a different console, SDK logs
  would bypass the progress-bar-aware _console.
- Document export_single_project's 4-tuple return value and explain
  how traces_had_errors affects the manifest state.
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] fix: prevent CI hang by passing show_progress=False in rate-limit tests
Rich's Progress spinner spawns a background render thread that calls
time.sleep(0.1) between redraws. When tests patch time.sleep globally to
a no-op, that thread spins at full CPU, flooding stdout with render output.
For tests that write 1 000 trace files (long Phase 3), the torrent of
console writes was enough to stall the CI runner until the 15-minute job
timeout fired.

Fix: pass show_progress=False to every export_traces call in the test file
so the nullcontext path is taken and no render thread is created.

Also corrects the stale assertion in
test_export_traces__page_fetch_429_retry_after_exceeds_cap: commit 3c14313
changed the behaviour from "fall back to default 30 s" to "clamp to the
120 s cap", but the test expectation was never updated. The test name and
assert range now reflect the actual clamped value (120–125 s).

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] fix: stop test suite from hanging due to real tenacity/span-fetch sleeps
Three changes to test_export_project_rate_limiting.py:
1. _run_with_first_call_raising: replace patch("time.sleep") with
  patch(f"{_MODULE}._fetch_traces_page.retry.sleep"). Tenacity captures
  time.sleep by reference at decoration time (self.sleep = time.sleep),
  so patching the Python attribute has no effect. Patching the Retrying
  instance's .sleep attribute directly is the only way to intercept the
  retry wait; this also makes the captured sleep values available for
  assertions.
2. test_export_traces__middle_page_fails: add
  patch(_fetch_spans_page, return_value=_make_page([])) to prevent the
  real (tenacity-decorated) span-fetch loop from running for all 1000
  traces and potentially triggering real network calls or sleeps.
3. test_export_traces__multiple_full_pages__sleep_called_between_pages:
  same _fetch_spans_page patch for the same reason (1000 traces).
All 17 tests now pass in ~1 second (was hanging/timing out in CI).

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] fix: reduce 1000-trace pages to 2-trace pages to prevent CI timeout
test_middle_page_fails and test_multiple_full_pages used _make_full_page(500)
which creates 500 MagicMock traces per page (1000 total), each flushed to disk
as a JSON file. On CI this caused a ~14-minute hang, exceeding the 15-minute
job timeout.

Fix: pass page_size=2 to export_traces and use _make_full_page(2) pages.
Two traces per page is still "full" relative to page_size, so the pagination
loop's short-page break does not fire early, exercising the same code paths
with only 4 file writes instead of 1000.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [NA] [SDK] fix: restore opik logger handlers after CLI export command to prevent test state leak
The handler-manipulation code in export_group() removed the StreamHandler
from the opik logger for the duration of the process. When CliRunner-based
unit tests invoked the export command, the StreamHandler was permanently
gone — causing test_track__nested_decorator_with_different_project to find
an empty captured.err (WARNING < ERROR-level RichHandler threshold).

Use ctx.call_on_close() to restore the original handlers on context exit.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com

Co-authored-by: Douglas Blank doug@comet.com
Co-authored-by: Claude Sonnet 4.6 noreply@anthropic.com
下载附件
- 源代码 (ZIP)
- 源代码 (TAR.GZ)

0 发布 8,340 Git 标签

[NA] [SDK] fix: harden opik export against server-side 429 rate limiting (#6087)