发布

RSS 订阅源

2.1.23-5992 a7a8dfed86

比较
[OPIK-7291] [SDK] fix: bound memory in project trace export with chunked flush (#7422)

frostbyte_neo 发布于 2026-07-10 14:09:30 +00:00
- [OPIK-7291] [SDK] fix: bound memory in project trace export with chunked flush
export_traces() buffered every trace and span in the project before
writing any file, so peak memory scaled with project size and OOM'd on
large projects during opik export ... all. Rework it to process traces
in bounded chunks: page until chunk_target (= page_size) traces need
downloading, fetch their spans, write them, then discard and continue.
Peak memory is now proportional to one chunk, and each flushed chunk is
persisted to the manifest so interrupted exports resume mid-project.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7291] [SDK] fix: address baz-reviewer comments on chunked trace export
- Flush the chunk buffer the moment it reaches chunk_target, mid-page if
  needed, so peak memory stays bounded to ~one chunk. Previously the flush
  check ran once per page after the whole page was appended, letting the
  buffer briefly hold almost two chunks (a nearly full buffer plus a fresh
  page) and breaking the bounded-memory guarantee.
- Rewrite the bounded-chunk regression test to exercise the public
  export_traces API and mock only the public REST client boundary
  (traces.get_traces_by_project / spans.get_spans_by_project) instead of
  patching private fetch* helpers. Assert on the public result tuple and
  final trace_*.json artifacts, replacing the brittle exact-call-order
  snapshot ([0, 2, 4]) with robust incremental invariants.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7291] [SDK] fix: cache unbounded span scan across chunks
Address baz-reviewer's high-severity comment on repeated full-project span
scans. With the new per-chunk flush, a chunk whose trace ids are not UUIDv7
makes _spans_trace_id_range_filter() return None, so each chunk fell back to
an unbounded project-wide span scan — turning a large export into
O(chunks × full-scan) and making timeouts far more likely.
- Factor the span pagination/grouping into _scan_and_group_spans().
- When no trace_id range filter can be built, run the unbounded full-project
  scan once, cache the grouped result, and serve every subsequent chunk from
  the cache. Bounded (UUIDv7) chunks keep their per-chunk primary-key-pruned
  scan and bounded memory. A partial (errored) scan is surfaced on every
  chunk it feeds so the manifest stays incomplete and the next run resumes.
- Tests: switch the bounded-chunk test to UUIDv7 ids (exercises the per-chunk
  path) and add a regression test asserting the unbounded scan runs once
  across three chunks rather than per chunk.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7291] [SDK] fix: don't cache a partial unbounded span scan
Address review feedback on the unbounded span-scan cache. Previously
unbounded_spans_cache was assigned an empty dict before the scan ran, so
a scan that errored midway left the dict partially populated but non-None —
every subsequent chunk then reused that partial cache, letting one transient
429/5xx permanently drop spans for the rest of the export.
- Scan into a local dict and only publish it as the shared cache when the
  scan completes cleanly. A scan that errors is used for the current chunk
  (partial spans beat none) and flags had_errors, but is left unpublished so
  the next chunk retries the scan instead of reusing partial data.
- Drop the now-unnecessary unbounded_scan_had_errors state variable.
- Add a regression test asserting a transient scan failure is retried by a
  later chunk (two scan sessions) rather than poisoning the cache.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7291] [SDK] fix: bound span-scan fallback memory to one chunk
Remove the whole-project span cache from the non-UUIDv7 fallback path.
Caching the full span set to avoid re-scanning reintroduced the unbounded
memory peak that chunked export exists to prevent (and could leave a
partial cache after a transient error). Each chunk now scans and keeps
only its own traces' spans, so peak memory stays bounded to one chunk
regardless of project size; the rare non-UUIDv7 path trades scan count
for memory safety. Also collapses the two span-fetch branches and drops
the now-dead only_known_ids parameter.

Rework the fallback tests accordingly: assert per-chunk scanning (bounded
memory) instead of single-cached-scan, and inject the transient span-scan
503 at the public spans.get_spans_by_project boundary rather than the
private _fetch_spans_page helper, per SDK testing guidelines.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

Co-authored-by: Douglas Blank doug@comet.com
Co-authored-by: Claude Opus 4.8 (1M context) noreply@anthropic.com
下载附件
- 源代码 (ZIP)
- 源代码 (TAR.GZ)

0 发布 8,340 Git 标签

[OPIK-7291] [SDK] fix: bound memory in project trace export with chunked flush (#7422)