发布

RSS 订阅源

1.10.33-4453 38d4eab91b

比较
[OPIK-4801] [SDK] Add migration manifest for resumable opik import (#5528)

frostbyte_neo 发布于 2026-03-10 10:43:43 +00:00
- [SDK] Add migration manifest for resumable opik import
Adds a migration_manifest.json to the import directory so that
interrupted imports can be resumed without creating duplicates.
- New MigrationManifest class: atomic JSON state file tracking
  completed files, failed files, and src→dest trace ID mappings
- Auto-detected on import startup: resumes if in_progress, blocks
  re-import if completed (unless --force is given)
- Added --force flag to all four import subcommands
- Trace ID map is persisted to disk after each file so experiment
  cross-references survive a resume across separate invocations
- 20 new unit tests covering lifecycle, file tracking, ID persistence,
  atomic writes, and interrupted+resumed migration scenarios
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [DOCS] Update import/export CLI docs for migration manifest and --force
- Documents the new --force flag on all opik import subcommands
- Explains the migration_manifest.json and automatic resume behaviour
- Adds the migration_manifest.json to the file structure diagram
- Replaces the "Project Migration" use case with a proper two-installation
  migration section that shows how to use different API keys for source
  and destination
- Adds three new troubleshooting entries for interrupted imports,
  "already completed" message, and missing experiment items
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- [SDK] Address PR review comments on migration manifest
- Remove redundant if debug: guard around debug_print in experiment.py
- Replace direct console.print with debug_print in project.py for consistency
- Restore removed explanatory comments in project.py span processing logic
- Restore removed examples in import_group docstring (dataset import, custom path)
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- Address PR review comments from petrotiurin
- Restore missing project import examples (name pattern and --debug) in __init__.py
- Restore removed "# Create span with parent_span_id if available" comment in project.py
- Replace redundant if debug: console.print(...) blocks with debug_print() in project.py
- Remove redundant if debug: guards wrapping single debug_print() calls in experiment.py
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- Batch manifest writes to reduce disk I/O during large imports
Instead of rewriting migration_manifest.json on every file completion,
accumulate changes and flush every 10 files (DEFAULT_BATCH_SIZE).
State-transition calls (start, complete, reset, save) always flush
immediately to keep lifecycle semantics correct.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- Replace JSON migration manifest with SQLite database
- Use migration_manifest.db (SQLite) instead of migration_manifest.json
- WAL mode + synchronous=NORMAL: crash-safe, fast appends, no full-file rewrites
- Batched writes: buffer up to 50 ops in memory, commit in one transaction
- INSERT OR IGNORE / INSERT OR REPLACE: idempotent — no duplicate rows possible
- In-memory buffers use set/dict: deduplication before hitting SQLite, O(1) lookup
- is_file_completed checks buffer first then keyed DB lookup (O(log n) vs O(n) list scan)
- del provides best-effort flush on GC for clean process exits
- Public API unchanged: all callers require no modification
- Update docs and __init__.py docstring to reference .db filename
- Rewrite tests: replace JSON-parsing assertions with SQLite queries and API calls;
  add TestDatabaseIntegrity suite covering idempotency and last-write-wins semantics
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- Update import docs: SQLite manifest, remove JSON corruption note
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com
- Address baz review comments on migration manifest
High: document and test the batching crash trade-off
- Add TestBatchingBehavior suite with 4 tests: buffered writes visible via API,
  crash simulation (unflushed data lost), batch_size=1 per-file durability,
  auto-flush at batch threshold
- Expand module docstring with explicit crash trade-off section
Medium: do not create manifest DB during --dry-run
- MigrationManifest is now constructed only inside if not dry_run: in
  _import_by_type; --dry-run passes manifest=None to all import functions
  (they already guard with if manifest:)
- Add assert to satisfy mypy at the manifest.complete() call site
Low: make DEFAULT_BATCH_SIZE env-configurable
- Read OPIK_MIGRATION_BATCH_SIZE env var at import time; validate range 1-10000;
  log a warning and fall back to 50 on invalid input
Low: rename tests to follow test_WHAT__CASE__EXPECTED_RESULT convention
- All 26 test methods now follow the pattern required by design/TESTING.md
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com

Co-authored-by: Claude Sonnet 4.6 noreply@anthropic.com
Co-authored-by: Douglas Blank doug@comet.com
下载附件
- 源代码 (ZIP)
- 源代码 (TAR.GZ)

0 发布 8,340 Git 标签

[OPIK-4801] [SDK] Add migration manifest for resumable opik import (#5528)