发布

RSS 订阅源

2.0.35-5335 d8d8bc552f

比较
[OPIK-6416] [SDK] feat: cascade experiments and traces and spans in opik migrate dataset (#6658)

frostbyte_neo 发布于 2026-05-14 11:12:41 +00:00
- [OPIK-6414] [SDK] feat: opik migrate dataset slice 1 — core command + current items
Add opik migrate Click command group with dataset and plan subcommands.
First slice of the SDK CLI Migration Tool epic (OPIK-5859): a dataset entity
plus its current items can now be migrated into a destination project within
the same workspace via a single CLI invocation, with a JSON audit-log
skeleton, fail-fast pre-flight, and a single-version target output.

Implements OPIK-6414.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(sdk): preserve full dataset/item fidelity + add test-suite support to opik migrate
Smoke-testing surfaced gaps in slice 1 that the unit tests with dict-equality
assertions hadn't caught:
- Per-item top-level fields (description, source, trace_id, span_id,
  evaluators, execution_policy) were silently dropped because the copy step
  routed through get_items()/insert(dicts) instead of the dataclass form.
  Switching to __internal_api__stream_items_as_dataclasses__ ↔
  __internal_api__insert_items_as_dataclasses__ round-trips them.
- Dataset-level description was wiped on rename: update_dataset(name=...)
  with description=OMIT is treated by the BE as null. Re-passing
  description/visibility/tags on the rename PUT preserves them.
- Dataset-level visibility/tags weren't forwarded to the target on create.
- Test suites (type='evaluation_suite') were initially refused. Slice 1 now
  migrates them: target is created with the same type, and the latest
  source version's suite-level evaluators + execution_policy
  (runs_per_item, pass_threshold) are applied to the target via
  apply_dataset_item_changes(override=True). The suite-config action runs
  BEFORE item copy so the target's first version carries the config (the BE
  rejects apply_dataset_item_changes without base_version once the
  dataset already has versions).
Workspace-mutating writes (rename, create, suite-config apply, delete) are
now wrapped with rest_helpers.ensure_rest_api_call_respecting_rate_limit
so a transient 429 doesn't abort a half-finished migration. Item-copy
already inherits the SDK's per-batch retry. Reads stay unwrapped.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(sdk): address PR review on opik migrate (sanitize errors, paginate, conventions)
Addresses 8 review comments from baz-reviewer[bot] on PR #6628:
- MigrationError now inherits from opik.exceptions.OpikException so shared
  shutdown/error-tracking code that does isinstance(..., OpikException)
  classifies migration failures correctly.
- ApiError sanitization: introduces errors.safe_error_envelope /
  safe_error_string so neither audit-log JSON nor terminal output leaks
  ApiError response bodies, headers, or tokens. Both migrate_dataset and
  migrate_plan now route exceptions through a shared _finalize_and_fail
  helper that reuses the sanitizer for non-MigrationError exceptions.
- resolve_source paginates find_datasets to exhaustion and derives
  ResolvedDataset.project_name from the matched row's project_id (resolved
  to a project name) instead of the --from-project flag — so workspace-
  scoped lookups still produce the correct project context for downstream
  Opik.get_dataset / delete_dataset calls.
- name_taken_in_workspace paginates similarly and emits collision project
  names ("project 'X'") in the resulting message.
- _suggest_project_names narrows the broad except to ApiError and logs a
  warning for unexpected exceptions instead of silently returning [].
- Stale comment on ResolvedDataset.type updated to reflect that suites are
  supported (with CopyTestSuiteConfig copying their suite-level config).
- Test names migrated to the SDK testing convention
  test_ (or __happyflow), per
  .agents/skills/python-sdk/testing.md.
Skipped (with on-PR rationale):
- TestPlanBuilding routing through the public CLI: planner is the unit
  under test in those cases; routing every assertion through Click adds
  mock setup and obscures the action-ordering contract.
- Extracting CopyTestSuiteConfig request build into rest_operations: the
  shapes differ (this side reads raw REST EvaluatorItemPublic, the other
  takes typed LLMJudge SDK objects) — a shared helper would force one to
  convert formats.
- Asserting internal_api__insert_items_as_dataclasses.call_args once:
  the whole point of that test is the one-target-version contract; the
  testing.md "explicit exception" allowance applies.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- test(sdk): cover workspace-scoped source dataset migration
Adds a unit test that pins the workspace-scoped source path (V1 datasets
or anything left at workspace scope after auto-migration, where the
source row has project_id=None on the BE):
- Resolver returns ResolvedDataset(project_name=None)
- Rename happens against the row id, no project context needed
- Destination is created under --to-project regardless of the source's
  (missing) project
- Item copy still produces exactly one target version
- Source-side reads go through Opik.get_dataset(project_name=None) — the
  workspace-scoped lookup path
Pinning this so a future refactor that silently routes the source read
through a default-project fallback (or breaks for project_id=None
sources) trips the test first.

_DatasetRow grew an explicit project_id field so the test surface is
honest about workspace-scoped vs project-scoped fixtures.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(sdk): address second-round PR review on opik migrate
Two fixes from baz-reviewer[bot]'s review of commit 3c167de:
- _project_name_for_row now narrows its catch to ApiError (recoverable
  — typically 404 if the project was deleted between listing and lookup,
  yields None and downstream treats as workspace-scoped). Other
  exceptions log a warning at WARNING level so auth/transport/config
  failures aren't silently masked, then still return None so a
  best-effort lookup doesn't abort the surrounding migration check.
- Extract iter_dataset_pages (no longer private) as the single source
  of truth for find_datasets pagination. The plan command's loop in
  migrate_plan_command reuses it instead of duplicating the same
  pagination structure. Optional name=None covers the listing case
  for the workspace survey; named lookups still pass name=str for
  resolver.
Skipped (with on-PR rationale): a third comment asked us to drop
ApiError.body text from safe_error_envelope and emit only HTTP <code>.
We already gate to scalar string fields (message / errors[0]) and
never include headers or raw body — the actionable text is what users
need at a CLI. Kept the current behavior with a reply explaining the
gating.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- refactor(sdk): trim opik migrate slice 1 surface
Per PM scope review on the epic Notion doc, slice 1 ships a smaller
surface so it can land sooner. Each removed surface element will return
in a follow-up if a real user need surfaces.

Removed:
- opik migrate plan subcommand (workspace survey) — defers to a future
  slice 4 once optimizations and experiments cascade.
- --target-name flag — users can rename via UI after the migration if
  they want a different name on the target.
- --delete-source flag — users can delete via UI after they've verified
  the migration. Reduces blast radius of the CLI.
- --source-suffix flag — replaced by a hardcoded _v1 constant in
  planner.py. Keeps the user-facing surface minimal.
Internal:
- iter_dataset_pages made private again (_iter_dataset_pages) since
  the only external reuse was the now-removed migrate plan command.
- DeleteSource action and its executor branch / audit-log entry deleted.
- Help text and PR-template examples updated to reflect the trimmed
  command shape.
Trimmed surface:
opik migrate dataset NAME --to-project=B [--from-project=A]
[--dry-run] [--audit-log=PATH]
[--workspace=...]

21/21 unit tests pass; pre-commit clean; real-BE smoke green.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- refactor(sdk): split opik migrate into generic + dataset-specific layers
Address @alexkuzmik's review on PR #6628 about the cli/migrate/ package
mixing general-purpose abstractions with dataset specifics. Applies the
parts of the suggestion we have direct evidence will generalise, defers
the rest until slice 2 / slice 3 give us a concrete second implementation
to extract real abstractions from.

Layer split (full B):
- cli/migrate/ — generic primitives + Click group
  _base.py — BaseMigrationPlan + execute_plan_loop +
  record_planned_loop (audit-bracketed)
  audit.py — entity-agnostic audit log JSON
  errors.py — generic exception hierarchy + sanitiser
  main.py — Click group, --workspace handling
- cli/migrate/datasets/ — slice 1's dataset-specific code
  resolver.py — name → ResolvedDataset, did-you-mean
  planner.py — DatasetMigrationPlan + action records
  executor.py — _apply_action dispatch, _action_details
Narrow A: only the abstractions we have evidence for:
- BaseMigrationPlan: minimal "source + ordered actions" shape. Subclasses
  narrow source/actions to their own types. kw_only=True so subclasses
  can add required fields without "non-default after default" errors.
- execute_plan_loop / record_planned_loop: the audit-bracketed for-loop.
  Caller supplies apply_fn + details_fn closures so each entity stays in
  charge of its own action types.
Deferred until slice 2/3 land concrete second implementations:
- BaseEntityResolver: slice 2 has no resolver step (it operates on the
  already-resolved dataset id); slice 3's experiment resolver has
  different validation rules. Designing a base now would either pin the
  wrong contract or force later slices to contort.
- BaseAction class hierarchy: slice 1's actions are stateless;
  slice 2's are chained (each version's apply depends on the previous);
  slice 3's are fan-out. Inheritance shape will be obvious once those
  exist.
User-facing imports (opik.cli.migrate.migrate_group) are unchanged. All
21 unit tests pass. Real-BE smoke on staging green.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-6415] [SDK] feat: replay full dataset version history in opik migrate
Slice 2 of the SDK CLI Migration Tool epic. opik migrate dataset now
replays the source's full version history onto the target with full per-
version fidelity. Items, item-level fields (data, description, tags,
evaluators, execution_policy, trace_id, span_id, source), version-level
fields (suite evaluators, execution_policy, user tags, metadata), and
display order are all preserved per version.

--exclude-versions falls back to Slice 1's current-items-only path.

Slice 3 (experiment cascade) reads MigrationPlan.version_remap and
item_id_remap to remap experiment FK references.

Implements OPIK-6415.
- feat(migrate): per-version Rich Progress bar for replay
- fix(migrate): address baz-reviewer feedback on version replay
- Swap RuntimeError → ReplayError (new MigrationError subtype) at the four
  BE-unexpected-response sites in version_replay.py so failures route
  through migrate_dataset_command's user-facing MigrationError branch.
- Forward clear_execution_policy=true at the version level when source
  drops the suite-level execution_policy between versions.
- Switch user_tags/metadata gating from truthiness to is not None so
  explicit clears ([] / {}) round-trip instead of being collapsed to
  omission (which the BE inherits from base_version).
- Switch target_items_by_hash from Dict[hash, str] to Dict[hash, List[str]]
  with FIFO consumption so source items with identical content (legal under
  the BE) each remap to a distinct target row instead of collapsing onto
  one.
Three new unit tests pin the bugs:
- test_replay__clears_version_level_execution_policy_when_dropped
- test_replay__forwards_empty_user_tags_and_metadata_explicitly
- test_replay__duplicate_content_items_remap_to_distinct_target_ids
- refactor(migrate): extract suite-payload conversion helpers
baz-reviewer PR comment 3217063444 — the suite-level evaluators /
execution_policy dict shapes were duplicated across four call sites
(_create_first_version_with_items, _create_first_version_config_only,
_apply_delta_and_collect_new_ids in version_replay.py and
_copy_test_suite_config in executor.py).

Adds two narrow conversion helpers in version_replay.py:
- _suite_evaluators_payload(evaluators)
- _suite_execution_policy_payload(execution_policy)
The helpers only convert wire types -> payload dicts; each call site
keeps its own gating (Slice 1's _copy_test_suite_config uses truthy
gating, Slice 2 uses 'is not None') so the dedup doesn't bleed
behavior across slices.
- refactor(migrate): reuse evaluator/policy payload helpers for per-item sites
baz-reviewer PR comment 3217168575 — since the suite-level helpers
already centralize the wire-shape conversion, the per-item dict
literals in _added_item_payload and _edited_item_payload should call
them too.
- Rename _suite_evaluators_payload -> _evaluators_payload and
  _suite_execution_policy_payload -> _execution_policy_payload (BE uses
  the same wire types at version + item level, so the prefix was misleading).
- Wire them into _added_item_payload and _edited_item_payload.
- Document why _content_hash_for stays uncoupled: it's a hash function,
  not a wire payload, and a future BE field added to the wire payload
  should not silently rehash everything (would break idempotency on re-runs).
- [OPIK-6416] [SDK] feat: cascade experiments and traces in opik migrate dataset
Adds Slice 3 of the SDK CLI migration tool: after the dataset and its
version history are at the destination (Slice 1/2), every experiment
referencing the source dataset is recreated under the destination project
with its traces and spans riding along.

Cascade-copy semantics

Like the dataset copy, this is a copy -- not a move. Source experiments
stay in their original projects with traces and spans intact; the
destination project gets brand-new experiments (new ids) that reference
the destination's dataset / version / item ids and carry independent
copies of the trace + span data.

Cross-project follow

find_experiments(dataset_id=...) is project-agnostic at the REST layer,
so every experiment referencing the source dataset cascades to
--to-project regardless of which project it originally lived in. This
is the epic's "baseline follow" default; it never produces dangling
references, but does duplicate when the source dataset was referenced by
experiments in multiple projects. Slice 4 (OPIK-6417) layers detection +
reporting on top.

FK remap during recreation

source dataset_id -> dest_dataset_id (Slice 1)
source dataset_version_id -> plan.version_remap[old] (Slice 2)
source dataset_item_id -> plan.item_id_remap[old] (Slice 2)
source trace_id -> built here as traces copy (this slice)
source project_id -> target_project_name (this slice)

Stripped on the destination experiment (the entities aren't migrated; per
Jacques's "strip the link, document loss" position from the epic
discussion, we drop the pointers rather than leave them dangling):

prompt_versions -- prompt entity isn't cascaded in v1 (epic open question)
optimization_id -- optimization entity is cascaded in Slice 4

Spans cascade with their parent trace; tree ordering is preserved via
sort_spans_topologically (existing helper from imports/) so parent_span_id
remap entries always exist by the time a child span is processed.

CLI surface

--exclude-experiments Skip the cascade entirely (dataset moves alone)

Validation rule: --exclude-versions requires --exclude-experiments,
because experiments reference specific dataset versions and a
current-items-only copy doesn't preserve the version IDs the experiments
need.

Reuses

recreate_experiment widened with optional target_* params (migrate
path); import-from-disk path unchanged
ReplayResult shape mirrored by ExperimentCascadeResult
execute_plan_loop wraps CascadeExperiments via existing audit-bracket
sort_spans_topologically preserves parent_span_id remap ordering

Out of scope (Slice 4, OPIK-6417)
- Cross-project dependent detection + reporting
- Optimization entity cascade + optimization_id remap
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- test(migrate-experiments): unit tests for experiment + trace/span cascade
Adds tests covering Slice 3's cascade in isolation. Mocks the REST surface
(find_experiments, stream_experiment_items, get_trace_by_id, create_traces,
get_spans_by_project, create_spans) and the recreate_experiment client path
so each acceptance criterion has a focused assertion:
- empty cascade no-ops
- full fidelity copy (name, tags, type, evaluation_method, inline prompts)
- prompt_versions stripped on destination
- optimization_id stripped on destination
- FK remap correctness (dataset_version_id, project_name, trace_id)
- span tree topological order preserves parent_span_id remap
- missing trace / missing dataset_item handled without crashing
- trace_id_remap accumulates across experiments sharing a trace
- id=None experiment raises ExperimentCascadeError
- planner gate (--exclude-experiments omits the cascade action)
- CLI validation (--exclude-versions without --exclude-experiments fails)
- dataclass / module-shape smoke checks
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- test(migrate-experiments): e2e test for multi-version dataset cascade
Adds an end-to-end test exercising the full Slice 3 pipeline against a
real Opik backend:
- Source: dataset with 2 versions under source_project; one experiment
  attached to v1's items; one trace + one LLM span per item.
- Action: opik migrate dataset --from-project
  --to-project
- Assertions:
  - destination dataset exists under --to-project
  - destination experiment exists, references the destination
    dataset's id, has a remapped dataset_version_id
  - destination experiment items use FRESH trace ids (disjoint from
    source trace ids) -- proves the cascade minted new ids rather
    than reusing source ones
  - each destination trace exists in --to-project and carries at
    least one span (the source had one LLM span per trace)
  - --exclude-experiments leaves no destination experiments
Skipped unless an Opik backend is reachable -- the e2e suite runs
against the docker-compose stack per
.agents/skills/python-sdk/testing.md.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- refactor(migrate): drop --exclude-versions/--exclude-experiments + slice-1 dead code
Per review feedback, the user-facing CLI shouldn't expose either flag --
power users won't need them and they encourage broken combinations. Since
the feature branches all stack into a single eventual merge, the planner-
level paths that only existed to support those flags are dead code too.

Removed from the CLI surface (main.py):
- --exclude-versions
- --exclude-experiments
- the "you must pair them" UsageError validation
Removed from the planner / executor (now-unreachable):
- build_dataset_plan(exclude_versions, exclude_experiments) parameters
- CopyCurrentItems dataclass + executor branch + helper
- CopyTestSuiteConfig dataclass + executor branch + helper
- the ValueError that gated the bad flag combination
Test changes:
- test_migrate_dataset_exclude_versions.py renamed to
  test_migrate_dataset_planner.py (the file now only covers planner
  edge cases + CLI help; nothing slice-1-specific)
- Dropped the entire TestMigrateDatasetCommand class (slice-1 CLI tests)
- Dropped 3 cascade-flag tests from
  test_migrate_dataset_experiments_cascade.py
Result: every migrate invocation now does the full sequence -- rename
source, create destination, replay all versions, cascade experiments +
traces + spans. There is no "dataset-only" or "versions-only" path
left in the code, in line with the simplified CLI surface.

243 unit tests pass (down from 262 -- the diff is the slice-1 dead-code
tests removed).

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- test+feat(migrate): port e2e infra, add trace/span feedback-score cascade
Two coordinated changes -- the e2e infrastructure surfaces a real
fidelity gap in the cascade (trace feedback scores not copied), and
fixing that gap is the natural prerequisite to the cascade e2e
scenarios that go on top.

E2E infrastructure (new under tests/e2e/cli/)

conftest.py:
- source_project_name / target_project_name ephemeral fixtures
with best-effort teardown
- run_migrate_cli helper (subprocess, real entrypoint, real env)
- create_dataset_shell + apply_changes for multi-version seeding
- chronological_versions, stream_items_wire, item_hashes,
display_order, normalize_evaluators, normalize_policy,
strip_be_managed_version_tags read-side helpers
- seed_experiment_with_trace_tree + verifiers for the cascade
scenarios (find_destination_experiment, destination_*)

test_migrate_dataset_e2e.py:
- TestMigrateDatasetVersionReplay::test_three_version_dataset_with_mixed_deltas_round_trips
- Single test exercises rename -> create destination -> replay
three versions with mixed adds/edits/deletes; asserts target
version count, per-version content set-equality under hash,
display order, audit log per-version records.

test_migrate_test_suite_e2e.py:
- test_test_suite_full_fidelity_round_trip
- Single test covers test-suite-specific fidelity: 4 versions,
per-version suite evaluators + execution_policy that change
across versions, per-version metadata + user tags, per-item
evaluators + policy + tags.

Trace + span feedback-score cascade (experiments.py)
- _copy_trace_feedback_scores: after the trace batch create, read
  feedback_scores off the source trace payload (already fetched
  during _copy_traces_and_spans) and re-emit via
  traces.score_batch_of_traces against the new destination trace id
  with the destination project.
- _copy_spans_for_trace: collect span-level feedback_scores from
  each source span's read payload and emit via
  spans.score_batch_of_spans against the new span ids after the
  span create batch.
Trace creates and span creates don't accept feedback scores on their
write payloads -- they live in separate per-trace / per-span tables
that the score_batch_of_* endpoints write to. The cascade now hits
both.

Unit tests
- _Trace stand-in: feedback_scores=None default so existing tests
  still pass.
- 3 new tests on TestCascadeExperiments:
  - test_trace_feedback_scores__copied_to_destination_trace
  - test_trace_without_feedback_scores__skips_score_batch_call
  - test_span_feedback_scores__copied_to_destination_spans
Removed
- sdks/python/tests/e2e/test_cli_migrate_experiments_cascade.py
  (superseded by tests/e2e/cli/* which uses the shared infra)
Known cascade gaps deferred to a follow-up

This commit does NOT include:
- Fix for _stream_experiment_items in experiments.py: it currently
  passes experiment_id= (wrong; endpoint takes experiment_name=)
  and treats the response as parsed objects (wrong; it's raw NDJSON
  bytes that need rest_stream_parser). Unit tests mock around this;
  e2e tests will fail until the fix lands.
- Full ExperimentItem write-side fidelity: the BE accepts input,
  output, feedback_scores, assertion_results, comments,
  execution_policy, description, status, total_estimated_cost,
  duration, usage on per-item creates -- the cascade currently
  forwards only id/experiment_id/dataset_item_id/trace_id.
- The actual cascade e2e scenarios on top of the existing dataset
  - test-suite tests (experiments + traces + spans assertions).
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(migrate): correct _stream_experiment_items REST call
The cascade was passing experiment_id= to
rest_client.experiments.stream_experiment_items, but the endpoint
takes experiment_name=. It also treated the response as already-
parsed items, when the endpoint actually returns Iterator[bytes]
of NDJSON that must go through rest_stream_parser.read_and_parse_stream
to become typed objects.

Both mistakes had been hidden by the unit-test mocks; they would have
surfaced on the first e2e run.

Changes
- _stream_experiment_items: takes experiment_name and routes the
  raw byte stream through rest_stream_parser into
  ExperimentItemPublic. The parser preserves BE-returned fields
  outside the typed schema on model_extra (input, output,
  feedback_scores, assertion_results, execution_policy, etc.), which
  the next commit will consume for full destination-side fidelity.
- cascade_one_experiment: pulls experiment_name off the source
  experiment and passes it to the streamer. Raises
  ExperimentCascadeError if the BE returns an experiment without a
  name (defensive; shouldn't happen in practice).
Unit-test mock updates

The cascade-rest-client builder in
test_migrate_dataset_experiments_cascade.py now keys
items_by_experiment by experiment NAME and emits NDJSON bytes
(one JSON line per item, terminated by \n) so the parser path is
exercised end-to-end. The cascade tests that used a default-named
_Experiment(...) were updated to key on "experiment"; the
two-experiment test was given explicit exp-a / exp-b names.

All 246 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- feat(migrate): full ExperimentItem write-side fidelity on cascade
The cascade was building destination experiment items from only the four
typed FK fields on ExperimentItemPublic (id, experiment_id,
dataset_item_id, trace_id). The BE returns a much richer per-item
payload via the extra="allow" schema; we now consume it through
model_extra and forward every field the destination
ExperimentItem write surface accepts:
- input / output -- per-item I/O snapshots (UI displays these)
- feedback_scores -- regular-dataset experiment scores
- assertion_results -- test-suite (evaluation_suite) pass/fail data
- execution_policy -- per-item runs / pass-threshold overrides
- description, status, usage, total_estimated_cost,
  duration
Some of these are dataset-type-specific in practice (feedback_scores on
regular datasets; assertion_results + execution_policy on test suites).
The cascade doesn't branch on dataset type -- the BE-returned shape is
the source of truth, and the is not None guard naturally drops
fields the source item never carried. Single code path; robust to
schema evolution. Comment on the constant spells this out.

Implementation
- _build_experiment_data in cli/migrate/datasets/experiments.py:
  builds per-item dicts via a new _experiment_item_to_dict helper
  that copies the FK fields and pulls each fidelity field off the
  ExperimentItemPublic.model_extra dict.
- recreate_experiment in cli/imports/experiment.py: on the
  migrate path, surfaces those same fields from item_data into
  the ExperimentItem(...) kwargs. Import-from-disk path is
  unchanged (gated on is_migrate_path).
Test coverage

New unit test:
test_experiment_item_fidelity__write_side_fields_forwarded

Seeds a source item with all ten fields populated, runs the cascade,
asserts that create_experiment_items was called with each field
forwarded verbatim (and FK fields remapped to the destination).

_ExperimentItem test stand-in gained an extras dict; the
NDJSON mock serialises it alongside the typed fields so pydantic's
extra='allow' surfaces them on model_extra.

All 247 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- test(migrate): cascade scenarios on dataset + test-suite e2e tests
Extends both existing e2e tests with experiment + trace + span
cascade assertions so the full pipeline -- rename, create destination,
replay versions, cascade -- gets exercised against a real backend
in one run per dataset type.

Dataset e2e (test_migrate_dataset_e2e.py)

On top of the existing 3-version replay scenario, the test now seeds
a regular-dataset experiment attached to v1 items:
- 3 traces (one per v1 item) each with a root + 1 LLM child span
  (exercises sort_spans_topologically and parent_span_id remap)
- Per-trace feedback scores (correctness + latency_p95) -- the
  feedback-score copy added in the previous commit
- Per-item input/output via per_item_extras
After the migration, asserts:
- Destination experiment exists with FKs remapped (dataset_id,
  dataset_version_id resolves to one of the target versions)
- Items use fresh trace ids (disjoint from source)
- Per-item input/output round-trip via model_extra
- Each destination trace carries 2 spans with parent_span_id remap
  pointing at the new root id
- Trace-level feedback scores re-emitted under the destination project
Test-suite e2e (test_migrate_test_suite_e2e.py)

On top of the existing 4-version suite replay scenario, seeds a
suite-driven experiment (type='trial',
evaluation_method='evaluation_suite') attached to v2 items:
- Per-item assertion_results with passed/failed/reason
- Per-item execution_policy (runs_per_item=3, pass_threshold=2)
- Per-item status, input, output
After the migration, asserts:
- Destination experiment type + evaluation_method preserved
- Per-item assertion_results round-trip (value/passed/reason all match)
- Per-item execution_policy round-trips
- Per-item status preserved
- Span tree shape preserved with parent_span_id remap
Conftest helpers
- seed_experiment_with_trace_tree gained per_item_extras: List[Dict]
  so callers can seed per-item write-side fields (input/output/
  assertion_results/execution_policy/status/...). Each per-item dict
  is splatted into the ExperimentItem(...) constructor;
  extra='allow' accepts unknown keys.
- destination_experiment_items corrected to take experiment_name
  and route through rest_stream_parser (same bug as the production
  cascade had; would have surfaced on the first e2e run).
All 247 unit tests pass; 2 e2e tests collect cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(migrate): thread source_project_name through cascade so source-side reads scope correctly
Running the e2e tests against a real backend (localhost:5174) surfaced
that the cascade's source-side reads stream_experiment_items and
get_spans_by_project 400 when called without project_name:

ApiError (400) Either 'project_name' or 'project_id' query
params must be provided

The cascade had no path to that value. Plumbing:
- CascadeExperiments action gains source_project_name: Optional[str];
  populated from source.project_name (which Slice 1 already resolves).
  May be None for workspace-scoped sources -- the BE accepts the
  omission there.
- executor._cascade_experiments forwards it to cascade_experiments.
- cascade_experiments -> cascade_one_experiment -> the two source-
  side helpers (_stream_experiment_items, _fetch_spans_for_trace)
  all carry the parameter; both REST calls now pass project_name on
  the request.
E2E test conftest mirrors the same fix for destination_experiment_items
and destination_spans_for_trace -- they hit the same BE endpoints
against the destination project for verification.

Unit-test mock signatures updated to accept the new kwargs.

Results against localhost:5174:
- tests/e2e/cli/test_migrate_dataset_e2e.py PASSED
- tests/e2e/cli/test_migrate_test_suite_e2e.py still fails on
  assertion_results round-trip (separate root cause, deferred to a
  follow-up: the BE's ExperimentItem write surface silently drops
  write-side fields like assertion_results / feedback_scores /
  execution_policy; the cascade needs to switch its source read to
  datasets.find_dataset_items_with_experiment_items Compare view
  and write assertions via the dedicated
  assertion_results.store_assertions_batch(entity_type='TRACE')
  endpoint scoped to the new trace id).
All 247 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(migrate): cascade assertion_results via store_assertions_batch + read source via Compare view
E2E discovery: ExperimentItem write-side fidelity fields
(input, output, feedback_scores, assertion_results,
execution_policy, description, status, usage,
total_estimated_cost, duration) are READ-ONLY on the BE Java
schema -- only id / experiment_id / dataset_item_id /
trace_id (+ project_name) appear in the Write JsonView. The BE
silently drops every other field on create_experiment_items. The
previous commit's "full ExperimentItem write-side fidelity" was
therefore dead code: writes succeeded but persisted nothing.

What the BE actually expects:
- Trace-scoped assertion results are written via the dedicated
  assertion_results.store_assertions_batch(entity_type='TRACE', entity_id=<trace_id>, ...) endpoint -- they're a separate entity
  table, not an ExperimentItem field. The Compare view surfaces them
  on ExperimentItemCompare on read.
- Trace + span feedback scores (already correct in a previous commit)
  go through score_batch_of_traces / score_batch_of_spans.
- Per-item input / output / usage / cost / duration are BE-computed
  read aggregates from the underlying trace + span entities; the
  cascade ensures those entities are correctly populated and the BE
  surfaces the aggregates on read.
Implementation
- recreate_experiment: dropped the migrate-path item_kwargs
  block that forwarded write-side fidelity fields. Destination
  ExperimentItems carry only the four FK fields; the rest is
  reconstructed from underlying entities.
- _stream_experiment_items -> _read_source_experiment_items:
  switched source read from the slim stream_experiment_items
  Public view to datasets.find_dataset_items_with_experiment_items
  Compare view, which exposes assertion_results. Endpoint takes
  experiment_ids as a JSON-array string.
- _copy_trace_assertion_results: new helper that re-emits source
  assertion results scoped to the new destination trace ids via
  assertion_results.store_assertions_batch(entity_type='TRACE', ...).
  Wired into _copy_traces_and_spans.
Test surface

Unit tests:
- Replaced the now-obsolete fidelity-writes test with two new tests:
  - test_trace_assertion_results__copied_to_destination_trace
  - test_no_assertion_results__skips_store_assertions_call
- Updated _cascade_rest_client mock to drive the Compare-view
  endpoint instead of stream_experiment_items, and pre-attach the
  assertion_results sub-mock (MagicMock auto-blocks the
  assertion_results attribute name).
E2E tests:
- destination_experiment_items switched to the Compare view
  (takes experiment_id + dataset_id) so tests can assert on
  ExperimentItemCompare.assertion_results directly via the typed shape.
- Test-suite seed (seed_experiment_with_trace_tree) now writes
  source assertions via store_assertions_batch(entity_type='TRACE')
  instead of the dead ExperimentItem.assertion_results field.
- Test-suite e2e asserts on the new Compare-view shape:
  dest_item.assertion_results[i].value round-trips the source's
  per-trace assertion name.
Verification against localhost:5174
- tests/e2e/cli/test_migrate_dataset_e2e.py PASSED
- tests/e2e/cli/test_migrate_test_suite_e2e.py PASSED
- 248 unit tests pass
Both e2e tests now exercise the full slice end-to-end against a real
backend.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- test(migrate-experiments): deep-equal source vs destination + align assertion names
Two coordinated changes that tighten the test-suite e2e:
1. New _cascade_comparison module
Adds compare_cascade(...) -- a field-by-field deep-equal walk of the
source and destination experiment + items + traces + spans, with all
remapped IDs (id / experiment_id / dataset_id / dataset_version_id /
dataset_item_id / trace_id / span_id / parent_span_id / project_id)
intentionally ignored.

What it compares:
- Experiment: name, type, evaluation_method, tags, metadata
  (after stripping project_name + prompt_versions; both intentionally
  differ between source and destination per epic decision).
  Asserts prompt_versions + optimization_id are stripped on
  destination.
- Items (Compare view): assertion_results + feedback_scores compared
  as sets. status skipped (BE-computed from assertion_results).
- Traces: name, input, output, metadata, tags, start_time, end_time,
  thread_id, error_info, ttft, feedback_scores (as set).
- Spans: tree-aware comparison -- both sides topologically sorted +
  stably sub-ordered, walked in lockstep. Per-span fields (name,
  type, input/output/metadata/model/provider/tags/usage/cost/ttft/
  timestamps/error_info/feedback_scores) compared, parent_span_id
  remap verified.
Wired into both e2e tests. Pairing strategy: both sides sorted by trace
name (assigned by the seed as "task-0", "task-1", "task-2" and
carried verbatim through the cascade), giving stable positional
correspondence.
1. Test-suite assertion names match suite-level evaluators
The previous seed used arbitrary runtime assertion result names
("check-0", "check-1", "check-2") that had no relationship to the
suite-level evaluators defined on the dataset versions. That made the
test data semantically incoherent: "this test suite ran some unnamed
assertions on items, here are results".

New shape mirrors what a real test-suite run produces:
- Each item carries a v1-judge runtime result (matches the v2
  suite-level evaluator's name); Q1 + Q3 pass, Q2 fails.
- Q2 additionally carries a q2-item-judge runtime result (matches
  Q2's per-item evaluator override).
This is the data shape an opik.evaluate() call against the suite
would produce -- one assertion result per evaluator that ran on the
item. Makes the test self-documenting and ties the runtime layer
(Slice 3's cascade) to the evaluator-definition layer (Slice 2's
replay).

Both tests verified against localhost:5174 -- 2/2 pass with deep-equal
active. 248 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(migrate): address PR review on slice 3 cascade
- Scope source-side span reads per-experiment via source_experiment.project_id
  instead of plumbing dataset-level source_project_name through the cascade
  (cross-project experiments referencing the same dataset legitimately live
  in different projects)
- Document ExperimentCascadeResult fields in cascade_experiments Returns
- Fix progress bar's final-tick 2/1 display (label="done" skips the +1
  mid-loop offset) and drop the post-cascade overwrite that would have
  dropped the bar below 100% when experiments were skipped
- Use generate_project_name("e2e", name) helper for PROJECT_NAME in
  the migrate e2e modules
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(migrate): gate v1 followup on is-not-None, not truthiness
needs_version_field_followup used bool(metadata)/bool(user_tags), which
treats explicit metadata={} / user_tags=[] as "absent" and skips the
follow-up apply_dataset_item_changes call. An explicit empty value on
the source means "clear the field" and must round-trip to the target;
silently skipping diverges from the delta path which already gates on
is not None for these same fields.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(migrate): scope span reads per-trace, not per-experiment
ExperimentItem.project_id is BE-derived from the trace, not validated
against the experiment, so an experiment's traces can legitimately live
in different projects from the experiment (and from each other). Using
the experiment's project_id for the span-list query would silently drop
spans of traces in any other project.

Switch the per-trace span read to use each trace's own project_id (already
fetched as part of get_trace_by_id). Experiment-level source_project_id
remains as a defensive fallback for the unlikely case where a trace's
project_id field is null. Adds a unit test covering an experiment whose
two traces live in distinct projects Y and Z, asserting the span query
uses Y/Z (not the experiment's project X).

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(migrate): rate-limit wrap + evaluate-shape e2e + empty/trace-origin coverage
Addresses PR review feedback (alexkuzmik):
- fix: wrap client._rest_client.experiments.create_experiment_items in
  imports/experiment.py:625 with ensure_rest_api_call_respecting_rate_limit
  -- the only unwrapped write in the migrate/import path. Cascade and
  version-replay writes were already wrapped.
- test (e2e): add a new module test_migrate_dataset_evaluate_shape_e2e.py
  that seeds via the natural opik.evaluate(...) flow rather than
  low-level REST. Two evaluations against the same dataset, the second
  with traces under a DIFFERENT project from the first -- pins the
  cross-project cascade behavior and the per-trace project_id span
  scoping shipped in the previous commit. Complements (doesn't replace)
  the existing low-level e2e which retains precise per-version delta
  coverage evaluate can't reproduce.
- test (unit): cover the empty-experiment-items edge case
  (test_experiment_with_zero_items__recreates_empty_shell) -- a
  source experiment whose items list is empty must not crash the
  cascade and should recreate the empty shell at the destination.
- test (unit): cover datasets-built-from-traces
  (test_replay__items_built_from_traces__source_and_trace_id_round_trip)
  -- items with source="trace" / source="span" keep their source
  - trace_id + span_id fields at the destination. Trace-id refs
    stay valid because traces are workspace-globally addressable;
    migration runs within one workspace.
Durability feedback (alexkuzmik) is deferred to a separate offline
discussion -- audit log already captures completed actions for manual
recovery; resume-from-audit-log is Slice 4 / OPIK-6417 territory.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(migrate): wrap all migration REST reads with rate-limit helper too
Extending PR review fix scope per alexkuzmik: "double check other places
in the code, especially the ones related to dataset items/experiment
items/spans/traces retrieval or upload".

Previously the convention was "writes wrapped, reads fail fast on 429".
That signal-loud-on-load argument is too strict mid-cascade: aborting
a long-running migration because a read hit a 429 wastes the work
already done. experiments.py was already wrapping reads "defensively"
for this reason; now we apply the same to:
- resolver.py: find_datasets (paginated pre-flight), get_project_by_id
  (name resolution), find_projects (did-you-mean suggestions)
- executor.py: get_dataset_by_identifier (read-back after CreateDestination)
- version_replay.py: list_dataset_versions (both call sites),
  stream_dataset_items (both call sites)
The outdated "Reads stay unwrapped" comment in executor.py is updated
to document the new convention: every REST call in the migrate path
is wrapped.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(migrate): PR review polish — verifier layer, fixture conventions, dedup
Addresses three baz-reviewer comments on the previous push:
- test (e2e): switch destination-side assertions to the shared verifier
  layer (tests.e2e.verifiers.verify_experiment / verify_trace) per
  .agents/skills/python-sdk/SKILL.md. The verifiers have built-in
  synchronization.until retry loops covering the BE's eventual-
  consistency window right after migrate completes -- spurious failures
  from not-yet-readable rows are eliminated. Bespoke
  destination_feedback_scores_for_trace call dropped from the test
  body; find_destination_experiment + destination_experiment_items
  retained for the two name->id lookups only.
- test (e2e): drop the unused trace_project_name_two fixture (V2's
  opik.evaluate(...) ignores per-call project_name overrides, so
  the cross-project shape it tried to produce is unreachable from the
  SDK). Per AGENTS.md, project names must come from
  generate_project_name("e2e", __name__) (already declared at module
  top); raw random_chars() for project names is forbidden. Reuse the
  shared experiment_name fixture from tests/e2e/conftest.py and
  derive E2 as f"{experiment_name}-second" instead of inventing
  parallel per-test fixtures.
- refactor (version_replay): extract _stream_version_items_raw(...)
  so the rate-limit-wrapped REST stream + read_and_parse_stream block
  isn't duplicated between _load_version_items and
  _read_back_target_items. Both callers now share the call/parse and
  keep their divergent post-processing (id-keyed vs hash-keyed).
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(migrate): unbreak evaluate-shape e2e after verifier refactor
Two assertions added in c6bb3d646 were too strict against actual
cascade behavior:
- recreate_experiment deliberately injects project_name into
  the destination experiment's metadata (recorded for future imports
  to re-derive the project context). verify_experiment's metadata
  check is exact-equality, so include project_name=target_project_name
  in the expected metadata.
- verify_trace(feedback_scores=[...]) requires exact-equality on
  the feedback_scores list, but the equals scorer produces a mixed
  set across items (1.0 for "Capital of France?/Paris", 0.0 for the
  others). Drop the per-trace feedback_scores argument so
  verify_trace still checks project_name + retry behavior; the
  experiment-level feedback_scores_amount=1 aggregate already pins
  that the cascade re-emitted the equals_scoring_function score.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- perf(migrate): batch span writes across all traces in an experiment
Previously _copy_traces_and_spans called _copy_spans_for_trace per
source trace, which issued one create_spans (and optionally one
score_batch_of_spans) HTTP request per trace. With ~1000 source
traces × ~4 spans each that's ~1000 tiny create_spans batches of 4
spans, each one consuming a workspace rate-limit token. Empirically the
per-trace pattern was the dominant cost in the cascade phase: each
experiment spent tens of minutes in the per-trace span write loop even
though the actual span payload is small.

Restructure the span phase so writes batch across ALL traces in the
experiment, not per-trace:
- Split _copy_spans_for_trace into _prepare_spans_for_trace,
  which fetches source spans + mints the per-trace span_id remap + builds
  the per-trace contribution to the cross-trace span_writes and
  span_feedback_batch lists, but DOES NOT issue any writes itself.
- _copy_traces_and_spans accumulates the per-trace contributions
  across the whole experiment, then issues a single batched loop over
  span_writes (chunked at _SPAN_BATCH_SIZE=100) and a single
  batched loop over span_feedback_batch (chunked at
  _FEEDBACK_BATCH_SIZE).
REST call count for the span phase drops from O(num_traces) to
O(total_spans / _SPAN_BATCH_SIZE). For the typical 1000-trace × 4-span
experiment that's ~1000 create_spans calls -> ~40, roughly a 25x
reduction. Wall-time speedup is larger than the call ratio because the
workspace rate-limit retry sleeps compound multiplicatively with call
count.

Correctness invariants preserved:
- Topological sort + span_id remap stays PER-TRACE. Parents must precede
  children within a trace tree, and span ids only collide within a
  trace.
- Spans across different traces are independent at the API layer:
  parent_span_id references stay within a trace, and the
  create_spans payload accepts mixed-trace spans.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- refactor(migrate): switch cascade writes to high-level opik client API
Move every write in the cascade off rest_client.X.create_* /
score_batch_of_* / store_assertions_batch and onto the
high-level Opik client API:
- Traces -> client.__internal_api__trace__(...). Uses the
  __internal_api__ variant rather than public client.trace()
  so we can pass source="experiment" (the public trace() defaults
  to source="sdk", which would diverge from what
  opik.evaluate(...) wrote on the source).
- Trace feedback scores -> client.log_traces_feedback_scores(...)
  (batched, routes through streamer).
- Trace assertion results -> client.log_assertion_results(...).
- Spans -> client._streamer.put(messages.CreateSpanMessage(...))
  directly. This bypasses the public client.span() and
  span_client.create_span() paths because both call
  helpers.add_usage_to_metadata which merges usage into
  metadata["usage"] -- a user-facing convenience for fresh writes
  that breaks the cascade's round-trip metadata-fidelity contract.
- Span feedback scores -> client.log_spans_feedback_scores(...).
Add explicit client.flush() between phases (traces -> assertions /
spans -> span feedback scores) because the streamer doesn't guarantee
inter-message ordering within a flush window.

Drops the explicit batching helpers, wire-shape builders, and
rate-limit wrapping for these writes -- the streamer handles batching
- retry internally.
Reads stay on rest_client where the high-level wrapper would require
extra lookups (project_id -> project_name) or doesn't exist (Compare
view). Each remaining rest_client read carries a docstring explaining
why.

Unit tests updated to assert on the high-level surfaces:
- client.__internal_api__trace__.call_args.kwargs
- client._streamer.put.call_args_list for CreateSpanMessage
- client.log_* for feedback scores and assertions
251 unit tests pass; 3 e2e tests pass against localhost:5174
(deep-compare verifies span.metadata round-trips exactly).

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- refactor(migrate): route remaining cascade reads through high-level Opik client
Following 726935f9c which routed the cascade writes via the high-level client,
this commit pushes the read path further toward the public SDK surface:
- traces.get_trace_by_id -> client.get_trace_content (paper-thin
  wrapper, same call underneath).
- datasets.find_dataset_items_with_experiment_items ->
  api_objects.experiment.rest_operations.find_experiment_items_for_dataset
  (SDK-blessed read; preserves assertion_results).
  max_results=sys.maxsize lets the helper's underlying pagination
  walk every page -- a migration must be lossless.
- spans.get_spans_by_project -> client.search_spans with a
  per-cascade project_id -> project_name cache (typically 1 lookup
  per experiment) so search_spans' name-only filter doesn't cost a
  round-trip per trace.
- projects.get_project_by_id -> client.get_project. To make that
  swap clean, the planner/resolver now take client: opik.Opik instead
  of rest_client: OpikApi and drop down to client.rest_client only
  for the calls that have no high-level wrapper.
Also pulls executor's _replay_versions destination lookup from
rest_client.datasets.get_dataset_by_identifier to client.get_dataset.

The 12 remaining direct Fern calls in cli/migrate/ are documented inline
with the specific reason each one stays (either no SDK wrapper exists, or
the wrapper drops fields the migration needs to preserve -- tags,
visibility, type, project_id, full ExperimentPublic shape, etc.).
version_replay.py gets a module-level note explaining why all 7 of its
calls stay on the Fern surface (BE-shape parameters the high-level
Dataset helper hides: base_version chaining, override=true,
batch_group_id, clear_execution_policy, raw tag preservation).

Tests: 251/251 CLI unit tests pass. _client_with_recreate_capture now
wires client.get_trace_content, client.search_spans, and
client.get_project to delegate to the rest_client stubs so per-test
fixtures keep working unchanged. _planner_client is a new helper that
wraps a rest_client mock as an opik.Opik-shaped client for the planner
tests.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- feat(migrate): nested progress bar for experiment cascade
Long-running experiments (hundreds of traces + spans each) made the outer
"experiments (N/M)" bar look frozen for minutes between ticks. Add a
second, inner Rich progress bar that ticks on every per-experiment
read/write/flush phase so the user always sees motion.
- cascade_experiments gains an inner_progress_callback parameter
  alongside the existing progress_callback. Same
  (completed, total, label) shape; total is computed per
  experiment as 1 + 2N + 5 for N traces (one tick for the
  Compare-view read, two per trace -- one read-and-emit, one
  span-fetch-and-emit -- plus five fixed phase ticks for flushes /
  feedback-score batches / assertion batches / recreate).
- _InnerProgress adapter encapsulates the per-experiment counter:
  clamps overshoots at total (idempotent-skip can remove traces, so
  the pre-computed estimate may run hot) and snaps to 100% on
  finish() so the bar finishes cleanly regardless.
- Executor's _cascade_experiments adds a second Rich task to the
  same Progress instance; outer task tracks experiments, inner task
  resets at every outer tick so its 0-100% sweep represents the current
  experiment's work. Inner labels describe the phase ("trace 47/150",
  "spans for trace 47/150", "flushed traces", etc.) so the user sees
  what is actually happening.
Tests: 252/252 CLI unit tests pass. Added
test_inner_progress_callback__fires_per_trace_and_phase which pins
the contract (each phase fires at least once, final tick snaps to 100%
with a terminal label).

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(migrate): narrow types on cascade write payloads for mypy
After merging main, mypy started enforcing the SDK's TypedDict shapes on
the high-level write surface (__internal_api__trace__,
CreateSpanMessage, log_traces_feedback_scores,
log_spans_feedback_scores, log_assertion_results). The cascade
was building plain dict[str, Any] containers that structurally match
those TypedDicts but mypy couldn't narrow them automatically.
- Annotate _to_error_info_dict return as Optional[ErrorInfoDict]
  and cast at the boundary -- the runtime shape (exception_type,
  traceback, optional message) matches the TypedDict's required keys but
  mypy can't infer that from dict / model_dump results.
- Type _emit_spans_for_trace's feedback-score return as
  List[BatchFeedbackScoreDict] and the local entry builder
  inside the loop as the same TypedDict literal.
- Same treatment for _log_trace_feedback_scores' batch and entry
  builders.
- Same for _log_trace_assertion_results with
  BatchAssertionResultDict; the status field is
  Literal["passed", "failed"] so the ternary result is hoisted into
  a typed local to keep mypy's narrowing intact.
No behavior change -- the runtime dicts already had the right shape;
this just spells the types out for the type checker. Pre-commit's full
--all-files mypy + ruff + ruff-format all pass locally now (matching
how CI invokes pre-commit).

Tests: 252/252 CLI unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- perf(migrate): bulk-read traces via search_traces(filter=experiment_id) per experiment
Replaces the per-trace get_trace_content loop with a single
client.search_traces(filter="experiment_id=<id>", truncate=False) call
per source experiment. The BE exposes TraceField.EXPERIMENT_ID as a
first-class filter (joined through experiment_items), so one paginated
read returns every trace linked to the experiment regardless of trace
count.

For the 1000-trace staging run this collapses 1000 get_trace_content
calls (each costing one searchTraces:{workspaceId} rate-limit token)
to 1 call -- eliminating the trace-half of the 30-then-pause-30 rate-limit
pattern. The span-read half stays per-trace (no experiment_id filter
on SpanField); the rate-limit pressure on spans is unchanged.
- _copy_traces_and_spans gains a source_experiment_id parameter
  and a new "Phase 1" bulk read at the top, with a defensive fallback to
  per-trace get_trace_content for any trace_ids that the join-based
  filter doesn't return (rare; preserves correctness when
  experiment_items is inconsistent).
- truncate=False is required: the SDK's client.search_traces
  default of True would replace inline base64 image data in
  input/output/metadata with the placeholder "[image]" and break
  round-trip fidelity.
- max_results=sys.maxsize lets the SDK's internal pagination
  (PAGE_SIZE=2000 via last_retrieved_id) walk every page; the cap is
  a caller-side "stop at N" UI knob, not a safety limit -- a migration
  must be lossless.
- New _resolve_project_name helper lifts the project_id -> project_name cache earlier so the bulk trace read and the per-trace
  span reads share the same cache. _fetch_spans_for_trace is
  refactored to call the helper.
- Inner progress total adjusted: was 1 + 2N + 5 (items+per-trace-
  read+per-trace-span+fixed), now 1 + 1 + 2N + 5 adding one tick for
  the bulk trace read.
Tests: 253/253 CLI unit tests pass (was 252). Added
test_bulk_trace_read__one_search_traces_call_per_experiment which
pins the contract: exactly one search_traces call per experiment,
filter is experiment_id = "<id>", truncate=False, fallback to
get_trace_content not used when bulk read returns the expected set.

_client_with_recreate_capture and _cascade_rest_client are
extended to wire client.search_traces against the test's existing
traces_by_id map automatically, so the bulk path is exercised by
default; tests that want to exercise the fallback path can omit the
map.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- perf(migrate): bulk-read spans via search_spans(filter_string=time-bounded) per experiment
Replaces the per-trace search_spans(trace_id=...) loop with a single
client.search_spans(filter_string="start_time >= X AND start_time <= Y", project_name=..., truncate=False) call per source experiment. The
[from_time, to_time] window is derived from the bulk-read traces'
own start_time/end_time (+/- a 5-minute buffer for late-arriving
spans and clock skew), so the bulk read is scoped to the experiment's
time window -- not the entire project's span history.

For the 1000-trace staging run this collapses 1000 search_spans
calls to 1 -- eliminating the second half of the 30-then-pause-30
rate-limit pattern on search_spans:{workspaceId}. Combined with the
prior bulk trace read (ff77be7ed), reads per experiment are now
constant: 1 trace search + 1 span search, regardless of trace count.

Implementation notes:
- _bulk_fetch_spans_for_experiment (new) issues the single read,
  filters client-side by span.trace_id in expected_trace_ids to
  discard over-fetch from concurrent activity in the same time window,
  and returns a {trace_id: [SpanPublic]} bucket dict.
- _emit_spans_for_trace (refactored) now takes pre-fetched
  source_spans instead of fetching per trace. The
  topological-sort + per-trace span_id_remap logic is unchanged --
  parents still precede children within each trace tree.
- _fetch_spans_for_trace deleted.
- New _compute_span_time_window derives (min(start_time)-5m, max(end_time | last_updated_at)+5m) from the trace bucket; returns
  None when no usable timestamps so the bulk read can fall back
  to "no time bound" (still correct, just over-fetches more).
- New _to_iso_z formats UTC datetimes as "2024-01-01T00:00:00Z"
  to match the BE filter grammar's date-time literal format.
- Zero-bucket warning: any expected trace_id with no matched spans is
  logged with a hint that the destination trace is still copied but
  without spans. Could be legitimate (genuinely no spans) or a missed
  late-arriving span; we don't distinguish today.
Trade-off (documented in test):
- The bulk read scopes to the experiment's project_id. If an
  experiment's traces live in different projects (rare; BE allows it
  but opik.evaluate(...) never produces it), spans for the
  cross-project traces are silently missed. The prior per-trace path
  used each trace's own project, defending against this case.
  test_bulk_span_read_is_scoped_to_experiment_project pins the new
  contract; the old test_span_read_uses_trace_project_not_experiment_project
  is replaced.
Inner progress total adjusted to 2N + 8 (added one tick for the bulk
span read between the assertion-log and per-trace span-emit phases).

Tests: 253/253 CLI unit tests pass. Test fixture extended so
_Span.__getattr__ exposes _fields items as attributes, and
_client_with_recreate_capture wires client.search_spans to
return the flat union of every test trace's spans (so the cascade's
client-side trace_id filter exercises correctly).

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- fix(migrate): unblock 10k-scale runs (chunk items, quiet logs, milestone trail)
Four small fixes motivated by an upcoming 10k-item × 3-experiment staging
test. Each one would have blocked or degraded the run; bundling them as
they're each <50 lines and all UX/correctness on the same cascade path.
- Chunk create_experiment_items to respect the BE's
  ExperimentItemsBatch @Size(min=1, max=1000) cap (see
  apps/opik-backend/.../ExperimentItemsBatch.java). For experiments
  with >1000 items the prior single-call POST returned 400 and aborted
  the cascade mid-experiment. Now we split via
  sequence_splitter.split_into_batches and post each chunk through
  the existing rate-limit-aware retry. Fixes both
  opik migrate dataset and opik import experiment for large
  experiments. Constant _EXPERIMENT_ITEMS_INSERT_BATCH_SIZE is
  documented as tracking the BE cap, not pinned in test names.
- Suppress the streamer's per-message ingestion-rate-limit INFO log
  during opik migrate dataset. The queue_consumer logs every 429
  retry with the full HTTP response headers dict (cookies, AWS LB
  markers, rate-limit telemetry) -- on a cascade that writes thousands
  of traces and spans, that drowns out the Rich progress bars with
  hundreds of long lines. Localized via _quiet_streamer_rate_limit_logs
  context manager: WARNING level for that one logger for the duration
  of execute_plan, restored on exit. Other SDK consumers and other
  CLI commands are unaffected.
- Print persistent milestone lines as the cascade walks per-experiment
  phases. The inner Rich bar overwrites itself, so per-phase history is
  otherwise lost; for a 100k-item experiment the bar might sit on
  "fetched 100000 traces in bulk" for 10+ minutes with no scrollback
  indication of which phase is slow. Now each milestone -- bulk reads,
  flushes, log-scores, log-assertions, recreate/skipped -- emits a
  ✓ <label> (took N.Ns) line. Per-trace ticks (trace 47/1000,
  spans for trace 47/1000) stay bar-only because they fire
  thousands of times per experiment and would flood the terminal.
  Scrollback volume is bounded by experiment count (~8 lines each),
  not item count. Per-milestone wall-clock anchored to the previous
  milestone of the SAME experiment (reset at every outer tick).
- Pin the chunking contract with test_recreate_experiment_chunks_items_within_be_cap
  -- reads the cap from the module constant rather than hardcoding 1000,
  exercises cap * 5 // 2 items so the multi-batch + partial-last-
  batch path is covered. Asserts ceil-division batch count, every batch
  within cap, and total items conserved.
Tests: 254/254 CLI unit tests pass (was 253). Pre-commit clean.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- feat(migrate): cover cross-project experiments via streamExperimentItems
Experiments may carry traces in projects different from the experiment's
own project_id (BE populates experiment_items.project_id from
traces.project_id at write time). search_traces / search_spans are
single-project-scoped, so the old cascade silently dropped traces living
outside the experiment's project.

Add _discover_trace_projects which streams the experiment's items via
streamExperimentItems (no JsonView restriction, so project_id
is surfaced per row), groups source trace_ids by their actual project_id,
and drives one bulk search_traces + search_spans per distinct
project. The experiment's own project_id is the fallback when the
stream's per-row project is null.

E2E coverage: a new test seeds an experiment with traces spread across
three distinct source projects (A: same-project, B + C: cross-project)
and asserts all four destination experiment items round-trip into the
single target project with fresh trace ids.
- fix(migrate): paginate per-version item stream to fix phantom deletions
/items/stream caps each response at 2000 items
(DatasetItemStreamRequest.steamLimit @Max(2000) + 2000 default).
_stream_version_items_raw called it once via
read_and_parse_stream, so versions with >2000 items were silently
truncated. _compute_delta then misclassified the missing tail as
deletions on every subsequent source version -- a 7-version source
with 500/1000/.../3500 items replayed onto a target stuck at 2000,
with v5/v6/v7 each appearing to "delete" 500 items they never had.

Switch to read_and_parse_full_stream so the helper walks the
lastRetrievedId cursor until a short page comes back, then add
focused unit tests pinning both the single-page (under cap) and
multi-page (over cap) shapes -- the existing replay tests stub the
parser so they wouldn't have caught a missing paginate-on-cap.
- fix(migrate): chunk _mint_v1 at BE 1000-item cap, share batch_group_id
DatasetItemBatch.@Size(max=1000) rejects oversized
create_or_update_dataset_items payloads with HTTP 422. _mint_v1
was sending a source v1's entire item list in one POST, so any source
dataset whose v1 carried >1000 items (legal -- apply_dataset_item_ changes has no @Size cap and produces such versions) failed at the
target with items size must be between 1 and 1000.

Chunk into batches of 1000 items via sequence_splitter.split_into_ batches and send every chunk under the SAME batch_group_id;
DatasetItemService.applyToLatestVersion rolls multiple POSTs with a
shared group id into one target version, so target v1 stays a single
version regardless of source size.

Test pins both invariants: the call count splits exactly at the cap,
all chunks share one batch_group_id, and the reversed display-order
sequencing survives across chunk boundaries.
- feat(migrate): print wall-clock elapsed on success and failure
10k-scale migrate runs can take many minutes; the success / failure
line previously gave no sense of how long it actually ran. Capture a
time.monotonic() anchor at the start of migrate_dataset_command
and render the delta on both exit paths -- success appends
Took 5m 12s., failure appends (after 5m 12s) so operators can
tell whether a 422 fired at the cascade tail vs the first _mint_v1
POST.

Formatter shows 12.3s under a minute, integer Mm Ss past, and
Hh Mm Ss once you cross an hour -- once you're in minutes you don't
care about fractional seconds.
- refactor(migrate): rename project_ids_by_trace -> trace_ids_by_project
The dict's key is project_id and the value is Set[trace_id]
(the trace_ids that live in that project). The old name read as
"project_ids keyed by trace," which is the opposite shape — a reader
seeing project_ids_by_trace[trace_id] would expect a project_id
back but actually gets a set of trace_ids. trace_ids_by_project
matches the actual indexing: [project_id] -> Set[trace_id].

Same data, same goal (discover every project the experiment's traces
live in), name just describes the shape correctly. Addresses PR #6658
review comment 3234828874.
- test(migrate): assert per-span fidelity per source project in cross-project e2e
The cross-project cascade discovers projects via streamExperimentItems
and loops per project for both search_traces AND search_spans.
The e2e already verified each destination trace lands in the target
project, but didn't pin that spans came along with the cross-project
traces -- if the per-project search_spans loop skipped B or C, the
trace would land at the destination with zero spans and the test would
still pass.

Read destination spans per item via destination_spans_for_trace and
assert the expected per-trace span count, keyed on trace name as the
stable handle for "which source project did this come from":
task-N -> A, 2 spans (root + LLM child)
task-cross-{b,c} -> B/C, 1 span (root)

Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
下载附件
- 源代码 (ZIP)
- 源代码 (TAR.GZ)

0 发布 8,340 Git 标签

[OPIK-6416] [SDK] feat: cascade experiments and traces and spans in opik migrate dataset (#6658)

Cascade-copy semantics

Cross-project follow

FK remap during recreation

CLI surface

Reuses

Out of scope (Slice 4, OPIK-6417)

E2E infrastructure (new under tests/e2e/cli/)

Trace + span feedback-score cascade (experiments.py)

Unit tests

Removed

Known cascade gaps deferred to a follow-up

Changes

Unit-test mock updates

Implementation

Test coverage

Dataset e2e (test_migrate_dataset_e2e.py)

Test-suite e2e (test_migrate_test_suite_e2e.py)

Conftest helpers

Implementation

Test surface

Verification against localhost:5174