github-featured / comet-ml--opik

发布

0 发布 8,340 Git 标签

RSS 订阅源
  • 2.0.60-5579 9544153ece
    比较

    [OPIK-5269] [DOCS] docs: add page on resuming interrupted evaluations (#6950)

    frostbyte_neo 发布于 2026-06-09 10:34:29 +00:00

    • [OPIK-5269] [DOCS] docs: add page on resuming interrupted evaluations

    Documents opik.evaluate_resume(experiment_id, ...): when it applies,
    the replayed-vs-reconstructed contract, what happens with custom
    samplers / explicit dataset_item_ids, and which existing tools to
    reach for when resume isn't the right fit (evaluate_experiment for
    re-scoring; a fresh evaluate() for new items).

    The page lands under Evaluation → Advanced, between Datasets &
    Experiments and Manage datasets — placement that matches when a user
    is most likely to need it.

    Pairs with the SDK feature shipping in #6941.

    Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

    • docs(resume): narrow scope note to evaluate() only

    Drop the parenthetical mention of evaluate_prompt and
    evaluate_optimization_trial — keeps the note focused on the entrypoint
    users actually reach for.

    Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

    • docs(resume): use canonical 'run' vocabulary; simplify resume requirements; reflect missing-runs-only semantics
    • Terminology: replace SDK-internal 'trial' with the user-facing 'run'
      vocabulary used in evaluation/concepts.mdx ("each item is run N
      times", "a run passes if...").
    • "When you can resume" section: drop the JSON-blob / experiment_config
      internals; keep the two requirements the user actually needs to know
      (resume-aware SDK + versioned dataset) and point at the
      ExperimentNotResumable error for everything else.
    • "What gets replayed vs reconstructed" multi-run paragraph: replace
      the old "redo all if any trial is missing" rule with the new
      "replay only the missing runs" semantics that ship with #6941.
    • "Wrong tool" bullets: tighten wording (existing-run → existing
      experiment; original run → original evaluation) so 'run' consistently
      means a single execution.

    Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

    • docs(resume): make the page scannable; cut implementation chatter

    Reorders the page around what a user wants to scan in order: what it
    is → quick start → what it preserves vs replays → requirements →
    same-machine caveat → when it's the wrong tool → reference.

    Drops:

    • The detailed "outcome of the original run → what resume does" table
      (replaced with three short bullets).
    • The "engine writes the trace's output only at the end of the happy
      path" implementation note — users don't need to know the marker
      mechanism.
    • The "JSON blob in experiment_config / resume reads it back" plumbing.
    • The two-script "Putting it together" section (the quick start already
      shows the call).
    • The local-checkpoint code block and the long explanation of why a
      sampler needs one — replaced with two short sentences.

    Result: 80 lines vs 164. Same content, faster to scan.

    Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

    • docs(resume): move same-machine caveat below 'when it's the wrong tool'

    It's a niche caveat (only the sampler / explicit-ids paths hit it), so
    it belongs after the broader "wrong tool" decision points rather than
    in the main reading flow.

    Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

    • docs(resume): widen task-change caveat to metrics; link evaluate_experiment; drop Reference section
    • Third 'wrong tool' bullet now also covers metrics: providing the same
      task and scoring_metrics between calls is the caller's
      responsibility. The same already-completed-runs-keep-their-original-
      outputs warning applies to both.
    • Link to the Python SDK reference page for evaluate_experiment so
      users have a single click to learn the alternative.
    • Drop the Reference section at the bottom — the function and
      exception names already appear inline where users see them.

    Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

    • docs(resume): reorder — guidance first, requirements + same-machine caveat at the bottom

    New flow: intro → quick start → what resume does → when it's the wrong
    tool → requirements (now including the same-machine sampler caveat as
    a follow-up paragraph in the same section). Decision-help moves up,
    gotchas move down.

    Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

    Co-authored-by: Claude Opus 4.7 noreply@anthropic.com

    下载附件