-
[OPIK-7511] [SDK] [PYBE] fix: make no-improvement runs honest — perfect_score, mini-batch sizing, finish_reason (#7650)
frostbyte_neo 发布于
2026-07-31 12:11:06 +00:00 - [OPIK-7511] [SDK] [PYBE] fix: make no-improvement runs honest — perfect_score, mini-batch sizing, finish_reason
- perfect_score default 0.95 -> 1.0: it gates both GEPA's iteration skip and
(since #7591) the run-level ScoreThresholdStopper, so anything below 1.0
ends runs with a strong-but-imperfect baseline at zero candidates. Matches
the gepa package's own default. - Studio's GEPA reflection_minibatch_size is no longer pinned to 5: it scales
as min(max_trials, dataset_size, max(5, ceil(dataset_size / 5))) so coarse
0/1 metrics get a usable gradient; the env var stays as operator override. - finish_reason is now populated on every early-stop path: budget-exhausted
GEPA runs report "max_trials" instead of falling through to "completed",
the baseline-perfect skip writes finish_reason="perfect_score" into details
(BaseOptimizer + ParameterOptimizer), and the python-backend forwards
finish_reason to the Java backend as metadata.finish_reason alongside
scoring_health so OPIK-7458 can render the real stop cause.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7511] [SDK] style: ruff-format fix in gepa stop-condition tests
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7511] [SDK] [PYBE] fix: carry scoring_health on the baseline-perfect path + real e2e for finish_reason
Found by running the new e2e against a live stack: build_early_stop_details
never included scoring_health, so a perfect-baseline run completed with no
scoring_health in metadata — exactly the case the UI's 'No usable scores'
heuristic misfires on (OPIK_7458). Align it with build_final_result, which
always carries the key.The new e2e (tests/e2e/test_studio_finish_reason.py) drives the real
process_optimizer_job -> optimizer_runner subprocess -> gateway pipeline and
asserts finish_reason is returned by the subprocess AND persisted as
metadata.finish_reason, with scoring_health riding the same completion update.Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7511] [PYBE] test: gate finish_reason e2e on the pinned SDK containing the fix
The python-backend CI installs the released opik_optimizer pin, which
predates the SDK half of OPIK-7511 — the perfect-baseline path completes
without finish_reason there, failing the e2e on the pin rather than on the
backend code under test. Feature-detect via DEFAULT_PERFECT_SCORE (ships in
the same SDK changeset) and skip until the pin catches up; the test then
activates automatically.Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7511] [PYBE] fix: address code-review findings
- resolve_reflection_minibatch_size: a malformed env override warns and
falls back to the dataset-scaled policy instead of failing every run. - run_optimization takes the item count from load_and_validate_dataset
instead of re-fetching the whole dataset a second time. - extract_finish_reason logs unknown reasons instead of dropping them
silently, so a new SDK FinishReason surfaces in logs. - extract_completion_metadata combines both extractors; the runner's
forwarding block is now unit-testable (and unit-tested) instead of only
covered by the pin-gated e2e. - The e2e imports KNOWN_FINISH_REASONS instead of a third copy of the
allowlist.
Verified: 229 backend unit tests pass; the live e2e (real gateway LLM
calls) passes against this exact tree.Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7511] [SDK] [PYBE] fix: replace the max_trials minibatch cap with a budget-based one
The max_trials cap silenced exactly the scaling this PR introduces: with
the default max_trials=10 the reflection mini-batch was pinned at 10 for
any dataset above ~50 items. The cap's premise was false — the gepa
engine never gates a reflection iteration on the remaining budget (the
only budget read, metric_calls_remaining, feeds a callback event), so a
large batch runs fine, it just costs ~2*batch metric calls per iteration.- resolve_reflection_minibatch_size: drop the max_trials cap; keep the
dataset_size cap; add a budget cap that guarantees at least
GEPA_MIN_REFLECTION_ITERATIONS (5) reflection iterations within
max_metric_calls = max_trials * n_samples. - SDK: rewrite the 'GEPA reflection will not run' warning as
_warn_if_reflection_minibatch_exhausts_budget — it now reports the
real iteration estimate (max_metric_calls // (2*b)). - Validate OPTIMIZER_GEPA_REFLECTION_BATCH_SIZE at service startup
(integer >= 1, error names the var, value truncated) instead of
lazily warning-and-falling-back per run. - Bound dataset size counting to DATASET_SAMPLES (nb_samples fetch)
instead of materializing the full dataset. - Tests: new policy table incl. 1-item/tiny datasets and budget-cap
cases; warning tests assert the logged iteration count; bound the
NoImprovementStopper loop; rename new tests to the WHAT__CASE__RESULT
convention.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7511] [SDK] [PYBE] fix: scope perfect_score=1.0 to Studio runs, not the SDK default
The ticket scopes the perfect_score fix to Studio runs; changing the
SDK-wide DEFAULT_PERFECT_SCORE also silently changed behaviour for every
SDK user of every optimizer, and would not reach Studio until the
python-backend's opik_optimizer pin is bumped. Instead: revert the SDK
constant to 0.95 and have OptimizerFactory inject perfect_score=1.0 as a
constructor default (explicit run params still win) — effective on the
currently pinned opik-optimizer==3.1.0, no pin bump needed. The e2e's
feature probe for the SDK finish_reason half now keys on
MIN_EXPECTED_REFLECTION_ITERATIONS (same changeset) instead of the
reverted constant.Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7511] [PYBE] test: make the studio e2e runnable against a cloud backend
Three gaps kept tests/e2e from running anywhere but the unauthenticated
CI stack: the provider fixture was hardcoded to Anthropic (contradicting
the OPTSTUDIO_E2E_MODEL override the finish_reason e2e documents), and
the job message never carried opik_api_key, so on a cloud backend the
subprocess's status updates and gateway LLM calls authenticated with a
placeholder and got 401. The fixture is now provider-aware
(workspace_provider_key: derives the required provider from
OPTSTUDIO_E2E_MODEL, no-ops when it is already configured in the
workspace) and the job message passes the workspace API key when
OPIK_API_KEY is set — local CI stacks keep today's behaviour.Verified live: test_finish_reason_is_returned_and_persisted passes
against dev.comet.com (gpt-5-nano via the workspace OpenAI key) with the
factory-injected perfect_score=1.0 mechanism from the previous commit.Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7511] [SDK] [PYBE] fix: stop deciding runs on a single noisy eval
Live A/B runs exposed that raising perfect_score to 1.0 removed the
deterministic abort on a strong baseline but left a stochastic one a step
later. Measured on dev: repeating the SAME eval of the SAME prompt over
the SAME 20-item dataset scored 0.95/1.00 across 6 repeats on gpt-5-nano
and 0.90/0.95 on gpt-4o-mini (flipping even on unambiguous items), so
whether a run searched at all was luck. When the seed program's own full
eval happened to reach 1.0, gepa's ScoreThresholdStopper ended the run at
iteration zero and it was reported as finish_reason='perfect_score' while
the UI showed 'no improvement' at 0.95 — the exact dishonesty this ticket
is about.Two changes:
-
CandidateScoreThresholdStopper ignores program_full_scores_val_set[0]
(the seed's own eval), so only a CANDIDATE reaching the threshold can
stop the run. 'Baseline is already good enough' stays owned by
should_skip_optimization on Opik's baseline eval, which runs earlier, so
no stop path is lost. _resolve_gepa_finish_reason skips index 0 for the
same reason — otherwise the label disagrees with the stopper. -
The task model (the one whose completions are scored) now runs at a
pinned temperature; pinning made the same eval reproduce 6/6. Not
applied to the reflection model, which needs sampling diversity to
propose varied candidates. Models that fix their temperature (the gpt-5
family accepts only 1) ignore it via litellm drop_params instead of
failing the run — which is why the stopper fix is needed regardless.
Verified live on dev: the exact config that previously aborted after 3
trials with finish_reason='perfect_score' now runs 12 trials and reports
finish_reason='max_trials' honestly at 0.95.Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7511] [PYBE] fix: validate the score/temperature config, fix e2e provider resolution
Review findings on the previous two commits, all four real:
-
OPTIMIZER_PERFECT_SCORE and OPTIMIZER_TASK_TEMPERATURE went through a bare
float(), which accepts 'nan' and 'inf'. Both feed score comparisons where
those degrade silently (baseline >= nan is never true, >= inf never stops),
and the sibling env var in the same file already failed fast — so they now
share _read_float_env: finite, in range, error names the variable, raised at
import so a bad deployment config fails at startup. Range is 0.0-1.0 because
every metric the Studio exposes is normalised; a custom metric on
another scale should set perfect_score per run instead. -
OptimizerFactory.build used setdefault for perfect_score, so an explicit
null from the studio config survived and crashed later inside
baseline_score >= perfect_score. _resolve_perfect_score now treats None as
absent, rejects non-finite/non-numeric (and bool, which would pass as 1.0)
with InvalidOptimizerError naming the field, and preserves 0 — a legal value
that disables threshold stopping. -
The e2e provider fixture recognised only bare 'gpt...' ids, so the normal
gateway-prefixed 'openai/' resolved to anthropic and the backend
would reject the run for a missing key of the wrong provider. It now honours
an explicit provider prefix. -
That fixture also always read ANTHROPIC_API_KEY, so an OpenAI model skipped
even with OPENAI_API_KEY set. The secret is selected by provider now, and the
skip message names the variable it looked for.
New tests: TestFloatEnvValidation (nan/inf/junk/bounds/import-time failure),
TestPerfectScoreValidation (null falls back, 0 kept, invalid types raise), and
test_studio_e2e_provider_resolution.py covering prefixed/bare/unknown models and
the openai branch the reviewer asked for.Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7511] [SDK] [PYBE] fix: address review — null handling, non-finite scores, error contracts
Findings from the self-review and the remaining reviewer threads:
-
ensure_default_model_params used setdefault, so an explicit null temperature
from the studio config was forwarded to litellm, and an explicit
drop_params=False could fail the run on a fixed-temperature model. None is now
treated as absent (temperature and max_tokens) and drop_params is forced: it is
our guard, not a user knob. Same bug class already fixed for perfect_score one
commit earlier. -
The threshold stopper and the finish_reason label both read the raw score list,
so an inf from a custom metric compared >= perfect_score as True and claimed a
stop that never happened. Both now share _candidate_full_eval_scores, which
excludes the seed's eval and drops None/non-numeric/bool/non-finite entries —
the two can no longer disagree. -
The stopper's broad except returned False silently, which disables threshold
stopping invisibly (a gepa rename would look like a target never reached). It
now logs a WARNING with exc_info. The catch stays broad on purpose: a stop
heuristic must not take a paid run down, and malformed inputs are handled
structurally rather than by exception. -
load_and_validate_dataset fetched items outside the try, so transport failures
escaped its documented DatasetNotFoundError contract. The bounded fetch moved
inside; EmptyDatasetError stays outside so it cannot be relabelled. -
The item count fed to the mini-batch counted rows the SDK drops: sampling keeps
only items with an id (utils/sampling._extract_ids), so the count could exceed
the real trainset. count_optimizable_items now counts by that rule and is
shared by both call sites. -
resolve_reflection_minibatch_size clamped budget_cap to 1 while the comment
promised at least GEPA_MIN_REFLECTION_ITERATIONS iterations. The clamp is
explicit and logs, and the comment says the guarantee holds only when the
budget allows it. -
Comments that overclaimed: the call-site comment implied pinning makes scoring
deterministic (it does not for gpt-5, which is the default model), and the
docstrings baked provider-specific measurements into durable docs. Both now
state the contract — best-effort pin, noise-tolerant stop conditions — with the
numbers left in the PR and ticket. The duplicated GEPA_MIN_REFLECTION_ITERATIONS
records why it cannot be imported from the SDK yet (not in the pinned release)
and where to collapse it. -
The temperature-pin test asserted a literal while config reads the env at
import, so a supported override would fail a correct implementation.
New tests: task-model pin asserted on the prompt itself (and absent on the
reflection model), non-finite/non-numeric scores for both stopper and label,
null/false normalisation, and tests/unit/test_studio_dataset_loading.py for the
count rule and both error contracts. SDK 994 passed, backend 438 passed.Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7511] [PYBE] fix: document the temperature range, stop leaking the e2e conftest module
Two smaller review findings on the previous commit:
-
OPTIMIZER_TASK_TEMPERATURE lost its units/range line when the comment was
trimmed; operators need it next to the constant, not only inside the
validation call. -
The provider-resolution test registered the e2e conftest in sys.modules and
never removed it, so a later import could pick up the stale copy and make
collection order-dependent. The fixture now restores the previous entry in a
finally block.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7511] [SDK] [PYBE] fix: reject id-less datasets, don't call an external stop "max_trials"
Two review findings:
-
load_and_validate_dataset() accepted a dataset whose rows the SDK's
sampling drops (no id), so the run reached optimization with an empty
trainset and a mini-batch sized from 0. Reject it up front with
EmptyDatasetError, which now carries a reason so the message says why
instead of claiming the dataset is empty. -
gepa auto-installs a FileStopper on <run_dir>/gepa.stop whenever run_dir
is set, so a run can exit with neither of our stoppers fired and budget
still on the clock. _resolve_gepa_finish_reason() labeled that
"max_trials" — a budget burn that never happened. Compare
GEPAResult.total_metric_calls against max_metric_calls and report
"cancelled" instead; an unknown spend keeps the old fallback.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7511] [SDK] [PYBE] fix: bound the reflection mini-batch, gate the external-stop label
Review follow-ups on #7650.
Mini-batch had no absolute ceiling: gepa serializes the whole mini-batch
into one reflection prompt (instruction_proposal.format_samples), so at
max_trials=10 a 1000-item dataset resolved to 200 traces per proposal —
neither the dataset_size cap nor the budget cap bounds the prompt.
GEPA_REFLECTION_MINIBATCH_MAX = 25 keeps the resolution win (26 distinct
mini-batch sums vs 6 at the old fixed 5) with a bounded prompt."cancelled" from an unspent metric-call budget is now gated on run_dir
being set. Only then does gepa wire the FileStopper that can produce it;
without one, an unspent-looking budget could only mean the counter and
MaxMetricCallsStopper's threshold drifted apart in a future gepa (the
dependency has no upper bound), which would relabel every ordinary
budget exit — including every Studio run, since Studio never sets
run_dir. Studio keeps the plain "max_trials" fallback.Dropped the per-call drop_params: opik_optimizer sets
litellm.drop_params process-wide on import (base_optimizer.py) and the
runner imports it, so forcing it here was dead weight that also
overrode an explicit false. A test now asserts the global instead.EmptyDatasetError no longer fires on a full page of id-less rows: the
fetch is capped at DATASET_SAMPLES but the SDK draws sample ids from the
whole dataset (sampling._extract_ids calls get_items() unbounded), so a
full page proves nothing about the rows behind it. Only a short page is
a complete verdict; a truncated one logs and continues.Also types EmptyDatasetError's reason as str | None.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
Co-authored-by: Claude Fable 5 noreply@anthropic.com
下载附件