发布

RSS 订阅源

2.2.14-6211 ad450965fd

比较
[OPIK-7510] [SDK] [PYBE] [FE] fix: stop the optimizer dropping prompt variables (#7656)

frostbyte_neo 发布于 2026-07-31 12:07:42 +00:00
- [OPIK-7510] [SDK] [PYBE] [FE] fix: stop the optimizer dropping prompt variables
Studio runs routinely destroyed the user's template variables. Three causes,
fixed cheapest-first:

(a) Programmatic guard (optimizer-agnostic). ChatPrompt substitutes dataset
values with a plain str.replace of "{key}", so a candidate that rewrites a
message without the token drops the input silently - no KeyError, no
warning, and if the metric happens to reward the corrupted prompt it wins
and the run keeps spending on it. Compare the seed's placeholder token set
against each rebuilt candidate and reject the edits that lost one.
```
Rejection is a per-message revert to seed content, mirroring how the
existing role-constraint path already substitutes seed content for
components it will not accept. Comparison is prompt-level, so a variable
merely moved between messages is not treated as a loss. The token shape is
restricted to identifier-like keys so JSON/code braces in a prompt are not
mistaken for variables (which would revert every legitimate edit).

Applied on both rebuild paths - the adapter's evaluate() hot path and the
shared candidate_ops helper used by rescoring and final-result assembly -
and recorded in the trial's GEPA metadata as well as the logs, so a
corrupted-candidate run is auditable rather than silent.
```
(b) Wire reflection_prompt_template through GepaOptimizer. GEPA's default
instruction-proposal prompt tells the reflection LM to inline "all niche and
domain specific factual information" from the examples, which on a prompt
holding a variable reads as an instruction to replace it with one row's
data. The new template is GEPA's default text unchanged plus an additive
verbatim-preservation block, so reflection quality is retained by
construction. prompt_overrides is now honoured instead of discarded, and the
template is validated up front so a bad override fails at setup.

(c) Change the Studio default prompt shape. The new-run form seeded a lone user
message and the backend made every present role optimizable, so the default
run handed the reflection LM the exact message holding the variables. Seed
system + user, and optimize only the system message when one exists. The
all-roles widening is kept as the fallback for system-less prompts, which
preserves the divide-by-zero guard GEPA needs.

Frontend change is v2 only; v1 is frozen.

Evidence for (b), 8 seed prompts x 6 repeats on openai/gpt-4o-mini through
gepa's real prompt_renderer and output_extractor:
```
arm       variable retention   fact carry-over   mean len
default   19/48 (40%)          72%               995
new       48/48 (100%)         72%               501
```
Retention goes from 40% to 100% with fact carry-over at parity, so the added
constraint does not cost reflection quality. (An early 8-sample run suggested a
fact-carry-over drop; it did not survive more repeats.)

Tests: reject-and-revert regression coverage on both rebuild paths, template
superset/validation/render tests that fail loudly if upstream's default drifts,
role-scoping tests that pin the divide-by-zero fallback, and a frontend test for
the seeded shape.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7510] [SDK] chore: apply ruff-format
Formatting only, no behaviour change: one blank line before a new method and
two assertion line-wraps flagged by the ruff-format pre-commit hook.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7510] [SDK] fix: validate the reflection template in init, cover the rejection metadata
Two review findings on the OPIK-7510 change.
1. The template was validated while building the gepa.optimize() kwargs, which
  happens after the baseline evaluation is resolved. A malformed
  prompt_overrides therefore billed the user a full dataset scoring pass before
  raising — the opposite of the fail-fast the docstring claimed. Validation now
  runs in init via a shared _validate_reflection_prompt_template helper, so
  a bad override fails when the optimizer is constructed. The resolve path keeps
  a re-check, which now covers only its real remaining case: a template swapped
  in afterwards through optimizer.prompts.set().
2. The acceptance criterion is that a rejection is recorded, but nothing
  asserted the event reaches the trial's experiment config — only that the
  adapter's internal list was populated. Added two tests that drive the real
  evaluate() path (rebuild deliberately not stubbed) and inspect what is handed
  to prepare_experiment_config: the component key is present on a rejection, and
  drop_none keeps the field absent on a clean candidate.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7510] [SDK] [PYBE] fix: address review — gepa>=0.1.0 floor, dataset-key-aware guard, local revert state
Review feedback on #7656:
- gepa floor -> >=0.1.0 (petrotiurin): init validates the default
  reflection template with gepa's own validator, and the
  <curr_param>/<side_info> markers only exist from 0.1.0 — on 0.0.x the
  optimizer could not even be constructed. A new test pins the installed
  gepa's default template to the markers we rely on, so the floor cannot
  silently regress. The python-backend gepa==0.0.17 pin is annotated to
  move in the same commit that bumps opik-optimizer past 3.1.0.
- Dataset columns as authoritative placeholder keys (petrotiurin): the
  guard now also protects the literal "{key}" of every dataset column —
  exactly what substitution replaces — so non-identifier keys like
  "{my key}" are covered. The adapter collects the columns itself; the
  rescoring and final-assembly rebuilds get them from the run's items.
  The identifier regex stays as the fallback when no columns are known.
- Placeholder reverts now flow through the rebuild's return value instead
  of shared instance state, so concurrent evaluate() calls can never
  attach another candidate's rejections to a trial's metadata (baz).
- A non-string reflection_prompt_template override now fails at
  construction with the documented ValueError instead of escaping as a
  bare TypeError, and a moved/renamed gepa validator symbol raises an
  explicit unsupported-version error instead of an opaque ImportError (baz).
- A guard revert keeps any extra fields a message carries instead of
  rebuilding it as role+content only.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7510] [SDK] refactor: single rebuild pipeline for GEPA candidates
Address review: OpikGEPAAdapter._rebuild_prompts_from_candidate duplicated
the candidate-to-messages rebuild flow already centralized in
candidate_ops.rebuild_prompts_from_candidate. The adapter now delegates to
the shared helper and keeps only its metadata/warning plumbing, so the
role-constraint and placeholder-guard logic lives in one place; the
guard's rejection warning is emitted by the helper alone.

Two side effects of converging on the shared path:
- the adapter rebuild now uses rebuild_content_with_new_text, preserving
  multimodal content structure instead of replacing content wholesale;
- the dropped-components warning counts actual candidate edits targeting
  disallowed roles (count_disallowed_candidate_components) rather than
  every constrained message, so it no longer fires on clean rebuilds.
The helper returns (rebuilt, reverted_component_keys) so the adapter's
evaluate() keeps its per-call rejection metadata.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7510] [SDK] fix: silence mypy on the deliberately-mistyped override test
The non-string-override test passes an int for a PromptOverrides value on
purpose — that wrong type is what the assertion is about. mypy flagged it
as dict-item; narrow ignore with a note on why the type is wrong.

This slipped in with c810b3a and went unnoticed because the branch was
conflicting with main at the time, so GitHub skipped every
pull_request-triggered workflow, including Code Quality.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7510] [SDK] fix: derive guard keys from unsampled dataset items
known_placeholder_keys was built from train_items/val_items, which
_apply_plan has already truncated or filtered by id. Rescoring then runs
against the full context.evaluation_dataset, so a column carried only by
rows this run did not sample was invisible to the guard: its "{key}" fell
back to identifier-shape matching, and a non-identifier token like
"{my key}" could be dropped by a candidate unnoticed — the exact failure
this PR exists to prevent.

Derive the keys from the unsampled get_items() results instead. No extra
fetch: the same call already ran, its result was just not retained. This
also matches OpikGEPAAdapter.init, which reads the full dataset.

Also documents the guard's rules on the shared rebuild helper (token
shape, prompt-wide comparison, per-message revert), per review.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7510] [SDK] [PYBE] fix: close the guard's own gaps found in final review
Five defects, four in the guard itself:
1. The guard could drop a variable. Reverting a message restores its seed
  text, discarding whatever the candidate put there — including a variable
  it had moved in. Seed (system "Use {context}", user "Answer {question}")
  with a candidate that moved {question} into system lost {question}
  entirely. Reverts are now re-checked to a fixed point; worst case the
  prompt equals the seed, so it always terminates with the contract intact.
2. Candidate-introduced dataset columns are rejected. A rewrite adding
  {answer} got the label substituted in, scored against data it will not
  have at inference, and won on a lie. Only dataset columns count as
  leakage — an invented {foo} is inert text and stays allowed.
3. Protection is now exact when the columns are known. Identifier-shaped
  prose like \frac{num}{den} or {TODO} is no longer treated as a variable,
  so a candidate removing it is kept instead of reverted forever. Without
  known columns we still fall back to the identifier shape.
4. A stale duplicate no longer excuses deleting the real input slot: a
  token counts as moved only when some message carries it where the seed
  did not.
5. Backend: a prompt with no optimizable role raised nothing and fell back
  to "system", handing GEPA zero components — the divide-by-zero the
  widening branch exists to avoid. It now fails with InvalidConfigError.
  The comment claiming the SDK guard protects the system-less path is
  corrected: that only holds once the backend pins a release carrying it.
Test gaps this closes, all mutation-verified (each new test fails when its
fix is reverted):
- nothing asserted the reflection template reaches gepa.optimize(); the
  kwarg could be deleted with the suite still green.
- nothing covered the optimizer-level placeholder-key derivation, so the
  unsampled-items fix was untested.
Harness for the gepa.optimize assertions is duplicated into
gepa_run_harness.py rather than imported from test_gepa_stop_conditions, to
keep that module's unrelated type errors out of this PR's mypy scope.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7510] [SDK] fix: empty known_keys is knowledge; stop retaining full datasets
Review round on 3e28f89:
- protected_tokens used if known_keys:, so an explicitly empty column
  set was treated as "columns unknown" and fell back to identifier-shaped
  tokens — reverting edits to literals a zero-column dataset can never
  substitute, and skipping the leakage check entirely. Both now test
  is not None, so an empty set stays authoritative.
- The unsampled item lists were held live through optimization and
  rescoring, undoing the memory benefit of n_samples. The guard's keys are
  now derived immediately after fetching and the full lists released, so
  only the small key set outlives that block.
- The guard loop delegates to _introduces_unseeded_column and
  _drops_an_unmoved_variable, so it orchestrates rather than inlines both
  policies and each is readable on its own.
- The override wiring test drives prompt_overrides through the public
  constructor instead of monkeypatching the private resolver; the harness
  takes optimizer kwargs for that.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7510] [SDK] fix: name an old gepa as the cause, not a phantom override
Review round on #7656. The version RuntimeError only fired when gepa lacked
validate_prompt_template entirely — true for <=0.0.17, but 0.0.18-0.0.27 do
expose it while checking the older <curr_instructions>/<inputs_outputs_feedback>
markers. Since __init__ validates our built-in default, plain
GepaOptimizer(model=...) on that band died with
```
ValueError: Invalid reflection_prompt_template override: Missing
placeholder(s) ... The template must contain both the <curr_param> and
<side_info> markers.
```
— blaming an override the caller never passed, and demanding markers their gepa
would reject anyway. The pyproject floor prevents this on clean installs, so it
is a diagnosis bug, not a correctness one; pinned/locked envs land here.

Verified against the real wheels (0.0.7/0.0.17: no validator; 0.0.18/0.0.24/
0.0.27: validator on the old dialect, reflection_prompt_template kwarg present
from 0.0.18; 0.1.0/0.1.4: <curr_param>).

Two attributions, each mutation-verified by its own test:
- Read the installed gepa's own default_prompt_template and raise the version
  error when it lacks <curr_param>. Only a readable str counts as evidence,
  so a future gepa that renames the attribute isn't misreported as too old.
- If validation of our own built-in template fails anyway, attribute it to the
  install rather than to the caller. This covers the unreadable-default case.
A malformed caller override still raises the documented ValueError.

Also from the same round:
- Delete an orphan comment above rescore_candidates — it described the
  known_placeholder_keys derivation ~130 lines earlier, where it is already
  commented, and both call sites pass the kwarg visibly.
- Record the Studio seeding trade-off in code (decision-check, no behavior
  change): two seeded cards mean isMessageEmpty requires content in both
  before submit. The form validates onSubmit, so an untouched form shows no
  errors; a user wanting a lone user prompt deletes the system card, and that
  shape is exactly the hazard this ticket fixes, so it should be explicit.
Tests: optimizer unit suite 1029 passed / 2 skipped; pre-commit hooks green on
the changed files (mypy, xenon/radon/lizard, eslint, frontend typecheck);
frontend schema.test.ts 11 passed.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

Co-authored-by: Claude Opus 5 (1M context) noreply@anthropic.com
下载附件
- 源代码 (ZIP)
- 源代码 (TAR.GZ)

0 发布 8,340 Git 标签

[OPIK-7510] [SDK] [PYBE] [FE] fix: stop the optimizer dropping prompt variables (#7656)