发布

RSS 订阅源

2.2.18-6243 1ed8e7c15b

比较
[OPIK-7521] [SDK] fix: trace optimizer LLM spend so the backend can price and attribute it (#7674)

frostbyte_neo 发布于 2026-08-04 12:29:31 +00:00
- [OPIK-7521] [SDK] fix: bound and report GEPA reflection-LLM spend
GEPA's reflection model was handed to gepa.optimize() as a bare model
string, so the library called litellm directly: those calls were not
traced, not counted, and not bounded by max_metric_calls — a user asking
for N trials paid for the trials plus an invisible, uncontrolled extra.
- Pass reflection_lm as a callable routed through core.llm_calls.call_model:
  every reflection call is now an Opik span (opik_call_type="reflection",
  cost visible per call), honors model_parameters (drops the stale
  "not surfaced for internal calls" warning), and increments both the
  run-wide llm_call_counter and a dedicated reflection counter.
- Give reflection its own budget, separate from the metric budget:
  extra_params["max_reflection_calls"], default max_trials (round-robin
  makes at most one reflection call per candidate, and the metric budget
  admits exactly max_trials candidates), 0 disables. Enforced via a stop
  callback; a run that exhausts it finishes with "reflection_budget".
- Report the spend: reflection_call_count and max_reflection_calls in
  result details, plus a summary log line.
Deliberately NOT folded into max_metric_calls: that would silently change
max_trials semantics (N trials would no longer mean N candidates), and
metric calls vs reflection calls have different unit costs.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7521] [SDK] chore: human-readable label for reflection_budget stop reason
Verified end-to-end (real GEPA + gpt-4o-mini): a capped run displays
"Stop: Reflection budget" instead of the raw enum value.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7521] [SDK] chore: fix mypy and vulture lint findings
Rename the unused stop-callback parameter to _gepa_state (vulture) and
type-ignore the SimpleNamespace test double passed to
_ReflectionBudgetStopper (mypy). No behavior change.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7521] [SDK] fix: hard-cap reflection calls in the callable; address review
Review follow-ups (PR #7674):
- The reflection-LM callable now refuses to spend past max_reflection_calls
  (returns an empty proposal instead of calling the model), covering
  configurations that reflect more than once per engine iteration where the
  loop-top stopper alone could overshoot. Refusal deliberately does not raise:
  with gepa's raise_on_exception=True an exception would abort the run.
- End-to-end budget test now drives the captured stop_callbacks like gepa's
  engine loop and asserts the third proposal is never attempted.
- Tests assert the public contract (result.details) instead of the private
  counter where both said the same thing.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
- [OPIK-7521] [SDK] chore: warn once per run when refusing reflection calls
The budget guard logged a WARNING on every reflection-LM call refused after
max_reflection_calls was exhausted. A component selector that keeps asking
within one engine iteration would flood the run logs with the identical line,
even though the refusals are already cheap (no model call) and the outcome is
reported as finish_reason='reflection_budget' in the result details.

Gate the warning on a run-scoped _reflection_budget_warned flag, reset next to
_reflection_call_count at the top of each optimize_prompt run so a reused
optimizer instance still warns once per run. The counter and the refusal
behaviour itself are unchanged.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7521] [SDK] fix: attribute the reflection-budget stop, claim slots atomically
Addresses the remaining review findings on the reflection-budget cap.
- finish_reason: the reflection counter reaching the cap was treated as the
  reason on its own, so an ordinary full-budget run — default cap is
  max_trials, which such a run always reaches — got relabelled
  "reflection_budget" instead of "max_trials". The stopper now records that it
  asked for the stop, and the resolver claims the reason only when that
  happened and metric-call budget was still unspent.
- Budget claim: the check and the counter increment now happen under a
  per-run lock, so concurrent proposals cannot both take the last slot; the
  lock is released before the model call.
- Tests: the disabled-cap (0) case now drives the callable and the stop
  callbacks to prove nothing refuses and nothing halts; new tests cover
  stopper attribution, the max_trials precedence, and a concurrent race for
  the last slot.
- Docs: note that a run which also spends its metric budget reports
  max_trials.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7521] [SDK] fix: consolidate reflection instrumentation; make the budget opt-in
Absorbs the SDK half of #7683 (same ticket, duplicate implementation of the
same seam) and fixes two defects found by running gepa for real.

Consolidation: one reflection callable, _build_reflection_lm, replacing this
branch's _make_reflection_lm and #7683's copy. It keeps that PR's tracked
gepa_reflection trace + _tag_trace (the backend cost aggregate attributes by
trace tag), seed passthrough, and empty-response handling, plus this branch's
atomic budget claim. Also brings over the SDK-side cost fixes it depended on:
response_cost extraction in llm_calls, token totals, the duplicate OpikLogger
strip in litellm_agent, and their tests. #7683 is now BE+FE only.

max_reflection_calls now defaults to 0 (uncapped) instead of max_trials. The
old default rested on "the metric budget admits exactly max_trials candidates",
which is not how gepa spends: an iteration whose candidate loses on the
minibatch costs only 2reflection_minibatch_size metric calls (the valset pass
is skipped, engine.py), while the budget is max_trialsn_samples. Measured
against gepa 0.1.1 with a stub adapter, runs make 15/38/75/150/300 reflection
calls where max_trials=10 - so the old default truncated ordinary searches by
1.5-30x and reported them as stopped early. The knob and its stopper are now
wired only when the caller asks for them; a default run is unchanged.

Budget refusal now raises ReflectionBudgetExceededError instead of returning
"". Returning "" is not a no-op: gepa's output_extractor maps it to an empty
instruction, and gepa then builds a candidate whose prompt is "" and evaluates
it - invoking the agent with no prompt and spending metric calls. Verified with
a real gepa run (6 evaluations of an empty instruction). The stated reason for
not raising does not hold either: propose() catches Exception around
propose_new_texts in both 0.0.17 and 0.1.x, so the iteration is skipped and the
run keeps its results. Confirmed stop_callbacks exists in 0.0.17 (the Studio
pin), so the opt-in cap works there too.

Tests: pytest tests/unit -> 1093 passed, 3 skipped. New coverage pins the
uncapped default and the raise-on-refusal contract; the stop-condition wiring
tests go back to expecting no reflection stopper by default. The gepa.lm
TrackingLM test now importorskips - that module does not exist in gepa 0.1.1
or in the 0.0.17 pin. ruff check/format clean; mypy error count in the touched
files unchanged from main (1 pre-existing).

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7521] [SDK] chore: satisfy pyupgrade on the test file carried over from #7683
The pre-commit pyupgrade hook (--py310-plus) rewrites from typing import Iterator to collections.abc; that import arrived with the reflection-LM test
file and failed the "🤖 pyupgrade — optimizer" CI leg. Whole hook set now passes
locally over every file this branch touches.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7521] [SDK] fix: count agent calls by owner, not call-stack walk; address review
Review follow-ups, each with a test that fails without its change.

Counter lost every threaded call (found while answering a review question, then
reproduced): evaluation dispatches agent calls onto a worker pool - the GEPA
adapter passes num_threads=n_threads, default 12 - and the optimizer's frames
are not on a worker thread's stack, so _increment_llm_counter_if_in_optimizer
found nothing and returned silently. Measured on LiteLLMAgent with a stubbed
provider: 4 calls on a pool gave llm_call_counter=0 while llm_cost_total=0.004,
because cost travels by the explicit _optimizer_owner reference and the counter
did not. Both LiteLLMAgent completion sites and the tool-call site now use the
owner. Once cost is reported truthfully, a zero call count beside a non-zero
spend is a visible contradiction, which is why this lands here.

The owner had two names - _attach_agent_owner sets optimizer and
_optimizer_owner, cost read one and the counters the other - so a caller
setting only one silently lost half the telemetry (exactly what the existing
cost-tracking test did). Resolved behind a single _owning_optimizer() accessor.

baz-reviewer findings:
- Negative provider costs are rejected instead of accumulated: add_llm_cost only
  adds, so a malformed figure would subtract from the run total and under-report
  the spend this accounting exists to measure.
- _extract_response_usage reads mapping-shaped responses and their usage, and
  returns None - not zeros - when no field is a usable number; a zero dict sets
  _llm_usage_recorded and makes the result claim a real zero-token run.
- The agent's cost/usage extraction logs at debug instead of swallowing silently.
- Dropped the gepa==0.0.17 compatibility claim. It was inherited from #7683 and
  is wrong: _validate_reflection_prompt_template already refuses gepa<0.1.0 at
  construction (older template dialect), and pyproject declares gepa>=0.1.0. The
  callable still accepts both prompt shapes, which is a protocol fact, not
  version support. No 0.0.x fallback path is added - the floor is deliberate.
Tests: pytest tests/unit -> 1101 passed, 3 skipped. Every new assertion verified
to fail against the pre-fix code. Full pre-commit hook set green.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7521] [SDK] fix: lock the usage reset; import llm_calls at module scope
Two more review points.

reset_usage now takes _usage_lock for the whole reset (baz-reviewer): clearing
the totals and the two "reported" flags outside the lock lets a concurrent
add_llm_cost/add_llm_usage be dropped, or - worse - have its recorded flag
cleared right after being set, which turns a real cost into "no provider
reported one" for the rest of the run.

litellm_agent imports llm_calls at module scope instead of inside the function
(alexkuzmik). Worth noting how this surfaced: the module-level import had become
unused earlier in this branch (the counter now goes through the owner), ruff
removed it, and dropping the function-local one left the name undefined - a
NameError that the old except Exception: pass would have swallowed, silently
zeroing cost for every agent call. The debug log added in the previous commit is
what made it visible; the unit suite caught it here.

Tests: pytest tests/unit -> 1101 passed, 3 skipped. Full pre-commit set green.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7521] [SDK] refactor: leave cost to the backend; keep the spans it prices
Per review (@alexkuzmik): the backend is the source of truth for cost, and SDK
integrations should only supply what it needs to price a call. This drops every
piece of cost arithmetic this branch carried and keeps the part that makes the
spend visible at all.

Removed: _extract_response_cost / _extract_response_usage / _coerce_cost /
_record_cost_usage_if_in_optimizer, the reported_llm_cost / reported_llm_usage
accessors with their llm*_recorded flags, the derived total_tokens, and the
agent's _hidden_params read (that block is back to main verbatim). The result's
llm_cost_total / llm_token_usage_total go back to what main reports.

Kept, because the backend needs them: reflection runs through call_model, so the
call produces a real span inside its own tracked trace tagged with the
optimization id — SpanDAO prices it from model/provider/usage via
CostService.calculateCost, exactly as it prices every other LLM span, and
#7683's aggregate attributes it by that tag. Also kept _strip_duplicate_opik_logger,
which matters more now than before: with the backend pricing spans, a duplicated
span doubles the reported cost directly.

Also kept from this branch: the worker-thread call-counter fix, the opt-in
max_reflection_calls with its stopper and finish reason, and the reflection
counts in result.details.

Trade-off recorded deliberately: the SDK total used to be an independent
cross-check of the backend number (it is what caught the double-logged span in
#7683). What replaces it is cheaper and catches the same class of defect — the
span count per trace against llm_call_counter, which is now correct under
threads. What is genuinely no longer cross-checked from here is mispricing of a
known model, which belongs to the backend's own tests.

Tests: pytest tests/unit -> 1080 passed, 3 skipped. Cost-specific tests reverted
to main; the reflection test now asserts the call is counted and its trace
tagged, not what it cost. Full pre-commit set green.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7521] [SDK] test: cover the duplicate-OpikLogger strip
Reverting the cost tests to main took the only coverage of
_strip_duplicate_opik_logger with it, and that function is load-bearing now that
the backend prices spans: a duplicated span doubles the reported cost directly,
and the same duplication was observed live (one call logged twice, trace cost
1.11e-05 for a 5.55e-06 call).

Pins both directions, since each fails differently: with a span open the
OpikLogger is dropped from both callback lists without mutating the caller's
params, and with no span open the params are returned untouched — there the
logger is the only thing stamping the optimization id onto the trace, so
stripping it would hide that spend from the run's cost aggregation.

Verified to fail with the strip neutralised.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7521] [SDK] test: decouple the counter test from GepaOptimizer
The counter fix is in LiteLLMAgent, not in GEPA, so a test under tests/unit/utils
should not construct a full GepaOptimizer to exercise it. Uses a minimal
BaseOptimizer subclass instead, matching the DummyOptimizer pattern already used
by test_cost_tracking.py in the same directory. Cheaper, and the test no longer
fails for reasons unrelated to what it pins.

Both assertions re-verified to fail against the previous call-stack-walk
implementation.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7521] [SDK] test: type the counter test's prompt properly
Follow-up to the previous commit: the stub prompt was a SimpleNamespace, which
mypy rejects against _run_completion's ChatPrompt parameter. Uses a real
ChatPrompt — it needs no optimizer and no gepa — and reformats.

Full pre-commit set verified green over every file this branch touches before
committing this time.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7521] [SDK] fix: own the LLM-call trace; attribute optimizer spend centrally
Drops opik_monitor (@alexkuzmik: use track_completion everywhere) and closes the
attribution hole that removing it would otherwise open.

opik_monitor did two things: it injected an OpikLogger next to our own
track_completion — logging the same call twice whenever a span was open, which
doubles the cost the backend computes from spans — and it was the only thing
carrying metadata["opik"]["tags"] onto a trace, because track_completion
hardcodes tags=["litellm"]. So it could not just be deleted; tracing had to move
here. _invoke_traced now takes both paths explicitly: with a span open it points
the call at that span so the LiteLLM span nests instead of forking a detached
trace (the hint opik_monitor used to set); with no span it opens one and stamps
the optimization tags on it. _strip_duplicate_opik_logger is gone with the
duplicate it worked around.

The attribution half is the bigger fix. 13 of 20 call_model sites never pass an
optimization_id — that is EVERY LLM call Evolutionary and HierarchicalReflective
make, two of the three optimizers Studio can run — so their traces carried no
optimization tag and the run's cost aggregation, which attributes optimizer-
internal spend by exactly that tag, counted none of it. Rather than thread the
id through a dozen ops signatures, _resolve_optimization_id falls back to the
optimizer on the call stack. An explicit argument always wins, and the failure
mode is losing a tag exactly as today — never attributing spend to the wrong run.

Fixed along the way, caught by an existing test once opik_monitor stopped
creating the metadata dict as a side effect: the reasoning call type was only
recorded when "metadata" already existed in params, so on the no-span path a
caller that passed no metadata lost it silently.

Verified live against a real Opik backend using litellm's mock_response (full
tracing stack, zero provider spend):
- no span open -> own trace, tags ["Prompt Optimization", "<optimization_id>"],
  llm_span_count=1 (no duplicate), total_estimated_cost priced by the backend;
- span open -> LiteLLM span nested under the caller's span, still one llm span,
  no second trace;
- a call passing NO optimization_id, shaped like Evolutionary's ops helpers: no
  tags before this change, correctly tagged after.
Tests: pytest tests/unit -> 1090 passed, 3 skipped. Full pre-commit set green.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7521] [SDK] style: ruff-format the tracing tests
Formatting only. My pre-push hook run took its file list from the committed
diff, so the newly added test file was not in it and ruff-format never saw it;
CI, which lists the PR's changed files, did. Re-verified over the PR's actual
file set this time.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7521] [SDK] refactor: drop the project_name copy the next line deletes
Code-review cleanup. _nest_under_current_span copied project_name off the span
into metadata["opik"], and both call sites pipe its result straight through
_strip_project_name, whose whole job is deleting that key — so the copy never
reached litellm. It mirrored what opik_monitor did, which had the same
redundancy. The span hint is all the function needs to set.

Re-verified end to end after the change: a full GEPA run still produces
gepa_reflection traces tagged [<optimization_id>, Reflection, GEPA], one llm
span each (no duplicate), priced by the backend, with the LiteLLM span nested
under the tracked span.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
- [OPIK-7521] [SDK] test: cover both tracing dispatchers and the attribution boundary
Addresses baz-reviewer on the tracing dispatchers.

Unit gaps closed: the async dispatcher's nesting path (it only had the no-span
one, and the nesting hint is exactly what the sync/async split exists to get
right), and project_name stripping through both dispatchers — track_completion
already sets the project, so leaving it in the LiteLLM params makes the two
disagree. Tags are asserted to survive that stripping in all four cases.

The unit tests also no longer emit live traces: the dispatchers open a real
trace through opik.track, so without a pass-through the suite was logging spans
to whatever backend the environment pointed at.

Added tests/e2e/tracing, marked e2e per AGENTS.md ("integration/e2e only when
cross-boundary behavior changes" — this changes exactly that boundary). It
asserts the record that actually reached the backend, which no unit test can:
a trace carrying the optimization id, named after the call type, with exactly
one LLM span — a second one is the duplicate-OpikLogger regression, and it would
double the cost the backend computes for the run. It finds the trace by tag, the
same way the backend's aggregate does, self-skips without credentials, and uses
litellm's mock_response so it spends nothing on a provider.

Verified against a real backend: passes, and fails (no tagged trace found) when
the tagging in _invoke_traced is neutralised.

Tests: pytest tests/unit -> 1094 passed, 3 skipped. Full pre-commit set green.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

Co-authored-by: Claude Fable 5 noreply@anthropic.com
下载附件
- 源代码 (ZIP)
- 源代码 (TAR.GZ)

0 发布 8,340 Git 标签

[OPIK-7521] [SDK] fix: trace optimizer LLM spend so the backend can price and attribute it (#7674)