-
[OPIK-6995] [FS] feat: add per-evaluation spend budget for LLM-as-judge evaluators (#7312)
frostbyte_neo 发布于
2026-07-10 09:03:58 +00:00 - [OPIK-6995] [FS] feat: add per-evaluation spend budget for LLM-as-judge evaluators
Add an optional per-rule USD spend budget for agentic (trace + thread)
LLM-as-judge online evaluators. The budget is a soft wrap-up trigger, not a
hard ceiling: once cumulative spend reaches the limit, the agentic tool loop
stops starting new turns and emits a best-effort verdict from the data
gathered so far (spend may overshoot by the wrap-up call, by design).- BudgetGuard encapsulates the limit, running spend, and all cost math; spend
is computed in-process via CostService (no provider round-trip) from each
response's token usage. A non-positive limit degrades to no-limit. - LlmUsageExtractor (extracted from EvaluationEntityFactory) builds the
cache-aware usage map for any provider Opik prices. - When the budget trips, the monitoring trace is tagged
budget_exceededand
the wrap-up call uses a distinct instruction (best-effort from partial data
rather than "investigation complete"). - Span evaluators are excluded: a single LLM call has no loop to wrap up.
- Frontend: optional "Max cost per evaluation (USD)" field on trace/thread
rules (hidden for span scope).
Implements OPIK-6995.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(online-scoring): keep terminal AiMessage when budget trips on a natural stop
Code-review follow-up on the spend-budget change:
- ToolCallLoop: check the no-tool-calls terminal case BEFORE the round/budget
gate. Previously a follow-up response that both finished naturally (no tool
calls) and tipped spend over the budget hit the budget gate first and returned
without appending the terminal AiMessage, so runWithWrapUp re-issued the
structured request missing the model's final reasoning turn. Regression test
added (appendsTerminalAiMessageWhenBudgetTripsOnANaturalStop). - LlmUsageExtractorTest: cover the null/zero cache-token guard branch (the
value>0 check that keeps cost unchanged when caching is off). - FE LLMJudgeRuleDetails: guard event.target.valueAsNumber against NaN so an
intermediate unparseable numeric input can't write NaN into maxCostUsd.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(fe): refine budget input — block unsupported symbols, hide spinners, drop min
Max cost per evaluation (USD) field:
- Block symbols type=number would otherwise accept (comma, sign, exponent) via
onKeyDown so only a positive decimal can be entered. - Hide the up/down spinner arrows (webkit inner/outer + Firefox textfield).
- Remove the min={0.000001} attribute so the field no longer surfaces that value
by default; positivity is still enforced by the Zod .positive() rule. - Use cn() for the conditional className (was string concatenation).
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(online-scoring): address PR review — budget flag placement, log level, FE input
- ToolCallLoop: flag budget_exceeded on the recorder at the point the budget gate
actually fires (agentic loop), instead of inferring it post-hoc from
shouldWrapUp() in the scorers. Fixes two issues: a natural stop that merely
crossed spend is no longer mislabelled, and the tag survives a downstream error
(it's set before the wrap-up/scoring chain can throw). Removed the inferred
flagBudgetExceeded() calls from both scorers' map(). - ToolCallLoop: demote the internal "budget reached" log to debug (the scorer's
user-facing warn is the primary signal). Tests added: flag set on mid-loop
budget trip, not set on a natural under-budget stop. - FE LLMJudgeRuleDetails: use FormLabel + FormDescription (forms.md) so the
label/input association and error styling are wired; reject non-plain-decimal
onChange input so pasted/IME exponent ("1e3") can't bypass the decimal-only rule.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(test-suites): preserve maxCostUsd through TestSuiteEvaluatorMapper transforms
The two transforms that rebuild LlmAsJudgeCode (renameSchemaToAssertionKeys,
applyTestSuitePrompt) used the positional constructor and dropped the new
maxCostUsd field, so a test-suite LLM-judge evaluator configured with a spend
budget was silently scored with BudgetGuard.UNLIMITED. Rebuild via toBuilder()
so maxCostUsd — and any field added later — is carried through. Regression tests
added (preserves maxCostUsd / leaves it null when unset).Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
- fix(online-scoring): address PR review on evaluation spend budget
- Enforce @Positive max_cost_usd at the API boundary: cascade validation into
the polymorphic evaluator code record via @Valid (create + update paths), so
a non-positive value is rejected with 422 instead of silently coerced to
"no limit". Teach PODAM to manufacture strictly-positive values for @Positive
BigDecimal fields so the newly-enforced constraint doesn't flake create tests. - Unify the budget signals off one authoritative event: add
BudgetGuard.wasBudgetEnforced(), set at the tool loop's budget gate alongside
the budget_exceeded trace tag, and drive the wrap-up instruction + user-facing
warn off it. A natural stop or inline single call that merely crosses spend is
no longer mislabelled as a budget cut-off. - Warn once per guard when a limited guard resolves an unpriced model (cost
stays zero, so the limit is silently ineffective). - MDC-wrap the budget-reached log line so it carries workspace_id / rule_id.
- Reword the span scorer's UNLIMITED rationale (it does run the agentic loop;
the real reason is the span DTO has no max_cost_usd field). - Test hardening: derive expected cost from CostService instead of hardcoding,
drop dead mock stubbing in the unlimited paths, cover the charge() fail-open
branch, assert a distinct total-token pass-through, split the bundled cache
test, rename overBudgetGuard -> tightBudgetGuard.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(online-scoring): address code-review findings on the spend budget
Three real bugs surfaced by the code-aware review:
-
Budget silently never enforced for Gemini/Vertex/OpenRouter judges. The
online-scoring provider is the LlmProvider value ("gemini", "vertex-ai"),
but CostService keys prices by the canonical provider ("google_ai",
"google_vertexai"/"anthropic_vertexai"), so the lookup missed and cost
resolved to zero. findModelPrice now canonicalizes the incoming provider and
tries candidate namespaces (Vertex hosts both Google and Anthropic models, so
both are tried and the model's own rows decide). Fixes it for every
cost-tracked provider and also fixes the monitoring trace's estimated cost;
backward compatible for callers already passing canonical names. OpenRouter
stays unpriced (no canonical namespace) and is surfaced by the unpriced warn. -
Decimal entry impossible in the "Max cost per evaluation" field. A controlled
type=number bound to the committed number erased an in-progress trailing "."
("1." committed as 1 and re-rendered "1"). Extracted a MaxCostInput that keeps
the raw text as the display source of truth and commits the parsed number
separately, re-syncing only on external value changes. -
Global @Positive -> BigDecimalStrategy PODAM registration would throw for any
manufactured @Positive int/long/Double field. Moved the positive-floor logic
into BigDecimalTypeManufacturer (scoped to BigDecimal fields), deriving the
floor from ValidationUtils.SCALE, and dropped the global attribute strategy.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(online-scoring): price judges by resolved provider only; fix manual-eval create test
Drop the CostService provider-canonicalization added while addressing an earlier
review round. The online evaluator now hands its resolved LlmProvider value
straight to CostService: OpenAI/Anthropic/Bedrock judges price on an exact
provider match, and any provider the price table doesn't cover (Gemini/Vertex/
OpenRouter/custom-llm/...) resolves to no cost — surfaced by BudgetGuard's
existing once-per-guard unpriced warn. No RUNTIME_PROVIDER_CANONICALS map, no
Vertex dual-namespace guessing, no candidate-provider fallback loop; CostService
is back to its main behavior, so all other cost callers are unchanged. Removes
the two canonicalization tests tied to that logic.Fix ManualEvaluationResourceTest: the @Valid cascade on the evaluator code field
now enforces @NotEmpty arguments on the hand-built span UserDefinedMetricPython
code (previously only set metric), so createEvaluator returned 422. Supply valid
arguments, matching the canonical AutomationRuleEvaluatorsResourceTest.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(cost): map online-eval provider names to canonical so every priced judge is cost-tracked
Online evaluation resolves models to LlmProvider serialized values ("gemini",
"vertex-ai") that differ from the canonical price-table vocabulary ("google_ai",
"google_vertexai"), so cost tracking and the spend budget silently priced those
judges at zero. Add a small, single-valued RUNTIME_PROVIDER_MAPPING applied at
lookup in CostService — a deterministic name normalization, no candidate-tries
loop and no fallback. Vertex is unambiguous because only Gemini models are
offered on Vertex for online evaluation. openai/anthropic/bedrock already equal
their canonical names; self-hosted ollama/custom-llm have no public pricing to
map to and stay at zero. Canonical names pass through unchanged, so all other
cost callers are unaffected.Covered by CostServiceTest (gemini -> google_ai, vertex_ai/gemini -> google_vertexai
both price > 0) and BudgetGuardTest (a Gemini judge now accrues spend and can trip
the budget).Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(online-scoring): add @NonNull to budget/usage entry-point params
Address review: annotate the non-nullable parameters flagged on the public
entry points — BudgetGuard.create(modelName), BudgetGuard.track(call), and
LlmUsageExtractor.toUsageMap(response). maxCostUsd stays nullable (null = no
limit).Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Co-authored-by: Claude Opus 4.8 noreply@anthropic.com
下载附件