发布

RSS 订阅源

2.0.38-5354 045e3f3a0d

比较
[OPIK-5333] [BE]: agentic tools online scoring threads; (#6719)

frostbyte_neo 发布于 2026-05-15 14:04:46 +00:00
- [OPIK-5333] [BE] feat: agentic-tools branch for thread LLM-as-judge scoring
Extends the size-based agentic-tools branch from trace-level scoring to
thread-level (OnlineScoringTraceThreadLlmAsJudgeScorer). Enormous threads
no longer try to inline every trace into the prompt: when the
inline-rendered context would exceed the configured threshold, the model
gets a compact per-trace skeleton (id, name, start/end, duration, span
counts) plus ReadTool/JqTool/SearchTool to drill into any specific trace
on demand. Spans for any one trace are fetched reactively, only when the
model actually asks for that trace.

Key pieces:
- OnlineScoringEngine: new estimateThreadContextTokens (size estimate on
  the trace-list JSON, no spans), prepareThreadLlmRequestWithTools
  (skeleton + drill-down hint replacing the {{context}} variable), and a
  ThreadTraceSkeleton record describing the per-trace summary shape.
- OnlineScoringTraceThreadLlmAsJudgeScorer: full reactive refactor of
  evaluate -> Mono<List<...>>, prepareEvaluation/handleToolCalls/
  toolCallLoop mirroring the trace-level scorer, scoreTraceReactive
  wrapping the sync LangChain4j chat on boundedElastic, shouldUseAgentic-
  Tools gating on toggle + threshold + provider-supports-tools.
- TraceToolContext: new forThread(workspaceId, userName) factory + has-
  ActiveTrace() accessor for thread-scoped evaluations that lack a single
  active trace.
- GetTraceSpansTool: returns a clear redirect error when invoked on a
  thread context, pointing the model at read(type=trace, id=X) instead
  of NPEing.
- Tests: thread-routing-gate truth table, handleToolCalls no-op short-
  circuit, existing scoring tests updated for the wider constructor.
Reuses the existing toggle (TOGGLE_AGENTIC_TOOLS_ENABLED), threshold
(ONLINE_SCORING_AGENTIC_TOOLS_THRESHOLD_TOKENS), and chars-per-token
(ONLINE_SCORING_AGENTIC_TOOLS_CHARS_PER_TOKEN) config; no new env vars.

Threads don't have an experimentId-driven branch (no test-suite-assertion
equivalent), so the routing decision is purely size-based + toggle +
provider-supports-tools.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] chore: defer handleToolCalls side effects to subscription time
Both scorers' handleToolCalls were allocating TraceToolContext + the
messages list + ToolOutputBudget at method-invocation time rather than
when the returned Mono was subscribed. Works correctly under the current
single-subscriber call shape, but a future caller that composes the
returned Mono differently (or subscribes twice) would observe the side
effects out of sync with the chain.

Wrap everything below the early-return guard in Mono.defer in both
OnlineScoringLlmAsJudgeScorer and OnlineScoringTraceThreadLlmAsJudgeScorer.
The early Mono.just(chatResponse) stays outside the defer because it's
cold and pure — only the mutable allocations move inside.

No behavior change for the current callers. 79 targeted tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] refactor: extract shared reactive tool-call loop + tighten thread skeleton naming + more thread tests
Three pre-merge cleanups for the thread agentic-tools branch:
1. Extract the reactive tool-call loop (toolCallLoop, executeToolOr-
  BudgetExhausted, the Budget holder, MAX_TOOL_CALL_ROUNDS,
  CUMULATIVE_TOOL_OUTPUT_BUDGET_CHARS, BUDGET_EXHAUSTED_MESSAGE) into
  a new ToolCallLoop utility class shared by both the trace- and
  thread-level LLM-as-judge scorers. Each scorer used to carry its own
  ~60 lines of identical recursive loop machinery; only the message
  type differed. The new helper takes a Function<ChatRequest, Mono<
  ChatResponse>> from each caller so the scorer threads its own
  message through scoreTraceReactive — clean separation of "what to
  call for the next LLM round" from "how to drive the loop."
2. Tighten ThreadTraceSkeleton field naming. Added @JsonNaming(snake_-
  case) and renamed durationMs -> duration so the wire shape matches
  Trace's serialization. The model previously saw durationMs in the
  skeleton but duration in any read(type=trace, id=X) response — a
  mild schema split that could confuse the agent. Both surfaces now
  emit the same field names.
3. Three new thread-side handleToolCalls tests mirroring the trace
  scorer's coverage: accumulate-and-finalize (multi-round happy path
  with structured wrap-up), propagate-scoreTrace-failure-mid-loop
  (failure escapes; redelivery contract holds), and cap-at-MAX_TOOL_-
  CALL_ROUNDS-and-still-wrap-up (loop terminates after 10 rounds and
  the wrap-up structured call still fires).
98 targeted unit tests pass + the 80-test OnlineScoringEngineTest
integration suite. No behavior change — pure refactor + naming +
test coverage.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] test: dedicated coverage for ToolCallLoop
Adds ToolCallLoopTest with five focused cases for the shared reactive
tool-call loop extracted in the previous commit:
- Early return when the initial response has no tool calls (skip the
  loop entirely; scoreTrace never invoked).
- Round cap at MAX_TOOL_CALL_ROUNDS — model keeps emitting tool calls
  every round; loop terminates after exactly 10 follow-up scoreTrace
  invocations and returns the cap-round response.
- Budget exhaustion — a single payload at CUMULATIVE_TOOL_OUTPUT_BUDGET
  forces the next round to skip the registry and emit the budget
  sentinel; registry dispatch counter stays at 1, exhaustedLogged flips.
- Defensive message copies — each follow-up ChatRequest holds a distinct
  list instance, post-loop mutation of the caller-side messages does
  not bleed into already-captured requests.
- Tool ordering — three tool calls in a single round produce
  ToolExecutionResultMessages in the same order via concatMap (OpenAI
  rejects out-of-order tool results).
Previously, ToolCallLoop was only covered indirectly via the trace and
thread scorer tests; this makes the contract independently checked.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] chore: tighten ToolCallLoopTest helpers
- Collapse the two stubTool variants into one parameterized
  stubTool(toolName, result) helper; call sites pass TOOL_NAME
  explicitly when they don't need a custom name.
- Drop a numeric "(3 vs 5 in the current shape)" aside from a comment
  that would rot if the loop's message-shape ever changes; the
  preceding assertion already guarantees the relationship.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] refactor: address Baz Reviewer comments on thread scorer
Five fixes prompted by PR #6719 review:
- Extract summarizeResponse(ChatResponse) and addToolSpecs(request,
  toolChoice, toolRegistry) to OnlineScoringEngine; both trace and
  thread scorers now share the helpers instead of duplicating ~20 lines.
- Replace the thread scorer's "Received response for threadId '{}':\n\n{}"
  log (which dumped the full ChatResponse, leaking assistant text + tool
  args into user-facing logs) with the shape-only summarizeResponse —
  same redaction the trace scorer already applies.
- Fix the error-log Throwable handling in both scorers' prepareEvaluation
  fallbacks: was passing exception.getMessage() (string), so SLF4J
  dropped the stack trace; now passes the exception as the last arg.
- Tighten ThreadTraceSkeleton with @Builder + @NonNull on the only
  required field (id) per .agents/skills/opik-backend/SKILL.md.
- Detect multimodal templates in OnlineScoringTraceThreadLlmAsJudgeScorer
  .prepareEvaluation and fall back to the inline path with a user-facing
  warn, instead of letting renderThreadMessagesWithReplacement throw
  UnsupportedOperationException downstream and break the whole
  evaluation. Mirrors the provider-doesn't-support-tools branch.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] refactor: fold multimodal check into shouldUseAgenticTools + cover with test
Two small follow-ups to the previous Baz Reviewer-comment commit:
- Move the multimodal-template check from the prepareEvaluation caller
  into shouldUseAgenticTools alongside the toggle / size / provider
  checks. Previously the info log "switching to agentic-tools mode for
  threadId X" could fire from shouldUseAgenticTools and then be
  invalidated by the caller flipping the decision back to inline on
  multimodal templates. Now all routing decisions live in one place and
  the info log is the last thing emitted, so the log is always accurate.
- Add a RoutingGateTests case for the multimodal branch — every other
  precondition holds but contentArray-bearing template messages force
  the inline fallback. The truth-table parameterized test still covers
  the size + provider permutations.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] chore: mark ThreadTraceSkeleton.startTime as @NonNull
Trace.startTime is @NotNull at the API boundary, so a persisted trace
always carries a non-null startTime. The skeleton field should mirror
that contract — failing fast on construction if a caller ever assembles
a ThreadTraceSkeleton without one. The other timestamp fields (endTime,
duration) remain nullable since in-flight traces legitimately lack them,
and name stays nullable to match the source.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] refactor: dedupe renderThreadMessages via renderMessagesWithReplacements
renderThreadMessages carried ~40 lines of rendering loop (string-content
branch, structured-content fan-out, role mapping, fall-through warn)
that exactly mirrored renderMessagesWithReplacements. The only thread-
specific piece is assembling the replacements map for the context
variable — everything after that is identical to the trace / span
flows. Delegate to the shared helper so role-fan-out and multimodal
handling stay in one place; future changes to the render loop won't
need to be made twice. No behavior change: the shared helper's 2-arg
overload defaults to PromptType.MUSTACHE, which is what
renderThreadMessages hard-coded.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] refactor: quote log placeholders + split user-facing vs internal error logger
Three more Baz Reviewer comments addressed:
- Wrap the summarizeResponse(...) placeholder in single quotes on both
  trace and thread "Received response" logs, per SKILL.md's "values in
  single quotes" convention (log.info("...: '{}'", value)).
- Update the matching integration-test regex in
  AutomationRuleEvaluatorsResourceTest to expect the quoted shape.
- Extract OnlineScoringEngine.logPreparingLlmRequestError that emits a
  sanitized one-liner to userFacingLogger (no Throwable, so the stack
  trace doesn't leak into the user-facing sink) AND the full stack
  trace to the scorer's standard slf4j logger. Both scorers' catch
  blocks in prepareEvaluation now call the helper — dedups the catch
  body and cleans up the privacy boundary at the same time.
Skipped: the shouldUseAgenticTools dedup comment. The truly-shared
computation is two one-liners; the decision logic and log conditions
genuinely differ between trace (experimentId-override) and thread
(multimodal-block), so any helper would be roughly break-even on
lines and add indirection.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] refactor: address 3 Baz Reviewer comments on thread scorer
- Skip the thread-JSON serialization in prepareEvaluation when the
  agentic-tools toggle is off — the resulting token estimate would be
  thrown away (shouldUseAgenticTools re-checks the toggle and returns
  false). Avoids an MB-scale OBJECT_MAPPER.writeValueAsString per
  evaluation on the inline path.
- Extract OnlineScoringEngine.summarizeRequest (model-name-as-string
  variant); replace the thread scorer's full-ChatRequest "Sending
  threadId" user-facing log with the shape-only summary. Matches the
  trace scorer's existing redaction and stops leaking the rendered
  prompt + skeleton into the user-facing log sink. Trace scorer's
  private summarizeRequest now delegates to the shared helper.
- Move the post-loop wrap-up (force-closure UserMessage + final
  structured re-issue) into ToolCallLoop.runWithWrapUp. Both scorers
  shed ~15 lines of identical wrap-up flatMap. The force-closure
  message itself becomes a single private constant in ToolCallLoop.
  The lower-level run(...) overload stays for ToolCallLoopTest.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] fix: keep terminal AiMessage in wrap-up + ID thread fallback warns
- ToolCallLoop: when the model stops on its own (no-tool-calls early
  return), append the terminal AiMessage to messages before returning.
  Otherwise runWithWrapUp's structured re-issue is missing the
  assistant's last turn, and the forcing user message lands in a
  conversation history where the model's final reasoning is gone.
  
  Kept the cap-reached early return as-is: that response may carry
  unfulfilled tool_executions_requests, and appending an AiMessage with
  tool_calls but no matching ToolExecutionResultMessage would produce
  a malformed message sequence that OpenAI / Anthropic reject.
  Documented the asymmetry in both branches.
- OnlineScoringTraceThreadLlmAsJudgeScorer.shouldUseAgenticTools: add
  threadId to the two fallback warn messages (provider doesn't support
  tools, multimodal template). Operators chasing why a specific thread
  fell back to inline can now grep by id.
- Updated ToolCallLoopTest + both handleToolCalls integration tests for
  the new "append on no-tool-calls" behavior (final structured re-issue
  now carries one extra message: the terminal AiMessage).
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] fix: split user-facing vs internal error logger in Python scorer prep
logAndPrepareEvaluatorInput was the missing instance of the same fix
already applied to logPreparingLlmRequestError on the LLM-as-judge
path: passing exception.getMessage() (string) dropped the stack trace
AND duplicated the exception text in the body. Apply the same split —
userFacingLogger gets a sanitized one-liner (no Throwable, so internal
class names / paths don't leak into the user-facing log sink) and the
scorer's internal slf4j logger gets the full Throwable for diagnosis.

Both Python scorer call sites (trace + span) updated to pass log as
the new internalLogger arg.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] refactor: address 3 LifeXplorer review nits
- summarizeRequest: drop the "~N chars" field that called m.toString()
  on every ChatMessage just to measure its length. On a multi-MB
  rendered prompt that allocated the full string (~2x prompt-size heap
  churn per evaluation) even at INFO. Kept the cheap shape info
  (model, message count, tool count, useTools). Integration-test
  regex updated to match.
- Move toolRegistry.execute(...) inside the try-with-resources MDC
  scope in ToolCallLoop so the synchronous logs from ToolRegistry
  itself (e.g. the "Unknown tool requested by judge" warn) carry the
  trace/thread/rule tags. Async logs from inside the tool's subscribed
  Mono still need reactor-context propagation — documented inline.
- Replace the fully-qualified @JsonNaming annotation on
  ThreadTraceSkeleton with proper imports for JsonNaming +
  PropertyNamingStrategies.
Skipped the "drop this.trace = null in TraceToolContext's private
ctor" suggestion: trace and spans are final fields, so the compiler
requires explicit initialization in every constructor.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] refactor: collapse TraceToolContext to one ctor + 2 factories
Addresses LifeXplorer's "no need to set nulls" comment more directly:
the two-constructor pattern had a thread-scoped private ctor that
assigned this.trace = null; and this.spans = null; purely to satisfy
the Java compiler's definite-assignment rule for final fields. Code
smell even though the assignments were required as-written.

Collapsed to a single nullable-accepting private constructor + two
static factories:
- forActiveTrace(trace, spans, ws, user) for trace-scoped contexts
- forThread(ws, user) for thread-scoped contexts
The explicit nulls now live at the forThread call site where the
absence of an active trace is the point, not buried in a parallel
constructor. All 8 call sites (1 prod + 7 test) updated to use the
factory; the 4-arg public constructor is gone.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
- [OPIK-5333] [BE] refactor: dedup renderThreadMessagesWithReplacement + tidy Sending log
- renderThreadMessagesWithReplacement no longer carries its own string-
  content rendering loop. It now asserts string-only templates (the
  caller's shouldUseAgenticTools already detects multimodal and falls
  back to inline upstream) and delegates to renderMessagesWithReplacements
  so role-switch / template-engine logic stays in one place. Same dedup
  pattern as renderThreadMessages.
- Drop the now-stale isInfoEnabled guard + comment on the "Sending
  traceId/threadId to LLM" log lines. The guard existed because the
  earlier summarizeRequest streamed over the message list to total up
  character counts; that field was already dropped, so the helper is
  cheap and the guard is dead weight.
LifeXplorer also suggested demoting the "Sending" log to DEBUG. Keeping
at INFO: the cost concern that motivated the suggestion is resolved by
the prior chars-count drop, the line completes the
Evaluating → Sending → Received chain operators rely on in the UI logs,
and the integration test (AutomationRuleEvaluatorsResourceTest$GetLogs)
asserts 4 INFO entries per evaluation — demoting would break that
contract.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Co-authored-by: Sasha sasha@Sashas-MacBook-Pro.local
Co-authored-by: Claude Opus 4.7 (1M context) noreply@anthropic.com
下载附件
- 源代码 (ZIP)
- 源代码 (TAR.GZ)

0 发布 8,340 Git 标签

[OPIK-5333] [BE]: agentic tools online scoring threads; (#6719)