-
[OPIK-4992] [Python SDK] fix: use indexed keys in LLMJudge response schema for cross-provider compatibility (#5690)
frostbyte_neo 发布于
2026-03-17 14:04:40 +00:00 - refactor(sdk): clarify LLM judge prompt with explicit delimiters
Use ---BEGIN/END--- delimiters around input and output sections so
short agent responses don't blend into the description text.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4992] [P SDK] fix: sanitize JSON schema field names for Anthropic compatibility
Anthropic's tool-use API (used by litellm for response_format) requires
property names to match ^[a-zA-Z0-9_]+$. Assertion text like "Response is
accurate" was used directly as JSON schema property names, causing API errors
with Anthropic models.Now sanitizes field names to valid identifiers (e.g. Response_is_accurate)
and maps them back to original assertion text when parsing results.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- test(sdk): add test for underscore-vs-space collision in field mapping
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- test(sdk): parametrize LLMJudge integration tests across OpenAI, Anthropic, Gemini
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(sdk): restore detailed INPUT/OUTPUT section descriptions in LLM judge prompt
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(sdk): replace free functions with ResponseSchema class in LLM judge parsers
Encapsulates field mapping, response format model, prompt formatting,
and output parsing into a single ResponseSchema object. Removes the
need to thread field_mapping through multiple functions.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(sdk): use indexed keys in LLMJudge response schema
Switch from sanitized assertion text keys to short indexed keys
(assertion_1, assertion_2, ...) with original text as field descriptions.
This avoids provider property name limits (Anthropic 64 char, OpenAI 15k
combined) while keeping assertion meaning in the JSON schema.Add unit and integration tests for many assertions (5-10).
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- ci: add Anthropic and Google API keys to LLM judge test workflow
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- test(sdk): add JSON schema structure snapshot tests for LLMJudge
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(sdk): strip noise from LLMJudge response schema sent to providers
Remove Pydantic-generated title fields and internal docstring description
from the JSON schema. Only assertion descriptions remain — everything
the provider receives is now essential for producing correct output.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(sdk): keep titles in schema, only strip leaked docstring
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(sdk): remove docstring from AssertionResultItem instead of stripping it
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(sdk): use comment instead of docstring on AssertionResultItem
Pydantic leaks docstrings into the JSON schema as "description" fields.
Use a comment to document the class without polluting the schema sent
to providers.Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- Fix lint errors
Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
下载附件