-
[OPIK-7160] [BE] fix: fail loudly on unresolvable metric reference key (#7364)
frostbyte_neo 发布于
2026-07-08 08:56:04 +00:00 - [OPIK-7160] [BE] fix: fail loudly on unresolvable metric reference key
A metric reference_key that matched no dataset field silently scored every
item 0. No candidate could beat the baseline, so the optimizer returned the
seed prompt while the run reported "completed" -- indistinguishable from a
legitimate "no improvement over baseline" run (which gates OPIK-7038).- Add _validate_reference_key_resolves: raise InvalidMetricError at build time
when the reference key resolves against zero dataset items, listing the
available fields. Wired into equals, levenshtein_ratio, numerical_similarity. - Score missing-per-item references 0.0 with a "Missing reference value" reason
and a warning log, instead of silently defaulting to "" (which spuriously
matched empty output as a perfect 1.0). Matches the numerical_similarity
precedent already in this file. - Tests: build-time validation (total mismatch, sparse data, no provider, empty
dataset, JSONPath), per-item missing reference, and updated the JSONPath
no-match regression to assert the new loud behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7160] [BE] fix: address code-review findings on reference-key validation
Follow-up to the reference-key fix, resolving issues from a high-effort review:
- Present-but-null reference field no longer hard-fails the run. Validation now
checks key presence via a _MISSING sentinel, not non-None value, so a field
that exists but holds null (a data issue) builds instead of aborting with a
self-contradictory "did not resolve ... available fields: "
message. Only a genuinely absent field/non-matching JSONPath raises. - numerical_similarity now fails loudly when the key resolves everywhere but to
non-numeric values (the silent flat-0 case its validation previously missed),
folded into the existing scale-inference pass so the dataset is fetched once,
not twice. - Non-dict dataset items yield a clean InvalidMetricError instead of an
AttributeError. - Extracted _missing_reference_result (shared by all three scoring functions),
_available_fields_hint, and _reference_key_error to remove duplication. - Updated MetricFactory.build docstring: reference-based metrics now invoke the
provider at build time to validate (the old "only numerical_similarity" note
no longer held).
Tests: added present-null, non-numeric-reference, non-dict, sparse-numeric, and
present-empty-string cases. Metric suite 86 passed.Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- [OPIK-7160] [BE] fix: address PR review comments on reference-key validation
- Malformed JSONPath (Baz medium): a JSONPath-shaped reference_key whose
syntax is invalid and that matches no literal field now fails with a clear
"not a valid JSONPath expression" message instead of the misleading
"did not resolve against any dataset item". The literal-key fallback is
preserved, so a real field named e.g. "user@email" still builds. Applied to
both _validate_reference_key_resolves and the numerical_similarity loop. - Duplicate reference-metric builders (Baz low): extracted
_build_reference_metric shared by equals and levenshtein_ratio. - Missing behavioral assertion (Baz low): the sparse-numeric test now invokes
the metric (exact match -> 1.0, unit-off -> 0.0, non-numeric sibling -> 0.0). - Added tests for the malformed-JSONPath error on both metric paths.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Co-authored-by: Claude Opus 4.8 (1M context) noreply@anthropic.com
下载附件