-
feat(benchmark): add evaluator info and optional examples to YAML export (#2296)
frostbyte_neo 发布于
2026-02-21 12:23:34 +00:00 - feat(benchmark): add evaluator info and optional examples to YAML export
Add evaluator/grader model details to benchmark YAML exports so
reviewers can verify evaluation quality. Individual question results
are now optionally included via a confirm dialog, with PII warnings
for custom datasets.- New lightweight /benchmark/api/results//export endpoint
- Evaluator section built from evaluation_config (allowlisted keys)
- Confirm dialog before including individual examples
- Updated template and README with evaluator + examples sections
- fix: remove unused allowedEvalKeys, improve yamlEscape, add export tests
Address AI code review feedback:
- Remove unused allowedEvalKeys variable (keys are accessed explicitly)
- Add tab character escaping to yamlEscape helper
- Add tests for the new export_benchmark_results endpoint
- refactor: compact YAML export — data only, no bloat
Remove comments, empty placeholders, duplicate fields, and boilerplate
notes from generated benchmark YAML. Cuts file size from ~1.5 KB to
~500 bytes per export (~3x smaller).- fix: restore data-provenance comments on context_window and max_tokens
These values come from current settings at download time, not from the
benchmark run itself. The inline comments warn users the data may not
reflect the actual benchmark configuration.- fix: show context_window_unrestricted for cloud providers in YAML export
Local providers (ollama/llamacpp/lmstudio) show context_window from
local_context_window_size. Cloud providers show context_window_unrestricted
flag, plus context_window only when unrestricted is disabled.-
Add audit logging to benchmark export endpoint
-
fix: snapshot LLM settings into search_config at benchmark start
The YAML export was fetching max_tokens, context_window_size,
context_window_unrestricted, and local_context_window_size live from
/settings/api at download time. If settings changed between running
and exporting, the YAML contained wrong values.Now these 4 settings are captured into the search_config JSON column
when the benchmark starts (both start_benchmark and
start_benchmark_simple routes), and the HTML template reads them
from search_config instead of making a live fetch.- fix: replace confirm() dialog with separate YAML/+Examples buttons
The confirm() popup was misleading — users click OK reflexively.
Now there are two explicit buttons: "YAML" for summary-only export
(the default/common case) and "+ Examples" for the full export with
individual Q&A pairs. Also add flex-wrap to prevent button overflow.下载附件