发布

RSS 订阅源

1.10.43-4538 3a7b9f4dce

比较
[OPIK-4727] [FE] feat: optimizer screens face-lift (#5554)

frostbyte_neo 发布于 2026-03-17 18:48:53 +00:00
- design doc
- FE communication and ERD additions
- ui reporting events in flow
- changes
- add reason to TrialItemRun
- [NA] [SDK] feat: add greenfield optimization framework package
Implements a new optimization framework (apps/opik-optimizer) that
decouples optimizer algorithms from experiment execution, persistence,
and UI concerns. Integrates via the existing optimization studio pipeline
(Redis queue → Python backend → subprocess).

Key components:
- Orchestrator: central lifecycle controller with sampler, validator,
  materializer, result aggregator, and event emitter
- StupidOptimizer: 2-step test optimizer (3 candidates → best → 2 more)
- EvaluationAdapter: wraps SDK evaluate_optimization_suite_trial()
- Backend integration: new Redis queue, framework_optimizer job processor,
  framework_runner subprocess entry point
Also adds evaluate_optimization_suite_trial() to the Python SDK, combining
optimization trial linkage with evaluation suite behavior (evaluators and
execution policy from the dataset).

53 unit + integration tests passing. Verified end-to-end against Comet cloud
with real LLM calls, UI progress chart, prompt display, and score tracking.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- Adjustments for UI and framework review
- fix: address PR review comments - dict access bug and theme color
- Fix AttributeError in framework_runner.py: dataset.get_items() returns
  dicts, use item["id"] instead of item.id
- Fix hard-coded hex color in TrialPassedCell.tsx: use text-success CSS
  class instead of text-[#12B76A] for proper dark theme support
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: address remaining PR review comments
- Add opik:optimizer-framework to default RQ queue names so framework
  jobs actually get consumed by workers
- Add dataset size guard in orchestrator before sample_split to provide
  a clear error message for datasets with fewer than 2 items
- Extract shared optimizer_job_helper.py to deduplicate identical logic
  between optimizer.py and framework_optimizer.py
- Extract checkIsEvaluationSuite helper in optimizations.ts to
  deduplicate predicate shared between CompareTrialsPage and
  useCompareOptimizationsData
- Fix hardcoded "pass_rate" in experiment_executor.py to use the actual
  metric_type parameter
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: separate experiment scores from feedback scores and handle single-item datasets
Splits the combined feedback/experiment scores into distinct fields in the
Optimization API and DAO so the frontend can fall back to experiment_scores
when feedback_scores lack the objective. Allows single-item datasets by
returning a train-only split instead of raising. Extracts shared runner
environment setup into runner_common.py.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: extract shared getBestOptimizationScore helper to deduplicate logic
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: evaluate baseline on full dataset instead of validation split only
The baseline was evaluated on split.validation_item_ids, which with an
80/20 split ratio meant only 1 out of 5 items was used. This gave an
unrepresentative baseline score. Now uses the full dataset_item_ids list.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat: enrich GEPA experiment metadata for optimization visualization
Add rich metadata to each experiment so the UI can aggregate and
visualize the optimization trajectory. Key changes:
- step_index increments only when candidate changes (not per eval)
- candidate_id is stable across re-evaluations of the same prompt
- parent_candidate_ids always set correctly for derived candidates
- New metadata fields: batch_index, num_items, capture_traces, eval_purpose
- Refactor optimizer package: protocol + factory pattern for registration
- Add GEPA adapter bridging GEPA callbacks to framework metadata
- Fix BE tests for experimentScores null and queue routing
- Add docs: ADDING_AN_OPTIMIZER.md and GEPA_IMPLEMENTATION.md
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: address PR review comments and simplify optimizer factory
- Remove register_optimizer public API and OptimizerFactory class;
  replace with a simple dict in _load_registry()
- framework_runner: avoid holding full dataset items in memory
- Update docs and tests to match simplified factory
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: lineage-based step_index and parent_candidate_ids for GEPA experiments
- Replace sequential step_index counter with parent-lineage derivation
  (max parent step + 1), so all re-evaluations of the same candidate
  share the same step_index
- Ensure every non-baseline experiment carries parent_candidate_ids,
  enabling the UI to draw lineage graphs
- Pass batch_index, num_items, capture_traces, and eval_purpose through
  to experiment metadata for richer visualization
- Revert runner scripts to direct invocation (remove runner_common.py)
- Update unit tests to match new metadata contract
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: remove unused config_hash and merge event emitters
- Remove canonical_config_hash from Candidate and TrialResult types,
  candidate_materializer, experiment_executor, and all tests
- Delete util/hashing.py module (unused — GEPA does minibatching so
  config-hash dedup would block valid re-evaluations)
- Merge SdkEventEmitter and LoggingEventEmitter into a single
  EventEmitter class with optional optimization_id
- Update GEPA_IMPLEMENTATION.md to reflect parent_ids tracking fixes
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: make CandidateConfig a plain dict and pass baseline_config through context
- Replace CandidateConfig dataclass with dict[str, Any] type alias
- Add baseline_config field to OptimizationContext (caller-provided, opaque)
- Orchestrator passes baseline_config through without knowing its structure
- Optimizers copy baseline_config and override prompt_messages only
- Remove result_aggregator module (inlined into evaluation_adapter)
- Move gepa imports to runtime (lazy) for optional dependency
- Fix protocol.py training_set/validation_set types to list[dict]
- Update ADDING_AN_OPTIMIZER.md to reflect all changes
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: move gepa tests to library_integration to avoid unit suite dependency on gepa
The gepa tests patch gepa.core.adapter.EvaluationBatch and gepa.optimize,
requiring the optional gepa package at import time. Moving them to
tests/library_integration/gepa/ with pytest.importorskip("gepa") keeps
the unit suite fast and dependency-free.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: remove event_emitter from optimizer interface, auto-emit step progress
Optimizers no longer receive or call event_emitter directly. The
EvaluationAdapter now auto-detects step_index changes during evaluate()
and emits on_step_started internally. GEPAProgressCallback simplified
to only forward GEPA events to the adapter.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- test: assert on actual log messages in event emitter tests
Use caplog to verify logger.info output includes optimization ID and
event details, instead of just checking calls don't crash.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: set evaluation_method on optimizer trial experiments for correct UI detection
evaluate_optimization_suite_trial was creating experiments without
evaluation_method="evaluation_suite", causing the backend to default
to "dataset". The frontend checkIsEvaluationSuite now uses the explicit
evaluation_method field instead of heuristic score detection.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: validate dataset is evaluation suite before running suite evaluation
Adds a guard to evaluate_suite and evaluate_optimization_suite_trial that
checks dataset.dataset_type == "evaluation_suite" before proceeding. This
prevents silently running an ineffective suite trial on a plain dataset
with no scoring rules.
- Add dataset_type param to Dataset constructor, populated at all call sites
- Add dataset_type property to Dataset
- Add _validate_dataset_is_evaluation_suite in evaluator.py
- Update tests and add rejection test
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: extract _run_suite_evaluation to deduplicate suite evaluation flow
evaluate_suite and evaluate_optimization_suite_trial had their entire body
duplicated. Extract shared logic into _run_suite_evaluation, parameterized
by optimization_id and dataset filters.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4727] [FE][BE] feat: optimization studio UI improvements
Comprehensive face-lift for optimizer screens including new KPI cards,
metric comparison cells, configuration diff views, progress charts,
trial status indicators, and backend dataset_item_count support.
Also adds backward compatibility for SDK-based optimizations.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4727] [FE] feat: optimizer screens face-lift
- Dataset name column: hover icon instead of clickable link
- Split Accuracy into Pass rate + Accuracy columns with compact metric display
- Conditionally hide Accuracy column when no old-type optimizations exist
- Remove Logs/Configuration tabs from single optimization page
- Fall back to studio_config for configuration display on old optimizations
- Chart tooltip: remove pass rate percentage background color
- Fix dataset hover icon vertical centering
- Restore feature toggle for optimization studio
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4727] [FE] fix: center trend arrow icons and rename tooltip label
- Fix arrow icon vertical centering in compact metric Tag
- Rename "Avg. runtime cost" to "Runtime cost" in chart tooltip
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4727] [FE] fix: polish optimizer screens UI consistency
- Fix chart tooltip background (use --background instead of --popover)
- Align column types with correct icons (cost, duration, numberDictionary)
- Align KPI card icons to match table column type icons
- Lowercase labels: Evaluation results, Best configuration, Runtime cost, Opt. cost, Optimization cost
- Darken success green color for better readability
- Remove Traces KPI card from trial view
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4687] [SDK] feat: GEPA v2 optimizer with reflection-based prompt evolution (#5547)
- [OPIK-4687] [SDK] feat: integrate GEPA v2 optimizer into framework
Add GepaV2Optimizer that delegates to the external gepa library (v0.1.0+)
for genetic-Pareto prompt optimization. Includes adapter bridging GEPA's
evaluate/reflect interface to the framework's EvaluationAdapter, lifecycle
event tracking via callbacks, result caching, and a reflection prompt that
encourages generalizable instructions while preserving template variables.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(gepa-v2): improve reflection feedback with structured assertions and dynamic inputs
- Extract template variables from prompt messages for dynamic input field mapping
- Store per-assertion structure (name, value, reason) instead of flat reason strings
- Show only failed assertions in reflection feedback for focused improvement signals
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(gepa-v2): adapter reflection control, FE chart filtering, experiment typing
- Move reflection to adapter's propose_new_texts with custom prompt template
- Use msg["name"] as candidate key when provided, fallback to {role}_{index}
- Strip echoed parameter prefix from reflection LLM output
- Disable GEPA evaluation cache so validations produce full-dataset experiments
- Tag exploration evals as mini-batch, only baseline/init/validation as trial
- FE: filter mini-batch experiments from optimization progress chart
- FE: show individual assertion score columns alongside "passed" for eval suites
- Update E2E script: no dataset split, max_candidates=10, reflection log capture
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(gepa-v2): improve reflection quality with structured feedback and template filtering
- Show FAILED and PASSED assertions separately in reflection feedback
- Keep worst run per item (most failed assertions) for reflection
- Sort reflective dataset records by failure count (most failures first)
- Exclude template-only messages (e.g. {question}) from GEPA seed candidate
- Rewrite reflection prompt: focus on failures, preserve what works, 500-word limit
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(gepa-v2): classify experiment type by batch size, not eval purpose
The purpose-based classification was unreliable: GEPA calls evaluate()
with capture_traces=False for both full validations and minibatch
evaluations of new candidates, making them indistinguishable by purpose.

Now records the full dataset size on the first evaluate call (initialization)
and classifies any call with fewer items as mini-batch.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(gepa-v2): improve scoring, stopping, and reflection quality
- Use mean instead of min for per-item assertion scores, giving GEPA
  granular signal instead of binary 0/1
- Track total_runs/passed_runs per item so reflection prompt shows
  whether failures are consistent or intermittent
- Stop on trial.score (framework experiment score) instead of GEPA's
  internal mean, so pass_threshold semantics are respected
- Rewrite reflection template with 4-step structure: diagnose, keep
  what works, write assertion-matched rules, generalize
- Increase max_metric_calls multiplier to 5x for deeper exploration
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(gepa-v2): hide mini-batch trials from table, use domain-neutral examples
- Filter mini-batch experiments from the trials table rows so only
  full evaluation trials are shown
- Replace customer-support-specific examples in the reflection
  template with domain-neutral ones
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(gepa-v2): show all runs per item in reflection feedback, log rendered prompt
Previously kept only the worst run per item for reflection. Now all runs
are preserved and shown separately (Run 1/3, Run 2/3, etc.) so the
reflection LLM can see what varies across attempts. Also captures the
fully rendered reflection prompt in the reflection log for debugging.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(gepa-v2): consolidate runs per input in reflection dataset, label assertion/reason
Consolidate multiple runs for the same input into a single record with
a Runs field and per-item Summary (pass count + consistent failures).
This eliminates input duplication (~40% token savings) and makes cross-run
comparison trivial. Also separates Assertion/Reason onto labeled lines
for clearer parsing by the reflection LLM.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- test(gepa-v2): add feedback format coverage for reflection dataset
Add tests for single-run flat keys, multi-run Assertion/Reason labels
in Runs field, and failed assertions with empty reason.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(gepa-v2): tell reflection LLM that examples are sorted by priority
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(gepa-v2): use flat config instead of prompt_messages
GEPA adapter now works with flat dict[str, str] candidates instead of
knowing about message roles. baseline_config is the single source of
truth with system_prompt and user_message keys. Added LLMChatTask that
constructs LLM messages from flat config keys, replacing the
prompt_messages reconstruction path.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(gepa-v2): use TrialResult.config instead of prompt_messages, fix validator and UI prompt display
Replace TrialResult.prompt_messages with TrialResult.config so config is
the single source of truth. Update candidate_validator to accept flat
message keys (system_prompt, user_message) in addition to prompt_messages.
Populate experiment metadata "prompt" from flat keys so the UI displays
prompts correctly.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(gepa-v2): add optimizable_keys to OptimizationContext
Replace hardcoded PROMPT_KEYS in GepaV2Optimizer with
context.optimizable_keys so the caller explicitly controls which
baseline config keys get optimized.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(gepa-v2): add failure-aware batch sampler for minibatch item selection
Replace GEPA's default uniform sampler with FailureAwareBatchSampler that
guarantees failed items from the last full eval appear in subsequent
minibatches, giving the reflection LLM actionable signal instead of wasting
iterations on easy items.

Parameters: min_failed_per_batch, min_unseen_per_batch, failure_threshold.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(gepa-v2): strict types in sampler, worst-first failed selection, update implementation doc
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(experiment): surface optimizable keys in experiment configuration
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(gepa-v2): failure streak tracking, history-aware reflection prompt, optimizable keys in config
- Track per-item failure streaks and failing assertion names in sampler
- Annotate reflective dataset records with "Failure History" for stuck items
- Rewrite reflection prompt: failure history step, structured output, topic headers
- Surface optimizable_keys in experiment config and baseline evaluation
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(gepa-v2): balanced reflection prompt, lower failure history threshold
- Rewrite reflection prompt to balance conservative and aggressive approaches:
  preserve working rules while encouraging grouped topic headers (## Empathy,
  Resolution, etc.) instead of flat numbered lists
- Lower failure history threshold from streak >= 2 to >= 1 so the reflection
  LLM sees failure context from the first repeated failure
- Guard failure history annotation with if stuck to avoid empty annotations
- Relax "3 unreturned callbacks" assertion to "multiple unreturned callbacks"
  (the exact-count version was too brittle for gpt-4o-mini to satisfy reliably)
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(gepa-v2): balanced 50/50 minibatch sampling, 20-item e2e suite
Balanced sampling: split minibatches ~50/50 between failed (worst-first)
and passed (random) items. Previously batches were almost entirely failed
items, causing the reflection LLM to over-correct and regress passing
behaviors (catastrophic 0.0 scores). Passed items now act as behavioral
anchors.
- Remove unseen item tracking (mark_seen, min_unseen_per_batch)
- Default min_failed_per_batch=1 (was batch_size-1)
- Minimum reflection_minibatch_size=4 (ensures 2+2 split)
- Redesign e2e suite: 20 items (5 easy, 7 medium, 8 hard)
- Fix contradicting assertions (hedging language vs no promises)
- Remove impossible assertions (specific loyalty benefits)
- Add problematic items summary to reflection log
- Save reflection log from orchestrator finally block
- Update GEPA_IMPLEMENTATION.md
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(gepa-v2): extract collaborators from adapter with DI
Split FrameworkGEPAAdapter into three injectable collaborators:
- CandidateTracker: candidate identity, parent lineage, GEPA index mapping
- ReflectiveDatasetBuilder: feedback dataset construction for reflection LLM
- ReflectionProposer: reflection LLM interaction and logging
The adapter is now a thin facade (~300 lines, down from 664) that
orchestrates evaluation and delegates to collaborators. Compatibility
properties ensure all existing tests pass unchanged.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(gepa-v2): move reflection template to ReflectionProposer
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(gepa-v2): task-agnostic reflection template with prompt descriptions and sibling awareness
Rewrite the reflection template to be domain-neutral, add optional
prompt_descriptions to OptimizationContext so the reflection LLM
understands what each parameter does, and include sibling parameter
context so the LLM knows what other params exist without modifying them.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- docs(gepa-v2): update implementation doc and reflection prompt algorithm
Update GEPA_IMPLEMENTATION.md with prompt descriptions, sibling awareness,
and task-agnostic template details. Rewrite REFLECTION_PROMPT_EXAMPLE.md
to document the full reflection prompt assembly algorithm with a rendered
example showing the new header format.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(gepa-v2): robust header stripping, markdown formatting in reflection template
The LLM sometimes echoes header metadata (Parameter:, Description:,
param name) in reformulated form. Replace exact-prefix matching with
line-by-line stripping of metadata patterns. Add IMPORTANT instruction
to not include metadata in output. Request markdown ## headers in STEP 4.

Add 11 unit tests for ReflectionProposer: header stripping edge cases,
build_header with/without descriptions, template content assertions.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- revert(fe): remove debug FE changes (mini-batch filtering, column reorder)
These were temporary UI tweaks for debugging the GEPA v2 optimizer.
They'll be re-implemented properly in a separate FE PR.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(gepa-v2): preserve template variables in reflection prompt
Instruct the reflection LLM to keep all template variables (e.g.
{var}, {{var}}, , {% var %}) intact during prompt rewriting.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: rename gepa_v2 to gepa, clean up OptimizationContext
- Rename gepa_v2/ -> gepa/ (now the primary optimizer)
- Rename gepa/ -> gepa_old/ (legacy optimizer)
- Rename GepaV2Optimizer -> GepaOptimizer
- Rename GepaOptimizer -> GepaLegacyOptimizer
- Remove unused fields from OptimizationContext: prompt_messages,
  metric_parameters, model_parameters
- Rename prompt_descriptions -> config_descriptions
- Delete SimpleOptimizer and its tests
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat: add configurable split_strategy to OptimizationContext
Add split_strategy field ("80_20" default, "no_split" for GEPA) so the
orchestrator handles dataset splitting instead of individual optimizers.
Remove internal train+val dedup logic from GepaOptimizer.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(gepa): clean up adapter API and fix code quality issues
- Make adapter facade properties public (remove underscore prefix)
- Add standalone reflection_log fallback to prevent silent data loss
- Rename consume_pending_capture_traces → get_pending_capture_traces
- Remove dead guard in _build_evaluation_batch
- Move SYSTEM_PROMPT_KEY constant to test file
- Fix update_scores type annotation in failure_aware_sampler
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: remove prompt_messages logic, validate optimizable_keys
- Adapt LLMTask to use config dict, remove LLMChatTask duplicate
- Simplify candidate_validator to check optimizable_keys from adapter
- Remove prompt_messages fallback from experiment_executor metadata
- Update all tests, fixtures, scripts, and docs to flat key format
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(scripts): remove stale prompt_messages and API references
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(gepa): remove optimizable_keys from config dict to fix caching
optimizable_keys was being injected into CandidateConfig by both
_make_config_builder and the orchestrator, causing cache key mismatches
between baseline and initialization evaluations. Pass it as an explicit
parameter through the evaluation chain instead.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(scripts): rename gepa_v2 scripts to gepa, delete run_optimization_e2e
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- docs: use optimizable_keys generically in ADDING_AN_OPTIMIZER guide
Remove hardcoded system_prompt references from code examples.
Optimizers should iterate over context.optimizable_keys instead
of assuming specific key names.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
- fix(fe): address baz review comments
- Use Tag component variants instead of hard-coded color spans for
  theme-aware diff badges (Added/Removed/Changed)
- Clamp formatAsPercentage input to [0, 1] range to prevent >100% or
  negative percentage display
- Read baseline score from experiment_scores as fallback when
  feedback_scores lacks the objective (evaluation-suite support)
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(fe): extract getObjectiveScoreValue shared helper
Move the feedback_scores -> experiment_scores fallback into a reusable
getObjectiveScoreValue helper in feedback-scores.tsx. Replace all 4 call
sites (CompareTrialsPage, TrialKPICards, useOptimizationScores,
useCompareOptimizationsData) with the shared helper.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(be): resolve CI failures - migration conflict and test ignored fields
- Rename migration 000063 → 000064 to avoid prefix conflict with main
- Add datasetItemCount to EXPERIMENT_IGNORED_FIELDS and test builder
- Add datasetName to OPTIMIZATION_IGNORED_FIELDS (transient field)
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(fe): extract shared aggregateExperimentMetrics helper
Deduplicate weighted score/cost/latency accumulation logic that was
duplicated between TrialKPICards and useCompareOptimizationsData.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: add optimization_id index on experiments and remove dead code
- Add minmax index on experiments.optimization_id to speed up
  optimization queries that join experiments by optimization_id
- Remove unused OptimizationDiffView component (dead code from iteration)
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- Update Helm documentation
- [OPIK-4383] [BE] Redis stream subscriber for debounced experiment aggregates recomputation (#5371)
- [OPIK-4380] [BE] Add experiment aggregates for denormalized metrics
- Add experiment_aggregates and experiment_item_aggregates tables
- Implement ExperimentAggregatesDAO with population and query methods
- Add ExperimentAggregatesService for aggregation management
- Refactor DTOs into organized model classes:
  - ExperimentAggregatesModel: aggregation results
  - ExperimentEntityData: entity models
  - ExperimentSourceData: raw source data
  - ExperimentAggregatesUtils: utilities
- Add FEEDBACK_SCORES_AGGREGATED filter strategy for map-based filtering
- Add comprehensive integration tests (10/10 passing)
- Configure batch size and parallelism settings
- [OPIK-4380] [BE] Add MySQL deadlock retry mechanism for concurrent dataset operations
Problem:
- MySQL deadlock on dataset_version_tags composite PRIMARY KEY (workspace_id, dataset_id, tag)
- Occurred during parallel dataset creation in same workspace
- Multiple threads inserting "latest" tag for different datasets caused lock contention
- Experiments with parallel execution were failing with MySQLTransactionRollbackException
Solution:
- Add handleOnDeadLocks() method in RetryUtils with:
  - 5 retry attempts with exponential backoff (250ms to 2s)
  - 0.5 jitter to reduce thundering herd effect
  - Recursive isDatabaseDeadlock() detection for MySQLTransactionRollbackException
- Apply retry logic in DatasetItemService.setDatasetItemVersion()
- Enables concurrent dataset creation for same workspace
Impact:
- Supports parallel experiment execution with proper deadlock handling
- Test success rate improved from 0/10 to 10/10 in ExperimentAggregatesIntegrationTest
- Fix visibility
- [OPIK-4380] [BE] Address PR review comments for experiment aggregates
Fixed 11 automated review comments from baz-reviewer:

CRITICAL fixes:
- Prevent NPE on null span aggregations by adding coalesce() in SQL
- Handle multi-project experiments with LIMIT 1 in GET_PROJECT_ID
- Handle zero-item experiments with empty aggregation helpers
- Bind feedback_scores_percentiles map instead of empty CAST
HIGH priority fixes:
- Use toDecimal128(12) instead of toDecimal64(9) for cost percentiles
- Add null-safe tags handling with Optional.ofNullable()
- Include exception objects in retry logging for stack traces
MEDIUM priority fixes:
- Add missing log_comment to SELECT_EXPERIMENT_BY_ID query
- Add missing log_comment to GET_PROJECT_ID query
LOW priority fixes:
- Remove duplicate "id" binding in bindItemsParameters
- Enhance batchSize config documentation with details
All 11 integration tests passing.
- [OPIK-4382] [BE] Refactor experiment aggregates with import cleanup and Optional patterns
- Add missing imports for IntStream, ProjectStats, and other dependencies
- Replace fully-qualified class names with proper imports across DAO and Service classes
- Fix IS_NOT_EMPTY filter handling for FEEDBACK_SCORES_AGGREGATED strategies
- Refactor null checks to use Optional in mapping methods:
  - mapFeedbackScoreAggregations, mapExperimentFromAggregates
  - mapFeedbackScoreData, mapExperimentGroupAggregationItem
  - Batch insert preparation with Optional chains
- Improve code readability and maintainability with functional patterns
- [OPIK-4380] [BE] Fix table definition
- [OPIK-4380] [BE] Address PR comments and consolidate DatasetItemService methods
- Fix tags NPE in ExperimentAggregatesDAO with defaultIfNull
- Remove unnecessary FINAL clause from GET_EXPERIMENT_DATA query
- Fix test naming in ExperimentAggregatesIntegrationTest
- Consolidate 7 duplicate createVersionFromDelta methods into single canonical implementation
- Remove debug logger from config-test.yml
- [OPIK-4380] [BE] Fix missing log_comment and centralize search criteria binding
- Fix SELECT_EXPERIMENT_BY_ID to properly render log_comment metadata
  - Use getSTWithLogComment pattern in getExperimentFromAggregates
  - Ensures ClickHouse query logging populates workspace/experiment IDs
- Centralize ExperimentSearchCriteria binding logic
  - Create ExperimentSearchCriteriaBinder utility class
  - Parameterize filter strategies to support both DAO variants
  - Eliminate 29-line duplication between ExperimentDAO and ExperimentAggregatesDAO
  - Single source of truth prevents DAOs from getting out of sync
- [OPIK-4380] [BE] Fix createVersionFromDelta consolidation after rebase
- Update canonical method signature to include new parameters:
  - List evaluators
  - ExecutionPolicy executionPolicy
  - boolean clearExecutionPolicy
- Update all 5 caller sites to pass new parameters:
  - Use changes.evaluators(), changes.executionPolicy(), changes.clearExecutionPolicy() when available
  - Pass null/false for auto-generated versions that inherit from base
- Add imports for EvaluatorItem and ExecutionPolicy
Fixes compilation errors introduced by rebase with upstream changes to DatasetVersionService
- [OPIK-4380] [BE] Address PR review comments - fix type mismatch, extract constants, remove DAO logging
- Fixed BigDecimal[] to Double[] conversion for experiment_scores (matches ClickHouse Float64)
- Extracted FilterStrategy lists to static final constants to avoid repeated allocations
- Added @NonNull validation to populateExperimentAggregate parameter
- Removed DAO layer logging, keeping service-level logging only
- [OPIK-4380] [BE] Extract shared helper for experiment data pagination
Extract streamWithExperimentPagination() helper method to eliminate
duplication in getTracesData(), getSpansData(), and getFeedbackScoresData().

All three methods followed identical pattern:
- asyncTemplate.stream with connection
- getSTWithLogComment with cursor flag
- Bind workspace_id, experiment_id, project_id, limit
- Optional cursor binding
- Result mapping
Benefits:
- Single source of truth for pagination binding logic
- Prevents divergence when tweaking cursor/limit bindings
- Reduces code from ~20 lines to ~10 lines per method
- Type-safe generic implementation
Note: CTE redundancy (3x experiment_items scan) is intentional to avoid
passing large trace ID lists as parameters, which would cause performance
issues with 10K+ traces.
- [OPIK-4383] [BE] Add Redis stream subscriber for debounced experiment aggregates recomputation
- Add ExperimentDenormalizationConfig implementing StreamConfiguration with debounce, job lock, and per-experiment aggregation lock settings
- Add ExperimentAggregationMessage as stream message record
- Add ExperimentAggregatesSubscriber consuming from the denormalization stream; acquires a workspace-scoped distributed lock per experiment before calling populateAggregations()
- Add experimentDenormalizationEnabled feature flag to ServiceTogglesConfig and FeatureFlags
- Wire ExperimentDenormalizationConfig into OpikConfiguration
- Update config.yml and config-test.yml with full experimentDenormalization block (enabled for tests)
- Add ExperimentAggregatesSubscriberTest covering lifecycle gating and processEvent success/error paths
- Revision 2: Address PR comments - add config defaults, remove toggle, rename tests
- ExperimentDenormalizationConfig: add sensible defaults to all fields so
  Dropwizard validation doesn't fail when the config block is absent from
  old deployments (config.isEnabled()=false still gates the subscriber)
- Remove experimentDenormalizationEnabled service toggle from
  ServiceTogglesConfig, FeatureFlags, config.yml and config-test.yml -
  the infrastructure gate (config.isEnabled()) is the single control point
- Rename lifecycle test methods to camelCase per project conventions:
  startSkipsStartupWhenDisabled / stopSkipsShutdownWhenDisabled
- Revision 3: Add @Max(500) to consumerBatchSize and @NotNull to jobLockWaitTime
- [OPIK-4380] [BE] Address PR review comments - fix TYPE_REFERENCE visibility, redundant IN subquery, hardcoded context keys, Instant.now in loop, and inline defaultIfNull
- Revision 4: Address remaining JetoPistola review comments (#7, #8, #10)
- #7: Remove "Used for testing and verification" from getExperimentFromAggregates javadoc
- #8: Replace recursive flatMap with Mono.expand() in populateExperimentItemsInBatches
- #10: Remove unrelated subscribeOn addition from DatasetItemService.createVersionFromDelta
- Revision 3: Add switchIfEmpty fallback for deleted traces in populateExperimentAggregate
- Fix tests
- Revision 6: Move countTotal log from DAO to service layer
Operational logs belong in the service layer, not the DAO.
- Revision 7: Apply Spotless formatting
- Revision 8: Make populateAggregations(UUID, int) private
Removes the uncapped public batch size entry point. All callers now go
through the public no-arg overload which reads batchSize safely from config.
- [OPIK-4380] [BE] Add evaluation_method support to experiment_aggregates pipeline
- Add ClickHouse migration (000062) to add evaluation_method column to experiment_aggregates table
- Add evaluationMethod field to ExperimentData record
- Update GET_EXPERIMENT_DATA query to read evaluation_method from experiments
- Update INSERT_EXPERIMENT_AGGREGATE to write evaluation_method to experiment_aggregates
- Update SELECT_EXPERIMENT_BY_ID to read evaluation_method from experiment_aggregates
- Fix Experiment record constructor call: insert EvaluationMethod at correct position (10)
- [OPIK-4380] [BE] Extract shared helper for experiment aggregation queries
Reduce copy-paste in getTraceAggregations, getSpanAggregations, and
getFeedbackScoreAggregations by extracting queryExperimentAggregation,
which centralises the context-aware execution, workspace/experiment/project
parameter binding, and singleOrEmpty pattern shared by all three methods.
- [OPIK-4380] [BE] Enforce non-null contract on countTotal criteria parameter
Add @NonNull to ExperimentSearchCriteria in the interface and implementation
so that a null argument fails fast with an explicit NullPointerException at
the DAO boundary instead of crashing deep inside buildCountTemplate.
- [OPIK-4380] [BE] Fix countTotal ignoring target project IDs in normal path
target_project_ids was only applied inside the project_deleted LEFT JOIN
subquery; the main WHERE had no project restriction, so counts were
workspace-wide. Reuse has_target_projects in the main WHERE so
project_id IN :target_project_ids always takes effect. Also replace
manual null/empty checks with CollectionUtils.isNotEmpty.
- [OPIK-4380] [BE] Apply Spotless formatting
- [OPIK-4382] [BE] Address PR review comments on experiment aggregates
- Fix :versionId → :version_id parameter naming in SQL templates and bindings
- Fix last_updated_at binding to use item.lastUpdatedAt() instead of Instant.now()
- Fix FEEDBACK_SCORES_AGGREGATED_IS_EMPTY filter: embed generated SQL into templates
  instead of hard-coded static condition, and add missing bind calls
- Fix RetryUtils log duplication (remove getMessage() + pass exception directly)
- Add batchSize = 1000 default in ExperimentAggregatesConfig
- Extract resolveVersionIdForCriteria helper to deduplicate version-id resolution
- Add null/blank/ClickHouse placeholder guards in extractUuidsFromGroupValues
- Extract loadEntityMap helper to deduplicate getEnrichInfoHolder enrichment logic
- Revision 3: Address PR comments E, F, G, H
- Fix E: Extract shared template/bind helpers in ExperimentAggregatesDAO
- Fix F: Bind experiment_ids as UUID[] instead of String[]
- Fix G+H: Extract getEnrichInfoHolder logic into ExperimentGroupEnricher,
  eliminating duplication between ExperimentService and ExperimentAggregatesService
  without introducing a direct dependency between them
- Revision 4: Fix ExperimentServiceTest to include ExperimentGroupEnricher mock
- [OPIK-4382] [BE] Extract shared Row→ExperimentGroup mappers into ExperimentGroupMappers
Pull the duplicated Row→ExperimentGroupItem and Row→ExperimentGroupAggregationItem
conversion logic from ExperimentDAO and ExperimentAggregatesDAO into a shared
ExperimentGroupMappers utility class. Both DAOs now delegate to the same
toExperimentGroupItem / toExperimentGroupAggregationItem helpers, eliminating
the need to mirror DTO mapping changes in two places.
- [OPIK-4382] [BE] Deduplicate bindGroupCriteria into ExperimentGroupMappers
Moves the shared group-criteria binding logic out of ExperimentDAO and
ExperimentAggregatesDAO into ExperimentGroupMappers.bindGroupCriteria(),
following the same pattern as ExperimentSearchCriteriaBinder. Adding or
fixing a criteria binding now only requires a change in one place.
- [OPIK-4382] [BE] Extract streamGroupQuery helper and fix null percentiles
- Deduplicate findGroups/findGroupsAggregations into a single private
  streamGroupQuery(queryTemplate, criteria, rowMapper) that differs only
  by the query constant and BiFunction row mapper.
- Fix convertToBigDecimal to return null for null/unsupported inputs so
  absent p50/p90/p99 entries in getDuration propagate as null rather than
  BigDecimal.ZERO, preserving the semantic-null that lets callers apply
  COALESCE/fallback logic correctly.
- [OPIK-4382] [BE] Consolidate cost/duration helpers into ExperimentGroupMappers
Promote getCostValue and getDuration to public static in
ExperimentGroupMappers and delete the private copies in ExperimentDAO.
ExperimentDAO.mapToDto now delegates to the shared helpers, so any
future change to cost filtering, duration percentile extraction, or the
BigDecimal conversion only needs to be made in one place.

Side-effect: ExperimentDAO.mapToDto also picks up the null-percentile
fix (convertToBigDecimal returns null for absent/unsupported inputs)
that was previously applied only to ExperimentGroupMappers.
- [OPIK-4382] [BE] Fix pagination count and add criteria filter tests
- Remove count() OVER () window function from paged query (returned
  page-scoped count instead of full result-set count)
- Replace with dedicated count query + short-circuit: skip items query
  when count == 0, use DatasetItemPage.empty() for that case
- Extract DatasetItemResultMapper.buildItemFromRow as public static
  helper reused by ExperimentAggregatesDAO
- Add parameterized integration tests for ExperimentGroupCriteria
  filters (name, types, projectId, combined, empty-result) covering
  both findGroups and findGroupsAggregations aggregate paths
- [OPIK-4380] [BE] Extract shared filter helpers into FilterQueryBuilder
Add FilterStrategyParam record, applyFiltersToTemplate and bindFilters
static helpers to FilterQueryBuilder, then replace duplicated per-strategy
loops in DatasetItemVersionDAO and ExperimentAggregatesDAO with single
delegating calls backed by per-DAO strategy constants.
- [OPIK-4382] [BE] Consolidate filter helpers in getExperimentItemsStatsFromAggregates
Add EXPERIMENT_ITEMS_STATS_FILTER_STRATEGY_PARAMS and
EXPERIMENT_ITEMS_STATS_BIND_STRATEGIES constants and replace the
per-strategy toAnalyticsDbFilters/bind blocks in
getExperimentItemsStatsFromAggregates with single delegating calls to
FilterQueryBuilder.applyFiltersToTemplate and bindFilters.
- Revision 9: Extract shared helpers to eliminate duplication across DAOs
- Create DatasetItemSearchCriteriaMapper: centralizes filters + search flag
  wiring for DatasetItemSearchCriteria, shared by DatasetItemVersionDAO and
  ExperimentAggregatesDAO
- Add ExperimentGroupMappers.applyGroupCriteriaToTemplate: centralizes
  ExperimentGroupCriteria → ST template wiring, now shared by ExperimentDAO
  and ExperimentAggregatesDAO
- Update DatasetItemVersionDAO.addFiltersToTemplate and bindSearchAndFilters
  to delegate to DatasetItemSearchCriteriaMapper
- Update ExperimentAggregatesDAO.applyDatasetItemFiltersToTemplate and
  bindDatasetItemSearchParams to delegate to DatasetItemSearchCriteriaMapper
- Update ExperimentDAO.newGroupTemplate and ExperimentAggregatesDAO.newGroupTemplate
  to delegate to ExperimentGroupMappers.applyGroupCriteriaToTemplate
- [OPIK-4383] [BE] Fix project_deleted filter and comments_dedup scope in ExperimentAggregatesDAO
- Fix project_deleted filter: use zero UUID sentinel instead of empty string
  for FixedString(36) column comparison in FIND_GROUPS and FIND_GROUPS_AGGREGATIONS
- Fix comments_dedup CTE: scope trace_id subquery by dataset_id to avoid
  scanning the entire workspace's comments table
- Add missing streamMaxLen and streamTrimLimit fields to
  ExperimentDenormalizationConfig (implements StreamConfiguration interface)
- [OPIK-4383] [BE] Address PR review comments: extract ZERO_UUID constant and fix config comment
- Promote zero UUID sentinel to shared constant in ExperimentGroupMappers
- Use parameterized :zero_uuid binding in SQL templates instead of hardcoded string
- Fix config.yml comment from "Default: 120s" to "Default: 1m"
- [OPIK-4383] [BE] Add streamMaxLen and streamTrimLimit to experimentDenormalization config
- [OPIK-4727] fix: remove old GEPA code, fix aggregates test, add migration rollback docs
- Remove gepa_old/ optimizer source and tests, clean factory registry
- Add datasetItemCount to EXPERIMENT_AGGREGATED_FIELDS_TO_IGNORE (not stored in aggregates table)
- Add rollback documentation to mutation experiment type migration
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4727] [FE] refactor: deduplicate KPI cards, metric cells, and cleanup
- Extract shared KPICard/MetricKPICard to pages-shared/experiments/KPICard
- Extract calcPercentageVsBaseline helper and TrialMetricCellContent to
  deduplicate percentage calculation across 3 trial metric cells
- Remove unused OptimizationUpdate interface from types
- Fix inconsistent color token (text-light-slate → text-muted-slate)
- Move IIFE out of JSX in MetricComparisonCell compact mode
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4727] [FE] refactor: rename Compare* to Optimization/Trial, simplify URL structure
- Rename CompareOptimizations* → Optimization* and CompareTrials* → Trial*
- Simplify optimization URL from /$datasetId/$optimizationId to /$optimizationId
- Change trial route from /compare to /trials
- Add OptimizationCompareRedirect for legacy URL backwards compatibility
- Update all navigation references across pages (OptimizationsPage, HomePage, BestPrompt, ResourceLink, etc.)
- Fix breadcrumbs: show raw optimization ID, "Trial #N" for trials
- Split optimization detail into Report & Trials tabs with underline style
- Replace ToggleGroup with underline Tabs on trial page
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4727] [FE] feat: rename tabs, add diff vs parent, fix word-level diffs
- Rename "Report" tab to "Overview" on optimization page
- Rename "Best configuration" to "Best trial configuration"
- Change "Diff" button to "Diff vs. baseline" in configuration sections
- Add "Diff vs. parent" option in trial configuration tab
- Fix prompt diff to use word-level mode for inline change highlights
- Fix TextDiff word-mode layout to flow inline instead of dropping lines
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4727] [FE] refactor: extract shared config flattening, add redirect tests
- Extract flattenConfig, EXCLUDED_CONFIG_KEYS, shouldSkipRedundantKey
  into configuration-renderer.ts (shared by TrialConfigurationSection
  and ConfigurationDiffContent)
- Convert ConfigViewMode string union to CONFIG_VIEW_MODE const object
- Add missing replace prop on fallback Navigate in
  OptimizationCompareRedirect
- Restore isArray guard in ConfigurationDiffContent collectPrompts
- Add unit tests for configuration-renderer (21 tests)
- Add unit tests for OptimizationCompareRedirect (4 tests)
- Add E2E Playwright test for legacy /compare URL redirect (2 tests)
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4727] [FE] fix: PR review fixes - generic flattenConfig, NamedPrompts diff, parent fallback
- Make flattenConfig accept generic skipKey callback instead of hardcoded
  filtering (addresses Baz review comment)
- Fix NamedPrompts format not recognized as "prompt" type in
  detectConfigValueType, causing JSON-level diff instead of word-level
- Add parent experiment fallback for old optimizations using chronological
  ordering (enables "Diff vs. parent" for non-GEPA v2)
- Fix PromptDiff fallback paths to use mode="words" for word-level diffs
- Add tests for NamedPrompts detection and generic skipKey behavior
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4727] [FE] fix: PR review - shared makeSkipKey, parent fallback, label tweak
- Extract makeSkipKey helper into configuration-renderer.ts to eliminate
  duplicate skipKey predicates in TrialConfigurationSection and
  ConfigurationDiffContent
- Fix parentCandidateIds lookup: fall through to chronological fallback
  when GEPA v2 metadata exists but no matching parent is found
- Use shared skipKey in collectPrompts instead of inline checks
- Remove period from "Diff vs." labels
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4727] [FE] fix: rename Config toggle label to Configuration
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4838] [SDK] feat: GEPA convergence improvements (#5570)
- [OPIK-4838] [SDK] feat: GEPA convergence improvements
- Add GepaConfig dataclass for centralized algorithm parameters
- Cache parent scores during minibatch gate with configurable tolerance
  to absorb LLM judge noise (gate_tolerance=0.1)
- Rewrite FailureAwareBatchSampler to use assertion-based pass/fail
  instead of score threshold, prioritize by failing assertion count
- Switch default candidate selection to current_best
- Add docs on scoring pipeline and candidate selection strategies
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- chore(scripts): use GepaConfig defaults in e2e scripts
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- chore: remove local-only docs from PR
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: address baz review comments, tighten e2e assertions
- Remove unused _global_assertion_failures Counter from sampler
- Update sampler docstring to match implementation (assertion count, not frequency)
- Bound _cached_full_eval_scores to max_candidates entries with FIFO eviction
- Tighten e2e assertions: add context-awareness checks to EASY tier, sharpen MEDIUM tier
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: remove gate_tolerance, require strict minibatch improvement
Cached parent scores are still used for deterministic comparison,
but mutations must now strictly beat the parent on the minibatch
without any tolerance cushion.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: shuffle failed items randomly instead of sorting by priority
Simplifies minibatch sampling — all failed items are shuffled equally
rather than sorted by assertion failure count then shuffled within tiers.
This gives better variety across iterations since most items share the
same tier anyway.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat: top-tier sampling, template var validation, improved reflection prompt
- Sampler splits failed items into top/rest by assertion failure count,
  draws randomly from top tier first (configurable top_failed_fraction)
- Reject reflection proposals that drop template variables (e.g. {question})
- Reflection prompt encourages surgical edits over full rewrites
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: only update best_trial from full evaluations, not minibatches
A minibatch scoring 1.0 on 4 items was being reported as best_trial
even when the full evaluation only reached 0.9 on 20 items. Now
best_trial is only updated when experiment_type is None (full eval).

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- chore: remove accidentally committed reflection logs
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- docs: update GEPA docs for top-tier sampling, 5-step reflection, template var validation
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: increase default reflection_minibatch_size from 4 to 6
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: thread evaluator_model through pipeline, increase max_candidates to 25
The default LLMJudge model (gpt-5-nano) was too lenient, causing
pass_rate to always report 1.0. Thread evaluator_model from
OptimizationContext through EvaluationAdapter and experiment_executor
so callers can specify a more capable judge model.

Also increase GEPA max_candidates default from 5 to 25 to allow
longer optimization runs.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat: configurable blended scoring with assertion-level tiebreaker
Add ScoringConfig with strategy ("blended" | "pass_rate"), configurable
weights, and auto-computed epsilon (1/(num_items+1)) that guarantees
pass_rate always dominates while giving the algorithm gradient signal
from individual assertion progress.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: log raw pass_rate to UI, use blended score only for algorithm
_extract_score now returns (optimization_score, display_score) tuple.
The blended score drives the algorithm's acceptance gate, while the
raw pass_rate is logged as the experiment score for the UI chart.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: align GEPA per-item scoring with pass_rate, expose blended score as internal
- Replace GEPA's mean-of-assertions scoring with pass_rate-aligned formula:
  passing items score 1.0, failing items score ε × assertion_frac
  (where ε = 1/(num_items+1)), preserving gradient for subsample gate
- Use build_suite_result as source of truth for item pass/fail
- Store pass_rate in trial.score (user-facing), blended score in
  internal_optimization_score (algorithm-only)
- Use pass_rate for stop condition threshold comparison
- Add type hints and ScoreResult type to _extract_per_item_feedback
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: only record full evaluations as visible trials
Skip appending subsample/minibatch evals and cache hits to state.trials
so the UI only shows meaningful trial progression. Internal evals are
still returned to the optimizer for its scoring logic.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- docs: update GEPA docs for scoring contract, trial visibility, per-item scoring
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat: make reflection prompt less conservative to reduce stagnation
Relax the "surgical edits / MINIMAL EDIT" constraints that prevented the
reflection LLM from making meaningful structural changes when persistent
failures are detected. Key changes: graduated edit aggressiveness based
on Failure History, concrete escalation strategies (restructure, step-by-step
procedures, conditional logic, section rewrite), and removal of the
single-rule-per-failure cap.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat: cumulative assertion failure tracking with persistent failure threshold
- Track per-assertion total failures and total evaluations across the
  entire optimization run in FailureAwareBatchSampler
- Show Failure History only when an assertion has failed >=10 times
  (persistent failure threshold) AND failed again in current eval
- Include failures/evals ratio so the reflection LLM can gauge severity
- Remove streak-based logic in favor of cumulative counts
- Simplify reflection prompt: 5 steps → 4, merge write+apply steps,
  remove duplicated escalation, cleaner multiline formatting
- Remove duplicate cumulative info from Summary's Blocking assertions
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat: show only worst run for multi-run items, reorder Failure History before run output
Reduces feedback verbosity by showing only the worst run instead of all runs
for multi-run items. Places Failure History right after Inputs in the record
so the reflection LLM sees persistent failure context before the run details.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat: guide reflection LLM toward general rules, allow specificity for persistent failures
Step 3 now instructs to abstract specific examples into general categories.
Step 2 allows more specific rules for persistently failing assertions.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- docs: update GEPA reflection prompt docs for current algorithm
Updates template (4 steps), feedback format (worst-run-only, Failure History
with cumulative counts and Z=10 threshold), and generalization guidance.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat: reduce prompt overfitting — prefer updating existing rules, group by behavior pattern
Step 3: check whether an existing rule covers the failing behavior before
adding a new one. Step 4: group by behavior pattern, not scenario type.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat: make persistent failure threshold configurable, default to 7
Add persistent_failure_threshold to GepaConfig (default=7), thread through
GepaOptimizer → FrameworkGEPAAdapter → ReflectiveDatasetBuilder.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat: strengthen anti-overfitting in reflection prompt
Step 3: NEVER copy specific names/details/scenarios from feedback — they
are samples that change at runtime. Step 2: persistent failure specificity
still avoids non-generalizable details.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- docs: update GEPA docs for current algorithm state
Update both implementation guide and reflection prompt docs to match
current code: 4-step template, worst-run-only feedback, cumulative
failure threshold (configurable, default 7), anti-overfitting guidance,
and corrected config defaults.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: handle edge cases in scoring and trial recording
- _item_score: return 0.0 (not 1.0) for failed items with empty
  assertions, so they don't get scored as passes
- evaluation_adapter: guard trial is not None before appending to
  state.trials and accessing trial.optimization_score
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(BE): exclude minibatch/mutation experiments from pass_rate computation
The FIND query's best_objective_score was computed as a weighted average
across ALL experiments including minibatch and mutation, causing the UI
to show incorrect pass_rate during optimization. Filter experiment_candidates
to only include full-eval experiments (regular/trial types).

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(test): replace ambiguous XYZ headphones e2e item with clear bluetooth speaker scenario
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor(test): use assertions= shorthand and typed ExecutionPolicy in e2e scripts
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- feat(optimizer): add early stopping when pass_rate plateaus
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(test): update reflective dataset tests to use Worst Run instead of Runs
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix(optimizer): update assertion failure counters on minibatch evals too
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
- [NA] [SDK] refactor: convert reflection template to triple-quoted string
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- refactor: extract apps/opik-optimizer to comet-ml/opik-optimizer repo
Remove apps/opik-optimizer/ directory (moved to separate repo).
Clean up python-backend framework optimizer references:
- Remove framework_optimizer.py and framework_runner.py
- Remove OPTIMIZER_FRAMEWORK queue from Java Queue enum
- Remove opik-optimizer additional_contexts from docker-compose and CI
- Simplify resolveQueue to always use OPTIMIZER_CLOUD
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: revert python-backend DRY refactor that was only needed for framework optimizer
Restore optimizer.py and rq_worker_manager.py to main state, delete
optimizer_job_helper.py which only existed to share code with the
now-removed framework_optimizer.py.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [NA] [FE] fix: trials table sorting, metric trend precision, and word diff readability
- Implement client-side sorting for optimization trials table (all columns)
- Default sort by Trial # ascending
- Show 0% trend when formatted values are identical (below display resolution)
- Fall back to block diff when word changes exceed 60% of content
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [NA] [FE] fix: improve diff readability and optimization progress status
- Use hybrid line+word diff: line-level diff first to find changed regions,
  then word-level refinement within paired lines for precise highlights
- Use diffTrimmedLines to ignore trailing whitespace differences
- Fall back to separate removed/added blocks when line pairs are too different
- Add "Running initial calculations..." status for early GEPA phases
- Unify Changed/Added/Removed tag styling in prompt diff view
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [NA] [FE] fix: address PR review — sortCandidates tests, diffLines, formatter comments
- Add unit tests for sortCandidates covering all sort branches
- Switch TextDiff from diffTrimmedLines to diffLines for whitespace detection
- Add comments explaining intentional formatter comparison in percentage calc
- Export sortCandidates and CANDIDATE_SORT_FIELD_MAP for testability
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [NA] [FE] fix: address PR feedback — shared percentage helper, migration renumber, test timezone fixes, dataset button layout
- Extract calcFormatterAwarePercentage into shared lib/percentage.ts (review comment)
- Renumber migrations 000064→000065, 000065→000066 to avoid prefix conflict with main
- Fix timezone-sensitive test failures in MetricDateRangeSelect/utils.test.ts
- Fix dataset NavigationTag rendering as block in OptimizationHeader
- Fix mypy return type for _run_suite_evaluation in evaluator.py
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [NA] [FE] feat: dynamic chart legend, clickable ghost dot, dataset button width fix
- Chart legend now shows only statuses present in the data
- Ghost (in-progress) dot is clickable to select the trial
- Dataset NavigationTag constrained to content width with w-fit
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [NA] [FE] refactor: simplify trial statuses, ghost dot color, best candidate breathing animation
- Remove "evaluating" status — candidates are now passed (scored > parent) or pruned (scored <= parent)
- Simplify computeCandidateStatuses to compare against parent score
- Ghost dot uses running status color (yellow) instead of hardcoded blue
- Best candidate dot breathes (opacity pulse) when optimization is active but no ghost dot is shown
- Clean up unused isOptimizationFinished/inProgressStepIndex props from columns and cells
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [NA] [FE] fix: optimization cost duration from created_at, live timer, absolute time in trials table
- Duration now starts from optimization created_at instead of first experiment
- Live ticking timer while optimization is in progress
- Trials table "Created" column shows absolute date/time instead of relative
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [NA] [SDK] fix: correct return type of evaluate_optimization_suite_trial
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- Remove orphaned test for evaluate_optimization_suite_trial
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- Revert unused dataset_type additions in Python SDK
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4727] [FE] fix: improve non-eval-suite optimization display
- All chart dots blue for non-eval-suite optimizations (no pruned status)
- Chart legend shows metric name instead of status labels
- Table shows "Baseline" for step 0, "Passed" for scored candidates
- Column header shows metric name (e.g. "Accuracy (geval)")
- Add reason tooltip (speech bubble) to trial items score columns
- Remove jailbreak password demo template
- Revert temporary feature flag override
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: renumber migration prefixes 000065→000067, 000066→000068 to avoid conflicts with main
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- [OPIK-4928] [BE] fix: use correct column to lookup execution policies for experiment items
The execution policy lookup query was filtering by dataset_item_versions.dataset_item_id
instead of dataset_item_versions.id. With dataset versioning enabled, experiment items
reference dataset_item_versions.id as their datasetItemId, causing the lookup to miss
and fall back to the default policy {runs_per_item:1, pass_threshold:1}.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
- fix: add missing dataset_item_count to aggregated experiment CTEs
The experiments_from_aggregates_final CTEs in both FIND and
FIND_GROUPS_AGGREGATIONS queries were missing dataset_item_count,
causing ClickHouse UNKNOWN_IDENTIFIER errors when the aggregated
branch was used. Maps ea.experiment_items_count to dataset_item_count.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- fix: remove duplicate SELECT in FIND_GROUPS_AGGREGATIONS and deduplicate chart empty states
- The FIND_GROUPS_AGGREGATIONS query had two outer SELECTs after the
  subquery, causing a ClickHouse syntax error. Merged dataset_item_count
  into the single outer SELECT and removed the duplicate.
- Collapsed identical spinner/NoData branches in OptimizationProgressChartContainer.
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- [OPIK-4928] [BE] fix: remove feature flag gate preventing execution policy resolution
Root cause: ExperimentItemService.fetchItemPolicies() was gated behind
isDatasetVersioningEnabled(). When disabled, item-level execution
policies were never fetched from dataset_item_versions, so all
experiment items fell back to ExecutionPolicy.DEFAULT {1, 1}.

Also reverts the incorrect DatasetItemVersionDAO column change from
576791acc — experiment_items.dataset_item_id stores the logical
dataset_items.id which matches dataset_item_versions.dataset_item_id.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- fix: address PR review comments and fix execution_policy propagation
- Address 60+ PR review comments (dedup, types, performance, refactoring)
- Split useOptimizationData hook into useOptimizationExperiments + useOptimizationTableState
- Break down OptimizationProgressChartContent into ScatterDot, ChartEdges, GhostCandidate
- Extract shared utilities (STATUS_VARIANT_MAP, getBaselineCandidate, formatValue)
- Add useMemo to TrialMetricCells, TrialScoreCell, ToolsDiff, TrialStatusCell
- Fix KPI timer re-rendering (extract ElapsedDuration component)
- Fix pagination total when client-side filtering active
- Fix tooltip scroll issue (position: fixed → absolute)
- Remove dead code (OptimizationToolbar, OptimizationDeployCell, useGroupedOptimizationsList)
- Remove deploy column and playground/promote buttons (not ready)
- Use backend status/run_summaries for pass/fail display (PR 5634 integration)
- Fix SDK execution_policy propagation to experiment items
- Escape LIKE metacharacters in DatasetDAO
- Replace sentinel UUID with early return in OptimizationService
- Add MATERIALIZE INDEX to migration 000068
- Use total_count from experiment aggregates instead of custom dataset_item_count
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- fix: naming "Optimization runs" and add aggregation fields test
- Fix "Optimization Runs" → "Optimization runs" (lowercase r)
- Add test for optimization aggregation fields (baselineObjectiveScore,
  bestObjectiveScore, baselineDuration, bestDuration, baselineCost,
  bestCost, totalOptimizationCost, experimentScores)
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- fix: address baz review comments, fix migration prefixes and SDK lint
- Renumber migrations 000067→000068, 000068→000069 to avoid prefix conflict with main
- Fix ruff format: single-line model_dump call in engine.py
- Rename handleClick → handleTrialSelectClick in ScatterDot.tsx
- Fix step_index sentinel bug: allow non-negative step_index to overwrite -1
- Extract useBaselinePercentage hook to deduplicate TrialMetricCells
- Fix "Optimization Runs" → "Optimization runs" naming
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- fix: address remaining baz comments
- formatValue: return "null"/"undefined" literals instead of "" for nullish values
- Extract createTrialClickHandler to chartConstants.ts, used in ScatterDot and GhostCandidate
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- fix: use unique dataset name in aggregation test to avoid 409 on retry
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- fix: use server-generated dataset ID in aggregation test
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- fix: use Dataset.builder() instead of PODAM to avoid name collisions in CI
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- feat: overhaul pruning logic with "evaluating" state
- Add "evaluating" status for scored candidates not yet selected
- Best candidate always "passed" (never "evaluating")
- Score < best → immediately pruned
- Sibling with children (including ghost) → pruned
- Pulsing animation on last passed candidate at highest step
  (not always on best score)
- After completion: descendants or best = passed, rest = pruned
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- fix: ghost dot overlap, trend from zero, and self-review cleanup
- Ghost dot participates in same overlap grouping as regular dots
- Remove dead GHOST_OVERLAP_OFFSET_PER_DOT constant
- Move GHOST_ID to module scope
- Add null guard in computeInProgressStatus (remove non-null assertion)
- Show trend arrow when baseline is 0% (icon-only for infinite change)
- Refactor computeCandidateStatuses into focused helpers
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- test: add Vitest suite for computeCandidateStatuses and buildCandidateChartData
14 tests covering: baseline, running, evaluating, passed, pruned states,
ghost parent pruning, descendant detection, in-progress vs completed logic

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- fix: scope sibling pruning to candidates sharing same parent
Candidates at the same step but from different parents are independent
branches and should not prune each other. Changed grouping key from
stepIndex to sorted parentCandidateIds.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- fix: show metric name, optimize baseline lookup, rename ancestors
- Show objective metric name instead of "Accuracy" in KPI card and table
- Pass baselineCandidate via column meta instead of scanning per cell
  (reduces O(3RN) to O(1) baseline lookups per render)
- Rename buildDescendantsSet → buildAncestorSet (matches traversal direction)
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- refactor: extract getObjectiveLabel to lib/optimizations
Shared helper for objective column/card label used by both
useOptimizationColumns and getMetricKPICardConfigs.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- refactor: extract ChartTooltip component and move helpers to lib/optimizations
- Extract hover tooltip into ChartTooltip.tsx component
- Move getOptimizationMetadata, aggregateCandidates, mergeExperimentScores
  from useOptimizationExperiments to lib/optimizations.ts
- Move CANDIDATE_SORT_FIELD_MAP, sortCandidates to lib/optimizations.ts
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com
- fix: default feedbackScores to [] in mergeExperimentScores
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Co-authored-by: Claude Opus 4.6 noreply@anthropic.com
Co-authored-by: Aliaksandr Kuzmik 98702584+alexkuzmik@users.noreply.github.com
Co-authored-by: CometActions github-actions@comet.com
Co-authored-by: Thiago dos Santos Hora thiagoh@comet.com
下载附件
- 源代码 (ZIP)
- 源代码 (TAR.GZ)

0 发布 8,340 Git 标签

[OPIK-4727] [FE] feat: optimizer screens face-lift (#5554)

Resolution, etc.) instead of flat numbered lists