-
[issue-3764] [P SDK] [FE] [BE] [Docs] Introduce experiment scoring functions (#3989)
frostbyte_neo 发布于
2025-12-02 13:44:01 +00:00 -
Hide experiment_scores columns in the single experiment table
-
Add SDK support for experiment_scores
-
Add SDK support for experiment_scores
-
Add BE functionality
-
Typescript autogenerated code
-
Documentation and FE update
-
Address PR comments
-
Address PR comments
-
Fix PR comments
-
Address PR comments
-
Fix merge conflicts
-
Fix tests
-
Fix failing tests
-
Fix failing tests
-
Fix UI colors and column names
-
Refactor: Extract common score averaging logic to eliminate duplication
-
Harmonize experiment scores sorting to use map access from CTE
- Add experiment_scores_agg LEFT JOIN to non-grouped queries
- Simplify SortingQueryBuilder to use coalesce(map[key]) instead of complex JSON extraction
- Remove special case handling for experiment_scores in null direction logic
- Addresses PR review comments about query harmonization
- Remove early return for empty test results in experiment scores
- Allow experiment score functions to handle empty test results
- Some functions may want to return baseline/default scores with no data
- Addresses PR review comment about preventing score function execution
- Add E2E test for experiment scores functionality
- Test verifies experiment scoring functions work end-to-end
- Validates experiment scores appear in evaluation result
- Validates experiment scores are retrievable via SDK API
- Uses compute_max_score function to test score aggregation
- Addresses PR review comment about E2E test coverage
- Enhance experiment score computation to handle empty test results gracefully
- Update condition to return empty list if either scoring functions or test results are absent
- Ensures robustness in score computation logic
- Add Python SDK E2E test for experiment scores
- Tests experiment_scoring_functions parameter in evaluate()
- Verifies experiment scores are computed and returned in result
- Validates scores are persisted to backend API
- Tests aggregate metrics (max, min, avg) computation
- Addresses PR review comment about SDK test coverage
- Revert "Add E2E test for experiment scores functionality"
This reverts commit
50f9f8da59.- Apply DRY principle to score type mapping in ExperimentFeedbackScoresTab
- Extract addScoresToMap helper function to avoid duplication
- Works for both feedback_scores and experiment_scores
- Reduces code duplication and improves maintainability
- Fix parameter ordering (required before optional)
- [FE] Apply DRY principle to feedback/experiment scores handling
- useExperimentsTableConfig: Extract getScoreByName helper, eliminate duplicate accessorFn logic
- useCompareExperimentsChartsData: Extract createScoresMap helper for both score types
- CompareExperimentsDetails: Extract markScores helper to avoid duplicate map calls
- ExperimentsPage: Extract createScoresMap and getScoreNames helpers
- EvaluationSection: Use shared transformExperimentScores utility
- experimentScoreUtils: Refactor with formatScores helper to eliminate duplication
All changes maintain type safety and pass linting/typecheck
-
Revision 7: Add missing experiment_scores_agg CTE to FIND query
-
Revision 8: Fix experiment_scores sorting to use correct CTE alias 'es'
-
Revision 9: Address all 9 PR review comments
Python SDK improvements:
- Import module instead of name for ExperimentScore (comment #2)
- Allow ExperimentScoreFunction to return single or List of ScoreResults (comment #3)
- Move experiment score verification to verify_experiment utility (comment #4)
Backend code quality:
- Simplify TypeReference diamond operator in ExperimentScore.java (comment #5)
- Remove overloaded constructor in FeedbackScoreNames.ScoreName (comment #6)
- Reuse ScoreName instead of ScoreNameWithType in DAO (comment #7)
- Add TODO for full primary key in ORDER BY (comment #8)
- Revert flakiness fix in TemplateUtilsTest.java (comment #9)
-
Update return type of get_experiment_data method to use rest_api_types for consistency
-
Revision 10: Add full primary key to ORDER BY clause
-
Refactor test for standard deviation calculation in experiment scoring functions
Replaced hardcoded expected standard deviation value with a dynamic calculation using the statistics.stdev function for improved accuracy and maintainability.
- Add experiment_scores column to experiments table in migration 000048
This migration introduces a new column, experiment_scores, to the experiments table to store precomputed metrics. The column is added with a default value of an empty string. A rollback statement is also included to drop the column if necessary.
-
Update import statement for Prompt in evaluator.py to reflect new module structure
-
Refactor whitespace in verifiers.py for improved readability
This commit removes unnecessary blank lines in the verify_experiment and _verify_experiment_scores functions, enhancing the overall clarity of the code without altering functionality.
- Enhance type hinting in dataset and experiment modules
This commit adds future annotations to the dataset REST operations and introduces TYPE_CHECKING for conditional imports in the experiment module, improving type hinting and code clarity without affecting functionality.
-
Update documentation to replace
experiment_scoreswithexperiment_scoring_functionsfor consistency across evaluation methods -
Refactor score type handling in experiment feedback components
This commit replaces string literals for score types with constants, enhancing type safety and code clarity across various components, including ExperimentFeedbackScoresTab, ExperimentItemsTab, and related utility functions. The changes ensure consistent usage of SCORE_TYPE_FEEDBACK and SCORE_TYPE_EXPERIMENT throughout the codebase.
- Refactor column mapping for sorting functionality
This commit consolidates the logic for converting underscore-prefixed column IDs to dot notation into a single array of sortable prefixes. The
mapComplexColumnfunction is updated to iterate over this array, improving code clarity and maintainability while ensuring consistent handling of various column types.- Implement ExperimentScoreListCell and refactor score handling in data tables
This commit introduces the new ExperimentScoreListCell component for displaying experiment scores and updates the relevant data tables to utilize this component. Additionally, it refactors the handling of score types across various components, replacing string literals with constants for improved type safety and consistency. The changes affect the ExperimentsPage, ProjectsPage, and other related components, ensuring a unified approach to score type management.
- Refactor FeedbackScoresChartsWrapper and FeedbackScoreHoverCard for consistency
This commit updates the FeedbackScoresChartsWrapper component to rename the
isAggregationScoresprop toareAggregatedScoresfor improved clarity. Additionally, it modifies the subtitle text in the FeedbackScoreHoverCard component to use "Aggregated experiment scores" and "Average feedback scores" for consistency in terminology across the application.- Add experiment scores tab to CompareExperimentsPage and update score handling
This commit introduces a new tab for displaying experiment scores in the CompareExperimentsPage. It updates the ExperimentFeedbackScoresTab component to handle both feedback and experiment scores based on the selected tab. The score retrieval logic is modified to filter scores according to their type, enhancing clarity and usability in the comparison of experiments.
-
run fern generate
-
Refactor score handling in various components to unify feedback and experiment score logic. Removed experiment score references and updated feedback score components to handle aggregated scores. Adjusted column definitions and metadata across multiple pages for consistency.
-
Add migration to include experiment_scores column in experiments table
Co-authored-by: Daniel Dimenshtein danield@comet.com
Co-authored-by: Ido Berkovich ido@comet.com
Co-authored-by: Boris Feld boris@comet.com
Co-authored-by: YarivHashaiComet yarivh@comet.com下载附件
-