nikkie
babb11c83c
Merge https://github.com/google/adk-python/pull/4344 **Please ensure you have read the [contribution guide](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) before creating a pull request.** ### Link to Issue or Description of Change **1. Link to an existing issue (if applicable):** - Closes: #4343 **Problem:** `AgentEvaluator.evaluate` did not register custom metrics from `EvalConfig`, so custom metrics worked in `adk eval` but not in pytest-based evals. **Solution:** Align `AgentEvaluator` with the CLI eval flow by registering custom metrics via a per-run metric registry and a shared default `MetricInfo` helper. The per-run registry is a fork of `DEFAULT_METRIC_EVALUATOR_REGISTRY` (new `MetricEvaluatorRegistry.fork()`), so the custom metrics declared by one eval config never leak into another run. ### Testing Plan Add unit coverage for the registration behavior and a lightweight integration example that uses a custom metric. **Unit Tests:** - [x] I have added or updated unit tests for my change. - [x] All unit tests pass locally. ``` % pytest tests/unittests/evaluation tests/unittests/cli 614 passed, 308 warnings in 12.74s ``` **Manual End-to-End (E2E) Tests:** ``` % pytest tests/integration/test_with_test_file.py::test_with_custom_metric tests/integration/test_with_test_file.py . [100%] 1 passed, 12 warnings in 3.57s ``` ### Checklist - [x] I have read the [CONTRIBUTING.md](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) document. - [x] I have performed a self-review of my own code. - [x] I have commented my code, particularly in hard-to-understand areas. - [x] I have added tests that prove my fix is effective or that my feature works. - [x] New and existing unit tests pass locally with my changes. - [x] I have manually tested my changes end-to-end. - [x] Any dependent changes have been merged and published in downstream modules. ### Additional context This change keeps `AgentEvaluator` behavior consistent with `adk eval` while avoiding CLI-layer dependencies. `fork()` returns an isolated copy seeded with the source registry's contents, rather than a bare `MetricEvaluatorRegistry()`. The seeding is what keeps this backwards compatible: registering an `Evaluator` subclass on `DEFAULT_METRIC_EVALUATOR_REGISTRY` is the only way to plug one in, since an eval config can only name a scoring function. Callers who do that today (including ones replacing the evaluator behind a standard metric name) would otherwise silently fall back to the stock evaluator, with no error and a different score. Covered by `test_evaluate_eval_set_keeps_evaluators_from_the_default_registry`. Co-authored-by: Yi Liu <yiliuly@google.com> COPYBARA_INTEGRATE_REVIEW=https://github.com/google/adk-python/pull/4344 from ftnext:agent-evaluator-support-custom-metric 3c844a9817d16954c683b164a88a86a82e8f9cf1 PiperOrigin-RevId: 966416454
88 lines
3.0 KiB
Python
88 lines
3.0 KiB
Python
# Copyright 2026 Google LLC
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
from google.adk.evaluation.agent_evaluator import AgentEvaluator
|
|
from google.adk.evaluation.local_eval_set_results_manager import LocalEvalSetResultsManager
|
|
import pytest
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_with_single_test_file():
|
|
"""Test the agent's basic ability via session file."""
|
|
await AgentEvaluator.evaluate(
|
|
agent_module="tests.integration.fixture.home_automation_agent",
|
|
eval_dataset_file_path_or_dir="tests/integration/fixture/home_automation_agent/simple_test.test.json",
|
|
)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_with_custom_metric():
|
|
"""Test eval with a custom metric."""
|
|
await AgentEvaluator.evaluate(
|
|
agent_module="tests.integration.fixture.home_automation_agent",
|
|
eval_dataset_file_path_or_dir=(
|
|
"tests/integration/fixture/home_automation_agent/test_files/custom_metrics/simple_custom_metric.test.json"
|
|
),
|
|
num_runs=1,
|
|
)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_with_folder_of_test_files_long_running():
|
|
"""Test the agent's basic ability via a folder of session files."""
|
|
await AgentEvaluator.evaluate(
|
|
agent_module="tests.integration.fixture.home_automation_agent",
|
|
eval_dataset_file_path_or_dir=(
|
|
"tests/integration/fixture/home_automation_agent/test_files"
|
|
),
|
|
num_runs=4,
|
|
)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_with_single_test_file_saves_eval_set_result(
|
|
tmp_path,
|
|
):
|
|
"""Persists eval set results under the explicitly provided app_name."""
|
|
eval_set_results_manager = LocalEvalSetResultsManager(
|
|
agents_dir=str(tmp_path)
|
|
)
|
|
await AgentEvaluator.evaluate(
|
|
agent_module="tests.integration.fixture.home_automation_agent",
|
|
eval_dataset_file_path_or_dir=(
|
|
"tests/integration/fixture/home_automation_agent/simple_test.test.json"
|
|
),
|
|
num_runs=2,
|
|
app_name="home_automation_agent",
|
|
eval_set_results_manager=eval_set_results_manager,
|
|
)
|
|
|
|
# Results are aggregated into a single eval set result file (matching
|
|
# LocalEvalService), containing one EvalCaseResult per run.
|
|
saved_result_files = list(
|
|
(tmp_path / "home_automation_agent" / ".adk" / "eval_history").glob(
|
|
"*.evalset_result.json"
|
|
)
|
|
)
|
|
assert len(saved_result_files) == 1
|
|
|
|
saved_result_ids = eval_set_results_manager.list_eval_set_results(
|
|
"home_automation_agent"
|
|
)
|
|
assert len(saved_result_ids) == 1
|
|
eval_set_result = eval_set_results_manager.get_eval_set_result(
|
|
"home_automation_agent", saved_result_ids[0]
|
|
)
|
|
assert len(eval_set_result.eval_case_results) == 2
|