google--adk-python/contributing/samples/evaluation/user_simulation

User simulation

Overview

Instead of fixed user prompts, an LLM plays the user: it follows a conversation_plan and adopts a user_persona, generating each user turn dynamically in reaction to what the agent says. This tests the agent on realistic, branching multi-turn dialogue rather than a scripted exchange. For example, the user starts with a vague goal ("I want my bedroom to be comfortable.") and only reveals the target temperature once the agent asks for it, so the agent must actually run the clarification loop.

This sample evaluates the shared home-automation agent with two criteria that support user simulation:

• hallucinations_v1: checks that the agent's responses are grounded in the tool results and conversation (no invented device states or actions).
• per_turn_user_simulator_quality_v1: checks that the simulated user behaved correctly, i.e. it followed the conversation plan and stayed in persona each turn.

Because both the user simulator and the judges are LLMs, this sample needs a model credential (a Gemini API key or a Vertex project). Both the simulator and the judges resolve through the standard model registry, so they run with the same credentials the agent uses.

Sample Inputs

I want my bedroom to be comfortable.

A NOVICE user who actually wants the bedroom set to 21°C but only reveals the exact number when the agent asks, and is done once the agent confirms.

I need to check on my devices.

An EXPERT user who first asks which devices are on, then asks the agent to turn off any device that is on in the Living Room, and is done once the agent confirms.

How To

Run the sample from the workspace root:

adk eval contributing/samples/evaluation/home_automation_agent \
    contributing/samples/evaluation/user_simulation/home_automation.evalset.json \
    --config_file_path contributing/samples/evaluation/user_simulation/eval_config.json \
    --print_detailed_results

Each eval case in home_automation.evalset.json supplies a conversation_scenario instead of a static conversation (an EvalCase must have exactly one of the two). Because the user turns are generated at run time, you'll see a different multi-turn dialogue each run, and the scores will vary somewhat from run to run; that's expected for LLM-driven simulation and LLM-as-a-judge scoring.

conversation_scenario

Each scenario describes what the simulated user is trying to do:

• starting_prompt: the fixed first user message handed to the agent verbatim. Every later user turn is generated by the simulator.
• conversation_plan: the plan the simulator follows as the conversation plays out (the goals to accomplish, in order, and any details to reveal only when asked).
• user_persona: the persona the simulator adopts. You can pass one of the pre-built persona ids and it is resolved from the default persona registry:
  • NOVICE: relies on the agent for guidance, patient with the agent's questions, does not correct the agent or troubleshoot its mistakes, conversational tone.
  • EXPERT: knows exactly what they want, provides details up front, answers only relevant questions, corrects the agent's mistakes, professional tone.
  • EVALUATOR: a third pre-built persona for assessing whether the agent can accomplish the plan.

user_simulator_config

The eval config's user_simulator_config selects and tunes the simulator:

• type: the simulator implementation; llm_backed uses an LLM to play the user.
• model: the model the simulator uses to generate user turns (gemini-2.5-flash here).
• max_allowed_invocations: a safety cap on the number of turns, so a run-off loop between the agent and the simulated user can't continue forever (the fixed starting prompt counts as one invocation). Raise it if a scenario needs more turns to reach its goal; here 8 is plenty.

Why only certain criteria pair with user simulation

Not every metric works with a dynamically simulated conversation. The two used here (hallucinations_v1 and per_turn_user_simulator_quality_v1) resolve their judge model through the standard model registry, so they run with ordinary Gemini API key or Vertex credentials. safety_v1 and the multi_turn_* criteria also support user simulation, but they require a Google Cloud / Vertex project (they call a Vertex-only eval service), so they are omitted here.

Alternate flow: build an eval set from conversation_scenarios.json

Instead of hand-writing the eval set, you can build one from a list of scenarios plus a shared session input, using the committed conversation_scenarios.json and session_input.json:

adk eval_set create contributing/samples/evaluation/home_automation_agent eval_set_with_scenarios
adk eval_set add_eval_case contributing/samples/evaluation/home_automation_agent eval_set_with_scenarios \
    --scenarios_file contributing/samples/evaluation/user_simulation/conversation_scenarios.json \
    --session_input_file contributing/samples/evaluation/user_simulation/session_input.json
adk eval contributing/samples/evaluation/home_automation_agent eval_set_with_scenarios \
    --config_file_path contributing/samples/evaluation/user_simulation/eval_config.json \
    --print_detailed_results

Note: safety_v1 and the multi_turn_* criteria also support user simulation but require a Google Cloud / Vertex project, so they are omitted here.

Related Guides

• User simulation guide: https://adk.dev/evaluate/user-sim/
• Evaluation overview: https://adk.dev/evaluate/
• Evaluation criteria reference: https://adk.dev/evaluate/criteria/