refactor: move workflow execution out of repository skills

2026-08-18 19:34:26 +09:00
parent 9648a401a0
commit 82e3571fc5
17 changed files with 530 additions and 345 deletions
@@ -1,86 +0,0 @@
 ---
 name: examples-auto-run
 description: Run python examples in auto mode with logging, rerun helpers, and background control.
 ---
 # examples-auto-run
 ## What it does
 - Runs `uv run examples/run_examples.py` with:
  - Optional dependency extras enabled by default:
    `litellm`, `any-llm`, `sqlalchemy`, `redis`, `blaxel`, `modal`, `runloop`, and `temporal`.
  - `EXAMPLES_INTERACTIVE_MODE=auto` (auto-input/auto-approve).
  - Per-example logs under `.tmp/examples-start-logs/`.
  - Main summary log path passed via `--main-log` (also under `.tmp/examples-start-logs/`).
  - Generates a rerun list of failures at `.tmp/examples-rerun.txt` when `--write-rerun` is set.
 - Provides start/stop/status/logs/tail/collect/rerun helpers via `run.sh`.
 - Background option keeps the process running with a pidfile; `stop` cleans it up.
 ## Usage
 ```bash
 # Start (auto mode; interactive included by default)
 .agents/skills/examples-auto-run/scripts/run.sh start [extra args to run_examples.py]
 # Examples:
 .agents/skills/examples-auto-run/scripts/run.sh start --filter basic
 .agents/skills/examples-auto-run/scripts/run.sh start --include-server --include-audio
 # Check status
 .agents/skills/examples-auto-run/scripts/run.sh status
 # Stop running job
 .agents/skills/examples-auto-run/scripts/run.sh stop
 # List logs
 .agents/skills/examples-auto-run/scripts/run.sh logs
 # Tail latest log (or specify one)
 .agents/skills/examples-auto-run/scripts/run.sh tail
 .agents/skills/examples-auto-run/scripts/run.sh tail main_20260113-123000.log
 # Collect rerun list from a main log (defaults to latest main_*.log)
 .agents/skills/examples-auto-run/scripts/run.sh collect
 # Rerun only failed entries from rerun file (auto mode)
 .agents/skills/examples-auto-run/scripts/run.sh rerun
 ```
 ## Codex execution environment
 - When Codex invokes `start` or `rerun`, run the command outside the Codex sandbox by default (`sandbox_permissions=require_escalated`). Many examples start nested sandboxes, browsers, npm helpers, or cloud/local service processes; running from inside the Codex sandbox can produce environment-only failures such as `sandbox-exec: sandbox_apply: Operation not permitted`, Playwright cache permission errors, or npm cache permission errors.
 - Use sandboxed execution only when the user explicitly asks for it or when running a narrow dry-run / log inspection command that does not execute examples.
 ## Defaults (overridable via env)
 - `EXAMPLES_INTERACTIVE_MODE=auto`
 - `EXAMPLES_INCLUDE_INTERACTIVE=1`
 - `EXAMPLES_INCLUDE_SERVER=0`
 - `EXAMPLES_INCLUDE_AUDIO=0`
 - `EXAMPLES_INCLUDE_EXTERNAL=0`
 - `EXAMPLES_UV_EXTRAS="litellm any-llm sqlalchemy redis blaxel modal runloop temporal"` (set to an empty string to disable extras)
 - Auto-approvals in auto mode: `APPLY_PATCH_AUTO_APPROVE=1`, `SHELL_AUTO_APPROVE=1`, `AUTO_APPROVE_MCP=1`
 ## Log locations
 - Main logs: `.tmp/examples-start-logs/main_*.log`
 - Per-example logs (from `run_examples.py`): `.tmp/examples-start-logs/<module_path>.log`
 - Rerun list: `.tmp/examples-rerun.txt`
 - Stdout logs: `.tmp/examples-start-logs/stdout_*.log`
 ## Notes
 - The runner delegates to `uv run --extra ... examples/run_examples.py`, which already writes per-example logs and supports `--collect`, `--rerun-file`, and `--print-auto-skip`.
 - `examples/sandbox/extensions/vercel_runner.py` is temporarily excluded from auto runs due to credential issues. Do not force-run it until the credential setup is fixed.
 - `start` uses `--write-rerun` so failures are captured automatically.
 - If `.tmp/examples-rerun.txt` exists and is non-empty, invoking the skill with no args runs `rerun` by default.
 ## Behavioral validation (Codex/LLM responsibility)
 The runner does not perform any automated behavioral validation. After every foreground `start` or `rerun`, **Codex must manually validate** all exit-0 entries:
 1. Read the example source (and comments) to infer intended flow, tools used, and expected key outputs.
 2. Open the matching per-example log under `.tmp/examples-start-logs/`.
 3. Confirm the intended actions/results occurred; flag omissions or divergences.
 4. Do this for **all passed examples**, not just a sample.
 5. Report immediately after the run with concise citations to the exact log lines that justify the validation.
@@ -1,4 +0,0 @@
 interface:
  display_name: "Examples Auto Run"
  short_description: "Run examples in auto mode with logs and rerun helpers"
  default_prompt: "Use $examples-auto-run to run the repo examples in auto mode, collect logs, and summarize any failures."
@@ -0,0 +1,51 @@
 ---
 name: examples-run-analysis
 description: Analyze artifacts from the latest completed manual examples Make run. Read the main log, every relevant per-example log, and example source; validate every exit-0 example and classify failures, skips, and environment restrictions. Never execute or control examples.
 ---
 # Examples Run Analysis
 Use this skill only to analyze artifacts that already exist after a user has manually invoked an examples Make target. This skill is read-only and analysis-only.
 ## Hard boundary
 - Never start, retry, stop, or otherwise execute examples.
 - Never invoke an examples Make target or `.github/scripts/run_examples.sh`.
 - Never request elevated execution, alter an environment, remove a pid file, or own or signal a background process.
 - Never treat an older completed run as current when the newest run is active, incomplete, or stale.
 - If usable results are missing, stale, incomplete, or still running, stop the analysis and ask the user to run the appropriate Make target manually. Give the exact command but do not execute it.
 The supported workflow is an explicit manual Make invocation followed by analysis of the generated artifacts.
 ## Artifacts to inspect
 - Background pid file: `.tmp/examples-auto-run.pid`.
 - Main logs: `.tmp/examples-start-logs/main_*.log`.
 - Per-example logs named by each `log=` field in the selected main log.
 - Example sources named by `PASSED`, `FAILED`, and `SKIPPED` records.
 - Runner sources that define artifact meaning: `examples/run_examples.py`, `.github/scripts/run_examples.sh`, and the example source files included in the run.
 Use only read-only inspection commands such as `git status`, `git log`, `find`, `ls`, `stat`, `ps`, `sed`, and `rg`. Do not call a command that can update an artifact or process.
 ## Analysis workflow
 1. Inspect the process table and `.tmp/examples-auto-run.pid` without changing either. Treat a process as an active examples run only when its command line is rooted in the current repository and invokes `.github/scripts/run_examples.sh` or `examples/run_examples.py`, including foreground and background runs. Use the pid file only to correlate a background process; an absent or stale pid file does not prove that no run is active. If a matching process is live, stop the analysis. Tell the user to wait for a foreground Make run to finish, or ask the user to run `make examples-status` manually for a background run, before requesting analysis again.
 2. Select the newest `main_*.log`. Require exactly one terminal `# summary executed=<n> skipped=<n> failed=<n>` record. Treat a missing or malformed summary, a changing log, or a matching active examples process as incomplete.
 3. Treat the result as stale when relevant runner or selected example source content changed after the run. Use Git history and file timestamps as evidence. If freshness cannot be established, say so and request a new manual run instead of assuming the artifacts apply.
 4. Parse every `PASSED`, `FAILED`, and `SKIPPED` record. Reconcile their counts with the terminal summary. Confirm that every referenced per-example log exists.
 5. For every `PASSED` record, without sampling, read the complete example source and its per-example log. Infer the intended flow, tools, side effects, and key result from the source and comments, then verify that the log demonstrates those behaviors. Exit status 0 alone is not behavioral validation.
 6. Read the relevant per-example logs for failures and environment-related skips. Classify each result as an example or SDK defect, dependency or credential problem, provider or network failure, local service or platform restriction, intentional runner skip, or unresolved. Keep genuine product failures separate from environment restrictions.
 7. Report the selected main log, freshness and completeness evidence, summary counts, validation status for every exit-0 example, classified failures and skips, and exact source/log line references that support each conclusion.
 ## Manual commands to request when artifacts are unusable
 Choose the narrowest applicable command and ask the user to run it in a terminal:
 ```bash
 make examples-run
 make examples-run EXAMPLES_ARGS="--filter basic"
 make examples-run-background EXAMPLES_ARGS="--include-server --include-audio"
 make examples-status
 ```
 Do not execute any of these commands as part of this skill.
@@ -0,0 +1,4 @@
 interface:
  display_name: "Examples Run Analysis"
  short_description: "Analyze completed example-run artifacts"
  default_prompt: "Use $examples-run-analysis to inspect the latest completed manual examples run and validate every exit-0 example without executing or controlling any process."
@@ -1,66 +0,0 @@
 ---
 name: integration-tests
 description: Run the packaged OpenAI Agents Python SDK integration tests from clean wheel and source-distribution environments. Use for release readiness, live OpenAI regression checks, package import compatibility, optional-extra validation, or when asked to run integration tests after examples-auto-run.
 ---
 # Integration Tests
 ## Overview
 Run the release-oriented integration suite against the exact wheel and source distribution produced by `uv build`. The runner installs both artifacts into isolated environments and validates supported imports, optional extras, OpenAI model adapters, hosted tools, Realtime, and voice workflows.
 ## Execution requirements
 - Fresh isolated environments download optional dependencies from PyPI and connect to the configured API providers.
 - When the execution environment requires approval for package downloads or configured provider connections, request elevated command execution (`sandbox_permissions=require_escalated`). Retry with the required network permissions before classifying a connectivity failure as an SDK regression.
 ## Release workflow
 Run this command from the repository root:
 ```bash
 env UV_DEFAULT_INDEX=https://pypi.org/simple \
  OPENAI_AGENTS_INTEGRATION_STRICT=1 \
  OPENAI_AGENTS_INTEGRATION_EXTERNAL_PROVIDERS=1 \
  OPENAI_AGENTS_INTEGRATION_DIRECT_PROVIDERS=0 \
  make integration-tests-release
 ```
 - Use the release profile as the default whenever `$integration-tests` is invoked without a narrower request.
 - Use OpenRouter as the standard multi-provider gateway. Add provider-specific direct connections only when the user explicitly requests that additional credential matrix.
 - Use existing `OPENAI_API_KEY` and `OPENROUTER_API_KEY` values without printing them. The release target enforces strict mode, so missing required service configuration fails instead of skipping.
 - The command rebuilds the wheel and source distribution, creates isolated virtual environments, checks public imports and optional dependencies, runs the release-oriented live suites, and executes the local Docker security contract against both artifacts.
 - Do not run watch mode, modify source files, create a branch, commit, push, or open a pull request as part of this skill.
 ## Paired release validation
 When the user requests both pre-release checks, run `$examples-auto-run` first and follow that skill's required per-example behavioral validation. Then run the command above and report the examples and integration outcomes separately. Invoking `$integration-tests` alone does not implicitly start the examples suite.
 ## Focused commands
 Use a focused target only when the user specifically asks to narrow the run:
 ```bash
 env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-packaging
 env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-security
 env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-core
 env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-providers
 env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-hosted
 env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-realtime
 env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-voice
 env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-extras
 ```
 For the minimum supported Python package boundary, use:
 ```bash
 env UV_DEFAULT_INDEX=https://pypi.org/simple \
  OPENAI_AGENTS_INTEGRATION_PYTHON=3.10 \
  make integration-tests-packaging
 ```
 Nightly and manual profiles include additional capability-specific or higher-cost checks. Run them only when explicitly requested; use the configured OpenRouter matrix by default and include direct providers only when explicitly selected.
 ## Reporting
 Report the final pass, fail, skip, and deselection counts for each isolated environment. If a command fails, identify the exact profile, package environment, failing test, and actionable error. Separate product regressions from missing credentials, unsupported hosted features, dependency installation failures, and execution-environment restrictions.
@@ -1,4 +0,0 @@
 interface:
  display_name: "Integration Tests"
  short_description: "Run packaged Python SDK integration tests"
  default_prompt: "Use $integration-tests to run the packaged Python SDK integration suite."
@@ -46,7 +46,7 @@ changed_files=$(git diff --name-only "$base_sha" "$head_sha" || true)
 case "$mode" in
  code)
-    pattern='^(src/|tests/|integration_tests/|examples/|\.github/scripts/(detect-changes\.sh|run_integration_tests\.py|update_released_api_contract\.py)$|\.github/workflows/tests\.yml$|pyproject.toml$|uv.lock$|Makefile$)'
+    pattern='^(src/|tests/|integration_tests/|examples/|\.agents/skills/(examples-auto-run|examples-run-analysis|integration-tests)/|\.github/scripts/(detect-changes\.sh|run_examples\.sh|run_integration_tests\.py|update_released_api_contract\.py)$|\.github/workflows/tests\.yml$|pyproject.toml$|uv.lock$|Makefile$)'
    ;;
  docs)
    pattern='^(docs/|mkdocs.yml$)'
@@ -1,10 +1,9 @@
 #!/usr/bin/env bash
 set -euo pipefail
-ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." && pwd)"
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
 PID_FILE="$ROOT/.tmp/examples-auto-run.pid"
 LOG_DIR="$ROOT/.tmp/examples-start-logs"
 RERUN_FILE="$ROOT/.tmp/examples-rerun.txt"
 DEFAULT_UV_EXTRAS="litellm any-llm sqlalchemy redis blaxel modal runloop temporal"
 build_uv_prefix() {
@@ -49,7 +48,6 @@ cmd_start() {
  local run_cmd=(
    "${UV_RUN[@]}" examples/run_examples.py
    --auto-mode
    --write-rerun
    --main-log "$main_log"
    --logs-dir "$LOG_DIR"
  )
@@ -81,7 +79,7 @@ cmd_start() {
    echo "Started run_examples.py (pid=$pid)"
    echo "Main log: $main_log"
    echo "Stdout log: $stdout_log"
-    echo "Run '.agents/skills/examples-auto-run/scripts/run.sh validate \"$main_log\"' after it finishes."
+    echo "After the run completes, use examples-run-analysis to inspect its artifacts."
    return 0
  fi
@@ -159,78 +157,31 @@ cmd_tail() {
  tail -f "$LOG_DIR/$file"
 }
 collect_rerun() {
  ensure_dirs
  local log_file="${1:-}"
  if [[ -z "$log_file" ]]; then
    log_file="$(ls -1t "$LOG_DIR"/main_*.log 2>/dev/null | head -n1)"
  fi
  if [[ -z "$log_file" ]] || [[ ! -f "$log_file" ]]; then
    echo "No main log file found."
    exit 1
  fi
  cd "$ROOT"
  build_uv_prefix
  "${UV_RUN[@]}" examples/run_examples.py --collect "$log_file" --output "$RERUN_FILE"
 }
 cmd_rerun() {
  ensure_dirs
  local file="${1:-$RERUN_FILE}"
  if [[ ! -s "$file" ]]; then
    echo "Rerun list is empty: $file"
    exit 0
  fi
  local ts main_log stdout_log
  ts="$(date +%Y%m%d-%H%M%S)"
  main_log="$LOG_DIR/main_${ts}.log"
  stdout_log="$LOG_DIR/stdout_${ts}.log"
  cd "$ROOT"
  export EXAMPLES_INTERACTIVE_MODE="${EXAMPLES_INTERACTIVE_MODE:-auto}"
  export APPLY_PATCH_AUTO_APPROVE="${APPLY_PATCH_AUTO_APPROVE:-1}"
  export SHELL_AUTO_APPROVE="${SHELL_AUTO_APPROVE:-1}"
  export AUTO_APPROVE_MCP="${AUTO_APPROVE_MCP:-1}"
  build_uv_prefix
  set +e
  "${UV_RUN[@]}" examples/run_examples.py --auto-mode --rerun-file "$file" --write-rerun --main-log "$main_log" --logs-dir "$LOG_DIR" 2>&1 | tee "$stdout_log"
  local run_status=${PIPESTATUS[0]}
  set -e
  return "$run_status"
 }
 usage() {
  cat <<'EOF'
-Usage: run.sh <start|stop|status|logs|tail|collect|rerun> [args...]
+Usage: run_examples.sh <start|stop|status|logs|tail> [args...]
 Commands:
  start [--filter ... | other args]   Run examples in auto mode (foreground). Pass --background to run detached.
-  stop                                Kill the running auto-run (if any).
+  stop                                Kill the running examples job (if any).
-  status                              Show whether it is running.
+  status                              Show whether an examples job is running.
  logs                                List log files (.tmp/examples-start-logs).
  tail [logfile]                      Tail the latest (or specified) log.
  collect [main_log]                  Parse a main log and write failed examples to .tmp/examples-rerun.txt.
  rerun [rerun_file]                  Run only the examples listed in .tmp/examples-rerun.txt.
 Environment overrides:
  EXAMPLES_INTERACTIVE_MODE (default auto)
  EXAMPLES_INCLUDE_SERVER/INTERACTIVE/AUDIO/EXTERNAL (defaults: 0/1/0/0)
-  EXAMPLES_UV_EXTRAS (default: litellm any-llm sqlalchemy redis blaxel modal runloop; set empty to disable)
+  EXAMPLES_UV_EXTRAS (default: litellm any-llm sqlalchemy redis blaxel modal runloop temporal; set empty to disable)
  APPLY_PATCH_AUTO_APPROVE, SHELL_AUTO_APPROVE, AUTO_APPROVE_MCP (default 1 in auto mode)
 EOF
 }
-default_cmd="start"
+case "${1:-start}" in
 if [[ $# -eq 0 && -s "$RERUN_FILE" ]]; then
  default_cmd="rerun"
 fi
 case "${1:-$default_cmd}" in
  start) shift || true; cmd_start "$@" ;;
  stop) shift || true; cmd_stop ;;
  status) shift || true; cmd_status ;;
  logs) shift || true; cmd_logs ;;
  tail) shift; cmd_tail "${1:-}" ;;
-  collect) shift || true; collect_rerun "${1:-}" ;;
+  help | --help | -h) usage ;;
  rerun) shift || true; cmd_rerun "${1:-}" ;;
  *) usage; exit 1 ;;
 esac
@@ -7,16 +7,10 @@ import shutil
 import subprocess
 import sys
 import xml.etree.ElementTree as ET
 from collections.abc import Callable, MutableMapping, Sequence
 from pathlib import Path
 ROOT = Path(__file__).resolve().parents[2]
 sys.path.insert(0, str(ROOT))
 from integration_tests._contract_support import (  # noqa: E402
    SubmoduleExportPolicy,
    load_submodule_export_policy,
 )
 WORKSPACE = ROOT / ".tmp" / "integration-tests"
 DIST = WORKSPACE / "dist"
 RESULTS = WORKSPACE / "results"
@@ -36,22 +30,89 @@ OPTIONAL_EXTRAS = (
    "s3",
 )
 STRICT_PROFILES = frozenset({"release", "security"})
-PROFILES = (
+LOCAL_ONLY_CREDENTIAL_CLASS = "local-only"
-    "packaging",
+LIVE_CREDENTIAL_CLASS = "live"
-    "prospective-contract",
+PROFILE_CREDENTIAL_CLASSES = {
-    "prospective-platform",
+    "packaging": LOCAL_ONLY_CREDENTIAL_CLASS,
-    "security",
+    "prospective-contract": LOCAL_ONLY_CREDENTIAL_CLASS,
-    "mcp-v1",
+    "prospective-platform": LOCAL_ONLY_CREDENTIAL_CLASS,
-    "core",
+    "security": LOCAL_ONLY_CREDENTIAL_CLASS,
-    "providers",
+    "mcp-v1": LOCAL_ONLY_CREDENTIAL_CLASS,
-    "realtime",
+    "extras": LOCAL_ONLY_CREDENTIAL_CLASS,
-    "voice",
+    "core": LIVE_CREDENTIAL_CLASS,
-    "hosted",
+    "providers": LIVE_CREDENTIAL_CLASS,
-    "extras",
+    "realtime": LIVE_CREDENTIAL_CLASS,
-    "full",
+    "voice": LIVE_CREDENTIAL_CLASS,
-    "release",
+    "hosted": LIVE_CREDENTIAL_CLASS,
-    "nightly",
+    "full": LIVE_CREDENTIAL_CLASS,
-    "manual",
+    "release": LIVE_CREDENTIAL_CLASS,
    "nightly": LIVE_CREDENTIAL_CLASS,
    "manual": LIVE_CREDENTIAL_CLASS,
 }
 PROFILES = tuple(PROFILE_CREDENTIAL_CLASSES)
 BOOTSTRAPPED_ENV = "OPENAI_AGENTS_INTEGRATION_RUNNER_BOOTSTRAPPED"
 def parse_args(arguments: Sequence[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Run packaged openai-agents integration tests.")
    parser.add_argument("--profile", choices=PROFILES, default="full")
    parser.add_argument(
        "--all",
        action="store_true",
        help="Include configured direct Anthropic and Gemini providers alongside OpenRouter.",
    )
    return parser.parse_args(arguments)
 def prepare_profile_environment(
    profile: str,
    environ: MutableMapping[str, str] | None = None,
 ) -> str:
    environment = os.environ if environ is None else environ
    try:
        credential_class = PROFILE_CREDENTIAL_CLASSES[profile]
    except KeyError as error:
        raise RuntimeError(f"Integration profile {profile!r} has no credential class.") from error
    if credential_class == LIVE_CREDENTIAL_CLASS:
        if environment.get("OPENAI_API_KEY_SOURCE") != "service-account":
            raise RuntimeError(
                f"Live integration profile {profile!r} requires "
                "OPENAI_API_KEY_SOURCE=service-account before any build or subprocess starts. "
                "Load the approved service-account environment and retry the Make target."
            )
    elif credential_class == LOCAL_ONLY_CREDENTIAL_CLASS:
        environment.pop("OPENAI_API_KEY", None)
    else:
        raise RuntimeError(
            f"Integration profile {profile!r} has unknown credential class {credential_class!r}."
        )
    return credential_class
 def bootstrap_in_uv(
    arguments: Sequence[str],
    environ: MutableMapping[str, str],
    exec_function: Callable[[str, list[str], dict[str, str]], object] = os.execvpe,
 ) -> None:
    args = parse_args(arguments)
    prepare_profile_environment(args.profile, environ)
    child_env = dict(environ)
    child_env[BOOTSTRAPPED_ENV] = "1"
    command = ["uv", "run", "python", str(Path(__file__).resolve()), *arguments]
    exec_function(command[0], command, child_env)
    raise RuntimeError("The uv integration runner bootstrap returned unexpectedly.")
 if __name__ == "__main__" and os.environ.get(BOOTSTRAPPED_ENV) != "1":
    bootstrap_in_uv(sys.argv[1:], os.environ)
 sys.path.insert(0, str(ROOT))
 from integration_tests._contract_support import (  # noqa: E402
    SubmoduleExportPolicy,
    load_submodule_export_policy,
 )
@@ -355,14 +416,8 @@ def _sanitize_and_load_junit(result_path: Path) -> ET.Element | None:
 def main() -> None:
-    parser = argparse.ArgumentParser(description="Run packaged openai-agents integration tests.")
+    args = parse_args()
-    parser.add_argument("--profile", choices=PROFILES, default="full")
+    prepare_profile_environment(args.profile)
    parser.add_argument(
        "--all",
        action="store_true",
        help="Include configured direct Anthropic and Gemini providers alongside OpenRouter.",
    )
    args = parser.parse_args()
    prospective_policy: SubmoduleExportPolicy | None = None
    if args.profile in {"prospective-contract", "prospective-platform"}:
        prospective_contract = os.environ.get(PROSPECTIVE_CONTRACT_ENV)
@@ -257,7 +257,7 @@ jobs:
        if: steps.changes.outputs.run == 'true'
        env:
          OPENAI_AGENTS_PROSPECTIVE_RELEASE_CONTRACT: ${{ github.workspace }}/.tmp/prospective_released_api_contract.json
-        run: uv run python .github/scripts/run_integration_tests.py --profile prospective-platform
+        run: make integration-tests-prospective-platform
      - name: Skip Windows prospective contract smoke test
        if: steps.changes.outputs.run != 'true'
        run: echo "Skipping Windows prospective contract smoke test for non-code changes."
@@ -20,7 +20,8 @@ PROSPECTIVE_RELEASED_API_CONTRACT ?= .tmp/prospective_released_api_contract.json
 .PHONY: prepare-prospective-released-api-contract
 prepare-prospective-released-api-contract:
-	@version="$$(uv run python -c 'from importlib.metadata import version; print(version("openai-agents"))')"; \
+	@unset OPENAI_API_KEY; \
 	version="$$(uv run python -c 'from importlib.metadata import version; print(version("openai-agents"))')"; \
 	uv run python .github/scripts/update_released_api_contract.py \
 		--version "$$version" \
 		--output "$(PROSPECTIVE_RELEASED_API_CONTRACT)"
@@ -92,53 +93,86 @@ tests-serial:
 tests-serial-review:
 	uv run python .github/scripts/run_serial_tests.py --exclude-review-optional
 EXAMPLES_RUNNER := bash .github/scripts/run_examples.sh
 EXAMPLES_ARGS ?=
 EXAMPLES_LOG ?=
 INTEGRATION_TEST_RUNNER := python .github/scripts/run_integration_tests.py
 .PHONY: examples-run
 examples-run:
 	$(EXAMPLES_RUNNER) start $(EXAMPLES_ARGS)
 .PHONY: examples-run-background
 examples-run-background:
 	$(EXAMPLES_RUNNER) start --background $(EXAMPLES_ARGS)
 .PHONY: examples-status
 examples-status:
 	$(EXAMPLES_RUNNER) status
 .PHONY: examples-stop
 examples-stop:
 	$(EXAMPLES_RUNNER) stop
 .PHONY: examples-logs
 examples-logs:
 	$(EXAMPLES_RUNNER) logs
 .PHONY: examples-tail
 examples-tail:
 	$(EXAMPLES_RUNNER) tail $(EXAMPLES_LOG)
 .PHONY: integration-tests
 integration-tests:
-	uv run python .github/scripts/run_integration_tests.py --profile full $(filter --all,$(MAKECMDGOALS))
+	$(INTEGRATION_TEST_RUNNER) --profile full $(filter --all,$(MAKECMDGOALS))
 .PHONY: integration-tests-release
 integration-tests-release:
-	uv run python .github/scripts/run_integration_tests.py --profile release $(filter --all,$(MAKECMDGOALS))
+	$(INTEGRATION_TEST_RUNNER) --profile release $(filter --all,$(MAKECMDGOALS))
 .PHONY: integration-tests-nightly
 integration-tests-nightly:
-	uv run python .github/scripts/run_integration_tests.py --profile nightly $(filter --all,$(MAKECMDGOALS))
+	$(INTEGRATION_TEST_RUNNER) --profile nightly $(filter --all,$(MAKECMDGOALS))
 .PHONY: integration-tests-manual
 integration-tests-manual:
-	uv run python .github/scripts/run_integration_tests.py --profile manual $(filter --all,$(MAKECMDGOALS))
+	$(INTEGRATION_TEST_RUNNER) --profile manual $(filter --all,$(MAKECMDGOALS))
 .PHONY: integration-tests-packaging
 integration-tests-packaging:
-	uv run python .github/scripts/run_integration_tests.py --profile packaging
+	$(INTEGRATION_TEST_RUNNER) --profile packaging
 .PHONY: integration-tests-prospective-contract
 integration-tests-prospective-contract:
-	uv run python .github/scripts/run_integration_tests.py --profile prospective-contract
+	$(INTEGRATION_TEST_RUNNER) --profile prospective-contract
 .PHONY: integration-tests-prospective-platform
 integration-tests-prospective-platform:
 	$(INTEGRATION_TEST_RUNNER) --profile prospective-platform
 .PHONY: integration-tests-security
 integration-tests-security:
-	uv run python .github/scripts/run_integration_tests.py --profile security
+	$(INTEGRATION_TEST_RUNNER) --profile security
 .PHONY: integration-tests-mcp-v1
 integration-tests-mcp-v1:
-	uv run python .github/scripts/run_integration_tests.py --profile mcp-v1
+	$(INTEGRATION_TEST_RUNNER) --profile mcp-v1
 .PHONY: integration-tests-core
 integration-tests-core:
-	uv run python .github/scripts/run_integration_tests.py --profile core
+	$(INTEGRATION_TEST_RUNNER) --profile core
 .PHONY: integration-tests-providers
 integration-tests-providers:
-	uv run python .github/scripts/run_integration_tests.py --profile providers $(filter --all,$(MAKECMDGOALS))
+	$(INTEGRATION_TEST_RUNNER) --profile providers $(filter --all,$(MAKECMDGOALS))
 .PHONY: integration-tests-providers-external
 integration-tests-providers-external:
-	OPENAI_AGENTS_INTEGRATION_EXTERNAL_PROVIDERS=1 uv run python .github/scripts/run_integration_tests.py --profile providers $(filter --all,$(MAKECMDGOALS))
+	OPENAI_AGENTS_INTEGRATION_EXTERNAL_PROVIDERS=1 $(INTEGRATION_TEST_RUNNER) --profile providers $(filter --all,$(MAKECMDGOALS))
 .PHONY: integration-tests-providers-all
 integration-tests-providers-all:
-	uv run python .github/scripts/run_integration_tests.py --profile providers --all
+	$(INTEGRATION_TEST_RUNNER) --profile providers --all
 .PHONY: --all
 --all:
@@ -146,19 +180,19 @@ integration-tests-providers-all:
 .PHONY: integration-tests-realtime
 integration-tests-realtime:
-	uv run python .github/scripts/run_integration_tests.py --profile realtime
+	$(INTEGRATION_TEST_RUNNER) --profile realtime
 .PHONY: integration-tests-voice
 integration-tests-voice:
-	uv run python .github/scripts/run_integration_tests.py --profile voice
+	$(INTEGRATION_TEST_RUNNER) --profile voice
 .PHONY: integration-tests-hosted
 integration-tests-hosted:
-	uv run python .github/scripts/run_integration_tests.py --profile hosted
+	$(INTEGRATION_TEST_RUNNER) --profile hosted
 .PHONY: integration-tests-extras
 integration-tests-extras:
-	uv run python .github/scripts/run_integration_tests.py --profile extras
+	$(INTEGRATION_TEST_RUNNER) --profile extras
 .PHONY: coverage
 coverage:
@@ -0,0 +1,18 @@
 # Running the example suite
 Example execution is owned by the repository runner and Make targets. Run the complete auto-mode workflow in the foreground with:
    make examples-run
 Pass runner arguments through `EXAMPLES_ARGS`, for example:
    make examples-run EXAMPLES_ARGS="--filter basic"
    make examples-run EXAMPLES_ARGS="--include-server --include-audio"
 Use `make examples-run-background` for a background run. The remaining lifecycle targets are `make examples-status`, `make examples-stop`, `make examples-logs`, and `make examples-tail`. Set `EXAMPLES_LOG` to select a specific file for `examples-tail`.
 Every normal run writes a main log and per-example logs under `.tmp/examples-start-logs/`. Use `EXAMPLES_ARGS="--filter <substring>"` to run a focused subset again when needed.
 The defaults preserve auto input and approvals, include interactive examples, and exclude server, audio, and external examples unless selected. `EXAMPLES_UV_EXTRAS` controls the optional dependency extras installed by `uv`; set it to an empty value to disable extras. `EXAMPLES_INCLUDE_INTERACTIVE`, `EXAMPLES_INCLUDE_SERVER`, `EXAMPLES_INCLUDE_AUDIO`, and `EXAMPLES_INCLUDE_EXTERNAL` provide environment-based inclusion overrides.
 The repository skill `examples-run-analysis` is analysis-only. After a manual run completes, use it to inspect the main log, every relevant per-example log, and example source. The skill never starts, retries, stops, or controls the example process.
@@ -6,7 +6,6 @@ Features:
 * Auto mode (``EXAMPLES_INTERACTIVE_MODE=auto``) enables deterministic inputs,
  auto-approvals, and turns on interactive examples by default.
 * Writes per-example logs to ``.tmp/examples-start-logs`` and a main summary log.
 * Generates a rerun list of failures at ``.tmp/examples-rerun.txt``.
 """
 from __future__ import annotations
@@ -36,7 +35,6 @@ MAIN_PATTERN = re.compile(r"__name__\s*==\s*['\"]__main__['\"]")
 LOG_DIR_DEFAULT = ROOT_DIR / ".tmp" / "examples-start-logs"
 ARTIFACTS_DIR_DEFAULT = ROOT_DIR / ".tmp" / "examples-artifacts"
 RERUN_FILE_DEFAULT = ROOT_DIR / ".tmp" / "examples-rerun.txt"
 DEFAULT_MAIN_LOG = LOG_DIR_DEFAULT / f"main_{datetime.datetime.now().strftime('%Y%m%d-%H%M%S')}.log"
 REDIS_SESSION_EXAMPLE = "examples/memory/redis_session_example.py"
 DAPR_SESSION_EXAMPLE = "examples/memory/dapr_session_example.py"
@@ -422,23 +420,6 @@ def parse_args() -> argparse.Namespace:
        default=str(ARTIFACTS_DIR_DEFAULT),
        help="Directory for example-generated artifacts.",
    )
    parser.add_argument(
        "--rerun-file",
        help="Only run examples listed in this file (one relative path per line).",
    )
    parser.add_argument(
        "--write-rerun",
        action="store_true",
        help="Write failures to .tmp/examples-rerun.txt after the run.",
    )
    parser.add_argument(
        "--collect",
        help="Parse a previous main log to emit a rerun list instead of running examples.",
    )
    parser.add_argument(
        "--output",
        help="Output path for --collect rerun list (defaults to stdout).",
    )
    parser.add_argument(
        "--print-auto-skip",
        action="store_true",
@@ -592,24 +573,6 @@ def artifact_dir_for_example(relpath: str, artifacts_dir: Path) -> Path:
    return artifacts_dir / stem.replace("/", "__")
 def parse_rerun_from_log(log_path: Path) -> list[str]:
    if not log_path.exists():
        raise FileNotFoundError(log_path)
    rerun: list[str] = []
    with log_path.open("r", encoding="utf-8") as handle:
        for line in handle:
            stripped = line.strip()
            if not stripped or stripped.startswith("#"):
                continue
            parts = stripped.split()
            if len(parts) < 2:
                continue
            status, relpath = parts[0].upper(), parts[1]
            if status in {"FAILED", "ERROR", "UNKNOWN"}:
                rerun.append(normalize_relpath(relpath))
    return rerun
 def run_examples(examples: Sequence[ExampleScript], args: argparse.Namespace) -> int:
    overrides: set[str] = set()
    if args.include_interactive or env_flag("EXAMPLES_INCLUDE_INTERACTIVE"):
@@ -633,8 +596,6 @@ def run_examples(examples: Sequence[ExampleScript], args: argparse.Namespace) ->
    ensure_dirs(logs_dir, is_file=False)
    ensure_dirs(artifacts_dir, is_file=False)
    ensure_dirs(main_log_path, is_file=True)
    rerun_entries: list[str] = []
    if not examples:
        print("No example entry points found that match the filters.")
        return 0
@@ -841,18 +802,8 @@ def run_examples(examples: Sequence[ExampleScript], args: argparse.Namespace) ->
                executed += 1
            elif result.status == "failed":
                failed += 1
                rerun_entries.append(ex.relpath)
        safe_write_main(f"# summary executed={executed} skipped={skipped} failed={failed}")
    if args.write_rerun:
        ensure_dirs(RERUN_FILE_DEFAULT, is_file=True)
        if rerun_entries:
            contents = "\n".join(rerun_entries) + "\n"
        else:
            contents = ""
        RERUN_FILE_DEFAULT.write_text(contents, encoding="utf-8")
        print(f"Wrote rerun list to {RERUN_FILE_DEFAULT}")
    print(f"Main log: {main_log_path}")
    print(f"Done. Ran {executed} example(s), skipped {skipped}, failed {failed}.")
@@ -882,31 +833,7 @@ def main() -> int:
            print(entry)
        return 0
    if args.collect:
        paths = parse_rerun_from_log(Path(args.collect))
        if args.output:
            out = Path(args.output)
            ensure_dirs(out, is_file=True)
            out.write_text("\n".join(paths) + "\n", encoding="utf-8")
            print(f"Wrote {len(paths)} entries to {out}")
        else:
            for p in paths:
                print(p)
        return 0
    examples = discover_examples(args.filter)
    if args.rerun_file:
        rerun_set = {
            line.strip()
            for line in Path(args.rerun_file).read_text(encoding="utf-8").splitlines()
            if line.strip()
        }
        examples = [ex for ex in examples if ex.relpath in rerun_set]
        if not examples:
            print("Rerun list is empty; nothing to do.")
            return 0
        print(f"Rerun mode: {len(examples)} example(s) from {args.rerun_file}")
    return run_examples(examples, args)
@@ -1,21 +1,24 @@
-# Packaged live integration tests
+# Packaged integration tests
 These tests exercise the exact wheel produced by `uv build` after installing it into clean virtual environments. The `integration_tests/` directory, repository automation metadata, and local dependency/type-checking caches are excluded from published distributions.
 Run the complete release-oriented matrix with:
    export UV_DEFAULT_INDEX=https://pypi.org/simple
    test "${OPENAI_API_KEY_SOURCE:-}" = service-account
    make integration-tests
-`make integration-tests-release` runs the release-safe live matrix and the local Docker security contract in strict mode, so an unavailable daemon, image, credential, or required capability fails the release gate instead of becoming a skip. The focused `make integration-tests-security` target runs the same wheel and sdist security contract in strict mode without the live provider matrix; the security profile remains separate from the credential-free PR packaging job. `make integration-tests-nightly` also includes extended capability and transport checks, while `make integration-tests-manual` includes checks reserved for an intentionally configured manual run. Focused entry points are `make integration-tests-packaging`, `make integration-tests-security`, `make integration-tests-mcp-v1`, `make integration-tests-core`, `make integration-tests-providers`, `make integration-tests-providers-external`, `make integration-tests-providers-all`, `make integration-tests-realtime`, `make integration-tests-voice`, `make integration-tests-hosted`, and `make integration-tests-extras`. The packaging profile validates the released public API manifest and historical `RunState` corpus from base wheel and sdist environments, then validates the public API again from wheel and sdist environments with the Cloudflare extra installed so dependency-conditional exports are required. The security profile installs the Docker extra for both distribution formats, checks packaged credential redaction, and runs model-controlled environment, filesystem, and process inspection inside a local Docker sandbox through the public `Runner` lifecycle. The MCP v1 profile installs the built wheel with both the supported v1 floor and latest tested v1 release in clean environments; the regular test job validates the locked MCP v2 dependency.
+`make integration-tests-release` runs the release-safe live matrix and the local Docker security contract in strict mode, so an unavailable daemon, image, credential, or required capability fails the release gate instead of becoming a skip. The focused `make integration-tests-security` target runs the same wheel and sdist security contract in strict mode without the live provider matrix; the security profile remains separate from the credential-free PR packaging job. `make integration-tests-nightly` also includes extended capability and transport checks, while `make integration-tests-manual` includes checks reserved for an intentionally configured manual run. Focused entry points are `make integration-tests-packaging`, `make integration-tests-prospective-contract`, `make integration-tests-prospective-platform`, `make integration-tests-security`, `make integration-tests-mcp-v1`, `make integration-tests-core`, `make integration-tests-providers`, `make integration-tests-providers-external`, `make integration-tests-providers-all`, `make integration-tests-realtime`, `make integration-tests-voice`, `make integration-tests-hosted`, and `make integration-tests-extras`. The packaging profile validates the released public API manifest and historical `RunState` corpus from base wheel and sdist environments, then validates the public API again from wheel and sdist environments with the Cloudflare extra installed so dependency-conditional exports are required. The security profile installs the Docker extra for both distribution formats, checks packaged credential redaction, and runs model-controlled environment, filesystem, and process inspection inside a local Docker sandbox through the public `Runner` lifecycle. The MCP v1 profile installs the built wheel with both the supported v1 floor and latest tested v1 release in clean environments; the regular test job validates the locked MCP v2 dependency.
 Release PR preparation updates the rolling API manifest locally rather than in a credentialed GitHub workflow. After the release branch version bump, run `make update-released-api-contract VERSION=<version>`, review and commit the manifest diff, then run `make check-released-api-contract VERSION=<version>` after subsequent rebases. Promotion fails before writing if the candidate breaks the committed released contract. The prospective release-contract job performs this source validation in one dedicated Python process so provider behavior tests cannot change its import graph. Inspectable top-level classes and functions are promoted automatically; documented properties, intended submodule paths, and canonical aliases remain explicit review decisions recorded in the manifest. The packaged profiles remain the artifact-level verification that the committed contract holds for core and policy-declared optional surfaces across wheel, sdist, and supported platforms.
-Invoke the repository-local `$integration-tests` skill to run the release profile with configured OpenRouter-backed provider checks. OpenRouter provides a single configured gateway for the standard multi-provider matrix; provider-specific direct connections are optional extensions selected explicitly. When a release review also requires runnable examples, run `$examples-auto-run` first and then `$integration-tests`.
+Integration execution is available only through these Make targets and `.github/scripts/run_integration_tests.py`; there is no integration execution skill. When a release review also requires runnable examples, run the relevant `make examples-*` target manually, analyze its completed artifacts with `examples-run-analysis`, and then run the selected `make integration-tests-*` target.
-Set `OPENAI_API_KEY` for live OpenAI calls. Override `OPENAI_AGENTS_INTEGRATION_MODEL`, `OPENAI_AGENTS_INTEGRATION_REALTIME_MODEL`, `OPENAI_AGENTS_INTEGRATION_ANY_LLM_MODELS`, and `OPENAI_AGENTS_INTEGRATION_LITELLM_MODELS` when testing different models or configured providers. Provider model lists contain comma-separated adapter model names and require the credentials matching each selected provider. Set `OPENAI_AGENTS_INTEGRATION_MCP_SERVER_URL` to use another trusted DeepWiki-compatible hosted MCP server that exposes the `ask_question` tool and can answer questions about the `openai/openai-agents-python` repository.
+Every integration profile has one credential class. Local-only profiles are `packaging`, `prospective-contract`, `prospective-platform`, `security`, `mcp-v1`, and `extras`; the runner removes an inherited `OPENAI_API_KEY` before building distributions or starting child processes for these profiles. Live profiles are `core`, `providers`, `realtime`, `voice`, `hosted`, `full`, `release`, `nightly`, and `manual`; the runner refuses them before any build or child process unless `OPENAI_API_KEY_SOURCE=service-account`. Load the approved service-account environment before invoking a live Make target.
-Run `make integration-tests-providers-external` with `OPENROUTER_API_KEY` to exercise current OpenAI, Anthropic, and Google models through one provider gateway. To extend the matrix with separately configured direct-provider credentials, use `make integration-tests-providers-external -- --all`, `make integration-tests-providers-all`, or `uv run python .github/scripts/run_integration_tests.py --profile providers --all`. Set `ANTHROPIC_API_KEY` and `GEMINI_API_KEY` or `GOOGLE_API_KEY` for the direct providers you want to include. Override `OPENAI_AGENTS_INTEGRATION_ANTHROPIC_MODEL`, `OPENAI_AGENTS_INTEGRATION_GEMINI_MODEL`, or the comma-separated `OPENAI_AGENTS_INTEGRATION_OPENROUTER_MODELS` to select provider models.
+Set `OPENAI_API_KEY` to the approved service-account key and set `OPENAI_API_KEY_SOURCE=service-account` for live OpenAI calls. Override `OPENAI_AGENTS_INTEGRATION_MODEL`, `OPENAI_AGENTS_INTEGRATION_REALTIME_MODEL`, `OPENAI_AGENTS_INTEGRATION_ANY_LLM_MODELS`, and `OPENAI_AGENTS_INTEGRATION_LITELLM_MODELS` when testing different models or configured providers. Provider model lists contain comma-separated adapter model names and require the credentials matching each selected provider. Set `OPENAI_AGENTS_INTEGRATION_MCP_SERVER_URL` to use another trusted DeepWiki-compatible hosted MCP server that exposes the `ask_question` tool and can answer questions about the `openai/openai-agents-python` repository.
 Run `make integration-tests-providers-external` with `OPENROUTER_API_KEY` to exercise current OpenAI, Anthropic, and Google models through one provider gateway. To extend the matrix with separately configured direct-provider credentials, use `make integration-tests-providers-external -- --all` or `make integration-tests-providers-all`. Set `ANTHROPIC_API_KEY` and `GEMINI_API_KEY` or `GOOGLE_API_KEY` for the direct providers you want to include. Override `OPENAI_AGENTS_INTEGRATION_ANTHROPIC_MODEL`, `OPENAI_AGENTS_INTEGRATION_GEMINI_MODEL`, or the comma-separated `OPENAI_AGENTS_INTEGRATION_OPENROUTER_MODELS` to select provider models.
 The default general model is `gpt-5.6`, while LiteLLM function-tool cases use the Chat Completions-native `openai/gpt-4.1-mini`. This avoids LiteLLM's separate Responses API bridge and keeps the adapter regression focused on its actual Chat Completions contract.
@@ -23,6 +23,154 @@ def _run_suite() -> Callable[..., None]:
    return cast(Callable[..., None], runpy.run_path(str(RUNNER))["run_suite"])
 def test_every_integration_profile_has_exactly_one_credential_class() -> None:
    namespace = runpy.run_path(str(RUNNER))
    profile_classes = namespace["PROFILE_CREDENTIAL_CLASSES"]
    assert tuple(profile_classes) == namespace["PROFILES"]
    assert set(profile_classes.values()) == {
        namespace["LIVE_CREDENTIAL_CLASS"],
        namespace["LOCAL_ONLY_CREDENTIAL_CLASS"],
    }
    assert {
        profile
        for profile, credential_class in profile_classes.items()
        if credential_class == namespace["LOCAL_ONLY_CREDENTIAL_CLASS"]
    } == {
        "packaging",
        "prospective-contract",
        "prospective-platform",
        "security",
        "mcp-v1",
        "extras",
    }
    assert {
        profile
        for profile, credential_class in profile_classes.items()
        if credential_class == namespace["LIVE_CREDENTIAL_CLASS"]
    } == {
        "core",
        "providers",
        "realtime",
        "voice",
        "hosted",
        "full",
        "release",
        "nightly",
        "manual",
    }
@pytest.mark.parametrize(
    "profile",
    ["core", "providers", "realtime", "voice", "hosted", "full", "release", "nightly", "manual"],
 )
 def test_live_profiles_refuse_untrusted_credentials_before_side_effects(
    profile: str,
 ) -> None:
    namespace = runpy.run_path(str(RUNNER))
    bootstrap_in_uv = cast(Callable[..., None], namespace["bootstrap_in_uv"])
    child_processes: list[str] = []
    with pytest.raises(
        RuntimeError,
        match="requires OPENAI_API_KEY_SOURCE=service-account before any build or subprocess",
    ):
        bootstrap_in_uv(
            ["--profile", profile],
            {"OPENAI_API_KEY": "inherited-employee-key"},
            lambda *args: child_processes.append("uv"),
        )
    assert child_processes == []
@pytest.mark.parametrize(
    "profile",
    ["packaging", "prospective-contract", "prospective-platform", "security", "mcp-v1", "extras"],
 )
 def test_local_only_profiles_remove_key_before_uv_child_process(profile: str) -> None:
    namespace = runpy.run_path(str(RUNNER))
    bootstrap_in_uv = cast(Callable[..., None], namespace["bootstrap_in_uv"])
    environment = {
        "OPENAI_API_KEY": "inherited-employee-key",
        "OPENAI_API_KEY_SOURCE": "employee",
    }
    captured: list[tuple[str, list[str], dict[str, str]]] = []
    def capture_exec(file: str, command: list[str], child_env: dict[str, str]) -> None:
        captured.append((file, command, child_env))
    with pytest.raises(RuntimeError, match="bootstrap returned unexpectedly"):
        bootstrap_in_uv(["--profile", profile], environment, capture_exec)
    assert len(captured) == 1
    assert captured[0][0] == "uv"
    assert captured[0][1][0:3] == ["uv", "run", "python"]
    assert "OPENAI_API_KEY" not in captured[0][2]
    assert captured[0][2][namespace["BOOTSTRAPPED_ENV"]] == "1"
    assert "OPENAI_API_KEY" not in environment
 def test_local_only_profile_removes_key_before_cleanup_build_and_children(
    monkeypatch: pytest.MonkeyPatch,
    tmp_path: Path,
 ) -> None:
    namespace = runpy.run_path(str(RUNNER))
    main = cast(Callable[[], None], namespace["main"])
    observed_steps: list[str] = []
    def assert_sanitized(step: str) -> None:
        assert "OPENAI_API_KEY" not in os.environ
        observed_steps.append(step)
    def fake_build_distributions() -> tuple[Path, Path]:
        assert_sanitized("build")
        return tmp_path / "candidate.whl", tmp_path / "candidate.tar.gz"
    def fake_create_environment(
        name: str,
        distribution: Path,
        *,
        extras: bool = False,
        optional_extra: str | None = None,
        additional_requirements: tuple[str, ...] = (),
    ) -> Path:
        _ = (name, distribution, extras, optional_extra, additional_requirements)
        assert_sanitized("create-environment")
        return tmp_path / "python"
    def fake_run_suite(*args: object, **kwargs: Any) -> None:
        _ = (args, kwargs)
        assert_sanitized("run-suite")
    monkeypatch.setenv("OPENAI_API_KEY", "inherited-employee-key")
    monkeypatch.setenv("OPENAI_API_KEY_SOURCE", "employee")
    monkeypatch.setattr(sys, "argv", [str(RUNNER), "--profile", "packaging"])
    monkeypatch.setitem(main.__globals__, "build_distributions", fake_build_distributions)
    monkeypatch.setitem(main.__globals__, "create_environment", fake_create_environment)
    monkeypatch.setitem(main.__globals__, "run_suite", fake_run_suite)
    monkeypatch.setattr(
        main.__globals__["shutil"],
        "rmtree",
        lambda *args, **kwargs: assert_sanitized("cleanup"),
    )
    main()
    assert observed_steps[0:2] == ["cleanup", "build"]
    assert "create-environment" in observed_steps
    assert "run-suite" in observed_steps
 def test_unknown_profile_fails_closed_during_credential_classification() -> None:
    namespace = runpy.run_path(str(RUNNER))
    prepare_profile_environment = cast(Callable[..., str], namespace["prepare_profile_environment"])
    with pytest.raises(RuntimeError, match="has no credential class"):
        prepare_profile_environment("unclassified", {"OPENAI_API_KEY": "inherited"})
 def test_junit_sanitizer_removes_failure_details_and_captured_output(tmp_path: Path) -> None:
    sentinel = "JUNIT_SECRET_SENTINEL_42"
    report = tmp_path / "results.xml"
@@ -345,6 +493,8 @@ def test_code_change_detection_includes_packaged_contract_inputs() -> None:
    assert "integration_tests/" in detector
    assert "detect-changes\\.sh" in detector
    assert "run_integration_tests\\.py" in detector
    assert "run_examples\\.sh" in detector
    assert "examples-run-analysis" in detector
    assert "update_released_api_contract\\.py" in detector
    assert "\\.github/workflows/tests\\.yml" in detector
@@ -440,6 +590,7 @@ def test_release_profile_enforces_strict_security_for_wheel_and_sdist(
        suites.append(kwargs)
    monkeypatch.setenv("OPENAI_AGENTS_INTEGRATION_STRICT", "0")
    monkeypatch.setenv("OPENAI_API_KEY_SOURCE", "service-account")
    monkeypatch.setattr(sys, "argv", [str(RUNNER), "--profile", "release"])
    monkeypatch.setitem(main.__globals__, "build_distributions", fake_build_distributions)
    monkeypatch.setitem(main.__globals__, "create_environment", fake_create_environment)
@@ -0,0 +1,130 @@
 from __future__ import annotations
 import re
 import runpy
 from pathlib import Path
 ROOT = Path(__file__).resolve().parents[1]
 MAKEFILE = ROOT / "Makefile"
 EXAMPLE_RUNNER = ROOT / ".github" / "scripts" / "run_examples.sh"
 EXAMPLE_SUITE = ROOT / "examples" / "run_examples.py"
 SKILLS = ROOT / ".agents" / "skills"
 def _make_recipes() -> dict[str, str]:
    recipes: dict[str, str] = {}
    current_target: str | None = None
    for line in MAKEFILE.read_text(encoding="utf-8").splitlines():
        target_match = re.fullmatch(r"([A-Za-z0-9][A-Za-z0-9_-]*):(?:\s.*)?", line)
        if target_match:
            current_target = target_match.group(1)
            recipes[current_target] = ""
        elif current_target is not None and line.startswith("\t"):
            recipes[current_target] += line.removeprefix("\t") + "\n"
        elif line and not line.startswith((" ", "\t")):
            current_target = None
    return recipes
 def test_examples_run_analysis_skill_has_no_execution_path() -> None:
    analysis_skill = SKILLS / "examples-run-analysis"
    assert not (SKILLS / "examples-auto-run").exists()
    assert not (SKILLS / "integration-tests").exists()
    assert sorted(
        path.relative_to(analysis_skill).as_posix()
        for path in analysis_skill.rglob("*")
        if path.is_file()
    ) == ["SKILL.md", "agents/openai.yaml"]
    instructions = (analysis_skill / "SKILL.md").read_text(encoding="utf-8")
    prompt = (analysis_skill / "agents" / "openai.yaml").read_text(encoding="utf-8")
    assert "This skill is read-only and analysis-only." in instructions
    assert (
        "Never invoke an examples Make target or `.github/scripts/run_examples.sh`." in instructions
    )
    assert "Inspect the process table and `.tmp/examples-auto-run.pid`" in instructions
    assert "including foreground and background runs" in instructions
    assert "an absent or stale pid file does not prove that no run is active" in instructions
    assert ".tmp/examples-run.pid" not in instructions
    assert "Do not execute any of these commands as part of this skill." in instructions
    assert "without executing or controlling any process" in prompt
 def test_makefile_exposes_every_preserved_example_operation() -> None:
    recipes = _make_recipes()
    expected_commands = {
        "examples-run": "$(EXAMPLES_RUNNER) start $(EXAMPLES_ARGS)",
        "examples-run-background": "$(EXAMPLES_RUNNER) start --background $(EXAMPLES_ARGS)",
        "examples-status": "$(EXAMPLES_RUNNER) status",
        "examples-stop": "$(EXAMPLES_RUNNER) stop",
        "examples-logs": "$(EXAMPLES_RUNNER) logs",
        "examples-tail": "$(EXAMPLES_RUNNER) tail $(EXAMPLES_LOG)",
    }
    assert EXAMPLE_RUNNER.is_file()
    assert "EXAMPLES_RUNNER := bash .github/scripts/run_examples.sh" in MAKEFILE.read_text(
        encoding="utf-8"
    )
    for target, command in expected_commands.items():
        assert recipes[target].strip() == command
    assert "examples-rerun" not in recipes
    assert "examples-collect-rerun" not in recipes
 def test_repository_example_script_preserves_runner_contract() -> None:
    runner = EXAMPLE_RUNNER.read_text(encoding="utf-8")
    assert 'PID_FILE="$ROOT/.tmp/examples-auto-run.pid"' in runner
    assert 'LOG_DIR="$ROOT/.tmp/examples-start-logs"' in runner
    assert (
        'DEFAULT_UV_EXTRAS="litellm any-llm sqlalchemy redis blaxel modal runloop temporal"'
        in runner
    )
    for required_argument in ("--auto-mode", "--main-log", "--logs-dir"):
        assert required_argument in runner
    for optional_mode in (
        "EXAMPLES_INCLUDE_INTERACTIVE",
        "EXAMPLES_INCLUDE_SERVER",
        "EXAMPLES_INCLUDE_AUDIO",
        "EXAMPLES_INCLUDE_EXTERNAL",
    ):
        assert optional_mode in runner
    for operation in ("start", "status", "stop", "logs", "tail"):
        assert re.search(rf"(?:^|\n)  {operation}\)", runner)
    assert 'rm -f "$PID_FILE"' in runner
 def test_examples_rerun_mechanism_is_removed() -> None:
    sources = [
        MAKEFILE.read_text(encoding="utf-8"),
        EXAMPLE_RUNNER.read_text(encoding="utf-8"),
        EXAMPLE_SUITE.read_text(encoding="utf-8"),
        (ROOT / "examples" / "README.md").read_text(encoding="utf-8"),
        (SKILLS / "examples-run-analysis" / "SKILL.md").read_text(encoding="utf-8"),
    ]
    assert all("rerun" not in source.lower() for source in sources)
 def test_all_make_integration_entry_points_use_classified_profiles() -> None:
    namespace = runpy.run_path(str(ROOT / ".github" / "scripts" / "run_integration_tests.py"))
    classified_profiles = set(namespace["PROFILE_CREDENTIAL_CLASSES"])
    recipes = _make_recipes()
    integration_recipes = {
        target: recipe
        for target, recipe in recipes.items()
        if target == "integration-tests" or target.startswith("integration-tests-")
    }
    assert integration_recipes
    for target, recipe in integration_recipes.items():
        profile = re.search(r"--profile ([a-z0-9-]+)", recipe)
        assert profile is not None, target
        assert profile.group(1) in classified_profiles
 def test_prospective_contract_preparation_removes_api_key_before_uv() -> None:
    recipe = _make_recipes()["prepare-prospective-released-api-contract"]
    assert recipe.startswith("@unset OPENAI_API_KEY; \\\n")
    assert recipe.index("unset OPENAI_API_KEY") < recipe.index("uv run")
@@ -1,7 +1,10 @@
 from __future__ import annotations
 import sys
 from pathlib import Path
 import pytest
 import examples.run_examples as run_examples
@@ -77,6 +80,24 @@ def test_artifact_dir_for_example_uses_tmp_safe_stem(tmp_path: Path) -> None:
    assert artifact_dir == tmp_path / "examples__sandbox__tutorials__vision_website_clone__main"
@pytest.mark.parametrize(
    "removed_arguments",
    [
        ["--rerun-file", ".tmp/failed.txt"],
        ["--write-rerun"],
        ["--collect", ".tmp/main.log"],
        ["--output", ".tmp/failed.txt"],
    ],
 )
 def test_removed_rerun_arguments_are_rejected(
    monkeypatch: pytest.MonkeyPatch, removed_arguments: list[str]
 ) -> None:
    monkeypatch.setattr(sys, "argv", ["run_examples.py", *removed_arguments])
    with pytest.raises(SystemExit, match="2"):
        run_examples.parse_args()
 def test_prepare_redis_for_example_uses_existing_local_redis(monkeypatch) -> None:
    env: dict[str, str] = {}
    monkeypatch.setattr(run_examples, "redis_ping_url", lambda url, timeout=0.5: True)