feat(bench): add CLI startup latency benchmark (#4793)

* feat(bench): add CLI startup latency benchmark

Measures wall-clock time from omnigent claude --server invocation to the
Claude terminal being ready (signalled by 'Claude terminal ready.' spinner
message, emitted just before tmux attach).

Unlike the HTTP/API benchmarks in run.py, this drives the real CLI binary
end-to-end against a remote server — auth, daemon tunnel, session create,
runner launch, terminal boot — via pexpect.

Usage:
  uv run --no-sync dev/benchmarks/omnigent/cli_startup.py
  uv run --no-sync dev/benchmarks/omnigent/cli_startup.py --also-isaac-omni --runs 10
  uv run --no-sync dev/benchmarks/omnigent/cli_startup.py --output startup.json
  uv run --no-sync dev/benchmarks/omnigent/cli_startup.py --max-p50-ms 12000

JSON output is compatible with the existing benchmark schema.

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* ci(bench): add cli-startup job to benchmark workflow

Adds a new 'CLI startup latency' job that runs cli_startup.py against
the ai-devtools managed workspace (OMNIGENT_REMOTE_AUTH_TOKEN secret).

- Runs on nightly schedule (when secret is configured) and on
  workflow_dispatch with cli_startup_runs input (default 5, 0 = skip)
- Skips gracefully when OMNIGENT_REMOTE_AUTH_TOKEN secret is absent
- Uploads benchmark-results-cli-startup-{run_id}.json as an artifact
  for the Databricks trend dashboard (same schema as the HTTP benchmarks)
- Renders a job summary table via report_markdown.py

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* feat(bench): align cli_startup with existing journey schema

- Use RunResult/aggregate/print_results/check_thresholds/build_report
  from the existing framework instead of custom stats/output code
- Each run is now a RunResult with all latency samples (matching the
  HTTP/API journey shape), not one run-per-sample
- Journey names are cli_startup and isaac_omni (snake_case, no spaces)
- Output table uses the same renderer as run.py
- Add cli_startup_runs dispatch input and cli-startup job to benchmark.yml

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* refactor(bench): move cli_startup into journeys.py; use local bench server

The cli_startup journey now lives in journeys.py alongside the other
journeys, using env.base_url (the local bench server) instead of a
remote Databricks URL. This aligns it with the existing pattern:
needs_host=True boots the host daemon, and omnigent claude --server
<local-url> connects to it for the full startup sequence.

cli_startup.py becomes a thin shim that calls run.py --journeys cli_startup.

benchmark.yml cli-startup job now uses run.py directly — no
OMNIGENT_REMOTE_AUTH_TOKEN secret needed, just pexpect + claude CLI.

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* ci(bench): fold claude CLI install into Install dependencies step

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* ci(bench): merge cli_startup into existing benchmark job (sqlite leg only)

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* remove cli_startup.py shim — use run.py --journeys cli_startup directly

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* ci(bench): run cli_startup on all matrix backends

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* ci(bench): install pexpect+claude before Run benchmark so cli_startup does not skip

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* fix(bench): fix policy_evaluate setup (POST /v1/agents → /v1/sessions bundle); add needs_runner to cli_startup

- policy_evaluate setup was calling POST /v1/agents which is GET-only.
  Fix: use POST /v1/sessions multipart bundle upload (same as ensure_agent),
  with executor fields added to pass spec validation, and read session_id
  from the correct response key.

- cli_startup: add needs_runner=True so the test_runner_journeys_are_capped
  invariant passes (needs_host implies needs_runner in BenchEnvironment but
  not on the Journey dataclass itself).

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* ci(bench): add pexpect+claude install to benchmark-pr.yml

cli_startup is in ALL_JOURNEYS so it runs in the benchmark-pr regression
check too. Without pexpect and claude installed, every iteration fails
with RuntimeError.

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* fix(bench): replace test fixture function ref in policy_evaluate with self-contained one

tests.runtime.policies.conftest._always_allow is a test fixture that may
not be importable in the server subprocess's PYTHONPATH in CI, causing
HTTP 500 on every evaluate call. Replace with _bench_policy_allow defined
directly in journeys.py, which is always importable since dev/ is on
PYTHONPATH in the benchmark environment.

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* fix(bench): gate cli_startup on OMNIGENT_BENCH_SERVER; skip gracefully when not set

cli_startup conflicts with the bench environment's host daemon when run
against the local bench server — omnigent claude spawns its own daemon
which hits a 'host on another replica' error. Gate on OMNIGENT_BENCH_SERVER
env var instead: skip with a clear RuntimeError when unset, use the remote
server when set.

- Remove needs_runner/needs_host (no local server contact)
- Reduce max_iterations from 5 to 3 (each is ~10s)
- Set OMNIGENT_BENCH_SERVER in benchmark.yml and benchmark-pr.yml
- Relax test_runner_journeys_are_capped to allow non-runner journeys to cap

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* ci(bench): remove hardcoded OMNIGENT_BENCH_SERVER from workflows

cli_startup skips gracefully in CI (no OMNIGENT_BENCH_SERVER set).
Run it manually: OMNIGENT_BENCH_SERVER=<url> uv run --no-sync dev/benchmarks/omnigent/run.py --journeys cli_startup

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* fix(bench): run cli_startup against local bench server; drop OMNIGENT_BENCH_SERVER

The daemon conflict was caused by needs_host=True booting a bench daemon
alongside the CLI's own daemon. With needs_host=False the bench environment
starts only the server; omnigent claude spawns its own daemon freely — no
conflict.

Result: 5.3s local vs 11s remote. CI runs it as part of the default suite
with no remote credentials needed.

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* fix(bench): use omnigent polly instead of omnigent claude for cli_startup

claude-native requires the external claude CLI binary which:
- Takes too long to boot on CI (90s timeout → job gets stuck)
- Requires npm install of @anthropic-ai/claude-code

polly (omnigent run with the bundled openai-agents harness) exercises the
same startup path (daemon, session create, runner launch, runner connect)
without any external binary dependency. Signal: 'Launching your agent'
with a 30s timeout instead of 90s.

Remove @anthropic-ai/claude-code install from both benchmark workflows.

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* fix(bench): move cli_startup to OPT_IN_JOURNEYS; exclude from default run

cli_startup against the local bench server hangs in CI — the polly runner
can't complete its startup within 30s, burning 19 min (39 attempts × 30s
including warmup) before failing.

Move it to OPT_IN_JOURNEYS: excluded from the default set, must be run
explicitly via --journeys cli_startup. resolve_journeys() looks in both
registries so it still works when named. Remove pexpect install from CI
workflows since it's no longer needed for the default benchmark run.

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* fix(bench): cli_startup back in ALL_JOURNEYS; add skip_warmup flag; 60s timeout

- Move cli_startup back to ALL_JOURNEYS (not needs_host; spawns its own daemon)
- Add Journey.skip_warmup: when True, run_latency skips the warmup phase
  regardless of --warmup. Avoids 10x60s = 10min of wasted warmup hangs.
- Increase timeout from 30s to 60s (CI runner is slower than local Mac)
- Restore pexpect install in both benchmark workflows

With skip_warmup=True and max_iterations=3: 3 runs x 3 = 9 iterations max,
no warmup hangs. Worst case: 9 x 60s = 9min if all timeout (shouldn't happen).

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* debug(bench): include RuntimeError message in failure breakdown for CI visibility

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* fix(bench): stop stale daemons before each cli_startup iteration

A leftover host daemon from the previous iteration causes the next
omnigent polly to fail with 'runner tunnel rejection' or 'host is on
another replica'. Run omnigent stop before spawning polly to ensure
a clean slate each time.

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

* fix(bench): move omnigent stop to prepare hook so it's outside the latency timer

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

---------

Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>

.github/workflows/benchmark-pr.yml

@@ -55,7 +55,10 @@ jobs:
           enable-cache: true
 
       - name: Install dependencies
-        run: uv sync --extra databricks
+        # pexpect drives omnigent polly via PTY for the cli_startup journey.
+        run: |
+          uv sync --extra databricks
+          uv pip install pexpect
 
       # Use the same corpus size as the nightly so baseline numbers are
       # directly comparable. Cache the seeded DB on the schema head + seed

.github/workflows/benchmark.yml

@@ -180,6 +180,10 @@ jobs:
             --database-uri "${{ steps.db.outputs.uri }}" \
             --sessions "$SESSIONS" --items-per-session "$ITEMS"
 
+      # pexpect drives omnigent polly via PTY for the cli_startup journey.
+      - name: Install CLI startup dependencies
+        run: uv pip install pexpect
+
       - name: Run benchmark
         run: |
           uv run --no-sync dev/benchmarks/omnigent/run.py \

dev/benchmarks/omnigent/journeys.py

@@ -45,7 +45,9 @@ from __future__ import annotations
 import asyncio
 import contextlib
 import json
+import os
 import shutil
+import subprocess
 import sys
 import tempfile
 import time
@@ -101,6 +103,9 @@ class Journey:
         per op, so 100+ iterations would blow the CI time budget; they cap at a
         few samples per run and lean on ``--runs`` for repeats. ``None`` (HTTP
         journeys) means no cap.
+    :param skip_warmup: When ``True``, the warmup phase is skipped regardless
+        of ``--warmup``. Useful for expensive journeys where even a single
+        warmup iteration would waste significant time.
     :param description: Human-readable one-liner for ``--list``.
     """
 
@@ -114,6 +119,7 @@ class Journey:
     needs_runner: bool = False
     needs_host: bool = False
     max_iterations: int | None = None
+    skip_warmup: bool = False
     description: str = ""
 
     async def run_setup(self, env: BenchEnvironment) -> JourneyContext:
@@ -135,10 +141,13 @@ def _failure_reason(exc: Exception) -> str:
     """Classify an exception into a stable failure-breakdown label.
 
     HTTP status errors key off their status code (``"HTTP 500"``) so the same
-    server error groups across ops; anything else keys off its class name.
+    server error groups across ops; RuntimeErrors include the message so CI
+    failure breakdowns show the actual cause; anything else keys off class name.
     """
     if isinstance(exc, httpx.HTTPStatusError):
         return f"HTTP {exc.response.status_code}"
+    if isinstance(exc, RuntimeError):
+        return f"RuntimeError: {exc}"
     return exc.__class__.__name__
 
 
@@ -239,7 +248,8 @@ async def run_latency(
     except Exception as exc:  # noqa: BLE001 — a setup failure is a recorded data point
         return _setup_failed_result(exc)
     try:
-        for _ in range(warmup):
+        effective_warmup = 0 if journey.skip_warmup else warmup
+        for _ in range(effective_warmup):
             with contextlib.suppress(Exception):  # warmup errors are non-fatal
                 await journey.run_prepare(env, ctx)
                 await journey.measure(env, ctx)
@@ -685,6 +695,12 @@ async def _measure_read_runner_file(env: BenchEnvironment, ctx: JourneyContext)
 
 # ── policy evaluate ──────────────────────────────────────────
 
+
+def _bench_policy_allow(_event: dict) -> dict:  # type: ignore[type-arg]
+    """Benchmark policy function: always ALLOW. Self-contained in this module."""
+    return {"result": "allow"}
+
+
 _POLICY_EVALUATE_PAYLOAD = {
     "event": {
         "type": "PHASE_TOOL_CALL",
@@ -711,15 +727,24 @@ async def _setup_policy_evaluate_session(env: BenchEnvironment) -> str:
 
     import yaml
 
+    # Build a bundle like BenchEnvironment._agent_bundle but with a policy
+    # declared so any_policies_apply is true and the full engine runs.
+    executor: dict[str, object] = {
+        "type": "omnigent",
+        "model": env.model,
+        "config": {"harness": env.harness},
+    }
     config: dict[str, object] = {
         "spec_version": 1,
         "name": "bench-policy-agent",
+        "prompt": "benchmark",
+        "executor": executor,
         "guardrails": {
             "policies": {
                 "allow_all": {
                     "type": "function",
                     "on": ["tool_call"],
-                    "function": "tests.runtime.policies.conftest._always_allow",
+                    "function": "dev.benchmarks.omnigent.journeys._bench_policy_allow",
                 }
             }
         },
@@ -732,16 +757,17 @@ async def _setup_policy_evaluate_session(env: BenchEnvironment) -> str:
         tar.addfile(info, io.BytesIO(payload))
     bundle = buf.getvalue()
 
+    # Register the agent + create a session in one call via the bundle upload
+    # path (``POST /v1/sessions`` multipart). ``/v1/agents`` is GET-only.
     resp = await env.client.post(
-        "/v1/agents",
+        "/v1/sessions",
+        data={"metadata": "{}"},
         files={"bundle": ("agent.tar.gz", bundle, "application/gzip")},
     )
     resp.raise_for_status()
-    agent_id = resp.json()["id"]
+    body = resp.json()
-
+    # Bundle upload returns ``session_id`` (not ``id``).
-    session_resp = await env.client.post("/v1/sessions", json={"agent_id": agent_id})
+    session_id = body.get("session_id") or body["id"]
-    session_resp.raise_for_status()
-    session_id = session_resp.json()["id"]
 
     # Warm the spec + policy caches — the measured iteration is steady-state.
     for _ in range(2):
@@ -763,6 +789,95 @@ async def _measure_policy_evaluate(env: BenchEnvironment, ctx: JourneyContext) -
     resp.raise_for_status()
 
 
+# ── CLI startup (omnigent polly against the local bench server) ──────────────
+
+# Signal that the REPL is ready — the last spinner message before the prompt.
+# polly (omnigent run) emits this just before the agent REPL appears.
+_CLI_STARTUP_READY_SIGNAL = "Launching your agent"
+
+# Per-attempt timeout. With the bench host daemon pre-running (needs_host=True),
+# polly reuses it; remaining work is session + runner connect ~5-20s on CI.
+_CLI_STARTUP_TIMEOUT_S = 60
+
+# ~5s per attempt; cap so a large --iterations stays in budget.
+_CLI_STARTUP_MAX_ITERATIONS = 3
+
+
+async def _prepare_cli_startup(env: BenchEnvironment, _ctx: JourneyContext) -> None:
+    """Stop stale daemons before each timed cli_startup iteration.
+
+    A leftover host daemon from the previous iteration causes the next
+    ``omnigent polly`` to fail with "runner tunnel rejection (HTTP 401)"
+    or "host is on another replica". Runs outside the latency timer.
+    """
+    del env
+    omnigent_bin = os.environ.get("OMNIGENT_BIN") or shutil.which("omnigent")
+    if omnigent_bin is None:
+        return
+    await asyncio.to_thread(
+        subprocess.run,
+        [omnigent_bin, "stop"],
+        capture_output=True,
+        timeout=15,
+        check=False,
+    )
+
+
+async def _measure_cli_startup(env: BenchEnvironment, _ctx: JourneyContext) -> None:
+    """Time ``omnigent polly --server`` from invocation to REPL ready.
+
+    Spawns ``omnigent polly --server <local>`` via pexpect and times until
+    ``"Launching your agent…"`` appears — the last spinner message before the
+    agent REPL. Using polly (the bundled openai-agents harness) avoids any
+    external binary dependency while exercising the same startup path as
+    ``omnigent claude``: daemon start, session create, runner launch, and
+    runner connect.
+
+    Requires ``pexpect``. No external LLM binary needed.
+
+    :param env: Benchmark environment — ``env.base_url`` is the local server URL.
+    :param _ctx: Unused (no setup context).
+    :raises RuntimeError: On timeout or process exit before the ready signal.
+    """
+    try:
+        import pexpect
+    except ImportError as exc:
+        raise RuntimeError(
+            "pexpect is required for cli_startup. Install with: pip install pexpect"
+        ) from exc
+
+    omnigent_bin = os.environ.get("OMNIGENT_BIN") or shutil.which("omnigent")
+    if omnigent_bin is None:
+        raise RuntimeError("omnigent binary not found. Set OMNIGENT_BIN or add omnigent to PATH.")
+
+    child = pexpect.spawn(
+        omnigent_bin,
+        args=["polly", "--server", env.base_url],
+        timeout=_CLI_STARTUP_TIMEOUT_S,
+        encoding="utf-8",
+        codec_errors="ignore",
+        env=dict(os.environ),
+    )
+    try:
+        idx = child.expect([pexpect.TIMEOUT, pexpect.EOF, _CLI_STARTUP_READY_SIGNAL])
+        if idx == 0:
+            raise RuntimeError(
+                f"Timed out after {_CLI_STARTUP_TIMEOUT_S}s waiting for "
+                f"{_CLI_STARTUP_READY_SIGNAL!r}"
+            )
+        if idx == 1:
+            output = (child.before or "").strip()
+            raise RuntimeError(
+                f"Process exited before {_CLI_STARTUP_READY_SIGNAL!r}. "
+                f"Last output: {output[-200:]!r}"
+            )
+        child.sendline("/exit")
+        child.expect([pexpect.EOF, pexpect.TIMEOUT], timeout=10)
+    finally:
+        if child.isalive():
+            child.terminate(force=True)
+
+
 # ── native hook spawn (no server involved) ───────────────────
 
 # Claude Code blocks its TUI on command hooks, so one hook subprocess's whole
@@ -969,9 +1084,25 @@ ALL_JOURNEYS: dict[str, Journey] = {
             teardown=_teardown_hook_spawn,
             description="Spawn the per-chunk MessageDisplay hook exactly as Claude Code does.",
         ),
+        Journey(
+            name="cli_startup",
+            kind="latency",
+            measure=_measure_cli_startup,
+            prepare=_prepare_cli_startup,
+            max_iterations=_CLI_STARTUP_MAX_ITERATIONS,
+            skip_warmup=True,
+            description=(
+                "Spawn `omnigent polly --server` and time invocation → REPL ready "
+                "(daemon + session + runner connect). No LLM call needed. "
+                "Requires pexpect."
+            ),
+        ),
     )
 }
 
+# Registry alias — kept for callers that enumerate opt-in journeys explicitly.
+OPT_IN_JOURNEYS: dict[str, Journey] = {}
+
 
 def resolve_journeys(names: list[str] | None) -> list[Journey]:
     """Resolve requested journey *names* (or all when ``None``/empty).

tests/benchmarks/test_benchmark_smoke.py

@@ -292,12 +292,14 @@ def test_effective_iterations_uncapped_journey_passthrough() -> None:
 
 
 def test_runner_journeys_are_capped() -> None:
-    """Every full-turn journey caps its iterations; HTTP journeys do not."""
+    """Every full-turn journey caps its iterations; HTTP journeys do not.
+
+    Non-runner journeys may also declare a cap when they are inherently slow
+    (e.g. cli_startup which takes ~10s per iteration).
+    """
     for journey in ALL_JOURNEYS.values():
         if journey.needs_runner:
             assert journey.max_iterations is not None, journey.name
-        else:
-            assert journey.max_iterations is None, journey.name
 
 
 @pytest.mark.asyncio