* feat(bench): add CLI startup latency benchmark
Measures wall-clock time from omnigent claude --server invocation to the
Claude terminal being ready (signalled by 'Claude terminal ready.' spinner
message, emitted just before tmux attach).
Unlike the HTTP/API benchmarks in run.py, this drives the real CLI binary
end-to-end against a remote server — auth, daemon tunnel, session create,
runner launch, terminal boot — via pexpect.
Usage:
uv run --no-sync dev/benchmarks/omnigent/cli_startup.py
uv run --no-sync dev/benchmarks/omnigent/cli_startup.py --also-isaac-omni --runs 10
uv run --no-sync dev/benchmarks/omnigent/cli_startup.py --output startup.json
uv run --no-sync dev/benchmarks/omnigent/cli_startup.py --max-p50-ms 12000
JSON output is compatible with the existing benchmark schema.
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* ci(bench): add cli-startup job to benchmark workflow
Adds a new 'CLI startup latency' job that runs cli_startup.py against
the ai-devtools managed workspace (OMNIGENT_REMOTE_AUTH_TOKEN secret).
- Runs on nightly schedule (when secret is configured) and on
workflow_dispatch with cli_startup_runs input (default 5, 0 = skip)
- Skips gracefully when OMNIGENT_REMOTE_AUTH_TOKEN secret is absent
- Uploads benchmark-results-cli-startup-{run_id}.json as an artifact
for the Databricks trend dashboard (same schema as the HTTP benchmarks)
- Renders a job summary table via report_markdown.py
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* feat(bench): align cli_startup with existing journey schema
- Use RunResult/aggregate/print_results/check_thresholds/build_report
from the existing framework instead of custom stats/output code
- Each run is now a RunResult with all latency samples (matching the
HTTP/API journey shape), not one run-per-sample
- Journey names are cli_startup and isaac_omni (snake_case, no spaces)
- Output table uses the same renderer as run.py
- Add cli_startup_runs dispatch input and cli-startup job to benchmark.yml
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* refactor(bench): move cli_startup into journeys.py; use local bench server
The cli_startup journey now lives in journeys.py alongside the other
journeys, using env.base_url (the local bench server) instead of a
remote Databricks URL. This aligns it with the existing pattern:
needs_host=True boots the host daemon, and omnigent claude --server
<local-url> connects to it for the full startup sequence.
cli_startup.py becomes a thin shim that calls run.py --journeys cli_startup.
benchmark.yml cli-startup job now uses run.py directly — no
OMNIGENT_REMOTE_AUTH_TOKEN secret needed, just pexpect + claude CLI.
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* ci(bench): fold claude CLI install into Install dependencies step
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* ci(bench): merge cli_startup into existing benchmark job (sqlite leg only)
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* remove cli_startup.py shim — use run.py --journeys cli_startup directly
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* ci(bench): run cli_startup on all matrix backends
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* ci(bench): install pexpect+claude before Run benchmark so cli_startup does not skip
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* fix(bench): fix policy_evaluate setup (POST /v1/agents → /v1/sessions bundle); add needs_runner to cli_startup
- policy_evaluate setup was calling POST /v1/agents which is GET-only.
Fix: use POST /v1/sessions multipart bundle upload (same as ensure_agent),
with executor fields added to pass spec validation, and read session_id
from the correct response key.
- cli_startup: add needs_runner=True so the test_runner_journeys_are_capped
invariant passes (needs_host implies needs_runner in BenchEnvironment but
not on the Journey dataclass itself).
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* ci(bench): add pexpect+claude install to benchmark-pr.yml
cli_startup is in ALL_JOURNEYS so it runs in the benchmark-pr regression
check too. Without pexpect and claude installed, every iteration fails
with RuntimeError.
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* fix(bench): replace test fixture function ref in policy_evaluate with self-contained one
tests.runtime.policies.conftest._always_allow is a test fixture that may
not be importable in the server subprocess's PYTHONPATH in CI, causing
HTTP 500 on every evaluate call. Replace with _bench_policy_allow defined
directly in journeys.py, which is always importable since dev/ is on
PYTHONPATH in the benchmark environment.
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* fix(bench): gate cli_startup on OMNIGENT_BENCH_SERVER; skip gracefully when not set
cli_startup conflicts with the bench environment's host daemon when run
against the local bench server — omnigent claude spawns its own daemon
which hits a 'host on another replica' error. Gate on OMNIGENT_BENCH_SERVER
env var instead: skip with a clear RuntimeError when unset, use the remote
server when set.
- Remove needs_runner/needs_host (no local server contact)
- Reduce max_iterations from 5 to 3 (each is ~10s)
- Set OMNIGENT_BENCH_SERVER in benchmark.yml and benchmark-pr.yml
- Relax test_runner_journeys_are_capped to allow non-runner journeys to cap
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* ci(bench): remove hardcoded OMNIGENT_BENCH_SERVER from workflows
cli_startup skips gracefully in CI (no OMNIGENT_BENCH_SERVER set).
Run it manually: OMNIGENT_BENCH_SERVER=<url> uv run --no-sync dev/benchmarks/omnigent/run.py --journeys cli_startup
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* fix(bench): run cli_startup against local bench server; drop OMNIGENT_BENCH_SERVER
The daemon conflict was caused by needs_host=True booting a bench daemon
alongside the CLI's own daemon. With needs_host=False the bench environment
starts only the server; omnigent claude spawns its own daemon freely — no
conflict.
Result: 5.3s local vs 11s remote. CI runs it as part of the default suite
with no remote credentials needed.
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* fix(bench): use omnigent polly instead of omnigent claude for cli_startup
claude-native requires the external claude CLI binary which:
- Takes too long to boot on CI (90s timeout → job gets stuck)
- Requires npm install of @anthropic-ai/claude-code
polly (omnigent run with the bundled openai-agents harness) exercises the
same startup path (daemon, session create, runner launch, runner connect)
without any external binary dependency. Signal: 'Launching your agent'
with a 30s timeout instead of 90s.
Remove @anthropic-ai/claude-code install from both benchmark workflows.
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* fix(bench): move cli_startup to OPT_IN_JOURNEYS; exclude from default run
cli_startup against the local bench server hangs in CI — the polly runner
can't complete its startup within 30s, burning 19 min (39 attempts × 30s
including warmup) before failing.
Move it to OPT_IN_JOURNEYS: excluded from the default set, must be run
explicitly via --journeys cli_startup. resolve_journeys() looks in both
registries so it still works when named. Remove pexpect install from CI
workflows since it's no longer needed for the default benchmark run.
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* fix(bench): cli_startup back in ALL_JOURNEYS; add skip_warmup flag; 60s timeout
- Move cli_startup back to ALL_JOURNEYS (not needs_host; spawns its own daemon)
- Add Journey.skip_warmup: when True, run_latency skips the warmup phase
regardless of --warmup. Avoids 10x60s = 10min of wasted warmup hangs.
- Increase timeout from 30s to 60s (CI runner is slower than local Mac)
- Restore pexpect install in both benchmark workflows
With skip_warmup=True and max_iterations=3: 3 runs x 3 = 9 iterations max,
no warmup hangs. Worst case: 9 x 60s = 9min if all timeout (shouldn't happen).
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* debug(bench): include RuntimeError message in failure breakdown for CI visibility
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* fix(bench): stop stale daemons before each cli_startup iteration
A leftover host daemon from the previous iteration causes the next
omnigent polly to fail with 'runner tunnel rejection' or 'host is on
another replica'. Run omnigent stop before spawning polly to ensure
a clean slate each time.
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
* fix(bench): move omnigent stop to prepare hook so it's outside the latency timer
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
---------
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
Tomu Hirata
2aba5079d4