Files
omnigent-ai--omnigent/tests/test_claude_native_status.py
Corey Zumar bc3dc80200 perf(claude-native): Improve performance of claude native terminal typing, text streaming, etc. (#4582)
* perf(claude-native): stop taxing every hook spawn with the eager package init

Claude Code blocks its TUI on command hooks — once per streamed text
chunk (MessageDisplay), per statusline refresh, and per tool call — and
every 'python -m omnigent.<hook>' subprocess re-ran omnigent/__init__,
which eagerly imported the datamodel/executor/model-catalog graph. The
deliberately stdlib-only hot-path hooks paid ~250 ms per spawn for
imports they never use, capping visible streaming at ~4 chunks/s.

The package init now re-exports lazily (PEP 562): the FIPS md5 patch
and legacy-env mirror stay eager, every public name resolves on first
attribute access (optional executors keep their import-failure->None
contract), and submodule attribute access still works. Hot-path hook
spawns drop to ~30 ms (~interpreter cost).

A native_hook_spawn benchmark journey spawns the MessageDisplay hook
exactly as Claude Code does and rides the release/nightly regression
comparison; fresh-interpreter import-graph guards in the display-hook
test suite pin what each hook entrypoint may import so the regression
cannot silently return.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* perf(claude-native): keep the hook's hot path off the bridge's heavy imports

The observer hook — Claude blocks on it at every prompt submit, tool
call, Stop, and task event — imported claude_native_bridge, whose
module-level tools/spec/pydantic imports cost ~450 ms of interpreter
startup, plus httpx and the policy machinery besides. Enter and every
tool call paid roughly a second of subprocess overhead per event even
after the package init went lazy.

The bridge now defers its tools graph to the one launch-path function
that builds MCP tools (_build_tools) and its bundle-skills parse to
the launch args builder; the hook imports httpx and the policy
machinery inside the subcommands that actually speak HTTP. Module
import cost: bridge 450 -> ~70 ms, hook 360 -> ~70 ms, and the hook's
fresh-interpreter import graph now contains no third-party modules at
all — the import guard pins the allowance at exactly that.

Tests that reached httpx or create_os_environment through the hook's
or bridge's module attributes now patch the owning modules directly.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* perf(claude-native): cache the ungoverned policy verdict at the relay

Sessions with no policies at all still paid a full server round trip
(~0.5-1.3s measured against a Databricks App) on every policy hook
event — twice per tool call plus every prompt submit — with the server
answering the same fast-path ALLOW each time. Typing during agentic
turns stuttered in the gaps; vanilla Claude pays nothing there.

The evaluate endpoint now stamps 'governed': false on its existing
no-policies fast path (any_policies_apply's False is session-scoped —
its only phase-scoped rule forces True), and the native-harness
loopback relay caches that verdict for 30s, answering hook events
instantly. A governed response of any kind drops the cache, a
sys_add_policy call through the relay's own /tool path clears it
before the policy lands, and expiry re-validates upstream — so
enforcement for governed sessions is untouched and the attach delay
for out-of-band policy edits is bounded at the TTL.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* perf(claude-native): keep blocking Claude hooks off Python and off the WAN

Claude blocks its TUI on every command hook, and three of them still
spawned a Python interpreter per event (~30ms floor, ~77ms under EDR):
MessageDisplay once per streamed chunk, statusLine per refresh, and
evaluate-policy twice per tool call — the last one also paying a
0.5-1.3s WAN round trip whenever its 30s ungoverned-cache window
lapsed.

- MessageDisplay: a /bin/sh one-liner appends the payload (newline-
  stripped, so any valid JSON lands single-line) straight to
  message_deltas.jsonl; the deltas reader already parses by key and
  skips malformed lines.
- statusLine: the shim captures raw stdin to context_raw.json (atomic
  rename) and chains the user's own status command; the forwarder
  normalizes it into context.json on its poll loop
  (sync_raw_status_context), so the Python normalizer leaves the
  blocking path. The module entrypoint stays for older bridge dirs.
- evaluate-policy: hooks try a curl against the relay's new
  /hook/claude/evaluate-policy endpoint (advertised via a
  shell-sourceable tool_relay.env); the long-lived runner process owns
  payload→EvaluationRequest mapping, retries, the ungoverned cache,
  and verdict→hook-output shaping. When the relay is absent or
  unreachable the same stdin replays into the Python hook, which keeps
  the direct-server path and the phase-aware fail-closed contract —
  exactly the pre-curl behavior.
- The relay starts at session create (runner app) instead of at the
  first web-dispatched turn, so prompts typed directly in the TUI get
  the curl fast path too; it comes up in the background, and hooks
  that beat it use the Python fallback.

Typing during a live 25-tool-call turn against a Databricks App
measured 56.0ms median / 57.2ms p90 / 0 samples over 200ms, from
118ms median / 264ms p90 / 8 freezes before this branch.

Also pins the relay-close ownership test's trusted-parent monkeypatch
to tempfile.gettempdir() — the literal /tmp never contains the macOS
fixture root, so the test only passed on Linux.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* perf(onboarding): cache harness CLI version and login probes

Every readiness refresh on every host daemon execs vendor CLIs
(--version / auth status) whose answers change only when the binary is
swapped or a login flips; with a few dozen idle hosts that compounds
into a constant machine-wide subprocess storm (~116 spawns/min
observed) that competes with interactive terminals.

--version output is a pure function of the binary bytes, so successful
parses cache permanently against the binary's (path, mtime_ns, size)
signature; failures keep re-probing. Login verdicts can flip without a
binary change, so only positives cache, with a 120s TTL — negatives
always re-probe so the setup wizard sees a fresh login immediately, and
harness_logout invalidates its key so a successful logout is confirmed
live.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* revert(claude-native): drop the ungoverned-verdict relay cache

The cache required stamping 'governed': false on the evaluate
response so the relay could tell which ALLOWs were safe to reuse —
new response-field surface carried only by this optimization, which
we don't need right now. Remove the stamp and the relay cache
wholesale: every policy hook event consults the server again, the
relay's /policies/evaluate proxy is a plain pass-through, and the
evaluate response is byte-identical to its pre-branch shape. The
sh-shim/curl hook path (no interpreter spawns) is unchanged.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
2026-08-13 13:11:46 -07:00

257 lines
8.8 KiB
Python

"""Tests for the Claude Code statusLine wrapper."""
from __future__ import annotations
import io
import json
import subprocess
import sys
from pathlib import Path
import pytest
from omnigent import claude_native_status
def _run(
*,
stdin_payload: str,
bridge_dir: Path,
chain: str | None = None,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str] | None = None,
) -> int:
"""
Invoke the wrapper's ``main()`` with a stubbed stdin.
:param stdin_payload: Raw stdin text the wrapper will read.
:param bridge_dir: Bridge directory the wrapper writes into.
:param chain: Optional chained command, e.g. ``"echo claude-hud"``.
:param monkeypatch: Pytest fixture for patching ``sys.stdin``.
:param capsys: Pytest stdout/stderr capture fixture, unused here.
:returns: Process exit code from ``main()``.
"""
del capsys
monkeypatch.setattr(sys, "stdin", io.StringIO(stdin_payload))
argv = ["--bridge-dir", str(bridge_dir)]
if chain is not None:
argv.extend(["--chain", chain])
return claude_native_status.main(argv)
def test_status_wrapper_writes_context_json(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""
The wrapper persists ``context_window_size`` and ``current_usage``.
These are the two fields the forwarder consumes; missing either
silently breaks the ring on a real claude-native session, so this
test pins the exact wire shape Claude Code provides on statusLine
stdin (modeled on claude-hud's reverse-engineered schema).
"""
stdin = json.dumps(
{
"session_id": "abc",
"model": {"display_name": "Opus 4.7"},
"context_window": {
"context_window_size": 1_000_000,
"current_usage": {
"input_tokens": 6,
"cache_creation_input_tokens": 100,
"cache_read_input_tokens": 200,
"output_tokens": 50,
},
"used_percentage": 31,
},
}
)
rc = _run(stdin_payload=stdin, bridge_dir=tmp_path, monkeypatch=monkeypatch)
assert rc == 0
persisted = json.loads((tmp_path / "context.json").read_text(encoding="utf-8"))
assert persisted["context_window_size"] == 1_000_000
assert persisted["current_usage"]["input_tokens"] == 6
assert persisted["current_usage"]["cache_creation_input_tokens"] == 100
assert persisted["used_percentage"] == 31
def test_status_wrapper_captures_cost(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""
The wrapper persists Claude Code's cumulative ``cost.total_cost_usd``.
Claude Code's statusLine stdin carries a top-level ``cost`` block with its
own session billing. claude-native never produces a ``response.completed``
event, so the Omnigent relay's cost accumulation never runs for it — capturing
this is the only way native session cost reaches ``session_usage``. A
failure here means native Cost-Ask policies always see $0.
"""
stdin = json.dumps(
{
"session_id": "abc",
"context_window": {"context_window_size": 1_000_000},
"cost": {"total_cost_usd": 0.42, "total_duration_ms": 1234},
}
)
rc = _run(stdin_payload=stdin, bridge_dir=tmp_path, monkeypatch=monkeypatch)
assert rc == 0
persisted = json.loads((tmp_path / "context.json").read_text(encoding="utf-8"))
assert persisted["total_cost_usd"] == 0.42
def test_status_wrapper_drops_payload_without_context_window(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""
Payloads with no ``context_window`` block leave no file behind.
Older Claude Code versions (pre v2.x) didn't include ``context_window``
on statusLine stdin. The wrapper must degrade gracefully so the
ring keeps the spec default rather than rendering a half-init state.
"""
stdin = json.dumps({"session_id": "abc", "model": {"display_name": "Opus 4.7"}})
rc = _run(stdin_payload=stdin, bridge_dir=tmp_path, monkeypatch=monkeypatch)
assert rc == 0
assert not (tmp_path / "context.json").exists()
def test_status_wrapper_chains_to_user_command(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""
The chained command receives the original stdin and renders its stdout.
Claude Code only invokes a single statusLine command. Without
chaining, overriding for context capture would silently hide
claude-hud / any user-installed status bar.
"""
captured_calls: list[dict[str, object]] = []
class _FakeProc:
returncode = 0
stdout = "claude-hud line\n"
stderr = ""
def fake_run(*args: object, **kwargs: object) -> _FakeProc:
captured_calls.append({"args": args, "kwargs": kwargs})
return _FakeProc()
monkeypatch.setattr(subprocess, "run", fake_run)
stdin = json.dumps({"context_window": {"context_window_size": 200_000}})
rc = _run(
stdin_payload=stdin,
bridge_dir=tmp_path,
chain="echo claude-hud",
monkeypatch=monkeypatch,
)
assert rc == 0
assert len(captured_calls) == 1
assert captured_calls[0]["args"] == ("echo claude-hud",)
assert captured_calls[0]["kwargs"]["input"] == stdin
assert captured_calls[0]["kwargs"]["shell"] is True
out, _err = capsys.readouterr()
assert "claude-hud line" in out
def test_status_wrapper_chain_swallows_subprocess_errors(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""
Chained-command failures don't propagate — context capture stays best-effort.
The statusLine command runs on every Claude Code render tick. A
crash there would yank the user's terminal status bar away on
every tick; the wrapper logs to stderr and returns 0 instead.
"""
def fake_run(*args: object, **kwargs: object) -> None:
raise OSError("chain broken")
monkeypatch.setattr(subprocess, "run", fake_run)
stdin = json.dumps({"context_window": {"context_window_size": 200_000}})
rc = _run(
stdin_payload=stdin,
bridge_dir=tmp_path,
chain="bogus",
monkeypatch=monkeypatch,
)
assert rc == 0
_out, err = capsys.readouterr()
assert "chain failed" in err
def test_normalize_status_payload_extracts_record() -> None:
"""The normalizer extracts window/usage/cost/model; None without a window."""
from omnigent.claude_native_status import normalize_status_payload
record = normalize_status_payload(
{
"context_window": {
"context_window_size": 200_000,
"current_usage": {"input_tokens": 5},
"used_percentage": 2.5,
},
"cost": {"total_cost_usd": 0.42},
"model": {"id": "claude-opus-4-8", "display_name": "Opus"},
}
)
assert record == {
"context_window_size": 200_000,
"current_usage": {"input_tokens": 5},
"used_percentage": 2.5,
"total_cost_usd": 0.42,
"model": "claude-opus-4-8",
}
assert normalize_status_payload({"model": "claude-opus-4-8"}) is None
def test_sync_raw_status_context_normalizes_and_retries(tmp_path: Path) -> None:
"""The forwarder-side sync normalizes raw captures and tolerates junk.
Unchanged signatures are no-ops, a malformed raw file leaves the
signature untouched so the next poll retries, and a rewritten raw
file re-normalizes.
"""
import json as _json
from omnigent.claude_native_status import (
CONTEXT_RAW_FILE,
sync_raw_status_context,
)
raw_path = tmp_path / CONTEXT_RAW_FILE
payload = {
"context_window": {"context_window_size": 1000},
"model": {"id": "claude-opus-4-8"},
}
raw_path.write_text(_json.dumps(payload, indent=2), encoding="utf-8")
sig = sync_raw_status_context(tmp_path, None)
assert sig is not None
written = _json.loads((tmp_path / "context.json").read_text("utf-8"))
assert written == {"context_window_size": 1000, "model": "claude-opus-4-8"}
# Unchanged raw file: same signature back, nothing rewritten.
assert sync_raw_status_context(tmp_path, sig) == sig
# Malformed raw file: signature unchanged so the next poll retries.
raw_path.write_text("{not json", encoding="utf-8")
assert sync_raw_status_context(tmp_path, sig) == sig
# Rewritten raw file: re-normalized under a fresh signature.
payload["model"] = {"id": "claude-sonnet-4-6"}
raw_path.write_text(_json.dumps(payload), encoding="utf-8")
new_sig = sync_raw_status_context(tmp_path, sig)
assert new_sig != sig
written = _json.loads((tmp_path / "context.json").read_text("utf-8"))
assert written["model"] == "claude-sonnet-4-6"