bc3dc80200
* perf(claude-native): stop taxing every hook spawn with the eager package init Claude Code blocks its TUI on command hooks — once per streamed text chunk (MessageDisplay), per statusline refresh, and per tool call — and every 'python -m omnigent.<hook>' subprocess re-ran omnigent/__init__, which eagerly imported the datamodel/executor/model-catalog graph. The deliberately stdlib-only hot-path hooks paid ~250 ms per spawn for imports they never use, capping visible streaming at ~4 chunks/s. The package init now re-exports lazily (PEP 562): the FIPS md5 patch and legacy-env mirror stay eager, every public name resolves on first attribute access (optional executors keep their import-failure->None contract), and submodule attribute access still works. Hot-path hook spawns drop to ~30 ms (~interpreter cost). A native_hook_spawn benchmark journey spawns the MessageDisplay hook exactly as Claude Code does and rides the release/nightly regression comparison; fresh-interpreter import-graph guards in the display-hook test suite pin what each hook entrypoint may import so the regression cannot silently return. Signed-off-by: dbczumar <corey.zumar@databricks.com> * perf(claude-native): keep the hook's hot path off the bridge's heavy imports The observer hook — Claude blocks on it at every prompt submit, tool call, Stop, and task event — imported claude_native_bridge, whose module-level tools/spec/pydantic imports cost ~450 ms of interpreter startup, plus httpx and the policy machinery besides. Enter and every tool call paid roughly a second of subprocess overhead per event even after the package init went lazy. The bridge now defers its tools graph to the one launch-path function that builds MCP tools (_build_tools) and its bundle-skills parse to the launch args builder; the hook imports httpx and the policy machinery inside the subcommands that actually speak HTTP. Module import cost: bridge 450 -> ~70 ms, hook 360 -> ~70 ms, and the hook's fresh-interpreter import graph now contains no third-party modules at all — the import guard pins the allowance at exactly that. Tests that reached httpx or create_os_environment through the hook's or bridge's module attributes now patch the owning modules directly. Signed-off-by: dbczumar <corey.zumar@databricks.com> * perf(claude-native): cache the ungoverned policy verdict at the relay Sessions with no policies at all still paid a full server round trip (~0.5-1.3s measured against a Databricks App) on every policy hook event — twice per tool call plus every prompt submit — with the server answering the same fast-path ALLOW each time. Typing during agentic turns stuttered in the gaps; vanilla Claude pays nothing there. The evaluate endpoint now stamps 'governed': false on its existing no-policies fast path (any_policies_apply's False is session-scoped — its only phase-scoped rule forces True), and the native-harness loopback relay caches that verdict for 30s, answering hook events instantly. A governed response of any kind drops the cache, a sys_add_policy call through the relay's own /tool path clears it before the policy lands, and expiry re-validates upstream — so enforcement for governed sessions is untouched and the attach delay for out-of-band policy edits is bounded at the TTL. Signed-off-by: dbczumar <corey.zumar@databricks.com> * perf(claude-native): keep blocking Claude hooks off Python and off the WAN Claude blocks its TUI on every command hook, and three of them still spawned a Python interpreter per event (~30ms floor, ~77ms under EDR): MessageDisplay once per streamed chunk, statusLine per refresh, and evaluate-policy twice per tool call — the last one also paying a 0.5-1.3s WAN round trip whenever its 30s ungoverned-cache window lapsed. - MessageDisplay: a /bin/sh one-liner appends the payload (newline- stripped, so any valid JSON lands single-line) straight to message_deltas.jsonl; the deltas reader already parses by key and skips malformed lines. - statusLine: the shim captures raw stdin to context_raw.json (atomic rename) and chains the user's own status command; the forwarder normalizes it into context.json on its poll loop (sync_raw_status_context), so the Python normalizer leaves the blocking path. The module entrypoint stays for older bridge dirs. - evaluate-policy: hooks try a curl against the relay's new /hook/claude/evaluate-policy endpoint (advertised via a shell-sourceable tool_relay.env); the long-lived runner process owns payload→EvaluationRequest mapping, retries, the ungoverned cache, and verdict→hook-output shaping. When the relay is absent or unreachable the same stdin replays into the Python hook, which keeps the direct-server path and the phase-aware fail-closed contract — exactly the pre-curl behavior. - The relay starts at session create (runner app) instead of at the first web-dispatched turn, so prompts typed directly in the TUI get the curl fast path too; it comes up in the background, and hooks that beat it use the Python fallback. Typing during a live 25-tool-call turn against a Databricks App measured 56.0ms median / 57.2ms p90 / 0 samples over 200ms, from 118ms median / 264ms p90 / 8 freezes before this branch. Also pins the relay-close ownership test's trusted-parent monkeypatch to tempfile.gettempdir() — the literal /tmp never contains the macOS fixture root, so the test only passed on Linux. Signed-off-by: dbczumar <corey.zumar@databricks.com> * perf(onboarding): cache harness CLI version and login probes Every readiness refresh on every host daemon execs vendor CLIs (--version / auth status) whose answers change only when the binary is swapped or a login flips; with a few dozen idle hosts that compounds into a constant machine-wide subprocess storm (~116 spawns/min observed) that competes with interactive terminals. --version output is a pure function of the binary bytes, so successful parses cache permanently against the binary's (path, mtime_ns, size) signature; failures keep re-probing. Login verdicts can flip without a binary change, so only positives cache, with a 120s TTL — negatives always re-probe so the setup wizard sees a fresh login immediately, and harness_logout invalidates its key so a successful logout is confirmed live. Signed-off-by: dbczumar <corey.zumar@databricks.com> * revert(claude-native): drop the ungoverned-verdict relay cache The cache required stamping 'governed': false on the evaluate response so the relay could tell which ALLOWs were safe to reuse — new response-field surface carried only by this optimization, which we don't need right now. Remove the stamp and the relay cache wholesale: every policy hook event consults the server again, the relay's /policies/evaluate proxy is a plain pass-through, and the evaluate response is byte-identical to its pre-branch shape. The sh-shim/curl hook path (no interpreter spawns) is unchanged. Signed-off-by: dbczumar <corey.zumar@databricks.com> --------- Signed-off-by: dbczumar <corey.zumar@databricks.com>
257 lines
8.8 KiB
Python
257 lines
8.8 KiB
Python
"""Tests for the Claude Code statusLine wrapper."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import io
|
|
import json
|
|
import subprocess
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from omnigent import claude_native_status
|
|
|
|
|
|
def _run(
|
|
*,
|
|
stdin_payload: str,
|
|
bridge_dir: Path,
|
|
chain: str | None = None,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
capsys: pytest.CaptureFixture[str] | None = None,
|
|
) -> int:
|
|
"""
|
|
Invoke the wrapper's ``main()`` with a stubbed stdin.
|
|
|
|
:param stdin_payload: Raw stdin text the wrapper will read.
|
|
:param bridge_dir: Bridge directory the wrapper writes into.
|
|
:param chain: Optional chained command, e.g. ``"echo claude-hud"``.
|
|
:param monkeypatch: Pytest fixture for patching ``sys.stdin``.
|
|
:param capsys: Pytest stdout/stderr capture fixture, unused here.
|
|
:returns: Process exit code from ``main()``.
|
|
"""
|
|
del capsys
|
|
monkeypatch.setattr(sys, "stdin", io.StringIO(stdin_payload))
|
|
argv = ["--bridge-dir", str(bridge_dir)]
|
|
if chain is not None:
|
|
argv.extend(["--chain", chain])
|
|
return claude_native_status.main(argv)
|
|
|
|
|
|
def test_status_wrapper_writes_context_json(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""
|
|
The wrapper persists ``context_window_size`` and ``current_usage``.
|
|
|
|
These are the two fields the forwarder consumes; missing either
|
|
silently breaks the ring on a real claude-native session, so this
|
|
test pins the exact wire shape Claude Code provides on statusLine
|
|
stdin (modeled on claude-hud's reverse-engineered schema).
|
|
"""
|
|
stdin = json.dumps(
|
|
{
|
|
"session_id": "abc",
|
|
"model": {"display_name": "Opus 4.7"},
|
|
"context_window": {
|
|
"context_window_size": 1_000_000,
|
|
"current_usage": {
|
|
"input_tokens": 6,
|
|
"cache_creation_input_tokens": 100,
|
|
"cache_read_input_tokens": 200,
|
|
"output_tokens": 50,
|
|
},
|
|
"used_percentage": 31,
|
|
},
|
|
}
|
|
)
|
|
|
|
rc = _run(stdin_payload=stdin, bridge_dir=tmp_path, monkeypatch=monkeypatch)
|
|
assert rc == 0
|
|
|
|
persisted = json.loads((tmp_path / "context.json").read_text(encoding="utf-8"))
|
|
assert persisted["context_window_size"] == 1_000_000
|
|
assert persisted["current_usage"]["input_tokens"] == 6
|
|
assert persisted["current_usage"]["cache_creation_input_tokens"] == 100
|
|
assert persisted["used_percentage"] == 31
|
|
|
|
|
|
def test_status_wrapper_captures_cost(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""
|
|
The wrapper persists Claude Code's cumulative ``cost.total_cost_usd``.
|
|
|
|
Claude Code's statusLine stdin carries a top-level ``cost`` block with its
|
|
own session billing. claude-native never produces a ``response.completed``
|
|
event, so the Omnigent relay's cost accumulation never runs for it — capturing
|
|
this is the only way native session cost reaches ``session_usage``. A
|
|
failure here means native Cost-Ask policies always see $0.
|
|
"""
|
|
stdin = json.dumps(
|
|
{
|
|
"session_id": "abc",
|
|
"context_window": {"context_window_size": 1_000_000},
|
|
"cost": {"total_cost_usd": 0.42, "total_duration_ms": 1234},
|
|
}
|
|
)
|
|
|
|
rc = _run(stdin_payload=stdin, bridge_dir=tmp_path, monkeypatch=monkeypatch)
|
|
assert rc == 0
|
|
|
|
persisted = json.loads((tmp_path / "context.json").read_text(encoding="utf-8"))
|
|
assert persisted["total_cost_usd"] == 0.42
|
|
|
|
|
|
def test_status_wrapper_drops_payload_without_context_window(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""
|
|
Payloads with no ``context_window`` block leave no file behind.
|
|
|
|
Older Claude Code versions (pre v2.x) didn't include ``context_window``
|
|
on statusLine stdin. The wrapper must degrade gracefully so the
|
|
ring keeps the spec default rather than rendering a half-init state.
|
|
"""
|
|
stdin = json.dumps({"session_id": "abc", "model": {"display_name": "Opus 4.7"}})
|
|
|
|
rc = _run(stdin_payload=stdin, bridge_dir=tmp_path, monkeypatch=monkeypatch)
|
|
assert rc == 0
|
|
assert not (tmp_path / "context.json").exists()
|
|
|
|
|
|
def test_status_wrapper_chains_to_user_command(
|
|
tmp_path: Path,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
capsys: pytest.CaptureFixture[str],
|
|
) -> None:
|
|
"""
|
|
The chained command receives the original stdin and renders its stdout.
|
|
|
|
Claude Code only invokes a single statusLine command. Without
|
|
chaining, overriding for context capture would silently hide
|
|
claude-hud / any user-installed status bar.
|
|
"""
|
|
captured_calls: list[dict[str, object]] = []
|
|
|
|
class _FakeProc:
|
|
returncode = 0
|
|
stdout = "claude-hud line\n"
|
|
stderr = ""
|
|
|
|
def fake_run(*args: object, **kwargs: object) -> _FakeProc:
|
|
captured_calls.append({"args": args, "kwargs": kwargs})
|
|
return _FakeProc()
|
|
|
|
monkeypatch.setattr(subprocess, "run", fake_run)
|
|
|
|
stdin = json.dumps({"context_window": {"context_window_size": 200_000}})
|
|
rc = _run(
|
|
stdin_payload=stdin,
|
|
bridge_dir=tmp_path,
|
|
chain="echo claude-hud",
|
|
monkeypatch=monkeypatch,
|
|
)
|
|
assert rc == 0
|
|
assert len(captured_calls) == 1
|
|
assert captured_calls[0]["args"] == ("echo claude-hud",)
|
|
assert captured_calls[0]["kwargs"]["input"] == stdin
|
|
assert captured_calls[0]["kwargs"]["shell"] is True
|
|
out, _err = capsys.readouterr()
|
|
assert "claude-hud line" in out
|
|
|
|
|
|
def test_status_wrapper_chain_swallows_subprocess_errors(
|
|
tmp_path: Path,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
capsys: pytest.CaptureFixture[str],
|
|
) -> None:
|
|
"""
|
|
Chained-command failures don't propagate — context capture stays best-effort.
|
|
|
|
The statusLine command runs on every Claude Code render tick. A
|
|
crash there would yank the user's terminal status bar away on
|
|
every tick; the wrapper logs to stderr and returns 0 instead.
|
|
"""
|
|
|
|
def fake_run(*args: object, **kwargs: object) -> None:
|
|
raise OSError("chain broken")
|
|
|
|
monkeypatch.setattr(subprocess, "run", fake_run)
|
|
stdin = json.dumps({"context_window": {"context_window_size": 200_000}})
|
|
rc = _run(
|
|
stdin_payload=stdin,
|
|
bridge_dir=tmp_path,
|
|
chain="bogus",
|
|
monkeypatch=monkeypatch,
|
|
)
|
|
assert rc == 0
|
|
_out, err = capsys.readouterr()
|
|
assert "chain failed" in err
|
|
|
|
|
|
def test_normalize_status_payload_extracts_record() -> None:
|
|
"""The normalizer extracts window/usage/cost/model; None without a window."""
|
|
from omnigent.claude_native_status import normalize_status_payload
|
|
|
|
record = normalize_status_payload(
|
|
{
|
|
"context_window": {
|
|
"context_window_size": 200_000,
|
|
"current_usage": {"input_tokens": 5},
|
|
"used_percentage": 2.5,
|
|
},
|
|
"cost": {"total_cost_usd": 0.42},
|
|
"model": {"id": "claude-opus-4-8", "display_name": "Opus"},
|
|
}
|
|
)
|
|
assert record == {
|
|
"context_window_size": 200_000,
|
|
"current_usage": {"input_tokens": 5},
|
|
"used_percentage": 2.5,
|
|
"total_cost_usd": 0.42,
|
|
"model": "claude-opus-4-8",
|
|
}
|
|
assert normalize_status_payload({"model": "claude-opus-4-8"}) is None
|
|
|
|
|
|
def test_sync_raw_status_context_normalizes_and_retries(tmp_path: Path) -> None:
|
|
"""The forwarder-side sync normalizes raw captures and tolerates junk.
|
|
|
|
Unchanged signatures are no-ops, a malformed raw file leaves the
|
|
signature untouched so the next poll retries, and a rewritten raw
|
|
file re-normalizes.
|
|
"""
|
|
import json as _json
|
|
|
|
from omnigent.claude_native_status import (
|
|
CONTEXT_RAW_FILE,
|
|
sync_raw_status_context,
|
|
)
|
|
|
|
raw_path = tmp_path / CONTEXT_RAW_FILE
|
|
payload = {
|
|
"context_window": {"context_window_size": 1000},
|
|
"model": {"id": "claude-opus-4-8"},
|
|
}
|
|
raw_path.write_text(_json.dumps(payload, indent=2), encoding="utf-8")
|
|
|
|
sig = sync_raw_status_context(tmp_path, None)
|
|
assert sig is not None
|
|
written = _json.loads((tmp_path / "context.json").read_text("utf-8"))
|
|
assert written == {"context_window_size": 1000, "model": "claude-opus-4-8"}
|
|
|
|
# Unchanged raw file: same signature back, nothing rewritten.
|
|
assert sync_raw_status_context(tmp_path, sig) == sig
|
|
|
|
# Malformed raw file: signature unchanged so the next poll retries.
|
|
raw_path.write_text("{not json", encoding="utf-8")
|
|
assert sync_raw_status_context(tmp_path, sig) == sig
|
|
|
|
# Rewritten raw file: re-normalized under a fresh signature.
|
|
payload["model"] = {"id": "claude-sonnet-4-6"}
|
|
raw_path.write_text(_json.dumps(payload), encoding="utf-8")
|
|
new_sig = sync_raw_status_context(tmp_path, sig)
|
|
assert new_sig != sig
|
|
written = _json.loads((tmp_path / "context.json").read_text("utf-8"))
|
|
assert written["model"] == "claude-sonnet-4-6"
|