bc3dc80200
* perf(claude-native): stop taxing every hook spawn with the eager package init Claude Code blocks its TUI on command hooks — once per streamed text chunk (MessageDisplay), per statusline refresh, and per tool call — and every 'python -m omnigent.<hook>' subprocess re-ran omnigent/__init__, which eagerly imported the datamodel/executor/model-catalog graph. The deliberately stdlib-only hot-path hooks paid ~250 ms per spawn for imports they never use, capping visible streaming at ~4 chunks/s. The package init now re-exports lazily (PEP 562): the FIPS md5 patch and legacy-env mirror stay eager, every public name resolves on first attribute access (optional executors keep their import-failure->None contract), and submodule attribute access still works. Hot-path hook spawns drop to ~30 ms (~interpreter cost). A native_hook_spawn benchmark journey spawns the MessageDisplay hook exactly as Claude Code does and rides the release/nightly regression comparison; fresh-interpreter import-graph guards in the display-hook test suite pin what each hook entrypoint may import so the regression cannot silently return. Signed-off-by: dbczumar <corey.zumar@databricks.com> * perf(claude-native): keep the hook's hot path off the bridge's heavy imports The observer hook — Claude blocks on it at every prompt submit, tool call, Stop, and task event — imported claude_native_bridge, whose module-level tools/spec/pydantic imports cost ~450 ms of interpreter startup, plus httpx and the policy machinery besides. Enter and every tool call paid roughly a second of subprocess overhead per event even after the package init went lazy. The bridge now defers its tools graph to the one launch-path function that builds MCP tools (_build_tools) and its bundle-skills parse to the launch args builder; the hook imports httpx and the policy machinery inside the subcommands that actually speak HTTP. Module import cost: bridge 450 -> ~70 ms, hook 360 -> ~70 ms, and the hook's fresh-interpreter import graph now contains no third-party modules at all — the import guard pins the allowance at exactly that. Tests that reached httpx or create_os_environment through the hook's or bridge's module attributes now patch the owning modules directly. Signed-off-by: dbczumar <corey.zumar@databricks.com> * perf(claude-native): cache the ungoverned policy verdict at the relay Sessions with no policies at all still paid a full server round trip (~0.5-1.3s measured against a Databricks App) on every policy hook event — twice per tool call plus every prompt submit — with the server answering the same fast-path ALLOW each time. Typing during agentic turns stuttered in the gaps; vanilla Claude pays nothing there. The evaluate endpoint now stamps 'governed': false on its existing no-policies fast path (any_policies_apply's False is session-scoped — its only phase-scoped rule forces True), and the native-harness loopback relay caches that verdict for 30s, answering hook events instantly. A governed response of any kind drops the cache, a sys_add_policy call through the relay's own /tool path clears it before the policy lands, and expiry re-validates upstream — so enforcement for governed sessions is untouched and the attach delay for out-of-band policy edits is bounded at the TTL. Signed-off-by: dbczumar <corey.zumar@databricks.com> * perf(claude-native): keep blocking Claude hooks off Python and off the WAN Claude blocks its TUI on every command hook, and three of them still spawned a Python interpreter per event (~30ms floor, ~77ms under EDR): MessageDisplay once per streamed chunk, statusLine per refresh, and evaluate-policy twice per tool call — the last one also paying a 0.5-1.3s WAN round trip whenever its 30s ungoverned-cache window lapsed. - MessageDisplay: a /bin/sh one-liner appends the payload (newline- stripped, so any valid JSON lands single-line) straight to message_deltas.jsonl; the deltas reader already parses by key and skips malformed lines. - statusLine: the shim captures raw stdin to context_raw.json (atomic rename) and chains the user's own status command; the forwarder normalizes it into context.json on its poll loop (sync_raw_status_context), so the Python normalizer leaves the blocking path. The module entrypoint stays for older bridge dirs. - evaluate-policy: hooks try a curl against the relay's new /hook/claude/evaluate-policy endpoint (advertised via a shell-sourceable tool_relay.env); the long-lived runner process owns payload→EvaluationRequest mapping, retries, the ungoverned cache, and verdict→hook-output shaping. When the relay is absent or unreachable the same stdin replays into the Python hook, which keeps the direct-server path and the phase-aware fail-closed contract — exactly the pre-curl behavior. - The relay starts at session create (runner app) instead of at the first web-dispatched turn, so prompts typed directly in the TUI get the curl fast path too; it comes up in the background, and hooks that beat it use the Python fallback. Typing during a live 25-tool-call turn against a Databricks App measured 56.0ms median / 57.2ms p90 / 0 samples over 200ms, from 118ms median / 264ms p90 / 8 freezes before this branch. Also pins the relay-close ownership test's trusted-parent monkeypatch to tempfile.gettempdir() — the literal /tmp never contains the macOS fixture root, so the test only passed on Linux. Signed-off-by: dbczumar <corey.zumar@databricks.com> * perf(onboarding): cache harness CLI version and login probes Every readiness refresh on every host daemon execs vendor CLIs (--version / auth status) whose answers change only when the binary is swapped or a login flips; with a few dozen idle hosts that compounds into a constant machine-wide subprocess storm (~116 spawns/min observed) that competes with interactive terminals. --version output is a pure function of the binary bytes, so successful parses cache permanently against the binary's (path, mtime_ns, size) signature; failures keep re-probing. Login verdicts can flip without a binary change, so only positives cache, with a 120s TTL — negatives always re-probe so the setup wizard sees a fresh login immediately, and harness_logout invalidates its key so a successful logout is confirmed live. Signed-off-by: dbczumar <corey.zumar@databricks.com> * revert(claude-native): drop the ungoverned-verdict relay cache The cache required stamping 'governed': false on the evaluate response so the relay could tell which ALLOWs were safe to reuse — new response-field surface carried only by this optimization, which we don't need right now. Remove the stamp and the relay cache wholesale: every policy hook event consults the server again, the relay's /policies/evaluate proxy is a plain pass-through, and the evaluate response is byte-identical to its pre-branch shape. The sh-shim/curl hook path (no interpreter spawns) is unchanged. Signed-off-by: dbczumar <corey.zumar@databricks.com> --------- Signed-off-by: dbczumar <corey.zumar@databricks.com>
329 lines
12 KiB
Python
329 lines
12 KiB
Python
"""Tests for the fast ``MessageDisplay`` deltas-appender hook."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import io
|
|
import json
|
|
import os
|
|
import subprocess
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from omnigent import claude_native_message_display_hook as hook
|
|
from omnigent.claude_native_bridge import read_message_deltas_from_offset
|
|
|
|
|
|
def _run_hook(
|
|
payload: dict[str, object], bridge_dir: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> int:
|
|
"""
|
|
Drive the hook ``main`` with one payload on stdin.
|
|
|
|
:param payload: Hook JSON object to feed on stdin, e.g.
|
|
``{"hook_event_name": "MessageDisplay", "message_id": "m1",
|
|
"index": 0, "final": False, "delta": "hi"}``.
|
|
:param bridge_dir: Bridge directory the hook should append to.
|
|
:param monkeypatch: Pytest monkeypatch fixture used to set stdin.
|
|
:returns: The hook's process exit code.
|
|
"""
|
|
monkeypatch.setattr(sys, "stdin", io.StringIO(json.dumps(payload)))
|
|
return hook.main(["--bridge-dir", str(bridge_dir)])
|
|
|
|
|
|
def test_message_display_hook_appends_well_formed_record(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""
|
|
A valid ``MessageDisplay`` payload is appended verbatim, in order.
|
|
|
|
Fails if the hook drops fields, reorders, or writes a shape the
|
|
bridge reader can't parse — which would break live streaming end
|
|
to end (the forwarder would forward nothing).
|
|
"""
|
|
bridge_dir = tmp_path / "bridge"
|
|
bridge_dir.mkdir()
|
|
|
|
assert (
|
|
_run_hook(
|
|
{
|
|
"hook_event_name": "MessageDisplay",
|
|
"message_id": "m1",
|
|
"index": 0,
|
|
"final": False,
|
|
"delta": "Hello ",
|
|
},
|
|
bridge_dir,
|
|
monkeypatch,
|
|
)
|
|
== 0
|
|
)
|
|
assert (
|
|
_run_hook(
|
|
{
|
|
"hook_event_name": "MessageDisplay",
|
|
"message_id": "m1",
|
|
"index": 1,
|
|
"final": True,
|
|
"delta": "world",
|
|
},
|
|
bridge_dir,
|
|
monkeypatch,
|
|
)
|
|
== 0
|
|
)
|
|
|
|
result = read_message_deltas_from_offset(bridge_dir, 0)
|
|
# Both chunks land, in order, with every field preserved — proving
|
|
# the on-disk shape round-trips through the reader the forwarder uses.
|
|
assert [(d.message_id, d.index, d.final, d.delta) for d in result.deltas] == [
|
|
("m1", 0, False, "Hello "),
|
|
("m1", 1, True, "world"),
|
|
]
|
|
|
|
|
|
def test_message_display_hook_writes_owner_only_file(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""
|
|
The deltas file is created with owner-only (0600) permissions.
|
|
|
|
Fails if the streamed assistant text (same content as the message)
|
|
becomes world-readable on a shared host.
|
|
"""
|
|
bridge_dir = tmp_path / "bridge"
|
|
bridge_dir.mkdir()
|
|
_run_hook(
|
|
{"hook_event_name": "MessageDisplay", "message_id": "m1", "index": 0, "delta": "x"},
|
|
bridge_dir,
|
|
monkeypatch,
|
|
)
|
|
mode = (bridge_dir / hook.MESSAGE_DELTAS_FILE).stat().st_mode
|
|
assert oct(mode & 0o777) == "0o600"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"payload",
|
|
[
|
|
{"hook_event_name": "MessageDisplay", "delta": "no id"},
|
|
{"hook_event_name": "MessageDisplay", "message_id": "", "delta": "empty id"},
|
|
{"hook_event_name": "MessageDisplay", "message_id": "m1"},
|
|
{"hook_event_name": "MessageDisplay", "message_id": "m1", "delta": 123},
|
|
{"hook_event_name": "MessageDisplay", "message_id": "m1", "delta": None},
|
|
],
|
|
ids=["missing-id", "empty-id", "missing-delta", "non-string-delta", "null-delta"],
|
|
)
|
|
def test_message_display_hook_skips_unforwardable_payloads(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, payload: dict[str, object]
|
|
) -> None:
|
|
"""
|
|
Payloads lacking a usable ``message_id``/``delta`` write nothing.
|
|
|
|
Fails if the hook appends a record the forwarder couldn't turn into
|
|
a valid delta event (e.g. an empty message id or a non-string delta),
|
|
which would surface as a malformed SSE event downstream.
|
|
"""
|
|
bridge_dir = tmp_path / "bridge"
|
|
bridge_dir.mkdir()
|
|
assert _run_hook(payload, bridge_dir, monkeypatch) == 0
|
|
# No file at all — there was nothing forwardable to record.
|
|
assert not (bridge_dir / hook.MESSAGE_DELTAS_FILE).exists()
|
|
|
|
|
|
def test_message_display_hook_defaults_missing_index_to_zero(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""
|
|
A single-chunk message with no ``index`` still forwards (index 0).
|
|
|
|
Fails if a missing index is dropped or coerced to something the
|
|
reader rejects, which would lose the one-and-only chunk of a short
|
|
assistant message.
|
|
"""
|
|
bridge_dir = tmp_path / "bridge"
|
|
bridge_dir.mkdir()
|
|
_run_hook(
|
|
{"hook_event_name": "MessageDisplay", "message_id": "m1", "final": True, "delta": "hi"},
|
|
bridge_dir,
|
|
monkeypatch,
|
|
)
|
|
result = read_message_deltas_from_offset(bridge_dir, 0)
|
|
assert [(d.message_id, d.index, d.final, d.delta) for d in result.deltas] == [
|
|
("m1", 0, True, "hi")
|
|
]
|
|
|
|
|
|
def test_message_display_hook_swallows_malformed_json(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
|
|
) -> None:
|
|
"""
|
|
Malformed stdin exits 0 (never blocks Claude) and writes nothing.
|
|
|
|
Claude blocks on command hooks, so a parse error must be a silent
|
|
no-op for the TUI; fails if the hook raises or appends garbage.
|
|
"""
|
|
bridge_dir = tmp_path / "bridge"
|
|
bridge_dir.mkdir()
|
|
monkeypatch.setattr(sys, "stdin", io.StringIO("{not json"))
|
|
assert hook.main(["--bridge-dir", str(bridge_dir)]) == 0
|
|
assert not (bridge_dir / hook.MESSAGE_DELTAS_FILE).exists()
|
|
# Diagnostic goes to stderr so it never lands in Claude's stdout
|
|
# (which Claude would interpret as hook output).
|
|
assert "malformed JSON" in capsys.readouterr().err
|
|
|
|
|
|
def test_message_display_hook_many_appends_stay_line_clean(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""
|
|
Many successive appends to the shared file stay newline-framed.
|
|
|
|
Each ``MessageDisplay`` chunk is a separate hook invocation appending
|
|
one line; this asserts every line is independently parseable and the
|
|
reader's reported offset reaches EOF after consuming them all. Fails
|
|
if the hook ever wrote a record without a trailing newline (which
|
|
would make the next record's bytes glom onto it and break the
|
|
reader's per-line decode). NOTE: this exercises sequential appends,
|
|
not true concurrent subprocesses — the O_APPEND atomicity that makes
|
|
real per-chunk parallelism safe is a POSIX guarantee, not asserted
|
|
here.
|
|
"""
|
|
bridge_dir = tmp_path / "bridge"
|
|
bridge_dir.mkdir()
|
|
for i in range(50):
|
|
_run_hook(
|
|
{
|
|
"hook_event_name": "MessageDisplay",
|
|
"message_id": "m1",
|
|
"index": i,
|
|
"delta": f"c{i} ",
|
|
},
|
|
bridge_dir,
|
|
monkeypatch,
|
|
)
|
|
result = read_message_deltas_from_offset(bridge_dir, 0)
|
|
# All 50 chunks parse and arrive in index order — no line was torn.
|
|
assert [d.index for d in result.deltas] == list(range(50))
|
|
assert result.byte_offset == os.path.getsize(bridge_dir / hook.MESSAGE_DELTAS_FILE)
|
|
|
|
|
|
# ── import-cost regression guards ────────────────────────────
|
|
#
|
|
# Claude Code blocks its TUI on command hooks, so every module on a hook's
|
|
# import path is paid per streamed chunk / statusline tick / tool call.
|
|
# ``omnigent/__init__`` re-exports lazily (PEP 562) precisely to keep these
|
|
# subprocesses cheap. The guards below pin the import graph in a fresh
|
|
# interpreter — the deterministic form of the latency claim; the wall-clock
|
|
# form is the ``native_hook_spawn`` benchmark journey (dev/benchmarks).
|
|
|
|
_HEAVY_IMPORTS = (
|
|
"fastapi",
|
|
"httpx",
|
|
"omnigent.inner.databricks_executor",
|
|
"omnigent.inner.datamodel",
|
|
"omnigent.model_catalog",
|
|
"omnigent.spec.parser",
|
|
"pydantic",
|
|
)
|
|
|
|
|
|
def _heavy_imports_after(statements: str) -> list[str]:
|
|
"""
|
|
Run *statements* in a fresh interpreter and report loaded heavy modules.
|
|
|
|
:param statements: Newline-joined Python statements to execute, e.g.
|
|
``"import omnigent"``.
|
|
:returns: The subset of :data:`_HEAVY_IMPORTS` present in the child's
|
|
``sys.modules`` after *statements* ran.
|
|
"""
|
|
repo_root = Path(hook.__file__).resolve().parent.parent
|
|
code = "\n".join(
|
|
(
|
|
"import json, sys",
|
|
f"sys.path.insert(0, {str(repo_root)!r})",
|
|
statements,
|
|
f"heavy = [m for m in {list(_HEAVY_IMPORTS)!r} if m in sys.modules]",
|
|
"print(json.dumps(heavy))",
|
|
)
|
|
)
|
|
proc = subprocess.run(
|
|
[sys.executable, "-c", code],
|
|
capture_output=True,
|
|
text=True,
|
|
timeout=120,
|
|
check=False,
|
|
)
|
|
assert proc.returncode == 0, proc.stderr
|
|
return json.loads(proc.stdout.strip().splitlines()[-1])
|
|
|
|
|
|
def test_package_init_defers_the_heavy_import_graph() -> None:
|
|
"""
|
|
``import omnigent`` alone loads none of the heavy graph.
|
|
|
|
The package init used to eagerly import the datamodel/executor graph,
|
|
taxing every hook subprocess ~250 ms before its first line ran. The
|
|
same child also proves the init's one eager side effect (the FIPS md5
|
|
patch) still applies.
|
|
"""
|
|
statements = "\n".join(
|
|
(
|
|
"import omnigent",
|
|
"import hashlib",
|
|
"assert hashlib.md5.__name__ == '_fips_safe_md5', hashlib.md5.__name__",
|
|
)
|
|
)
|
|
assert _heavy_imports_after(statements) == []
|
|
|
|
|
|
def test_package_lazy_exports_resolve_on_access() -> None:
|
|
"""
|
|
The lazy re-exports keep the package's public import contract.
|
|
|
|
Plain and optional exports, ``from omnigent import`` forms, bare
|
|
submodule attribute access, and ``dir()`` all resolve exactly as the
|
|
eager init did — laziness must never be observable beyond timing.
|
|
"""
|
|
statements = "\n".join(
|
|
(
|
|
"import omnigent",
|
|
"from omnigent import AgentDef, Executor, load_agent_def",
|
|
"assert omnigent.TurnComplete is not None",
|
|
"assert 'omnigent.inner.executor' in sys.modules",
|
|
"_ = omnigent.DatabricksExecutor # optional: a class or None, never a raise",
|
|
"assert omnigent.inner is not None",
|
|
"assert 'TurnComplete' in dir(omnigent)",
|
|
)
|
|
)
|
|
_heavy_imports_after(statements) # the child's asserts are the test
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("module", "allowed"),
|
|
[
|
|
("omnigent.claude_native_message_display_hook", frozenset()),
|
|
("omnigent.claude_native_status", frozenset()),
|
|
(
|
|
"omnigent.claude_native_hook",
|
|
# The observer path (the most frequent invocation) is pure
|
|
# stdlib + light bridge state: httpx and the policy machinery
|
|
# are imported inside the subcommands that speak HTTP, and the
|
|
# bridge defers its tools/spec/pydantic graph to the launch
|
|
# path. Nothing heavy may ride the module import.
|
|
frozenset(),
|
|
),
|
|
],
|
|
)
|
|
def test_hook_entrypoints_stay_import_light(module: str, allowed: frozenset[str]) -> None:
|
|
"""
|
|
A hook entrypoint's fresh-interpreter import graph stays bounded.
|
|
|
|
Claude Code spawns these per streamed chunk / statusline tick / tool
|
|
call and blocks on them, so a heavy dependency creeping onto any of
|
|
these import paths is a direct TUI-latency regression even while
|
|
every functional test still passes.
|
|
"""
|
|
loaded = _heavy_imports_after(f"import {module}")
|
|
assert set(loaded) <= allowed, f"{module} newly imports {sorted(set(loaded) - allowed)}"
|