feat(cli): add omni bench to run the harness capability bench

Register a `bench` subcommand on the unified CLI that delegates to the bench's argparse main (`omnigent.harness_bench.__main__.main`) via an ignore-unknown-options pass-through, so `omni bench` and `python -m omnigent.harness_bench` share one arg surface (`omni bench --help` shows the bench's own options). The import is lazy inside the command body, so it adds no CLI startup cost and no import cycle. Flip the bench's live/offline gate to match `omni run`: a live run no longer requires --profile. It turns on whenever creds are resolvable — a --profile, a configured ~/.omnigent profile, or ambient OPENAI_* (via bench_creds_skip_reason from Stage 2) — and --no-live forces the offline declared matrix. Updated the --profile / --live help text accordingly. Add "bench" to _CLICK_SUBCOMMANDS (the ad-hoc-detection sync set) and CLI tests covering registration, verbatim arg delegation, and exit-code propagation. Docs (README + design doc) lead with `omni bench`. 194 CLI tests pass; bench suite green; ruff clean; no lockfile drift. Co-authored-by: Isaac
feat(harness-bench): derive env like omni run; --profile now optional
2026-07-09 18:06:35 +08:00 · 2026-07-09 18:06:12 +08:00 · 2026-07-09 18:05:21 +08:00 · 2026-07-09 17:54:16 +08:00 · 2026-07-09 17:19:30 +08:00
38 changed files with 905 additions and 459 deletions
@@ -1,6 +1,6 @@
 # Seam: harness capabilities → harness bench

-**Audience:** whoever wires the harness bench (`tests/harness_bench/`, the
+**Audience:** whoever wires the harness bench (`omnigent/harness_bench/`, the
 `#1787 → #1790 → #1792` stack) to consume the declarative capability model.
 **Status:** capability model is PR #1847 (open, base `main`). This note is the
 contract for the follow-up that makes the bench derive from it. No bench code
@@ -11,7 +11,7 @@ has been changed yet.
 ## The one-sentence idea

 The bench today hand-maintains a "declared support matrix" in
-`tests/harness_bench/manifest.py` (`_P0_ALL_SUPPORTED` verdicts + `_STATIC`
+`omnigent/harness_bench/manifest.py` (`_P0_ALL_SUPPORTED` verdicts + `_STATIC`
 columns). That is a *second copy* of "what each harness supports". PR #1847 adds
 the *first, canonical* copy — `harness_capabilities()`. **Make the manifest
 derive from `harness_capabilities()` and delete the hand-typed dicts**, so there
@@ -128,7 +128,7 @@ list" to "discover"; probes, profiles, and reports are untouched.
 Three layers plus a report step.

 ```
-tests/harness_bench/
+omnigent/harness_bench/   # ships in the wheel; `omni bench` runs it
  profile.py         # BenchProfile: per-harness self-declared facts
  manifest.py        # registry of official BenchProfiles (the spreadsheet as data)
  verdict.py         # Verdict enum, ProbeResult, priority (P0/P1)
@@ -164,7 +164,7 @@ tests/harness_bench/
  server, exactly like the existing e2e tests
  (`/v1/sessions` + `send_user_message_to_session` +
  `poll_session_until_terminal` + `final_assistant_text`).
- **Report.** `python -m tests.harness_bench --harness codex` prints one
+- **Report.** `python -m omnigent.harness_bench --harness codex` prints one
  harness's matrix; no filter regenerates the whole sheet with a `DRIFT` column
  diffing declared vs observed.

@@ -331,18 +331,25 @@ The MVP and most of phase-2 are landed. What exists on `main` today:

 ## Running the bench and reading the result

+`omni bench` is the entry point (a thin wrapper over
+`python -m omnigent.harness_bench`; both share one arg surface).
+
 ```
 # Offline: the declared matrix, no creds, every harness. Fast.
-python -m tests.harness_bench
+omni bench --no-live

-# Live: probe one harness against a gateway profile.
-python -m tests.harness_bench --harness codex-native --profile oss
+# Live if creds resolve (a configured ~/.omnigent profile or ambient
+# OPENAI_*, like `omni run`), else the declared matrix.
+omni bench
+
+# Live: probe one harness against a specific gateway profile.
+omni bench --harness codex-native --profile oss

 # Live: probe every official harness (SDK + native) sequentially.
-python -m tests.harness_bench --profile oss
+omni bench --profile oss

 # A community harness that ships its own BenchProfile.
-python -m tests.harness_bench --harness mypkg.harness:PROFILE --profile oss
+omni bench --harness mypkg.harness:PROFILE --profile oss
 ```

 **You do not need to live-probe every harness on every host — and you cannot.**
@@ -409,7 +416,7 @@ the **default** for SDK harnesses — a plain live run proves Tool calling and
 Policy DENY out of the box:

 ```
-python -m tests.harness_bench --harness claude-sdk --profile oss
+omni bench --harness claude-sdk --profile oss
 ```

 Live-verified: `claude-sdk` completes the full matrix on `full-server` —
@@ -1178,6 +1178,7 @@ _CLICK_SUBCOMMANDS: frozenset[str] = frozenset(
    {
        "antigravity",
        "attach",
+        "bench",
        "claude",
        "codex",
        "config",
@@ -3422,6 +3423,31 @@ def server_status(json_output: bool) -> None:
    click.echo(f"  host daemon attached: {'yes' if daemon_attached else 'no'}")


+@cli.command(
+    "bench",
+    context_settings={"ignore_unknown_options": True, "help_option_names": []},
+    add_help_option=False,
+)
+@click.argument("bench_args", nargs=-1, type=click.UNPROCESSED)
+def bench(bench_args: tuple[str, ...]) -> None:
+    """Probe harness capabilities and report a per-dimension verdict matrix.
+
+    A thin wrapper over ``python -m omnigent.harness_bench``; all flags pass
+    straight through (``omni bench --help`` shows the bench's own options). With
+    no ``--profile`` the bench derives creds the way ``omni run`` does (a
+    configured ~/.omnigent profile or ambient OPENAI_*); ``--profile NAME``
+    overrides. Examples::
+
+        omni bench --list
+        omni bench                       # live if creds resolve, else declared
+        omni bench --harness codex --profile oss --rich
+        omni bench --no-live             # offline declared matrix
+    """
+    from omnigent.harness_bench.__main__ import main as _bench_main
+
+    raise SystemExit(_bench_main(list(bench_args)))
+
+
@cli.command("stop")
@click.option(
    "--force",
@@ -7,18 +7,26 @@ against a self-declared profile to surface drift. Design and rationale:

 ## Run it

+`omni bench` is the entry point (a thin wrapper over
+`python -m omnigent.harness_bench`; both share one arg surface, so every flag
+below works either way).
+
 ```bash
 # List official harnesses (name, resolved transport, model).
-python -m tests.harness_bench --list
+omni bench --list
+
+# Live if creds resolve (a configured ~/.omnigent profile or ambient
+# OPENAI_*, exactly like `omni run`), else the declared matrix.
+omni bench

 # Offline (declared) matrix -- no turns, no creds.
-python -m tests.harness_bench
+omni bench --no-live

-# Live probe one harness against a gateway profile.
-python -m tests.harness_bench --harness codex --profile my-profile
+# Live probe one harness against a specific gateway profile.
+omni bench --harness codex --profile my-profile

 # Live probe every official harness, several at a time, with a live table.
-python -m tests.harness_bench --profile my-profile --jobs 4 --rich
+omni bench --profile my-profile --jobs 4 --rich
 ```

 A non-zero exit means a `DRIFT` cell was found (observed behavior disagrees
@@ -26,8 +34,10 @@ with the declared matrix).

 ### Flags

- `--profile NAME` -- Databricks gateway profile. Enables the live layer;
-  without it the bench renders the declared matrix offline.
+- `--profile NAME` -- Databricks gateway profile override. Optional: without it
+  the bench derives creds the way `omni run` does (a configured `~/.omnigent`
+  profile, or ambient `OPENAI_*`). The live layer turns on whenever creds are
+  resolvable; use `--no-live` for the offline declared matrix.
 - `--harness NAME` -- probe one harness (repeatable). An official name, or a
  `module:attr` / `module.ATTR` reference to a community `BenchProfile`.
  Defaults to every official harness.
@@ -66,7 +76,7 @@ the file is self-contained.
 A live `--rich` run of the four SDK harnesses on the `oss` profile:

 ```console
-$ uv run --no-sync python -m tests.harness_bench --profile oss --rich \
+$ uv run --no-sync python -m omnigent.harness_bench --profile oss --rich \
    --harness claude-sdk --harness codex --harness pi --harness openai-agents

                              Harness capability matrix (live)
@@ -158,8 +168,9 @@ so they show `·` on `sdk-inproc` and `native-tui`:

 ## Add a harness

- **Official SDK:** add a `BenchProfile` to `manifest.py` (base fields come
-  from `_harness_probes.HARNESS_PROBES`). No probe or driver edits.
+- **Official SDK:** add an `SdkSeed` to `seed.py` (the single source of truth the
+  e2e matrix also rebuilds from); `manifest.py` turns it into a `BenchProfile`.
+  No probe or driver edits.
 - **Native:** nothing to add -- every harness the capability model marks
  `NATIVE_TUI` (in-repo or a community plugin) is auto-derived into the matrix
  and drivable by name; `native_vendor()` derives what the driver needs.
@@ -0,0 +1,39 @@
+"""Harness capability test bench.
+
+A standardized, pluggable conformance suite that probes a harness and
+reports a verdict per capability dimension (basic turn, streaming,
+tool calling, interrupt, policy DENY, model override, ...), reconciling
+observed behavior against a self-declared :class:`BenchProfile` to
+surface drift.
+
+Design: ``docs/harness-bench-design.md``.
+
+Two entry points:
+
+- ``python -m omnigent.harness_bench --harness <name>`` renders the matrix
+  for one harness (or all official harnesses with no ``--harness``).
+- ``tests/harness_bench/test_bench.py`` runs the offline conformance
+  layer on every PR and the live-probe layer when a ``--profile`` and
+  the harness CLI are available.
+"""
+
+from __future__ import annotations
+
+from omnigent.harness_bench.profile import BenchProfile, resolve_profile
+from omnigent.harness_bench.verdict import (
+    Applicability,
+    Priority,
+    ProbeResult,
+    Verdict,
+    reconcile,
+)
+
+__all__ = [
+    "Applicability",
+    "BenchProfile",
+    "Priority",
+    "ProbeResult",
+    "Verdict",
+    "reconcile",
+    "resolve_profile",
+]
@@ -3,24 +3,24 @@
 Examples::

    # List official harnesses.
-    python -m tests.harness_bench --list
+    python -m omnigent.harness_bench --list

    # Dry (offline) render — declared matrix, no turns, no creds.
-    python -m tests.harness_bench
+    python -m omnigent.harness_bench

    # Live probe one harness against a gateway profile (SDK → full-server,
    # the default: covers Tool calling + Policy DENY).
-    python -m tests.harness_bench --harness codex --profile my-profile
+    python -m omnigent.harness_bench --harness codex --profile my-profile

    # Quicker run: SDK harnesses on sdk-inproc (skips the server boot; no
    # Tool calling / Policy DENY coverage).
-    python -m tests.harness_bench --harness codex --profile my-profile --fast
+    python -m omnigent.harness_bench --harness codex --profile my-profile --fast

    # Live probe all official harnesses, JSON out.
-    python -m tests.harness_bench --profile my-profile --json
+    python -m omnigent.harness_bench --profile my-profile --json

    # A community harness that ships its own BenchProfile.
-    python -m tests.harness_bench --harness mypkg.harness:PROFILE --profile my-profile
+    python -m omnigent.harness_bench --harness mypkg.harness:PROFILE --profile my-profile
 """

 from __future__ import annotations
@@ -29,17 +29,17 @@ import argparse
 import asyncio
 import sys

-from tests.harness_bench.bench import run_bench
-from tests.harness_bench.events import LineSink
-from tests.harness_bench.manifest import OFFICIAL_PROFILES
-from tests.harness_bench.profile import BenchProfile, resolve_profile
-from tests.harness_bench.report import render_json, render_markdown, render_table
-from tests.harness_bench.transport import driver_registry, resolve_transport_name
+from omnigent.harness_bench.bench import run_bench
+from omnigent.harness_bench.events import LineSink
+from omnigent.harness_bench.manifest import OFFICIAL_PROFILES
+from omnigent.harness_bench.profile import BenchProfile, resolve_profile
+from omnigent.harness_bench.report import render_json, render_markdown, render_table
+from omnigent.harness_bench.transport import driver_registry, resolve_transport_name


 def _parse_args(argv: list[str]) -> argparse.Namespace:
    parser = argparse.ArgumentParser(
-        prog="python -m tests.harness_bench",
+        prog="python -m omnigent.harness_bench",
        description="Probe a harness and report a verdict per capability dimension.",
    )
    parser.add_argument(
@@ -54,15 +54,18 @@ def _parse_args(argv: list[str]) -> argparse.Namespace:
        "--profile",
        metavar="NAME",
        default=None,
-        help="Databricks gateway profile. Enables the live layer; without "
-        "it the bench renders the declared matrix offline.",
+        help="Databricks gateway profile override. Optional: without it the "
+        "bench derives creds the way `omni run` does (a configured "
+        "~/.omnigent profile or ambient OPENAI_*). The live layer turns on "
+        "whenever creds are resolvable; use --no-live for the offline matrix.",
    )
    parser.add_argument(
        "--live",
        dest="live",
        action="store_true",
        default=None,
-        help="Force the live layer (requires --profile).",
+        help="Force the live layer (needs resolvable creds: --profile, a "
+        "configured ~/.omnigent profile, or ambient OPENAI_*).",
    )
    parser.add_argument(
        "--no-live",
@@ -169,10 +172,15 @@ def main(argv: list[str] | None = None) -> int:
        print("--jobs must be >= 1", file=sys.stderr)
        return 2

-    # Live if explicitly forced, or implied by a supplied profile.
-    live = args.live if args.live is not None else bool(args.profile)
-    if live and not args.profile:
-        print("--live requires --profile <name>", file=sys.stderr)
+    # Live layer: derive creds the way `omni run` does — a --profile, a
+    # configured ~/.omnigent profile, or ambient OPENAI_*. So a live run no
+    # longer requires --profile; it is implied whenever creds are resolvable.
+    from omnigent.harness_bench.runtime_env import bench_creds_skip_reason
+
+    creds_skip = bench_creds_skip_reason(args.profile)
+    live = args.live if args.live is not None else creds_skip is None
+    if live and creds_skip is not None:
+        print(f"--live needs resolvable gateway creds: {creds_skip}", file=sys.stderr)
        return 2

    # Progress sink: only for a live run (offline is instant). Prefer the rich
@@ -245,7 +253,7 @@ def _select_progress_sink(rich_flag: bool | None):
    if rich_flag is not False:
        # richreport is imported lazily: it is the only place that touches the
        # optional `rich` dependency, so a plain/no-rich run never imports it.
-        from tests.harness_bench.richreport import rich_sink_or_none
+        from omnigent.harness_bench.richreport import rich_sink_or_none

        rich_sink = rich_sink_or_none(force=bool(rich_flag))
        if rich_sink is not None:
@@ -16,8 +16,8 @@ import logging
 from collections.abc import Callable
 from dataclasses import dataclass, field

-from tests.harness_bench.driver import ProvisioningError
-from tests.harness_bench.events import (
+from omnigent.harness_bench.driver import ProvisioningError
+from omnigent.harness_bench.events import (
    HarnessFinished,
    HarnessSkipped,
    HarnessStarted,
@@ -26,11 +26,12 @@ from tests.harness_bench.events import (
    ProbeStarted,
    ProgressSink,
 )
-from tests.harness_bench.full_server import SharedFullServer
-from tests.harness_bench.probes import ALL_PROBES, CapabilityProbe
-from tests.harness_bench.profile import BenchProfile
-from tests.harness_bench.transport import resolve_driver_class, resolve_transport_name
-from tests.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict, reconcile
+from omnigent.harness_bench.full_server import SharedFullServer
+from omnigent.harness_bench.probes import ALL_PROBES, CapabilityProbe
+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.runtime_env import bench_creds_skip_reason, resolve_bench_env
+from omnigent.harness_bench.transport import resolve_driver_class, resolve_transport_name
+from omnigent.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict, reconcile

 _logger = logging.getLogger(__name__)

@@ -158,7 +159,7 @@ def _as_sink(progress: Progress | ProgressSink | None) -> ProgressSink | None:
    """Normalize the ``progress`` argument to a :class:`ProgressSink`.

    Accepts a structured sink (used as-is), a plain line callback (adapted to a
-    :class:`~tests.harness_bench.events.LineSink`), or ``None`` (silent).
+    :class:`~omnigent.harness_bench.events.LineSink`), or ``None`` (silent).
    """
    if progress is None:
        return None
@@ -199,7 +200,7 @@ async def run_harness(
    :param progress: A :class:`ProgressSink` (structured events), a plain
        per-line callback (adapted), or ``None`` (silent).
    :param shared_full_server: An optional shared
-        :class:`~tests.harness_bench.full_server_driver.SharedFullServer` to
+        :class:`~omnigent.harness_bench.full_server_driver.SharedFullServer` to
        register this harness on, instead of the driver spawning its own
        server+runner. Only used when the resolved driver is the full-server
        driver; ignored otherwise.
@@ -229,7 +230,9 @@ async def run_harness(
            transport=resolved_transport,
        )

-    assert databricks_profile is not None  # guaranteed by the unavailable() check
+    # databricks_profile may be None here — the driver derives creds like
+    # `omni run` (config profile / ambient OPENAI_*); the unavailable() check
+    # above already confirmed creds are resolvable.
    _emit(sink, HarnessStarted(profile.harness, driver_cls.transport, profile.model))
    cells: list[CellResult] = []
    # Only the full-server driver accepts a shared server; pass it through when
@@ -394,14 +397,17 @@ async def _maybe_shared_full_server(
    run still owns its own server, unchanged).
    """
    shared = None
-    if live and jobs > 1 and databricks_profile is not None:
+    if live and jobs > 1:
        full = [
            p
            for p in profiles
            if resolve_driver_class(p, override=transport, fast=fast).transport == "full-server"
        ]
-        if len(full) > 1:
-            shared = SharedFullServer(databricks_profile)
+        # Only stand one up when creds are actually resolvable (a --profile, a
+        # configured ~/.omnigent profile, or ambient OPENAI_*); otherwise let
+        # each harness's unavailable() report the clean no-creds skip.
+        if len(full) > 1 and bench_creds_skip_reason(databricks_profile) is None:
+            shared = SharedFullServer(resolve_bench_env(databricks_profile))
            await asyncio.to_thread(shared.__enter__)
    try:
        yield shared
@@ -0,0 +1,59 @@
+"""Cheap "is this CLI runnable?" probe for harness skip-gating.
+
+A harness whose vendor CLI is missing or broken should SKIP cleanly rather than
+fail deep inside an executor. Both the bench drivers and the e2e parametrize
+gate (:func:`tests.e2e._harness_probes.skip_if_harness_cli_missing`) call
+:func:`cli_unavailable_reason` for that decision, so it lives here in the shipped
+package and the test tree re-imports it.
+"""
+
+from __future__ import annotations
+
+import shutil
+import subprocess
+from functools import cache
+
+
+def _cli_probe_args(binary: str) -> list[str]:
+    """Return a cheap command that proves *binary* is runnable."""
+    if binary == "pi":
+        # ``shutil.which("pi")`` alone is not enough: pi's npm package
+        # may be installed under an older Node version than the package
+        # supports. ``pi --help`` exercises module loading without making
+        # model/network calls, so it catches broken installs early and lets
+        # rows skip instead of failing deep inside ``PiExecutor``.
+        return [binary, "--help"]
+    return [binary, "--version"]
+
+
+@cache
+def cli_unavailable_reason(binary: str) -> str | None:
+    """
+    Return ``None`` when *binary* exists and starts, else a skip reason.
+
+    The result is cached because callers gate many rows off it and CLI
+    startup can be non-trivial.
+    """
+    path = shutil.which(binary)
+    if path is None:
+        return f"{binary!r} CLI is not on PATH"
+
+    try:
+        proc = subprocess.run(
+            _cli_probe_args(binary),
+            capture_output=True,
+            text=True,
+            timeout=15,
+            check=False,
+        )
+    except (OSError, subprocess.TimeoutExpired) as exc:
+        return f"{binary!r} CLI at {path!r} is not runnable: {exc}"
+
+    if proc.returncode != 0:
+        detail = (proc.stderr or proc.stdout).strip().splitlines()
+        suffix = f": {detail[0]}" if detail else ""
+        return f"{binary!r} CLI at {path!r} exits {proc.returncode}{suffix}"
+    return None
+
+
+__all__ = ["cli_unavailable_reason"]
@@ -0,0 +1,30 @@
+"""Databricks workspace-host lookup for bench transports.
+
+A thin ``~/.databrickscfg`` reader the full-server and native-tui drivers use to
+skip-gate on a hostless profile. Kept package-local so the bench ships without a
+test-tree dependency.
+"""
+
+from __future__ import annotations
+
+import configparser
+from pathlib import Path
+
+_DATABRICKSCFG_PATH = Path.home() / ".databrickscfg"
+
+
+def lookup_databricks_host(profile: str) -> str | None:
+    """Return the workspace ``host`` for *profile* from ``~/.databrickscfg``.
+
+    :param profile: The Databricks profile name to look up.
+    :returns: The workspace host with any trailing ``/`` stripped, or ``None``
+        when the profile is absent or the section has no ``host`` key.
+    """
+    cfg = configparser.ConfigParser()
+    if _DATABRICKSCFG_PATH.exists():
+        cfg.read(_DATABRICKSCFG_PATH)
+    host = cfg[profile].get("host") if profile in cfg else None
+    return host.rstrip("/") if host else None
+
+
+__all__ = ["lookup_databricks_host"]
@@ -25,9 +25,10 @@ from typing import Any

 import httpx

+from omnigent.harness_bench.cli_probe import cli_unavailable_reason
+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.runtime_env import bench_creds_skip_reason, resolve_bench_env
 from omnigent.runtime.harnesses.process_manager import HarnessProcessManager
-from tests.e2e._harness_probes import cli_unavailable_reason
-from tests.harness_bench.profile import BenchProfile


 class ProvisioningError(RuntimeError):
@@ -248,7 +249,7 @@ class SdkInprocDriver:

    transport = "sdk-inproc"

-    def __init__(self, profile: BenchProfile, *, databricks_profile: str) -> None:
+    def __init__(self, profile: BenchProfile, *, databricks_profile: str | None) -> None:
        self._profile = profile
        self._databricks_profile = databricks_profile
        self._pm: HarnessProcessManager | None = None
@@ -259,20 +260,22 @@ class SdkInprocDriver:
    def unavailable(profile: BenchProfile, *, databricks_profile: str | None) -> str | None:
        """Return a skip reason if this driver cannot run *profile*, else ``None``.

-        Checks, in order: the profile's transport matches this driver, a
-        supplied Databricks profile (no gateway route without one), and a
-        runnable harness CLI binary. Mirrors the e2e suite's gating so the
-        bench skips — rather than errors — in environments missing creds or
-        a vendor CLI, or when a profile declares a transport this driver
-        does not implement (e.g. a native/community harness).
+        Checks, in order: the profile's transport matches this driver,
+        resolvable gateway credentials (``--profile``, a configured
+        ``~/.omnigent`` profile, or ambient ``OPENAI_*`` — like ``omni run``),
+        and a runnable harness CLI binary. Mirrors the e2e suite's gating so the
+        bench skips — rather than errors — in environments missing creds or a
+        vendor CLI, or when a profile declares a transport this driver does not
+        implement (e.g. a native/community harness).
        """
        if profile.transport != SdkInprocDriver.transport:
            return (
                f"transport {profile.transport!r} not supported by the "
                f"{SdkInprocDriver.transport!r} driver"
            )
-        if not databricks_profile:
-            return "no --profile / databricks profile provided; live probes need a gateway route"
+        creds_skip = bench_creds_skip_reason(databricks_profile)
+        if creds_skip is not None:
+            return creds_skip
        if profile.cli_binary is not None:
            reason = cli_unavailable_reason(profile.cli_binary)
            if reason is not None:
@@ -285,15 +288,17 @@ class SdkInprocDriver:
        self._pm = HarnessProcessManager(tmp_parent=self._tmp_parent)
        await self._pm.start()
        p = self._profile
-        self._client = await self._pm.get_client(
-            _CONV_ID,
-            p.harness,
-            env={
-                f"{p.env_prefix}GATEWAY": "true",
-                f"{p.env_prefix}DATABRICKS_PROFILE": self._databricks_profile,
-                f"{p.env_prefix}MODEL": p.model,
-            },
-        )
+        # Resolve the effective profile the way `omni run` does (the --profile
+        # override, else the config-derived one). May be None when auth comes
+        # from ambient OPENAI_*, in which case the wrap inherits that env.
+        resolved = resolve_bench_env(self._databricks_profile)
+        wrap_env = {
+            f"{p.env_prefix}GATEWAY": "true",
+            f"{p.env_prefix}MODEL": p.model,
+        }
+        if resolved.db_profile:
+            wrap_env[f"{p.env_prefix}DATABRICKS_PROFILE"] = resolved.db_profile
+        self._client = await self._pm.get_client(_CONV_ID, p.harness, env=wrap_env)
        return self

    async def __aexit__(self, *exc: object) -> None:
@@ -7,7 +7,7 @@ render progress in more than one way without the orchestrator knowing how:

 - :class:`LineSink` prints the plain ``[harness] Probe: VERDICT`` lines to a
  writer (the default; what CI / a piped run wants).
- a rich live-table sink (see :mod:`tests.harness_bench.richreport`) draws one
+- a rich live-table sink (see :mod:`omnigent.harness_bench.richreport`) draws one
  row per harness with per-dimension cells that fill in as events arrive.

 Events carry structured fields (harness id, probe name/title, verdict, note),
@@ -20,7 +20,7 @@ from __future__ import annotations
 from dataclasses import dataclass
 from typing import Protocol, runtime_checkable

-from tests.harness_bench.verdict import Verdict
+from omnigent.harness_bench.verdict import Verdict


@dataclass(frozen=True)
@@ -1,16 +1,17 @@
 """Shared full-server infrastructure: spawn a real Omnigent server + runner.

-Split from :mod:`tests.harness_bench.full_server_driver` so the *server
-lifecycle* (spawning the server/runner, minting a bearer, registering bench
-agents + sessions) lives apart from the *driver* that runs probes against it.
-Two consumers use this:
+Split from :mod:`omnigent.harness_bench.full_server_driver` so the *server
+lifecycle* (spawning the server/runner, registering bench agents + sessions)
+lives apart from the *driver* that runs probes against it. Credentials come from
+:func:`omnigent.harness_bench.runtime_env.resolve_bench_env` (the same layering
+``omni run`` uses), not a bench-local bearer mint. Two consumers use this:

- :class:`~tests.harness_bench.full_server_driver.FullServerDriver` — one
+- :class:`~omnigent.harness_bench.full_server_driver.FullServerDriver` — one
  harness per :class:`SharedFullServer` (solo run), or several harnesses on one
  shared server (parallel run; see ``bench.run_bench``).
- :mod:`tests.harness_bench.native_tui_driver` reuses the lower-level spawn
-  helpers (:func:`spawn_omnigent_server`, :func:`_mint_bearer`,
-  :func:`_find_free_port`) for its own server + host-daemon topology.
+- :mod:`omnigent.harness_bench.native_tui_driver` reuses the lower-level spawn
+  helpers (:func:`spawn_omnigent_server`, :func:`_find_free_port`) for its own
+  server + host-daemon topology.
 """

 from __future__ import annotations
@@ -22,6 +23,7 @@ import shutil
 import signal
 import socket
 import subprocess
+import sys
 import tarfile
 import time
 import uuid
@@ -31,18 +33,15 @@ from typing import Any
 import httpx
 import yaml

+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.runtime_env import BenchRuntimeEnv
 from omnigent.runner.identity import OMNIGENT_INTERNAL_WS_ORIGIN, token_bound_runner_id
-from tests._helpers.compat import (
-    apply_runner_env,
-    apply_server_env,
-    compat_runner_cwd,
-    compat_server_cwd,
-    runner_executable,
-    server_executable,
-)
-from tests.e2e.helpers import lookup_databricks_host
-from tests.harness_bench.profile import BenchProfile

+# Repo root when running from a source checkout (``.../omnigent/harness_bench/
+# full_server.py`` -> parents[2]). Prepended to the spawned server's PYTHONPATH
+# so it imports the working copy. In an installed wheel this points at
+# site-packages, where omnigent already resolves, so the prepend is a harmless
+# no-op.
 _REPO_ROOT = str(Path(__file__).resolve().parents[2])
 _HEALTH_TIMEOUT_S = 90.0
 _POLL_INTERVAL_S = 0.2
@@ -60,27 +59,6 @@ def _find_free_port() -> int:
        return int(s.getsockname()[1])


-def _mint_bearer(profile: str) -> str:
-    """Mint a Databricks bearer for *profile* via the CLI (isolated from ambient token env).
-
-    ``env -u DATABRICKS_TOKEN -u DATABRICKS_BEARER`` guards against a stale
-    ambient credential shadowing profile auth (see omnigent issue #1781).
-    """
-    proc = subprocess.run(
-        ["databricks", "auth", "token", "--profile", profile, "--output", "json"],
-        capture_output=True,
-        text=True,
-        timeout=30,
-        check=True,
-        env={
-            k: v
-            for k, v in os.environ.items()
-            if k not in ("DATABRICKS_TOKEN", "DATABRICKS_BEARER")
-        },
-    )
-    return str(json.loads(proc.stdout)["access_token"])
-
-
 def spawn_omnigent_server(
    tmp: Path, port: int, base_env: dict[str, str], binding_token: str
 ) -> subprocess.Popen[bytes]:
@@ -95,7 +73,7 @@ def spawn_omnigent_server(
    artifact_dir.mkdir(exist_ok=True)
    log = tmp / "server.log"
    args = [
-        server_executable(),
+        sys.executable,
        "-m",
        "omnigent.cli",
        "server",
@@ -109,7 +87,6 @@ def spawn_omnigent_server(
    return subprocess.Popen(
        args,
        env={**base_env, "OMNIGENT_RUNNER_TUNNEL_TOKEN": binding_token},
-        cwd=compat_server_cwd(),
        stdout=log.open("wb"),
        stderr=subprocess.STDOUT,
    )
@@ -120,19 +97,16 @@ def _spawn_bench_runner(
 ) -> subprocess.Popen[bytes]:
    """Spawn a bench runner bound to *base_url* (the full-server execution sandbox)."""
    log = tmp / "runner.log"
-    runner_env = apply_runner_env(
-        {
-            **base_env,
-            "OMNIGENT_RUNNER_ID": runner_id,
-            "OMNIGENT_RUNNER_TUNNEL_BINDING_TOKEN": binding_token,
-            "OMNIGENT_RUNNER_PARENT_PID": str(os.getpid()),
-            "RUNNER_SERVER_URL": base_url,
-        }
-    )
+    runner_env = {
+        **base_env,
+        "OMNIGENT_RUNNER_ID": runner_id,
+        "OMNIGENT_RUNNER_TUNNEL_BINDING_TOKEN": binding_token,
+        "OMNIGENT_RUNNER_PARENT_PID": str(os.getpid()),
+        "RUNNER_SERVER_URL": base_url,
+    }
    return subprocess.Popen(
-        [runner_executable(), "-m", "omnigent.runner._entry"],
+        [sys.executable, "-m", "omnigent.runner._entry"],
        env=runner_env,
-        cwd=compat_runner_cwd(),
        stdout=log.open("wb"),
        stderr=subprocess.STDOUT,
    )
@@ -162,7 +136,7 @@ def _wait_server_runner_ready(base_url: str, runner_id: str) -> None:


 def _build_bench_agent_config(
-    profile: BenchProfile, db_profile: str, *, deny: bool
+    profile: BenchProfile, db_profile: str | None, *, deny: bool
 ) -> dict[str, Any]:
    """The agent spec for a bench harness: the harness + the read-only builtin,
    plus (when *deny*) a baked tool_call-phase deny on that builtin."""
@@ -171,16 +145,21 @@ def _build_bench_agent_config(
    # the real id so the runner resolves the right ACP agent at spawn.
    safe_harness = profile.harness.replace(":", "-")
    name = f"bench-{safe_harness}" + ("-deny" if deny else "")
+    executor: dict[str, Any] = {
+        "type": "omnigent",
+        "model": profile.model,
+        "config": {"harness": profile.harness},
+    }
+    # Omit executor.profile when auth comes from the ambient env (no derived
+    # profile), so the runner uses the OPENAI_* already in its env instead of
+    # trying to resolve a profile that may not exist.
+    if db_profile:
+        executor["profile"] = db_profile
    config: dict[str, Any] = {
        "spec_version": 1,
        "name": name,
        "prompt": "You are a helpful assistant used for capability testing.",
-        "executor": {
-            "type": "omnigent",
-            "model": profile.model,
-            "profile": db_profile,
-            "config": {"harness": profile.harness},
-        },
+        "executor": executor,
        # A read-only builtin the server dispatches (and gates at the tool_call
        # phase). The tool/policy probes drive a call to it; harmless for basic
        # turns (the model just won't call it).
@@ -233,8 +212,9 @@ class SharedFullServer:
    are the per-harness operations a ``FullServerDriver`` calls against it.
    """

-    def __init__(self, db_profile: str) -> None:
-        self._db_profile = db_profile
+    def __init__(self, env: BenchRuntimeEnv) -> None:
+        self._env = env
+        self._db_profile = env.db_profile
        self._proc: subprocess.Popen[bytes] | None = None
        self._runner: subprocess.Popen[bytes] | None = None
        self.client: httpx.Client | None = None
@@ -244,20 +224,18 @@ class SharedFullServer:

    def __enter__(self) -> SharedFullServer:
        self._tmp.mkdir(mode=0o700, parents=True, exist_ok=True)
-        host = lookup_databricks_host(self._db_profile)
-        assert host is not None
-        bearer = _mint_bearer(self._db_profile)
        port = _find_free_port()
        self.base_url = f"http://localhost:{port}"
        binding_token = uuid.uuid4().hex
        self.runner_id = token_bound_runner_id(binding_token)
-        base_env = {
-            **os.environ,
-            "OPENAI_API_KEY": bearer,
-            "OPENAI_BASE_URL": f"{host}/serving-endpoints",
-            "DATABRICKS_CONFIG_PROFILE": self._db_profile,
-        }
-        apply_server_env(base_env, _REPO_ROOT)
+        # Credentials/profile were derived the way ``omni run`` does (ambient
+        # OPENAI_* wins, else resolve_databricks_workspace) in resolve_bench_env.
+        base_env = dict(self._env.base_env)
+        # Prepend the repo root so a source-checkout run imports the working
+        # copy; harmless in an installed wheel (where it points at site-packages).
+        base_env["PYTHONPATH"] = os.pathsep.join(
+            p for p in (_REPO_ROOT, os.environ.get("PYTHONPATH", "")) if p
+        )
        self._proc = spawn_omnigent_server(self._tmp, port, base_env, binding_token)
        self._runner = _spawn_bench_runner(
            self._tmp, base_env, self.runner_id, binding_token, self.base_url
@@ -1,6 +1,6 @@
 """Full-server transport driver (phase-2).

-Unlike :class:`tests.harness_bench.driver.SdkInprocDriver` (which drives a
+Unlike :class:`omnigent.harness_bench.driver.SdkInprocDriver` (which drives a
 harness wrap subprocess directly), this driver spins up a REAL Omnigent
 ``server`` + ``runner`` pair, registers an agent, and drives turns through
 the full session path — so policy enforcement and server-dispatched tools
@@ -35,16 +35,16 @@ from typing import Any

 import httpx

-from tests.e2e._harness_probes import cli_unavailable_reason
-from tests.e2e.helpers import lookup_databricks_host
-from tests.harness_bench.driver import TurnResult
-from tests.harness_bench.full_server import (
+from omnigent.harness_bench.cli_probe import cli_unavailable_reason
+from omnigent.harness_bench.driver import TurnResult
+from omnigent.harness_bench.full_server import (
    _DENY_REASON,
    _POLL_INTERVAL_S,
    _TOOL_NAME,
    SharedFullServer,
 )
-from tests.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.runtime_env import bench_creds_skip_reason, resolve_bench_env

 _TOOL_PROMPT = f"List the files using the {_TOOL_NAME} tool, then tell me how many there are."

@@ -111,12 +111,9 @@ class FullServerDriver:
                f"{profile.harness!r} is a native-tui harness; the full-server transport "
                "registers via an agent bundle and cannot drive it (use --transport native-tui)"
            )
-        if not databricks_profile:
-            return "no --profile / databricks profile provided; full-server needs a gateway route"
-        if lookup_databricks_host(databricks_profile) is None:
-            return (
-                f"databricks profile {databricks_profile!r} missing/hostless in ~/.databrickscfg"
-            )
+        creds_skip = bench_creds_skip_reason(databricks_profile)
+        if creds_skip is not None:
+            return creds_skip
        # Same CLI gate as the wrap driver (same binary requirement), but skip
        # its transport check — that is sdk-inproc-specific and would misreport
        # the driver name; the native case is already handled above.
@@ -126,7 +123,7 @@ class FullServerDriver:

    def __enter__(self) -> FullServerDriver:
        if self._shared is None:
-            self._shared = SharedFullServer(self._db_profile)
+            self._shared = SharedFullServer(resolve_bench_env(self._db_profile))
            self._shared.__enter__()
        agent_name = self._shared.register_agent(self._profile, deny=False)
        self._session_id = self._shared.create_session(agent_name)
@@ -3,13 +3,14 @@
 Each profile's descriptive columns and *declared* verdicts derive from the
 canonical capability model (:func:`omnigent.harness_plugins.harness_capabilities`),
 so there is a single source of truth for "what each harness supports". The
-base fields (model, env_prefix, marker, cli_binary) are reused from
-``tests.e2e._harness_probes.HARNESS_PROBES`` — a harness added to the e2e
-parametrize matrix flows into the bench without a second copy.
+base fields (model, env_prefix, marker, cli_binary) come from
+:data:`omnigent.harness_bench.seed.SDK_SEEDS` — the single source of truth the
+e2e parametrize matrix (``tests.e2e._harness_probes``) also rebuilds from, so a
+harness added once flows into both without a second copy.

 The declared matrix is the harness's *published capability*; the bench's
 probes measure live behavior. When they disagree,
-:func:`tests.harness_bench.verdict.reconcile` flags ``DRIFT`` — which means a
+:func:`omnigent.harness_bench.verdict.reconcile` flags ``DRIFT`` — which means a
 harness's capability declaration is false. That makes the capability table
 self-enforcing.

@@ -35,6 +36,9 @@ or correct them as transport coverage lands.
 from __future__ import annotations

 from omnigent.harness_aliases import is_native_harness
+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.seed import SDK_SEEDS, SdkSeed
+from omnigent.harness_bench.verdict import Verdict
 from omnigent.harness_capabilities import AuthModel, HarnessCapabilities, IntegrationMode
 from omnigent.harness_plugins import (
    harness_aliases,
@@ -44,9 +48,6 @@ from omnigent.harness_plugins import (
    install_specs,
    model_env_keys,
 )
-from tests.e2e._harness_probes import HARNESS_PROBES, HarnessProbe
-from tests.harness_bench.profile import BenchProfile
-from tests.harness_bench.verdict import Verdict

 # ── Group A: enum → prose for the descriptive columns ────────────

@@ -135,43 +136,39 @@ def _declared_from_capabilities(harness: str) -> dict[str, Verdict]:
    return declared


-def _profile_from_probe(probe: HarnessProbe) -> BenchProfile:
-    """Build an official :class:`BenchProfile` from an e2e ``HarnessProbe``.
+def _profile_from_seed(seed: SdkSeed) -> BenchProfile:
+    """Build an official :class:`BenchProfile` from an :class:`SdkSeed`.

    Descriptive columns and declared verdicts derive from the capability
-    model; only the transport and the e2e base fields are bench-local.
+    model; only the transport and the concrete seed fields are bench-local.
    """
-    caps = harness_capabilities().get(probe.harness)
+    caps = harness_capabilities().get(seed.harness)
    return BenchProfile(
-        harness=probe.harness,
-        model=probe.model,
-        env_prefix=probe.env_prefix,
-        marker=probe.marker,
-        cli_binary=probe.cli_binary,
+        harness=seed.harness,
+        model=seed.model,
+        env_prefix=seed.env_prefix,
+        marker=seed.marker,
+        cli_binary=seed.cli_binary,
        transport="sdk-inproc",
        owner="",
        auth=_auth_prose(caps),
        implementation=_implementation_prose(caps),
-        declared=_declared_from_capabilities(probe.harness),
+        declared=_declared_from_capabilities(seed.harness),
    )


 # Official harnesses the bench ships with: the P0 SDK harnesses the
-# sdk-inproc driver covers today. Built from HARNESS_PROBES so the e2e and
-# bench matrices never diverge.
-_OFFICIAL_HARNESSES = frozenset({"claude-sdk", "codex", "pi", "openai-agents"})
-
+# sdk-inproc driver covers today. Built from SDK_SEEDS (the single source of
+# truth the e2e matrix also rebuilds from) so the two never diverge.
 OFFICIAL_PROFILES: dict[str, BenchProfile] = {
-    probe.harness: _profile_from_probe(probe)
-    for probe in HARNESS_PROBES
-    if probe.harness in _OFFICIAL_HARNESSES
+    seed.harness: _profile_from_seed(seed) for seed in SDK_SEEDS
 }


 # ── native-tui harnesses ─────────────────────────────────────────
 #
-# Native harnesses are not in HARNESS_PROBES (that matrix is the SDK-wrap
-# e2e set), so their profiles are derived here directly from the capability
+# Native harnesses are not in SDK_SEEDS (that table is the SDK-wrap set), so
+# their profiles are derived here directly from the capability
 # model: every harness with integration_mode == NATIVE_TUI is registered, so
 # the shipped natives and any community-plugin native (harness_capabilities()
 # discovers plugins via entry points) are probeable by name with no bench edit.
@@ -2,14 +2,14 @@

 Drives a native-tui harness — a resident vendor CLI (``claude``, ``codex``,
 ``pi``, ``cursor-agent``, ...) running in a runner-owned tmux pane — through
-the bench's :class:`~tests.harness_bench.transport.Driver` protocol.
+the bench's :class:`~omnigent.harness_bench.transport.Driver` protocol.

 The research finding this is built on: a native-tui turn rides the *same*
 HTTP surface as the full server — ``POST /v1/sessions/{id}/events`` to send,
 ``GET /v1/sessions/{id}/stream`` for ``response.output_text.delta`` events,
 ``/v1/sessions/{id}/policies`` for a tool-call deny, and item polling for the
 assistant reply. So ~90% of this driver is shared with
-:class:`~tests.harness_bench.full_server_driver.FullServerDriver`. Three
+:class:`~omnigent.harness_bench.full_server_driver.FullServerDriver`. Three
 things genuinely diverge, and they are the entire reason this is a separate
 driver:

@@ -50,10 +50,10 @@ from __future__ import annotations

 import asyncio
 import contextlib
-import os
 import shutil
 import signal
 import subprocess
+import sys
 import threading
 import time
 import uuid
@@ -63,6 +63,18 @@ from typing import Any

 import httpx

+from omnigent.harness_bench.cli_probe import cli_unavailable_reason
+from omnigent.harness_bench.driver import ProvisioningError, TurnResult
+from omnigent.harness_bench.full_server import (
+    _find_free_port,
+    spawn_omnigent_server,
+)
+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.runtime_env import (
+    BenchRuntimeEnv,
+    bench_creds_skip_reason,
+    resolve_bench_env,
+)
 from omnigent.harness_capabilities import AuthModel, IntegrationMode
 from omnigent.harness_plugins import harness_capabilities
 from omnigent.host.daemon_launch import (
@@ -72,16 +84,6 @@ from omnigent.host.daemon_launch import (
 )
 from omnigent.native_terminal import bind_session_runner
 from omnigent.runner.identity import OMNIGENT_INTERNAL_WS_ORIGIN
-from tests._helpers.compat import apply_runner_env, compat_runner_cwd, runner_executable
-from tests.e2e._harness_probes import cli_unavailable_reason
-from tests.e2e.helpers import lookup_databricks_host
-from tests.harness_bench.driver import ProvisioningError, TurnResult
-from tests.harness_bench.full_server import (
-    _find_free_port,
-    _mint_bearer,
-    spawn_omnigent_server,
-)
-from tests.harness_bench.profile import BenchProfile

 _HEALTH_TIMEOUT_S = 90.0
 _HOST_ONLINE_TIMEOUT_S = 45.0
@@ -257,9 +259,10 @@ class NativeTuiDriver:

    transport = "native-tui"

-    def __init__(self, profile: BenchProfile, *, databricks_profile: str) -> None:
+    def __init__(self, profile: BenchProfile, *, databricks_profile: str | None) -> None:
        self._profile = profile
        self._db_profile = databricks_profile
+        self._resolved_env: BenchRuntimeEnv | None = None
        self._vendor = native_vendor(profile.harness)
        self._proc: subprocess.Popen[bytes] | None = None
        self._daemon: subprocess.Popen[bytes] | None = None
@@ -278,12 +281,9 @@ class NativeTuiDriver:
        vendor = native_vendor(profile.harness)
        if vendor is None:
            return f"{profile.harness!r} is not a native-tui harness"
-        if not databricks_profile:
-            return "no --profile / databricks profile provided; native-tui needs a gateway route"
-        if lookup_databricks_host(databricks_profile) is None:
-            return (
-                f"databricks profile {databricks_profile!r} missing/hostless in ~/.databrickscfg"
-            )
+        creds_skip = bench_creds_skip_reason(databricks_profile)
+        if creds_skip is not None:
+            return creds_skip
        # The vendor CLI must exist AND be interactively logged in on this
        # host; the bench cannot provision a login. Presence on PATH is the
        # cheapest precondition we can check — a missing login still fails the
@@ -333,17 +333,15 @@ class NativeTuiDriver:
    def _provision(self) -> None:
        self._tmp.mkdir(mode=0o700, parents=True, exist_ok=True)
        assert self._vendor is not None
-        host = lookup_databricks_host(self._db_profile)
-        assert host is not None
        port = _find_free_port()
        self._base_url = f"http://localhost:{port}"
        binding_token = uuid.uuid4().hex

+        # Credentials derived the way `omni run` does (ambient OPENAI_* wins,
+        # else resolve_databricks_workspace); plus the native tunnel token.
+        self._resolved_env = resolve_bench_env(self._db_profile)
        base_env = {
-            **os.environ,
-            "OPENAI_API_KEY": _mint_bearer(self._db_profile),
-            "OPENAI_BASE_URL": f"{host}/serving-endpoints",
-            "DATABRICKS_CONFIG_PROFILE": self._db_profile,
+            **self._resolved_env.base_env,
            "OMNIGENT_RUNNER_TUNNEL_TOKEN": binding_token,
        }
        # An omnigent-credential native resolves its provider from omnigent's
@@ -378,13 +376,17 @@ class NativeTuiDriver:

    def _write_provider_config(self) -> Path:
        """Write the ``OMNIGENT_CONFIG_HOME`` config that routes the vendor's
-        LLM provider through this run's Databricks profile; return its dir."""
+        LLM provider through this run's Databricks profile; return its dir.
+
+        Uses the profile resolved by :func:`resolve_bench_env` (``--profile`` or
+        the config-derived one). When auth came from the ambient env (no
+        profile), the vendor inherits the ambient ``OPENAI_*`` already in
+        ``base_env``, so no ``auth:`` block is written."""
        config_home = self._tmp / "omnigent-config"
        config_home.mkdir(exist_ok=True)
-        (config_home / "config.yaml").write_text(
-            f"auth:\n  type: databricks\n  profile: {self._db_profile}\n",
-            encoding="utf-8",
-        )
+        profile = self._resolved_env.db_profile if self._resolved_env is not None else None
+        body = f"auth:\n  type: databricks\n  profile: {profile}\n" if profile else "auth: {}\n"
+        (config_home / "config.yaml").write_text(body, encoding="utf-8")
        return config_home

    def _wire_native_forwarder(self, host_id: str, workspace: Path) -> None:
@@ -466,9 +468,8 @@ class NativeTuiDriver:
        # (auth cannot be relocated for native harnesses).
        log = (self._tmp / "host-daemon.log").open("wb")
        return subprocess.Popen(
-            [runner_executable(), "-m", "omnigent.host._daemon_entry", "--server", self._base_url],
-            env=apply_runner_env(base_env),
-            cwd=compat_runner_cwd(),
+            [sys.executable, "-m", "omnigent.host._daemon_entry", "--server", self._base_url],
+            env=base_env,
            stdout=subprocess.DEVNULL,
            stderr=log,
        )
@@ -7,13 +7,13 @@ here to add a dimension. Order is the report's column order:

 from __future__ import annotations

-from tests.harness_bench.probes.base import CapabilityProbe
-from tests.harness_bench.probes.basic_turn import BasicTurnProbe
-from tests.harness_bench.probes.interrupt import InterruptProbe
-from tests.harness_bench.probes.model_override import ModelOverrideProbe
-from tests.harness_bench.probes.policy_deny import PolicyDenyProbe
-from tests.harness_bench.probes.streaming import StreamingProbe
-from tests.harness_bench.probes.tool_calling import ToolCallingProbe
+from omnigent.harness_bench.probes.base import CapabilityProbe
+from omnigent.harness_bench.probes.basic_turn import BasicTurnProbe
+from omnigent.harness_bench.probes.interrupt import InterruptProbe
+from omnigent.harness_bench.probes.model_override import ModelOverrideProbe
+from omnigent.harness_bench.probes.policy_deny import PolicyDenyProbe
+from omnigent.harness_bench.probes.streaming import StreamingProbe
+from omnigent.harness_bench.probes.tool_calling import ToolCallingProbe

 # Order is the report's column order AND the run order. basic_turn is first
 # (the prerequisite short-circuit). interrupt is LAST because cancelling a
@@ -3,7 +3,7 @@
 A probe measures one dimension of the support matrix against one harness.
 Probes are harness-agnostic: they call the transport driver's small
 surface and return a :class:`ProbeResult`. Adding a dimension means adding
-a probe module and listing it in :data:`tests.harness_bench.probes.ALL_PROBES`
+a probe module and listing it in :data:`omnigent.harness_bench.probes.ALL_PROBES`
 — no per-harness code anywhere.
 """

@@ -11,9 +11,9 @@ from __future__ import annotations

 import abc

-from tests.harness_bench.profile import BenchProfile
-from tests.harness_bench.transport import Driver
-from tests.harness_bench.verdict import Applicability, Priority, ProbeResult
+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.transport import Driver
+from omnigent.harness_bench.verdict import Applicability, Priority, ProbeResult


 class CapabilityProbe(abc.ABC):
@@ -7,11 +7,11 @@ also tells a reader whether a red row is "this harness is broken" versus

 from __future__ import annotations

-from tests.harness_bench.driver import infra_failure_reason
-from tests.harness_bench.probes.base import CapabilityProbe
-from tests.harness_bench.profile import BenchProfile
-from tests.harness_bench.transport import Driver
-from tests.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict
+from omnigent.harness_bench.driver import infra_failure_reason
+from omnigent.harness_bench.probes.base import CapabilityProbe
+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.transport import Driver
+from omnigent.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict


 class BasicTurnProbe(CapabilityProbe):
@@ -8,11 +8,11 @@ long reply; one that honors it terminates with far less output.

 from __future__ import annotations

-from tests.harness_bench.driver import infra_failure_reason
-from tests.harness_bench.probes.base import CapabilityProbe
-from tests.harness_bench.profile import BenchProfile
-from tests.harness_bench.transport import Driver
-from tests.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict
+from omnigent.harness_bench.driver import infra_failure_reason
+from omnigent.harness_bench.probes.base import CapabilityProbe
+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.transport import Driver
+from omnigent.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict


 class InterruptProbe(CapabilityProbe):
@@ -17,12 +17,12 @@ gateway to reject unknown ids promptly.

 from __future__ import annotations

+from omnigent.harness_bench.driver import infra_failure_reason
+from omnigent.harness_bench.probes.base import CapabilityProbe
+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.transport import Driver
+from omnigent.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict
 from omnigent.model_override import model_family_mismatch, validate_model_override
-from tests.harness_bench.driver import infra_failure_reason
-from tests.harness_bench.probes.base import CapabilityProbe
-from tests.harness_bench.profile import BenchProfile
-from tests.harness_bench.transport import Driver
-from tests.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict


 class ModelOverrideProbe(CapabilityProbe):
@@ -14,10 +14,10 @@ was actually surfaced and that the DENY landed on ``PHASE_TOOL_CALL``.

 from __future__ import annotations

-from tests.harness_bench.probes.base import CapabilityProbe
-from tests.harness_bench.profile import BenchProfile
-from tests.harness_bench.transport import Driver
-from tests.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict
+from omnigent.harness_bench.probes.base import CapabilityProbe
+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.transport import Driver
+from omnigent.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict


 class PolicyDenyProbe(CapabilityProbe):
@@ -15,11 +15,11 @@ SUPPORTED, "never streams" to PARTIAL.

 from __future__ import annotations

-from tests.harness_bench.driver import TurnResult, infra_failure_reason
-from tests.harness_bench.probes.base import CapabilityProbe
-from tests.harness_bench.profile import BenchProfile
-from tests.harness_bench.transport import Driver
-from tests.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict
+from omnigent.harness_bench.driver import TurnResult, infra_failure_reason
+from omnigent.harness_bench.probes.base import CapabilityProbe
+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.transport import Driver
+from omnigent.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict


 class StreamingProbe(CapabilityProbe):
@@ -13,10 +13,10 @@ transport differs.

 from __future__ import annotations

-from tests.harness_bench.probes.base import CapabilityProbe
-from tests.harness_bench.profile import BenchProfile
-from tests.harness_bench.transport import Driver
-from tests.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict
+from omnigent.harness_bench.probes.base import CapabilityProbe
+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.transport import Driver
+from omnigent.harness_bench.verdict import Applicability, Priority, ProbeResult, Verdict


 class ToolCallingProbe(CapabilityProbe):
@@ -7,7 +7,7 @@ verdict for each dimension (the spreadsheet cell, as data).

 The bench never hard-codes a harness anywhere else — probes and drivers
 are harness-agnostic. Adding an official harness means adding a profile
-to :mod:`tests.harness_bench.manifest`; a community / out-of-repo harness
+to :mod:`omnigent.harness_bench.manifest`; a community / out-of-repo harness
 ships its own profile and is selected by name via :func:`resolve_profile`.
 """

@@ -17,7 +17,7 @@ import importlib
 from collections.abc import Mapping
 from dataclasses import dataclass, field

-from tests.harness_bench.verdict import Verdict
+from omnigent.harness_bench.verdict import Verdict


@dataclass(frozen=True)
@@ -42,7 +42,7 @@ class BenchProfile:
        e.g. ``"codex"``. ``None`` for pure-Python harnesses. Used to skip
        the live layer when the binary is absent.
    :param transport: Transport class name selecting a driver in
-        :mod:`tests.harness_bench.driver`, e.g. ``"sdk-inproc"``. A
+        :mod:`omnigent.harness_bench.driver`, e.g. ``"sdk-inproc"``. A
        harness on an unknown transport degrades its transport-dependent
        probes to ``SKIPPED``.
    :param owner: Static matrix column — who owns this harness.
@@ -76,7 +76,7 @@ def resolve_profile(name: str) -> BenchProfile:

    Resolution chain:

-    1. An official harness in :mod:`tests.harness_bench.manifest`.
+    1. An official harness in :mod:`omnigent.harness_bench.manifest`.
    2. A community harness that ships a profile: *name* is a dotted path
       to either a ``BenchProfile`` instance or a zero-arg
       ``bench_profile()`` factory (e.g.
@@ -84,7 +84,7 @@ def resolve_profile(name: str) -> BenchProfile:
    3. Any harness registered in the omnigent registry (in-repo or an
       entry-point plugin), resolved by name / alias — the profile is
       derived from the capability model (see
-       :func:`tests.harness_bench.manifest._registry_profile`). This is what
+       :func:`omnigent.harness_bench.manifest._registry_profile`). This is what
       lets ``--harness acp`` or ``--harness rovo`` run with no bench edit.

    This keeps the official list a convenience index, not a gate: any
@@ -96,7 +96,7 @@ def resolve_profile(name: str) -> BenchProfile:
    :raises KeyError: If *name* is not a registered harness nor an importable
        profile reference.
    """
-    from tests.harness_bench.manifest import OFFICIAL_PROFILES, _registry_profile
+    from omnigent.harness_bench.manifest import OFFICIAL_PROFILES, _registry_profile

    if name in OFFICIAL_PROFILES:
        return OFFICIAL_PROFILES[name]
@@ -19,9 +19,9 @@ from __future__ import annotations
 import json
 from typing import Any

-from tests.harness_bench.bench import BenchMatrix, CellResult, HarnessReport
-from tests.harness_bench.probes import ALL_PROBES
-from tests.harness_bench.verdict import Verdict
+from omnigent.harness_bench.bench import BenchMatrix, CellResult, HarnessReport
+from omnigent.harness_bench.probes import ALL_PROBES
+from omnigent.harness_bench.verdict import Verdict

 # ANSI colors per verdict, applied only when writing to a TTY.
 _ANSI: dict[Verdict, str] = {
@@ -8,20 +8,20 @@ table is for.

 Only usable on a TTY with ``rich`` installed. :func:`rich_sink_or_none`
 returns ``None`` when either precondition is missing, so the CLI falls back to
-the plain :class:`~tests.harness_bench.events.LineSink`.
+the plain :class:`~omnigent.harness_bench.events.LineSink`.
 """

 from __future__ import annotations

-from tests.harness_bench.events import (
+from omnigent.harness_bench.events import (
    BenchEvent,
    HarnessSkipped,
    HarnessStarted,
    ProbeFinished,
    ProbeStarted,
 )
-from tests.harness_bench.probes import ALL_PROBES
-from tests.harness_bench.verdict import Verdict
+from omnigent.harness_bench.probes import ALL_PROBES
+from omnigent.harness_bench.verdict import Verdict

 # Cell state → what the table shows. Verdicts reuse the report glyphs; the two
 # transient states (pending/running) are bench-live only.
@@ -60,7 +60,7 @@ def rich_sink_or_none(*, force: bool = False):


 class _RichLiveSink:
-    """A :class:`~tests.harness_bench.events.ProgressSink` backed by ``rich.Live``.
+    """A :class:`~omnigent.harness_bench.events.ProgressSink` backed by ``rich.Live``.

    Holds a ``{harness: {dimension: cell-markup}}`` grid and re-renders a table
    on every event. Rows appear as harnesses start; a whole-harness skip marks
@@ -0,0 +1,134 @@
+"""Derive the bench's server+runner environment the way ``omni run`` does.
+
+``omni run`` resolves credentials through the canonical
+:func:`omnigent.runtime.credentials.databricks.resolve_databricks_workspace`,
+takes its profile from ``~/.omnigent/config.yaml``'s ``auth:``/``profile`` block,
+and lets an ambient ``OPENAI_*`` env win when present. This module mirrors that
+layering for the bench so ``omni bench`` (no flag) behaves like ``omni run``,
+while an explicit ``--profile`` overrides the config-derived profile.
+
+The old bench path always minted its own bearer via a ``databricks auth token``
+subprocess (which did not handle OAuth profiles); this replaces it.
+"""
+
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class BenchRuntimeEnv:
+    """The environment a bench server/runner/host-daemon is spawned with.
+
+    :param base_env: The full env dict (``os.environ`` plus any derived
+        ``OPENAI_*`` / ``DATABRICKS_CONFIG_PROFILE``).
+    :param db_profile: The resolved Databricks profile name (for
+        ``spec.executor.profile`` / skip-gating), or ``None`` when auth came
+        from the ambient env or the SDK default.
+    """
+
+    base_env: dict[str, str]
+    db_profile: str | None
+
+
+def _profile_from_config() -> str | None:
+    """Return the Databricks profile ``omni run`` would use, or ``None``.
+
+    Reads the user-level ``auth:`` block first (what ``omni run`` consults), then
+    a top-level ``profile:`` key. Imported lazily so this module never imports
+    ``omnigent.cli`` at module load (``cli`` imports the bench command, which
+    would cycle).
+    """
+    try:
+        from omnigent.runtime.workflow import _load_global_auth
+
+        auth = _load_global_auth()
+    except Exception:
+        auth = None
+    profile = getattr(auth, "profile", None)
+    if profile:
+        return str(profile)
+
+    try:
+        from omnigent.cli import _load_effective_config
+
+        cfg_profile = _load_effective_config().get("profile")
+    except Exception:
+        cfg_profile = None
+    return str(cfg_profile) if cfg_profile else None
+
+
+def resolve_bench_env(explicit_profile: str | None) -> BenchRuntimeEnv:
+    """Build the bench runtime env, mirroring ``omni run``'s credential layering.
+
+    Precedence:
+
+    1. **Ambient wins.** If both ``OPENAI_BASE_URL`` and ``OPENAI_API_KEY`` are
+       already in the environment, use them and skip credential resolution
+       entirely — the same short-circuit ``omni run`` has
+       (``inner/databricks_executor.py``). This also lets a run with an
+       exported gateway token work with no profile configured.
+    2. **Profile.** ``explicit_profile`` (the ``--profile`` flag) wins; else the
+       config-derived profile (``auth:``/``profile`` in
+       ``~/.omnigent/config.yaml``). May be ``None`` (the resolver then uses the
+       SDK / ``[DEFAULT]`` path, as ``omni run`` does).
+    3. **Compose** ``OPENAI_*`` from
+       :func:`resolve_databricks_workspace` — OAuth-profile aware, fails loud on
+       a typo'd named profile — filling in only the vars not already ambient.
+
+    :param explicit_profile: The ``--profile`` value, or ``None`` to derive.
+    :returns: A :class:`BenchRuntimeEnv`.
+    :raises OSError: When credentials cannot be resolved and no ambient
+        ``OPENAI_*`` covers auth (surfaced by the caller as a clean skip).
+    """
+    base = dict(os.environ)
+    have_ambient = bool(base.get("OPENAI_BASE_URL")) and bool(base.get("OPENAI_API_KEY"))
+    profile = explicit_profile or _profile_from_config()
+
+    if have_ambient:
+        # Ambient OPENAI_* already routes the gateway; don't mint. Still stamp
+        # the profile so the runner's DATABRICKS_CONFIG_PROFILE matches.
+        if profile:
+            base["DATABRICKS_CONFIG_PROFILE"] = profile
+        return BenchRuntimeEnv(base_env=base, db_profile=profile)
+
+    from omnigent.runtime.credentials.databricks import resolve_databricks_workspace
+
+    creds = resolve_databricks_workspace(profile)
+    base["OPENAI_BASE_URL"] = f"{creds.host}/serving-endpoints"
+    base["OPENAI_API_KEY"] = creds.token
+    if profile:
+        base["DATABRICKS_CONFIG_PROFILE"] = profile
+    return BenchRuntimeEnv(base_env=base, db_profile=profile)
+
+
+def bench_creds_skip_reason(explicit_profile: str | None) -> str | None:
+    """Cheap gate: why the bench cannot get gateway creds, or ``None`` if it can.
+
+    Mirrors :func:`resolve_bench_env`'s precedence without minting a token, so a
+    driver's ``unavailable()`` can skip a live run cleanly (no creds) rather than
+    fail mid-provision. A ``--profile`` is no longer required: an ambient
+    ``OPENAI_*`` or a configured ``~/.omnigent`` profile is enough, matching
+    ``omni run``.
+
+    :param explicit_profile: The ``--profile`` value, or ``None`` to derive.
+    :returns: A skip reason, or ``None`` when creds are resolvable.
+    """
+    if os.environ.get("OPENAI_BASE_URL") and os.environ.get("OPENAI_API_KEY"):
+        return None
+    profile = explicit_profile or _profile_from_config()
+    if not profile:
+        return (
+            "no gateway creds: pass --profile, configure a profile in "
+            "~/.omnigent/config.yaml (like `omni run`), or export OPENAI_API_KEY + "
+            "OPENAI_BASE_URL"
+        )
+    from omnigent.harness_bench.creds_lookup import lookup_databricks_host
+
+    if lookup_databricks_host(profile) is None:
+        return f"databricks profile {profile!r} missing/hostless in ~/.databrickscfg"
+    return None
+
+
+__all__ = ["BenchRuntimeEnv", "bench_creds_skip_reason", "resolve_bench_env"]
@@ -0,0 +1,80 @@
+"""Plain-data seed for the official SDK-wrap harnesses the bench ships with.
+
+Each :class:`SdkSeed` names a harness plus the four concrete fields the
+capability model does NOT carry: the default ``model`` id, the ``env_prefix`` its
+wrap reads, the ``marker`` string a basic-turn probe echoes, and the ``cli_binary``
+to skip-gate on. Descriptive/declared columns still come from
+``harness_capabilities()`` (see :mod:`omnigent.harness_bench.manifest`).
+
+This is the single source of truth for the SDK-wrap set. The e2e parametrize
+matrix (``tests.e2e._harness_probes``) rebuilds its ``HARNESS_PROBES`` from these
+seeds, wrapping ``model`` through its env-tunable test pools, so the bench and
+e2e matrices cannot diverge. The ``model`` ids here are the deterministic
+defaults (the same base ids the e2e pools are seeded with).
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class SdkSeed:
+    """One SDK-wrap harness's bench-local concrete fields.
+
+    :param harness: Registry harness id, e.g. ``"claude-sdk"``.
+    :param model: Default gateway model id for a live probe.
+    :param env_prefix: The env-var prefix the wrap reads, e.g.
+        ``"HARNESS_CLAUDE_SDK_"``.
+    :param marker: The literal string a basic-turn probe asks the model to echo.
+    :param cli_binary: The CLI binary to skip-gate on, or ``None`` (pure-Python).
+    """
+
+    harness: str
+    model: str
+    env_prefix: str
+    marker: str
+    cli_binary: str | None = None
+
+
+# The P0 SDK harnesses the sdk-inproc / full-server drivers cover. Add a new SDK
+# wrap here and it flows into both the bench and the e2e parametrize matrix.
+SDK_SEEDS: tuple[SdkSeed, ...] = (
+    SdkSeed(
+        harness="claude-sdk",
+        model="databricks-claude-opus-4-6",
+        env_prefix="HARNESS_CLAUDE_SDK_",
+        marker="CLAUDE_E2E_OK",
+        cli_binary="claude",
+    ),
+    SdkSeed(
+        harness="codex",
+        # OpenAI-style model exposed via the Databricks gateway; Codex's executor
+        # speaks the OpenAI Responses API, lit up via HARNESS_CODEX_GATEWAY.
+        model="databricks-gpt-5-4-mini",
+        env_prefix="HARNESS_CODEX_",
+        marker="CODEX_E2E_OK",
+        cli_binary="codex",
+    ),
+    SdkSeed(
+        harness="pi",
+        # Pi speaks the OpenAI Responses API; the gateway exposes Claude through
+        # that endpoint too.
+        model="databricks-claude-sonnet-4-6",
+        env_prefix="HARNESS_PI_",
+        marker="PI_E2E_OK",
+        cli_binary="pi",
+    ),
+    SdkSeed(
+        harness="openai-agents",
+        # Registry key is ``openai-agents`` (not ``-sdk``) to match the Omnigent
+        # YAML ``executor.harness`` spelling. Pure-Python package; no CLI binary.
+        model="databricks-gpt-5-4-mini",
+        env_prefix="HARNESS_OPENAI_AGENTS_",
+        marker="OPENAI_AGENTS_E2E_OK",
+        cli_binary=None,
+    ),
+)
+
+
+__all__ = ["SDK_SEEDS", "SdkSeed"]
@@ -3,17 +3,17 @@
 A probe measures one capability dimension by calling a small set of
 *semantic* methods on a driver — ``run_basic_turn``, ``run_streaming_turn``,
 ``run_tool_turn``, ``run_interrupt_turn`` — each returning a
-:class:`~tests.harness_bench.driver.TurnResult`. The driver owns the
+:class:`~omnigent.harness_bench.driver.TurnResult`. The driver owns the
 *mechanism* (how a tool call is provoked, how a deny is enforced, how deltas
 are observed); the probe owns the *interpretation* (what verdict the result
 implies). This split is what lets one probe run over transports that reach
 the same capability by different means:

- ``sdk-inproc`` (:class:`~tests.harness_bench.driver.SdkInprocDriver`)
+- ``sdk-inproc`` (:class:`~omnigent.harness_bench.driver.SdkInprocDriver`)
  drives a harness wrap subprocess directly, with request-level tools and
  verdict-posted policy.
 - ``full-server``
-  (:class:`~tests.harness_bench.full_server_driver.FullServerDriver`) drives
+  (:class:`~omnigent.harness_bench.full_server_driver.FullServerDriver`) drives
  a real server+runner, with a builtin tool and a spec-baked policy.

 A kwargs-carrying ``run_turn`` could not bridge these: e.g. streaming is only
@@ -38,8 +38,8 @@ from __future__ import annotations

 from typing import Protocol

-from tests.harness_bench.driver import TurnResult
-from tests.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.driver import TurnResult
+from omnigent.harness_bench.profile import BenchProfile


 class Driver(Protocol):
@@ -93,9 +93,9 @@ def driver_registry() -> dict[str, type]:
    Imported lazily so the transport module stays cheap to import (the
    full-server driver pulls in server/runner spawn helpers).
    """
-    from tests.harness_bench.driver import SdkInprocDriver
-    from tests.harness_bench.full_server_driver import FullServerDriver
-    from tests.harness_bench.native_tui_driver import NativeTuiDriver
+    from omnigent.harness_bench.driver import SdkInprocDriver
+    from omnigent.harness_bench.full_server_driver import FullServerDriver
+    from omnigent.harness_bench.native_tui_driver import NativeTuiDriver

    return {
        SdkInprocDriver.transport: SdkInprocDriver,
@@ -453,6 +453,15 @@ known-first-party = ["omnigent"]
 # files have a backlog of unused-arg, blind-except, and bugbear issues
 # that were invisible before. Graduate each subpath out of this list
 # as it's cleaned up.
+# The harness bench (moved from tests/harness_bench/ so `omni bench` can ship
+# it) carries the same test-adjacent patterns the tests/** block waives: probes
+# take a uniform ``(driver, profile)`` signature where some don't read
+# ``profile`` (ARG002), and the orchestrator + drivers intercept broad
+# exceptions to turn a provisioning/probe failure into a capability-neutral SKIP
+# (BLE001) rather than aborting the whole matrix.
+"omnigent/harness_bench/**/*.py" = [
+    "ARG001", "ARG002", "ARG004", "ARG005", "BLE001",
+]
 "omnigent/llms/**/*.py" = [
    "ARG001", "ARG002", "BLE001", "B008", "RUF012",
 ]
@@ -127,6 +127,35 @@ def test_python_module_entrypoint_uses_unified_click_cli() -> None:
    assert "Omnigent quick chat" not in result.stdout


+def test_bench_command_registered() -> None:
+    """``bench`` is a real subcommand (and listed in the sync set)."""
+    assert "bench" in _CLICK_SUBCOMMANDS
+    assert "bench" in cli.commands
+
+
+def test_bench_command_delegates_to_harness_bench_main(monkeypatch: pytest.MonkeyPatch) -> None:
+    """``omni bench`` forwards its args verbatim to the bench's argparse main
+    and exits with its return code (thin pass-through wrapper)."""
+    captured: dict[str, object] = {}
+
+    def _fake_main(argv: list[str]) -> int:
+        captured["argv"] = argv
+        return 0
+
+    monkeypatch.setattr("omnigent.harness_bench.__main__.main", _fake_main)
+    result = CliRunner().invoke(cli, ["bench", "--list", "--harness", "codex"])
+    assert result.exit_code == 0
+    # Unknown-to-click options pass straight through (ignore_unknown_options).
+    assert captured["argv"] == ["--list", "--harness", "codex"]
+
+
+def test_bench_command_propagates_nonzero_exit(monkeypatch: pytest.MonkeyPatch) -> None:
+    """A DRIFT / error exit code from the bench surfaces as the command's exit."""
+    monkeypatch.setattr("omnigent.harness_bench.__main__.main", lambda argv: 1)
+    result = CliRunner().invoke(cli, ["bench"])
+    assert result.exit_code == 1
+
+
@pytest.mark.parametrize(
    ("argv", "expected"),
    [
@@ -16,15 +16,23 @@ per-file edits.

 from __future__ import annotations

-import shutil
-import subprocess
 from dataclasses import dataclass
-from functools import cache

 import pytest

+from omnigent.harness_bench.cli_probe import cli_unavailable_reason
+from omnigent.harness_bench.seed import SDK_SEEDS
 from tests._model_pools import resolve_model

+__all__ = [
+    "HARNESS_HARNESS_MODELS",
+    "HARNESS_IDS",
+    "HARNESS_PROBES",
+    "HarnessProbe",
+    "cli_unavailable_reason",
+    "skip_if_harness_cli_missing",
+]
+

@dataclass(frozen=True)
 class HarnessProbe:
@@ -75,55 +83,20 @@ class HarnessProbe:
 # so the OMNIGENT_TEST_MODEL_* env vars can rebalance a harness's
 # rows without code edits; pools stay within the API style each
 # harness supports.
+# Built from :data:`omnigent.harness_bench.seed.SDK_SEEDS` (the single source of
+# truth the bench also uses) so the e2e and bench matrices never diverge. The
+# only e2e-specific layer is routing each seed's default ``model`` through
+# ``resolve_model`` (the env-tunable test pools), which lets ``OMNIGENT_TEST_MODEL_*``
+# rebalance a harness's rows without editing the seed table.
 HARNESS_PROBES: list[HarnessProbe] = [
    HarnessProbe(
-        harness="claude-sdk",
-        model=resolve_model("databricks-claude-opus-4-6", key="probe:claude-sdk"),
-        env_prefix="HARNESS_CLAUDE_SDK_",
-        marker="CLAUDE_E2E_OK",
-        cli_binary="claude",
-    ),
-    HarnessProbe(
-        harness="codex",
-        # Per CLAUDE.md guidance: ``databricks-gpt-5-4-mini`` is
-        # the OpenAI-style model exposed via the Databricks
-        # gateway. Codex's executor speaks the OpenAI Responses
-        # API, so this model lights up via the
-        # ``HARNESS_CODEX_GATEWAY`` route.
-        model=resolve_model("databricks-gpt-5-4-mini", key="probe:codex"),
-        env_prefix="HARNESS_CODEX_",
-        marker="CODEX_E2E_OK",
-        cli_binary="codex",
-    ),
-    HarnessProbe(
-        harness="pi",
-        # Pi speaks the OpenAI Responses API and the Databricks
-        # gateway exposes Claude through that endpoint too. Per
-        # CLAUDE.md, ``databricks-claude-sonnet-4-6`` is the
-        # default Claude-via-Databricks model the per-harness
-        # pi suite uses.
-        model=resolve_model("databricks-claude-sonnet-4-6", key="probe:pi"),
-        env_prefix="HARNESS_PI_",
-        marker="PI_E2E_OK",
-        cli_binary="pi",
-    ),
-    HarnessProbe(
-        harness="openai-agents",
-        # The openai-agents SDK speaks the OpenAI Responses API
-        # via the Databricks gateway; the GPT model is the
-        # natural fit per CLAUDE.md (``databricks-gpt-5-4-mini``
-        # is the OpenAI-style Databricks model). Registry key is
-        # ``openai-agents`` (not ``-sdk``) to match the
-        # Omnigent YAML ``executor.harness`` spelling.
-        model=resolve_model("databricks-gpt-5-4-mini", key="probe:openai-agents"),
-        env_prefix="HARNESS_OPENAI_AGENTS_",
-        marker="OPENAI_AGENTS_E2E_OK",
-        # Pure-Python ``openai-agents`` package; no CLI binary
-        # to skip on. ``cli_binary=None`` means
-        # :func:`skip_if_harness_cli_missing` is a no-op for
-        # this row.
-        cli_binary=None,
-    ),
+        harness=seed.harness,
+        model=resolve_model(seed.model, key=f"probe:{seed.harness}"),
+        env_prefix=seed.env_prefix,
+        marker=seed.marker,
+        cli_binary=seed.cli_binary,
+    )
+    for seed in SDK_SEEDS
 ]


@@ -146,47 +119,9 @@ HARNESS_IDS: list[str] = [p.harness for p in HARNESS_PROBES]
 # helper below doesn't reconstruct it on every call.
 _CLI_BINARY_BY_HARNESS: dict[str, str | None] = {p.harness: p.cli_binary for p in HARNESS_PROBES}

-
-def _cli_probe_args(binary: str) -> list[str]:
-    """Return a cheap command that proves *binary* is runnable."""
-    if binary == "pi":
-        # ``shutil.which("pi")`` alone is not enough: pi's npm package
-        # may be installed under an older Node version than the package
-        # supports. ``pi --help`` exercises module loading without making
-        # model/network calls, so it catches broken installs early and lets
-        # e2e rows skip instead of failing deep inside ``PiExecutor``.
-        return [binary, "--help"]
-    return [binary, "--version"]
-
-
-@cache
-def cli_unavailable_reason(binary: str) -> str | None:
-    """
-    Return ``None`` when *binary* exists and starts, else a skip reason.
-
-    The result is cached because e2e suites call the harness gate from many
-    parametrized rows and CLI startup can be non-trivial.
-    """
-    path = shutil.which(binary)
-    if path is None:
-        return f"{binary!r} CLI is not on PATH"
-
-    try:
-        proc = subprocess.run(
-            _cli_probe_args(binary),
-            capture_output=True,
-            text=True,
-            timeout=15,
-            check=False,
-        )
-    except (OSError, subprocess.TimeoutExpired) as exc:
-        return f"{binary!r} CLI at {path!r} is not runnable: {exc}"
-
-    if proc.returncode != 0:
-        detail = (proc.stderr or proc.stdout).strip().splitlines()
-        suffix = f": {detail[0]}" if detail else ""
-        return f"{binary!r} CLI at {path!r} exits {proc.returncode}{suffix}"
-    return None
+# ``cli_unavailable_reason`` now lives in the shipped package
+# (:mod:`omnigent.harness_bench.cli_probe`) and is imported at the top; the e2e
+# gate below reuses it so there is one copy.


 def skip_if_harness_cli_missing(harness: str) -> None:
@@ -1,39 +1,5 @@
-"""Harness capability test bench.
+"""Tests for the harness capability bench.

-A standardized, pluggable conformance suite that probes a harness and
-reports a verdict per capability dimension (basic turn, streaming,
-tool calling, interrupt, policy DENY, model override, ...), reconciling
-observed behavior against a self-declared :class:`BenchProfile` to
-surface drift.
-
-Design: ``docs/harness-bench-design.md``.
-
-Two entry points:
-
- ``python -m tests.harness_bench --harness <name>`` renders the matrix
-  for one harness (or all official harnesses with no ``--harness``).
- ``tests/harness_bench/test_bench.py`` runs the offline conformance
-  layer on every PR and the live-probe layer when a ``--profile`` and
-  the harness CLI are available.
+The bench itself lives in the shipped package
+:mod:`omnigent.harness_bench`; these are its unit tests.
 """
-
-from __future__ import annotations
-
-from tests.harness_bench.profile import BenchProfile, resolve_profile
-from tests.harness_bench.verdict import (
-    Applicability,
-    Priority,
-    ProbeResult,
-    Verdict,
-    reconcile,
-)
-
-__all__ = [
-    "Applicability",
-    "BenchProfile",
-    "Priority",
-    "ProbeResult",
-    "Verdict",
-    "reconcile",
-    "resolve_profile",
-]
@@ -17,14 +17,14 @@ import json

 import pytest

+from omnigent.harness_bench.bench import run_bench, run_harness
+from omnigent.harness_bench.driver import SdkInprocDriver
+from omnigent.harness_bench.manifest import OFFICIAL_PROFILES
+from omnigent.harness_bench.probes import ALL_PROBES
+from omnigent.harness_bench.profile import BenchProfile, resolve_profile
+from omnigent.harness_bench.report import render_json, render_markdown
+from omnigent.harness_bench.verdict import Priority, Verdict, reconcile
 from omnigent.runtime.harnesses import _HARNESS_MODULES
-from tests.harness_bench.bench import run_bench, run_harness
-from tests.harness_bench.driver import SdkInprocDriver
-from tests.harness_bench.manifest import OFFICIAL_PROFILES
-from tests.harness_bench.probes import ALL_PROBES
-from tests.harness_bench.profile import BenchProfile, resolve_profile
-from tests.harness_bench.report import render_json, render_markdown
-from tests.harness_bench.verdict import Priority, Verdict, reconcile

 _OFFICIAL = list(OFFICIAL_PROFILES.values())
 _OFFICIAL_IDS = [p.harness for p in _OFFICIAL]
@@ -70,8 +70,8 @@ def test_declared_covers_every_p0_dimension(profile: BenchProfile) -> None:
 def test_streaming_capability_declares_binary_verdict() -> None:
    # Guards the kiro-native drift: streaming declares binary (True→SUPPORTED,
    # False→UNSUPPORTED), never PARTIAL.
+    from omnigent.harness_bench.manifest import _declared_from_capabilities
    from omnigent.harness_plugins import harness_capabilities
-    from tests.harness_bench.manifest import _declared_from_capabilities

    caps = harness_capabilities()
    for harness, cap in caps.items():
@@ -168,7 +168,7 @@ def test_registry_profile_happy_path_no_plugin(monkeypatch: pytest.MonkeyPatch)
    """
    from types import SimpleNamespace

-    import tests.harness_bench.manifest as man
+    from omnigent.harness_bench import manifest as man
    from omnigent.harness_capabilities import AuthModel, IntegrationMode

    class _Spec:
@@ -206,7 +206,7 @@ def test_registry_refuses_native_server_mode(monkeypatch: pytest.MonkeyPatch) ->
    """
    from types import SimpleNamespace

-    import tests.harness_bench.manifest as man
+    from omnigent.harness_bench import manifest as man
    from omnigent.harness_capabilities import AuthModel, IntegrationMode

    caps = SimpleNamespace(
@@ -222,7 +222,7 @@ def test_registry_refuses_native_server_mode(monkeypatch: pytest.MonkeyPatch) ->


 def test_infra_failure_reason_classifies_auth_and_ignores_capability_gaps() -> None:
-    from tests.harness_bench.driver import TurnResult, infra_failure_reason
+    from omnigent.harness_bench.driver import TurnResult, infra_failure_reason

    # A 403 gateway error is an environment problem -> yields a skip reason.
    auth = TurnResult(
@@ -279,8 +279,8 @@ async def test_offline_render_produces_matrix() -> None:

 def test_grid_already_shown_only_for_grid_drawing_sink() -> None:
    """_grid_already_shown is True only for a sink that painted the grid."""
-    from tests.harness_bench.__main__ import _grid_already_shown
-    from tests.harness_bench.events import LineSink
+    from omnigent.harness_bench.__main__ import _grid_already_shown
+    from omnigent.harness_bench.events import LineSink

    assert _grid_already_shown(None) is False
    assert _grid_already_shown(LineSink(lambda _m: None)) is False
@@ -298,7 +298,7 @@ async def test_render_table_grid_false_drops_grid_keeps_footer() -> None:
    on the same terminal: the report should add the per-cell explanations, not
    reprint the grid.
    """
-    from tests.harness_bench.report import render_table
+    from omnigent.harness_bench.report import render_table

    matrix = await run_bench(_OFFICIAL, live=False)
    full = render_table(matrix, declared=True, grid=True)
@@ -325,8 +325,8 @@ async def test_run_harness_emits_structured_events_and_linesink_adapts() -> None
    Uses a fake driver so no creds/subprocess are needed: a basic turn passes,
    which lets every probe run and produce a ProbeFinished.
    """
-    from tests.harness_bench.driver import TurnResult
-    from tests.harness_bench.events import (
+    from omnigent.harness_bench.driver import TurnResult
+    from omnigent.harness_bench.events import (
        HarnessFinished,
        HarnessStarted,
        ProbeFinished,
@@ -376,7 +376,7 @@ async def test_run_harness_emits_structured_events_and_linesink_adapts() -> None

    monkeypatch = pytest.MonkeyPatch()
    monkeypatch.setattr(
-        "tests.harness_bench.bench.resolve_driver_class",
+        "omnigent.harness_bench.bench.resolve_driver_class",
        lambda p, *, override=None, fast=False: _OKDriver,
    )
    try:
@@ -402,7 +402,7 @@ async def test_run_harness_emits_structured_events_and_linesink_adapts() -> None
    lines: list[str] = []
    monkeypatch = pytest.MonkeyPatch()
    monkeypatch.setattr(
-        "tests.harness_bench.bench.resolve_driver_class",
+        "omnigent.harness_bench.bench.resolve_driver_class",
        lambda p, *, override=None, fast=False: _OKDriver,
    )
    try:
@@ -416,7 +416,7 @@ async def test_run_bench_jobs_preserves_order(monkeypatch: pytest.MonkeyPatch) -
    """--jobs > 1 runs harnesses concurrently but keeps report order == input order."""
    import asyncio as _asyncio

-    from tests.harness_bench.driver import TurnResult
+    from omnigent.harness_bench.driver import TurnResult

    class _SlowDriver:
        transport = "sdk-inproc"
@@ -450,7 +450,7 @@ async def test_run_bench_jobs_preserves_order(monkeypatch: pytest.MonkeyPatch) -
            return TurnResult(cancelled=True)

    monkeypatch.setattr(
-        "tests.harness_bench.bench.resolve_driver_class",
+        "omnigent.harness_bench.bench.resolve_driver_class",
        lambda p, *, override=None, fast=False: _SlowDriver,
    )
    profiles = [
@@ -468,7 +468,7 @@ async def test_parallel_full_server_shares_one_server(monkeypatch: pytest.Monkey
    one SharedFullServer is entered once and each harness registers its own
    agent+session on it.
    """
-    from tests.harness_bench.driver import TurnResult
+    from omnigent.harness_bench.driver import TurnResult

    built: list[object] = []

@@ -524,9 +524,9 @@ async def test_parallel_full_server_shares_one_server(monkeypatch: pytest.Monkey
            return TurnResult(cancelled=True)

    # bench imports SharedFullServer into its own namespace, so patch it there.
-    monkeypatch.setattr("tests.harness_bench.bench.SharedFullServer", _FakeShared)
+    monkeypatch.setattr("omnigent.harness_bench.bench.SharedFullServer", _FakeShared)
    monkeypatch.setattr(
-        "tests.harness_bench.bench.resolve_driver_class",
+        "omnigent.harness_bench.bench.resolve_driver_class",
        lambda p, *, override=None, fast=False: _FSDriver,
    )

@@ -551,7 +551,7 @@ async def test_parallel_full_server_shares_one_server(monkeypatch: pytest.Monkey

 def test_cli_writes_report_file(tmp_path) -> None:
    """`--report PATH` writes the matrix; format follows the extension."""
-    from tests.harness_bench.__main__ import main
+    from omnigent.harness_bench.__main__ import main

    md = tmp_path / "matrix.md"
    rc = main(["--no-live", "--report", str(md)])
@@ -616,9 +616,9 @@ async def test_full_server_async_shims_delegate_to_sync(monkeypatch: pytest.Monk
    in the async binding is caught without a server+runner. Builds no driver
    state — every sync method is stubbed.
    """
-    from tests.harness_bench.driver import TurnResult
-    from tests.harness_bench.full_server_driver import FullServerDriver
-    from tests.harness_bench.profile import BenchProfile
+    from omnigent.harness_bench.driver import TurnResult
+    from omnigent.harness_bench.full_server_driver import FullServerDriver
+    from omnigent.harness_bench.profile import BenchProfile

    profile = BenchProfile(harness="stub", model="m", env_prefix="HARNESS_STUB_", marker="STUB_OK")
    driver = FullServerDriver(profile, databricks_profile="oss")
@@ -678,7 +678,7 @@ async def test_provisioning_failure_skips_and_tears_down(monkeypatch: pytest.Mon
        harness="stub-native", model="m", env_prefix="HARNESS_STUB_NATIVE_", marker="X"
    )
    monkeypatch.setattr(
-        "tests.harness_bench.bench.resolve_driver_class",
+        "omnigent.harness_bench.bench.resolve_driver_class",
        lambda p, *, override=None, fast=False: _FailingDriver,
    )

@@ -701,7 +701,7 @@ async def test_expected_provisioning_error_logged_quietly(
    """
    import logging

-    from tests.harness_bench.driver import ProvisioningError
+    from omnigent.harness_bench.driver import ProvisioningError

    def _driver_raising(exc: Exception):
        class _D:
@@ -726,12 +726,12 @@ async def test_expected_provisioning_error_logged_quietly(

    # Expected failure → a single INFO record, no exception/traceback attached.
    monkeypatch.setattr(
-        "tests.harness_bench.bench.resolve_driver_class",
+        "omnigent.harness_bench.bench.resolve_driver_class",
        lambda p, *, override=None, fast=False: _driver_raising(
            ProvisioningError("cli not logged in")
        ),
    )
-    with caplog.at_level(logging.INFO, logger="tests.harness_bench.bench"):
+    with caplog.at_level(logging.INFO, logger="omnigent.harness_bench.bench"):
        await run_harness(profile, databricks_profile="oss", live=True)
    provisioning_logs = [r for r in caplog.records if "stub" in r.getMessage()]
    assert provisioning_logs, "expected a log line for the skip"
@@ -740,10 +740,10 @@ async def test_expected_provisioning_error_logged_quietly(
    # Unexpected failure → WARNING with the traceback attached.
    caplog.clear()
    monkeypatch.setattr(
-        "tests.harness_bench.bench.resolve_driver_class",
+        "omnigent.harness_bench.bench.resolve_driver_class",
        lambda p, *, override=None, fast=False: _driver_raising(RuntimeError("boom")),
    )
-    with caplog.at_level(logging.INFO, logger="tests.harness_bench.bench"):
+    with caplog.at_level(logging.INFO, logger="omnigent.harness_bench.bench"):
        await run_harness(profile, databricks_profile="oss", live=True)
    warnings = [r for r in caplog.records if r.levelno == logging.WARNING]
    assert warnings and any(r.exc_info is not None for r in warnings), (
@@ -754,10 +754,10 @@ async def test_expected_provisioning_error_logged_quietly(
 # ── native-tui transport (offline) ──────────────────────────────


-def test_native_tui_registered_and_gates() -> None:
+def test_native_tui_registered_and_gates(monkeypatch: pytest.MonkeyPatch) -> None:
    """native-tui is in the registry and derives any native-tui harness."""
-    from tests.harness_bench.native_tui_driver import NativeTuiDriver, native_vendor
-    from tests.harness_bench.transport import driver_registry, resolve_driver_class
+    from omnigent.harness_bench.native_tui_driver import NativeTuiDriver, native_vendor
+    from omnigent.harness_bench.transport import driver_registry, resolve_driver_class

    assert driver_registry()["native-tui"] is NativeTuiDriver

@@ -781,7 +781,15 @@ def test_native_tui_registered_and_gates() -> None:
    codex_sdk = BenchProfile(harness="codex", model="m", env_prefix="X_", marker="X")
    assert NativeTuiDriver.unavailable(codex_sdk, databricks_profile="oss") is not None

-    # No profile → the same capability-neutral skip contract as other drivers.
+    # No creds anywhere (no --profile, no ambient OPENAI_*, no configured
+    # profile) → capability-neutral skip. Clear the ambient env and stub the
+    # config lookup so the contract is tested deterministically regardless of
+    # the host's environment.
+    monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+    monkeypatch.delenv("OPENAI_BASE_URL", raising=False)
+    monkeypatch.setattr(
+        "omnigent.harness_bench.runtime_env._profile_from_config", lambda: None
+    )
    assert NativeTuiDriver.unavailable(claude_native, databricks_profile=None) is not None


@@ -792,7 +800,7 @@ def test_transport_resolution_family_default_and_fast() -> None:
    the profile's transport is a family marker, and the effective driver comes
    from family + flags (see resolve_transport_name).
    """
-    from tests.harness_bench.transport import resolve_transport_name
+    from omnigent.harness_bench.transport import resolve_transport_name

    sdk = BenchProfile(
        harness="codex", model="m", env_prefix="X_", marker="X", transport="sdk-inproc"
@@ -827,8 +835,8 @@ async def test_native_provisioning_http_error_becomes_provisioning_error(
    """
    import httpx

-    from tests.harness_bench.driver import ProvisioningError
-    from tests.harness_bench.native_tui_driver import NativeTuiDriver
+    from omnigent.harness_bench.driver import ProvisioningError
+    from omnigent.harness_bench.native_tui_driver import NativeTuiDriver

    profile = BenchProfile(
        harness="claude-native",
@@ -857,7 +865,7 @@ def test_full_server_skips_native_with_accurate_message() -> None:
    there (bundle registration, not host-daemon provisioning). The skip must
    name native-tui as the answer, not misreport the 'sdk-inproc' driver.
    """
-    from tests.harness_bench.full_server_driver import FullServerDriver
+    from omnigent.harness_bench.full_server_driver import FullServerDriver

    # Real native profiles carry transport="native-tui" (set in the manifest);
    # that is what the full-server gate keys on.
@@ -13,8 +13,8 @@ from __future__ import annotations

 import pytest

-from tests.harness_bench.full_server_driver import FullServerDriver
-from tests.harness_bench.profile import resolve_profile
+from omnigent.harness_bench.full_server_driver import FullServerDriver
+from omnigent.harness_bench.profile import resolve_profile


@pytest.fixture
@@ -12,12 +12,12 @@ from __future__ import annotations
 import json
 from typing import Any

-from tests.harness_bench.driver import TurnResult
-from tests.harness_bench.native_tui_driver import NativeTuiDriver, native_vendor
-from tests.harness_bench.probes.policy_deny import PolicyDenyProbe
-from tests.harness_bench.probes.tool_calling import ToolCallingProbe
-from tests.harness_bench.profile import BenchProfile
-from tests.harness_bench.verdict import Verdict
+from omnigent.harness_bench.driver import TurnResult
+from omnigent.harness_bench.native_tui_driver import NativeTuiDriver, native_vendor
+from omnigent.harness_bench.probes.policy_deny import PolicyDenyProbe
+from omnigent.harness_bench.probes.tool_calling import ToolCallingProbe
+from omnigent.harness_bench.profile import BenchProfile
+from omnigent.harness_bench.verdict import Verdict


 class _FakeResponse:
@@ -233,8 +233,8 @@ async def test_probes_read_native_tool_result_as_supported() -> None:

 def test_format_matches_server_wire_name() -> None:
    """The driver keys on the exact wire name the server publishes."""
+    from omnigent.harness_bench.native_tui_driver import _POLICY_DENIED_EVENT
    from omnigent.server.routes.sessions import _format_sse
-    from tests.harness_bench.native_tui_driver import _POLICY_DENIED_EVENT

    sse = _format_sse(_POLICY_DENIED_EVENT, {"type": _POLICY_DENIED_EVENT})
    # The reader parses `event: <name>`; assert the driver's key is that name.
@@ -0,0 +1,121 @@
+"""Unit tests for the bench env resolver (derive creds like ``omni run``).
+
+Network-free: monkeypatches the config lookup and the canonical Databricks
+resolver, so these assert the *layering* (ambient wins, ``--profile`` overrides
+config, no-creds skips) without touching ``~/.databrickscfg`` or the gateway.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from omnigent.harness_bench import runtime_env
+from omnigent.harness_bench.runtime_env import (
+    BenchRuntimeEnv,
+    bench_creds_skip_reason,
+    resolve_bench_env,
+)
+
+
+class _Creds:
+    """Stand-in for ``WorkspaceCreds`` (host + token)."""
+
+    def __init__(self, host: str, token: str) -> None:
+        self.host = host
+        self.token = token
+
+
+@pytest.fixture(autouse=True)
+def _clean_env(monkeypatch: pytest.MonkeyPatch) -> None:
+    """Start each test from a known env: no ambient OPENAI_*, no config profile."""
+    monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+    monkeypatch.delenv("OPENAI_BASE_URL", raising=False)
+    monkeypatch.delenv("DATABRICKS_CONFIG_PROFILE", raising=False)
+    monkeypatch.setattr(runtime_env, "_profile_from_config", lambda: None)
+
+
+def _stub_resolver(monkeypatch: pytest.MonkeyPatch, host: str, token: str) -> None:
+    monkeypatch.setattr(
+        "omnigent.runtime.credentials.databricks.resolve_databricks_workspace",
+        lambda profile: _Creds(host, token),
+    )
+
+
+def test_explicit_profile_mints_via_canonical_resolver(monkeypatch: pytest.MonkeyPatch) -> None:
+    _stub_resolver(monkeypatch, "https://ws.example.com", "tok-123")
+    env = resolve_bench_env("oss")
+    assert isinstance(env, BenchRuntimeEnv)
+    assert env.db_profile == "oss"
+    assert env.base_env["OPENAI_BASE_URL"] == "https://ws.example.com/serving-endpoints"
+    assert env.base_env["OPENAI_API_KEY"] == "tok-123"
+    assert env.base_env["DATABRICKS_CONFIG_PROFILE"] == "oss"
+
+
+def test_ambient_openai_wins_and_skips_resolver(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setenv("OPENAI_BASE_URL", "https://ambient.example.com/serving-endpoints")
+    monkeypatch.setenv("OPENAI_API_KEY", "ambient-key")
+
+    def _boom(profile: str | None) -> _Creds:  # pragma: no cover - must not run
+        raise AssertionError("resolver must not be called when ambient OPENAI_* is set")
+
+    monkeypatch.setattr(
+        "omnigent.runtime.credentials.databricks.resolve_databricks_workspace", _boom
+    )
+    env = resolve_bench_env(None)
+    assert env.base_env["OPENAI_API_KEY"] == "ambient-key"
+    assert env.base_env["OPENAI_BASE_URL"] == "https://ambient.example.com/serving-endpoints"
+
+
+def test_config_profile_used_when_no_flag(monkeypatch: pytest.MonkeyPatch) -> None:
+    # No --profile, but ~/.omnigent config yields a profile (like `omni run`).
+    monkeypatch.setattr(runtime_env, "_profile_from_config", lambda: "from-config")
+    seen: dict[str, str | None] = {}
+
+    def _resolver(profile: str | None) -> _Creds:
+        seen["profile"] = profile
+        return _Creds("https://cfg.example.com", "cfg-tok")
+
+    monkeypatch.setattr(
+        "omnigent.runtime.credentials.databricks.resolve_databricks_workspace", _resolver
+    )
+    env = resolve_bench_env(None)
+    assert seen["profile"] == "from-config"
+    assert env.db_profile == "from-config"
+    assert env.base_env["OPENAI_API_KEY"] == "cfg-tok"
+
+
+def test_explicit_profile_overrides_config(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setattr(runtime_env, "_profile_from_config", lambda: "from-config")
+    seen: dict[str, str | None] = {}
+
+    def _resolver(profile: str | None) -> _Creds:
+        seen["profile"] = profile
+        return _Creds("https://x", "t")
+
+    monkeypatch.setattr(
+        "omnigent.runtime.credentials.databricks.resolve_databricks_workspace", _resolver
+    )
+    resolve_bench_env("explicit")
+    assert seen["profile"] == "explicit"  # flag wins over config
+
+
+def test_skip_reason_none_when_ambient(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setenv("OPENAI_BASE_URL", "https://a/serving-endpoints")
+    monkeypatch.setenv("OPENAI_API_KEY", "k")
+    assert bench_creds_skip_reason(None) is None
+
+
+def test_skip_reason_when_no_creds_anywhere() -> None:
+    # _clean_env stubbed _profile_from_config -> None and cleared OPENAI_*.
+    reason = bench_creds_skip_reason(None)
+    assert reason is not None
+    assert "--profile" in reason
+
+
+def test_skip_reason_when_profile_hostless(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setattr(
+        "omnigent.harness_bench.creds_lookup.lookup_databricks_host", lambda p: None
+    )
+    reason = bench_creds_skip_reason("typo-profile")
+    assert reason is not None
+    assert "typo-profile" in reason
Author	SHA1	Message	Date
Pat Sukprasert	edef6354ab	feat(cli): add `omni bench` to run the harness capability bench Register a `bench` subcommand on the unified CLI that delegates to the bench's argparse main (`omnigent.harness_bench.__main__.main`) via an ignore-unknown-options pass-through, so `omni bench` and `python -m omnigent.harness_bench` share one arg surface (`omni bench --help` shows the bench's own options). The import is lazy inside the command body, so it adds no CLI startup cost and no import cycle. Flip the bench's live/offline gate to match `omni run`: a live run no longer requires --profile. It turns on whenever creds are resolvable — a --profile, a configured ~/.omnigent profile, or ambient OPENAI_* (via bench_creds_skip_reason from Stage 2) — and --no-live forces the offline declared matrix. Updated the --profile / --live help text accordingly. Add "bench" to _CLICK_SUBCOMMANDS (the ad-hoc-detection sync set) and CLI tests covering registration, verbatim arg delegation, and exit-code propagation. Docs (README + design doc) lead with `omni bench`. 194 CLI tests pass; bench suite green; ruff clean; no lockfile drift. Co-authored-by: Isaac	2026-07-09 18:06:35 +08:00
Pat Sukprasert	d9cdff2742	feat(harness-bench): derive env like `omni run`; --profile now optional The bench used to always mint its own bearer via a `databricks auth token` subprocess (which did not handle OAuth profiles) and required --profile. Replace that with resolve_bench_env(), which mirrors `omni run`'s credential layering: 1. ambient OPENAI_BASE_URL + OPENAI_API_KEY win (skip resolution entirely), 2. else the profile from --profile, else the ~/.omnigent/config.yaml auth:/profile block (what `omni run` reads), 3. compose OPENAI_* via the canonical resolve_databricks_workspace() (OAuth-aware, fail-loud on a typo'd profile). So a no-flag run now derives creds exactly like `omni run`, and --profile overrides. A new bench_creds_skip_reason() gives every driver's unavailable() a cheap, token-free gate: a run skips cleanly when no creds are resolvable instead of requiring a flag. - SharedFullServer takes a BenchRuntimeEnv (was db_profile: str); __enter__ drops _mint_bearer + lookup_databricks_host and uses env.base_env. - FullServerDriver / NativeTuiDriver / SdkInprocDriver resolve via resolve_bench_env; databricks_profile is now Optional throughout (the --profile override, None = derive). run_bench keeps the kwarg for back-compat. - The full-server agent spec and the native provider-config omit executor.profile / the auth: block when auth came from the ambient env. Note: this drops the bench-only #1781 stale-token strip (env -u DATABRICKS_TOKEN). That is intentional -- `omni run` uses the same resolver and does not strip either; aligning with omni is the goal. New tests/harness_bench/test_runtime_env.py covers the layering (ambient wins, --profile overrides config, config-derived, no-creds skip, hostless profile). 80 passed / 18 skipped; ruff clean; e2e still collects (376). Co-authored-by: Isaac	2026-07-09 18:06:12 +08:00
Pat Sukprasert	cdb986e855	style(harness-bench): single import style for manifest in test_bench Code-quality review flagged that test_bench.py imported omnigent.harness_bench.manifest both ways: `from ...manifest import OFFICIAL_PROFILES` at the top and `import ...manifest as man` inside two tests (an artifact of the Stage-1 tests.->omnigent. rewrite). Switch the two in-test statements to `from omnigent.harness_bench import manifest as man`, keeping the `man` alias and all monkeypatch logic intact. No functional change. Co-authored-by: Isaac	2026-07-09 18:05:21 +08:00
Pat Sukprasert	37e5e1c83a	Merge branch 'main' into harness-bench-package-move	2026-07-09 17:54:16 +08:00
Pat Sukprasert	3c043f5425	refactor(harness-bench): move bench into omnigent package (ships in wheel) Relocate tests/harness_bench/ -> omnigent/harness_bench/ so the bench ships in the wheel and a future `omni bench` command works for any install (today pyproject packages only omnigent; tests/ is not in the wheel). Pure relocation, no behavior change. Sever the three test-tree dependencies so the shipped package has no pytest/tests import: - cli_unavailable_reason -> new omnigent/harness_bench/cli_probe.py; the e2e gate re-imports it (one copy). - The SDK-wrap seed data (harness/model/marker/cli_binary/env_prefix), which the capability registry does not carry, moves to a plain-data table in omnigent/harness_bench/seed.py (SDK_SEEDS). manifest.py seeds OFFICIAL_PROFILES from it; tests/e2e/_harness_probes.py rebuilds HARNESS_PROBES from the same table (wrapping model through resolve_model), so the bench and e2e matrices keep their single source of truth. - Inline the (inert-outside-CI) tests._helpers.compat helpers; a shipped bench never runs in the version-compat mode they guard. - lookup_databricks_host -> package-local creds_lookup.py (Stage 2 replaces it with resolve_databricks_workspace). The bench's own test_.py stay under tests/ (repo ships zero test_.py in omnigent/), repointed to omnigent.harness_bench.. Add a per-file ruff ignore for omnigent/harness_bench/ mirroring the tests/ waivers (uniform probe signature ARG002, capability-neutral boundary catches BLE001). Docs updated to the new path / python -m omnigent.harness_bench. Verified: python -m omnigent.harness_bench --list renders; 73 bench tests pass, 18 skip; tests/e2e collects clean (376 tests); ruff check + format clean. Co-authored-by: Isaac	2026-07-09 17:19:30 +08:00