test(harness-bench): derive declared matrix from the capability model

The bench hand-maintained a second copy of 'what each harness supports'
(manifest._P0_ALL_SUPPORTED verdicts + _STATIC auth/implementation). Make
it derive from the canonical harness_capabilities() (PR #1847) so there is
one source of truth, and the bench's job sharpens to 'does the harness do
what it publicly claims?'.

- Group A (descriptive columns): implementation from integration_mode, auth
  from auth, via small enum->prose maps.
- Group B (capability-backed verdicts): streaming from capabilities.streaming
  (True->SUPPORTED deltas, False->PARTIAL complete-only), interrupt from
  capabilities.interrupt, model_override from model_env_keys() membership.
- Group C (probe-only, kept explicit): basic_turn, tool_calling, policy_deny.
  policy_deny is enforcement, NOT the elicitation ASK surface — deliberately
  not derived from the elicitation axis.
- Deleted _P0_ALL_SUPPORTED and the derivable _STATIC dict.
- Tolerates sparse capabilities (community plugins): a harness with no
  declared capabilities gets only the probe-only dims, no KeyError.
- reconcile() phrasing now reads DRIFT as 'declared capability vs observed
  behavior' — the capability table is self-enforcing.

Reads the STATIC harness_capabilities(), not the runtime Executor.supports_*
methods (different layers). Verified live on oss: openai-agents (SDK) and
codex (CLI-subprocess) reconcile with no unexpected DRIFT on
streaming/interrupt/model_override; offline 17 passed, ruff+pre-commit clean.

tests/harness_bench/manifest.py

@@ -1,70 +1,125 @@
-"""The registry of official harness bench profiles — the spreadsheet, as data.
+"""The registry of official harness bench profiles.
 
-Each entry declares the static matrix columns and the *expected* verdict
-per P0 dimension for one official SDK harness. The base fields (model,
-env_prefix, marker, cli_binary) are reused from
-``tests.e2e._harness_probes.HARNESS_PROBES`` so a harness added to the e2e
-parametrize matrix flows into the bench without a second source of truth.
+Each profile's descriptive columns and *declared* verdicts derive from the
+canonical capability model (:func:`omnigent.harness_plugins.harness_capabilities`),
+so there is a single source of truth for "what each harness supports". The
+base fields (model, env_prefix, marker, cli_binary) are reused from
+``tests.e2e._harness_probes.HARNESS_PROBES`` — a harness added to the e2e
+parametrize matrix flows into the bench without a second copy.
 
-Declared verdicts encode the SDK support matrix. When a live probe
-observes something different, :func:`tests.harness_bench.verdict.reconcile`
-flags ``DRIFT`` — the whole point of the bench.
+The declared matrix is the harness's *published capability*; the bench's
+probes measure live behavior. When they disagree,
+:func:`tests.harness_bench.verdict.reconcile` flags ``DRIFT`` — which means a
+harness's capability declaration is false. That makes the capability table
+self-enforcing.
 
-Native harnesses and the remaining SDK harnesses (cursor, antigravity,
-kimi, qwen, goose, copilot, hermes) are phase-2: they need transport
-drivers and profile entries, tracked in the design doc.
+Axis mapping (see ``designs/harness-capabilities-bench-seam.md``):
+
+- **Group A — descriptive columns** derive from capabilities:
+  ``implementation`` from ``integration_mode``, ``auth`` from ``auth``.
+- **Group B — declared verdicts** derive where a capability backs the probe:
+  ``interrupt`` from ``capabilities.interrupt``, ``streaming`` from
+  ``capabilities.streaming``, ``model_override`` from membership in
+  ``model_env_keys()`` (the SDK model-override registry).
+- **Group C — probe-only** dimensions have no backing capability axis and
+  stay explicit: ``basic_turn`` (every harness completes a turn),
+  ``tool_calling`` (not a modeled axis), and ``policy_deny`` (enforcement,
+  distinct from the elicitation ASK surface — deliberately NOT derived from
+  ``elicitation``).
+
+Non-P0 harnesses' ``interrupt``/``streaming`` are declared best-effort by
+integration mode and not yet probe-verified; the bench's live probes confirm
+or correct them as transport coverage lands.
 """
 
 from __future__ import annotations
 
+from omnigent.harness_capabilities import AuthModel, HarnessCapabilities, IntegrationMode
+from omnigent.harness_plugins import harness_capabilities, model_env_keys
 from tests.e2e._harness_probes import HARNESS_PROBES, HarnessProbe
 from tests.harness_bench.profile import BenchProfile
 from tests.harness_bench.verdict import Verdict
 
-# All P0 SDK harnesses declare the same verdicts: they stream, call tools,
-# interrupt, enforce policy verdicts, and accept a model override. Drift
-# against this baseline is the signal we care about.
-_SUPPORTED = Verdict.SUPPORTED
-_P0_ALL_SUPPORTED: dict[str, Verdict] = {
-    "basic_turn": _SUPPORTED,
-    "streaming": _SUPPORTED,
-    "tool_calling": _SUPPORTED,
-    "interrupt": _SUPPORTED,
-    "policy_deny": _SUPPORTED,
-    "model_override": _SUPPORTED,
+# ── Group A: enum → prose for the descriptive columns ────────────
+
+_INTEGRATION_MODE_PROSE: dict[IntegrationMode, str] = {
+    IntegrationMode.SDK_IN_PROCESS: "SDK in-process",
+    IntegrationMode.CLI_SUBPROCESS: "CLI subprocess",
+    IntegrationMode.ACP_SUBPROCESS: "ACP subprocess",
+    IntegrationMode.NATIVE_TUI: "Native TUI",
+    IntegrationMode.NATIVE_SERVER: "Native server",
+}
+
+_AUTH_PROSE: dict[AuthModel, str] = {
+    AuthModel.OMNIGENT_CREDENTIAL: "Omnigent credential (gateway / provider config)",
+    AuthModel.OWN_AUTH: "Own auth (vendor login / API key)",
+    AuthModel.SESSION_SCOPED_CONFIG: "Session-scoped vendor config",
 }
 
 
-# Static matrix columns per official harness, keyed by harness name. Kept
-# beside the declared verdicts so the rendered report reproduces the
-# spreadsheet's descriptive columns, not just the ✓/✗ grid.
-_STATIC: dict[str, dict[str, str]] = {
-    "claude-sdk": {
-        "owner": "",
-        "auth": "Anthropic key / Databricks gateway",
-        "implementation": "SDK in-process",
-    },
-    "codex": {
-        "owner": "",
-        "auth": "Databricks gateway / codex auth.json",
-        "implementation": "CLI subprocess (app-server RPC)",
-    },
-    "pi": {
-        "owner": "",
-        "auth": "Databricks gateway / API keys",
-        "implementation": "CLI subprocess (JSONL RPC)",
-    },
-    "openai-agents": {
-        "owner": "",
-        "auth": "Databricks gateway / OpenAI key",
-        "implementation": "SDK in-process",
-    },
+# ── Group C: probe-only dimensions with no backing capability ────
+#
+# These stay explicitly SUPPORTED for the official (P0) harnesses: every one
+# completes a turn, calls tools, and enforces a policy DENY. They are NOT
+# derived from any capability axis (see the module docstring / seam brief).
+_PROBE_ONLY_DECLARED: dict[str, Verdict] = {
+    "basic_turn": Verdict.SUPPORTED,
+    "tool_calling": Verdict.SUPPORTED,
+    "policy_deny": Verdict.SUPPORTED,
 }
 
 
+def _implementation_prose(caps: HarnessCapabilities | None) -> str:
+    """Group A: the ``implementation`` column from ``integration_mode``."""
+    if caps is None:
+        return ""
+    return _INTEGRATION_MODE_PROSE.get(caps.integration_mode, caps.integration_mode.value)
+
+
+def _auth_prose(caps: HarnessCapabilities | None) -> str:
+    """Group A: the ``auth`` column from ``auth``."""
+    if caps is None:
+        return ""
+    return _AUTH_PROSE.get(caps.auth, caps.auth.value)
+
+
+def _declared_from_capabilities(harness: str) -> dict[str, Verdict]:
+    """Build a harness's declared verdicts from the capability model.
+
+    Group B (capability-backed) plus group C (probe-only, explicit).
+    Tolerant of a harness with no declared capabilities (a sparse
+    ``harness_capabilities()`` — e.g. a community plugin): the
+    capability-backed dimensions are simply omitted (left ``UNKNOWN`` by
+    :meth:`BenchProfile.declared_for`) rather than raising.
+
+    :param harness: Harness id, e.g. ``"codex"``.
+    :returns: A ``{dimension: Verdict}`` map for this harness.
+    """
+    declared: dict[str, Verdict] = dict(_PROBE_ONLY_DECLARED)
+
+    caps = harness_capabilities().get(harness)
+    if caps is not None:
+        # streaming: True → deltas (SUPPORTED); False → complete-only (PARTIAL).
+        declared["streaming"] = Verdict.SUPPORTED if caps.streaming else Verdict.PARTIAL
+        # interrupt: True → SUPPORTED; False → UNSUPPORTED.
+        declared["interrupt"] = Verdict.SUPPORTED if caps.interrupt else Verdict.UNSUPPORTED
+
+    # model_override is backed by the model-env-key registry (the SDK
+    # model-override set), not a capability field: a harness with a
+    # HARNESS_<H>_MODEL env key accepts a caller-specified model.
+    if harness in model_env_keys():
+        declared["model_override"] = Verdict.SUPPORTED
+
+    return declared
+
+
 def _profile_from_probe(probe: HarnessProbe) -> BenchProfile:
-    """Build an official :class:`BenchProfile` from an e2e ``HarnessProbe``."""
-    static = _STATIC.get(probe.harness, {})
+    """Build an official :class:`BenchProfile` from an e2e ``HarnessProbe``.
+
+    Descriptive columns and declared verdicts derive from the capability
+    model; only the transport and the e2e base fields are bench-local.
+    """
+    caps = harness_capabilities().get(probe.harness)
     return BenchProfile(
         harness=probe.harness,
         model=probe.model,
@@ -72,20 +127,22 @@ def _profile_from_probe(probe: HarnessProbe) -> BenchProfile:
         marker=probe.marker,
         cli_binary=probe.cli_binary,
         transport="sdk-inproc",
-        owner=static.get("owner", ""),
-        auth=static.get("auth", ""),
-        implementation=static.get("implementation", ""),
-        declared=dict(_P0_ALL_SUPPORTED),
+        owner="",
+        auth=_auth_prose(caps),
+        implementation=_implementation_prose(caps),
+        declared=_declared_from_capabilities(probe.harness),
     )
 
 
-# Official harnesses the bench ships with. Built from HARNESS_PROBES so the
-# two matrices never diverge; restricted to the harnesses the sdk-inproc
-# driver covers today.
+# Official harnesses the bench ships with: the P0 SDK harnesses the
+# sdk-inproc driver covers today. Built from HARNESS_PROBES so the e2e and
+# bench matrices never diverge.
+_OFFICIAL_HARNESSES = frozenset({"claude-sdk", "codex", "pi", "openai-agents"})
+
 OFFICIAL_PROFILES: dict[str, BenchProfile] = {
     probe.harness: _profile_from_probe(probe)
     for probe in HARNESS_PROBES
-    if probe.harness in _STATIC
+    if probe.harness in _OFFICIAL_HARNESSES
 }
 
 

tests/harness_bench/verdict.py

@@ -119,21 +119,26 @@ class ProbeResult:
 
 
 def reconcile(observed: Verdict, declared: Verdict) -> Verdict:
-    """Compare an observed verdict against the harness's declared verdict.
+    """Compare observed behavior against the harness's *declared capability*.
 
-    Returns :attr:`Verdict.DRIFT` when both sides assert a concrete fact
-    and those facts differ — the alarm the whole bench exists to raise
-    (a harness that *claims* a capability but no longer exhibits it, or
-    the reverse). Otherwise returns *observed* unchanged.
+    The declared verdict is derived from the harness's published capability
+    model (``harness_capabilities()``); the observed verdict is what a probe
+    measured live. Returns :attr:`Verdict.DRIFT` when both sides assert a
+    concrete fact and those facts differ — i.e. **the harness's capability
+    declaration is false** (it claims a capability it does not exhibit, or
+    exhibits one it does not claim). This makes the capability table
+    self-enforcing: a wrong entry in the model surfaces as DRIFT on the next
+    live run. Otherwise returns *observed* unchanged.
 
-    Drift is symmetric on purpose: a capability that regressed
-    (declared ``SUPPORTED``, observed ``UNSUPPORTED``) and one that
-    quietly gained coverage (declared ``UNSUPPORTED``, observed
-    ``SUPPORTED``) both mean the matrix is now lying, and both deserve a
-    human's attention.
+    Drift is symmetric on purpose: a declared capability that is not observed
+    (declared ``SUPPORTED``, observed ``UNSUPPORTED``) and an observed
+    behavior that was not declared (declared ``UNSUPPORTED``, observed
+    ``SUPPORTED``) both mean the declaration is out of sync with reality, and
+    both deserve a human's attention.
 
-    :param observed: The verdict a probe produced this run.
-    :param declared: The verdict the :class:`BenchProfile` claims.
+    :param observed: The verdict a probe measured this run.
+    :param declared: The verdict derived from the harness's declared
+        capability.
     :returns: ``DRIFT`` on a concrete mismatch, else *observed*.
     """
     if observed in _CONCRETE and declared in _CONCRETE and observed != declared: