Files
omnigent-ai--omnigent/tests/test_codex_native_bridge.py
Corey Zumar 1732faf3f3 feat: one harness-truth source for every model surface (listings, defaults, reports, confirmed switching) (#5022)
* feat(host): answer pre-launch model listings by probing the real harnesses

The pre-launch pickers were fed by catalog reconstruction — for a
Databricks-gateway codex host, serving-endpoint name enumeration: id
spellings the gateway's codex surface does not route, chat-only traps
(gpt-oss), no display names or effort ladders. The harness itself is
the only authority on what its /model picker would offer, so the host
now asks the harnesses:

- codex-native: probe_codex_model_options boots codex app-server with
  the SAME Databricks materialization a session launch gets (shared
  _databricks_launch_materialization, extracted from
  build_codex_native_server so the two cannot drift), a persistent
  probe CODEX_HOME (codex's own models_cache ETag makes refreshes
  cheap), and passes model/list rows through verbatim with a single
  default marker (launch pin first, else codex's own). Scoped to
  Databricks-profile launches; everything else — and every probe
  failure — falls open to the existing catalog path unchanged.
- claude-native: session launches (and the probe) now opt in to Claude
  Code's gateway model discovery
  (CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1 in the ucode env; the
  fetch 404s harmlessly until the gateway serves /v1/models).
  probe_claude_gateway_models runs claude -p "/model" with the launch
  env so the harness executes its own discovery, then reads the
  harness-written gateway-models.json artifact — no discovery
  semantics replicated. Rows union with the configured tier rows,
  exact-id deduped. The nonessential-traffic kill-switch is stripped
  from the probe env (Claude treats it as covering discovery).
- claude-sdk: SDK-mode claude is a pass-through client with no catalog
  of its own, so the endpoint listing is the harness truth — served
  via the existing list_models_for_worker in the exact wire spelling
  the SDK sends.

Serving stays off the probe path: a new host-side cache
(omnigent/host/model_options_cache.py) keys results by a resolved-
config fingerprint, serves stale-while-revalidating with single-flight
probes, and is prewarmed per tunnel connection — measured 65ms at the
REST route warm, ~1.3s joining the prewarm probe cold. The
model-options frame is now answered from a tracked task instead of
inline on the tunnel receive loop (a cold probe there stalled every
frame — same class as a83cc707); a filesystem frame answered in 22ms
mid-probe. The REST route also stops dropping the routable_models the
frame already carries (openapi regenerated).

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* feat(host): always probe Claude Code itself for the model list

claude -p "/model" makes the harness print its own alias enumeration
headlessly, so the curated static subscription list demotes from
first resort to failure fallback. probe_claude_gateway_models
generalizes to probe_claude_model_options: it runs for every config
shape (bare subscription launches included), parses the printed
"Available:" aliases verbatim (no alias names known to the parser, so
new Claude releases flow through), and still reads the discovery
artifact when the env opts in. The host lane serves configured tier
rows (the rich spelling for pinned aliases) unioned with the
harness's printed aliases and discovered gateway rows, exact-id
deduped; the configured/static rows stand alone only when the probe
itself fails.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* feat(host): probe Codex for every launch shape, not just Databricks routing

Same mandate as the Claude lane: the harness always answers. The probe
drops its Databricks-profile gate — a non-profile launch boots codex
app-server with whatever -c overrides the launch resolved (provider
routing, the dismissal pin, or nothing) and reads model/list verbatim,
so subscription/CLI-login and custom-provider shapes get Codex's real
visible catalog instead of the static curated list (the stale
hyphenated-id class of bug) or the raw enumeration. With no
launch-pinned model, Codex's own default marker stands. The legacy
catalog paths remain solely as the probe-failure fallback, pinned by
the existing handler tests now running with a failing probe stub.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* feat(host): resolve Claude model aliases to concrete versions in the picker

The probed alias list answered WHICH aliases exist but not what they land
on — 'opus' could be Opus 5.0 or 4.8 and the picker couldn't say. Ask the
harness that too: each printed alias gets its own headless
'--model <alias> -p /model' run in stream-json mode, whose init event
carries the exact resolved id and whose printed 'Current model:' line
carries the human label (only the effort suffix stripped). Rows become
{id: alias, model: exact id, displayName: 'alias — label'}; the web
picker already renders displayName, so no frontend change.

Resolution runs share the enumeration run's invocation assembly so the
two cannot drift, fan out under one bounded budget (startup dominates
and stretches with box load — measured 0.7s-17s for the same command —
so one wave covers a whole alias set), and fail per-alias back to the
bare row, never the probe. Live run resolves all 10 aliases in ~6s and
surfaces facts worth not guessing: fable[1m] resolves to plain
claude-fable-5, and 'best' pins to Fable rather than Opus.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* refactor(host): show Claude picker rows as resolved versions only

Presentation pass on the resolved alias rows: the display is the
harness's resolved label alone (the alias prefix was noise), 1M-context
resolutions always say '(1M context)' even where the harness's label
omits it (sonnet[1m] prints just 'Sonnet 5'), the 'default' alias never
becomes a row (the picker renders its own Default choice, so it was a
duplicate), and aliases resolving to an earlier row's exact (model,
label) are dropped — which removes 'best' and 'fable[1m]' as the
duplicates of fable's row they currently are, without hardcoding any
alias name. Launch ids are untouched; only displayName and row
membership change.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* refactor(host): dedupe Claude picker rows by resolved model alone

opusplan resolves to claude-sonnet-5 — a model the sonnet row already
lists — so the same duplicate-model rule that removes best and
fable[1m] now covers it: one picker row per resolved model, no alias
names hardcoded. A composite-mode alias would reappear only if it ever
resolved to a model no other alias offers.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* refactor(models): drop curated picker fallbacks; sessions ride the probe

The release-curated picker stand-ins in model_fallbacks are gone — the
live harness probes are the source of truth everywhere, and a path that
cannot probe now reports nothing rather than a plausible-but-stale list
(the codex entries even carried hyphenated spellings codex itself does
not use). Smart Routing's tables stay: rankings, arm menus, and probed
exclusions are router contract data no discovery API can provide, and
the ownership test now guards those records.

Companions so nothing regresses to empty:
- The subscription sonnet_5 pick degrades to Claude's own 'sonnet'
  alias instead of hunting a static list — the harness resolves it.
- The claude-sdk pre-launch lane rides the claude probe whenever the
  endpoint listing is empty (the SDK drives the claude CLI, so the
  CLI's aliases are its truth on subscription boxes).
- Existing sessions now match the new-session picker: the runner's
  claude-model-options endpoint resolves configured rows ∪ probe once
  per session via the new shared claude_model_options_with_probe (the
  host lane uses the same composition, so the two cannot drift),
  answering 503-pending while the probe is in flight (the server fetch
  already retries those) and falling back to configured rows past a
  grace so the catalog is never empty.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(web): friendly composer label for Claude [1m] aliases off-catalog

The composer chip prefers the session catalog's display name, but a
Claude bracket alias the catalog doesn't list (a pick made before the
catalog carried the row, e.g. on a session launched by an older runner)
fell through to the raw id — 'sonnet[1m] High'. Render that case as
'Sonnet (1M context)': title-cased family plus the context marker, no
version claimed, since only the harness knows which Sonnet the alias
lands on. Catalog hits keep the probed display name verbatim.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(web): version-agnostic 'Sonnet' fallback label kills the 4.6 flash

Cold-loading a claude session painted the composer chip 'Sonnet 4.6'
for the window before the session catalog arrived, then corrected to
'Sonnet 5' — the fallback label list pinned a version that only the
harness can know (reproduced via Playwright: 'Sonnet 4.6 High' at
3.96s → 'Sonnet 5 High' at 4.70s). The fallback now says just
'Sonnet'; the catalog's display name supersedes it wherever one has
arrived, so the pre-catalog window shows a coarser label, never a
wrong one. Same honesty for the sandbox new-chat picker and the
scheduled-task model dropdown, which render the same list.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* refactor(web): retire two redundant uses of the local Claude alias list

The composer chip's pre-catalog fallback now formats alias-shaped ids
mechanically (title-case family, '_N' → ' N', '[1m]' → ' (1M context)')
instead of looking them up in CLAUDE_NATIVE_MODELS — same rendering,
zero model knowledge. The sticky-model compatibility check collapses to
session-catalog membership alone: its isClaudeNativeModel conjunct was
subsumed by the catalog check it was AND-ed with, and would have
rejected catalog rows whose ids don't look Claude-ish even though the
session's own catalog offered them. The now-orphaned guard is deleted;
the list itself stays for the genuinely hostless surfaces (sandbox
picker, unpinned scheduled tasks, schema enums).

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* test(e2e_ui): composer label stays version-free until the catalog speaks

Covers the label behaviors the model-listing work changed: with the
session catalog held back, the composer chip renders the alias
mechanically ('sonnet[1m]' → 'Sonnet (1M context)'), and only the
arriving catalog upgrades it to its display name ('Sonnet 5 (1M
context)'). Every painted label is recorded via a MutationObserver so a
transient raw id or invented version ('Sonnet 4.6') cannot hide from a
retrying expect().

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* feat(web): offer Smart Routing in the in-session gear for native panes

The in-session composer gear withheld the Smart Routing model option
from native Claude Code / Codex sessions under a stale premise ('their
CLI bakes the model at launch') — the server has routed native panes
per turn via /model injection since the create-time gear gained the
option, and validates routing-on creates with a per-family rule. The
in-session gate now mirrors that exact rule: a router must answer for
the session's family — the external AI-Gateway router only when the
host runs the family through the gateway (read off the session's host
row; absent rows fail open like the landing), the built-in judge
anywhere. SDK/bundle sessions keep their existing flag-only gate.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* style(web): prettier over the routing-gate and label changes

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* test(runner): sys_list_models subscription row is an honest empty listing

The curated claude stand-ins are gone from the static subscription
path; the dispatch test now pins the empty-models shape with the
probing note, matching the model-catalog contract.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(web): composer gear rides the host probe until the session catalog lands

A fresh codex session's gear showed a sparse Model row and no Effort row
for ~15s: effort levels come from the session catalog's
supportedReasoningEfforts, and that catalog only resolves once codex
app-server answers model/list. The session's host already probed the
same harness for the new-chat picker, so the gear (and the composer
chip) now falls back to those cached rows — same ids the launch accepts,
~90ms warm — whenever the session's own catalog is empty; the runner's
per-session catalog supersedes them the moment it arrives. Claude
sessions get the same pre-catalog Model list for free (their effort
levels were already static).

Verified live on a fresh codex session: Effort visible 0.5s after
create+load with the session catalog still empty, offering the host
row's levels.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(web): keep model-options identity stable when the host fallback is idle

The pre-catalog host fallback returned a fresh empty array whenever the
session catalog was empty and no host rows existed — for EVERY session
shape, native or not. That new identity per render re-rendered each
options consumer (composer, gear, agent-info popover) on every
streaming/liveness tick, which under CI load tipped the agent-info
hover-open grace race (shard 1 failed the same popover test twice).
Substitute only when host rows actually exist; otherwise the store's own
stable array reference flows through untouched, restoring the exact
pre-fallback behavior for every session the feature doesn't apply to.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(claude-native): apply picked model aliases verbatim, never the default

Picking Fable in a gateway session's composer switched the pane to
Opus: resolve_claude_native_model_selection swapped an unpinned family
alias for the provider's default model (a degrade from the era when
the picker always showed every family alias), the vocabulary re-spelled
that default as its pinned alias, so the runner injected '/model opus'
for a 'fable' pick — and the statusLine mirror then recorded the wrong
model as the session override. Bracket aliases had sibling failures:
'/model sonnet[1m]' 503'd on a pinned env (no spelling for it) and
silently dropped the [1m] marker on a bare login (family-segment
step-down).

Picker rows are pin-backed or probe-vouched now, so a pick passes
through verbatim and Claude owns resolution:
- the resolver's no-pin gateway degrade is gone (an out-of-band
  unpinned pick now fails visibly at inference instead of silently
  running the default);
- bracket variants of the family aliases are their own /model
  arguments in the vocabulary — the harness enumerates them itself;
- the configured∪probe union drops probe rows whose resolved model is
  a bare canonical Anthropic id on an endpoint that routes its own ids
  only: the pick could never work there, so the row is not offered
  (a pinned family resolves to the endpoint's spelling and stays).

Reproduced at the runner layer (events → resolver → injected command):
picking 'fable' asserted '/model fable' and got '/model opus' before
the fix. An e2e_ui guard pins that the web PATCHes the picked row id
verbatim — the client layer was innocent.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(claude-native): mirror pane model switches in the catalog's vocabulary

Live verification of the verbatim-alias fix exposed the last surface in
the same family: after a web pick of 'sonnet[1m]' correctly switched the
pane, the statusLine mirror collapsed the observed model back to the
LEGACY picker vocabulary — 'databricks-claude-sonnet-5[1m]' became
'sonnet_5' — stomping the just-saved override with an id the session's
catalog doesn't list (and which a relaunch would resolve through the
custom-tier branch, silently dropping the 1M context).

_model_alias_for now speaks the catalog's row ids: 1M resolutions keep
their bracket marker ('sonnet[1m]'), and the legacy 'sonnet_5' opt-in
row is mirrored only on a config whose custom slot actually pins it —
read off the session's launch pins — since everywhere else the generic
sonnet row IS that model. This also restores the designed web→TUI
round-trip no-op: the mirrored alias now equals the persisted override,
so the server-side dedupe skips the write.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(codex-native): report the model a Default launch actually runs

A Default codex launch names no model, so nothing pinned the session's
config.toml — yet the profile still resolved a concrete model and passed it
as `-c model=`, which outranks the config copied from the user's shared
~/.codex home. The pane ran the resolved model while the session reported
the shared file's leftover one: the create dialog promised
"Default (GPT-5.6-Luna)" and the session then said GPT-5.4.

Pin the profile-resolved model in codex's own spelling, so the forwarder
mirror and the cost gate read the model this session runs. Mark that model
as the catalog default too — codex's own isDefault is its built-in
preference and named GPT-5.6-Sol on a session running Luna, which also fed
the composer gear an effort ladder the running model rejects.

Web side: fold catalog and codex spellings when resolving a session's model
onto a picker row, and stop borrowing the default row's effort levels for an
unresolved model.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* test(runner): keep the codex model-options test off the real launch

Session create launches Codex for real, and that launch owns the bridge
dir: it clears the state and its forwarder task rewrites both the state
and CODEX_HOME/config.toml after the response returns. On a machine
where Codex and a Databricks profile resolve, that wiped the seeded
state no matter which side of create seeded it, so the endpoint answered
503. Stub the launch; the endpoint, the bridge-state read, the
CODEX_HOME read, and the fake app-server client all stay real.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(web): name Codex's Default the same in both model gears

The composer gear and the new-session gear each built their own copy for
the Model row, so one session read a bare "Default" in the composer and
"Default (gpt-5.6-luna)" on the landing page, and the landing page
listed raw catalog ids where the composer listed display names. Neither
gear told the user which model Codex would actually run.

Move both labels into HarnessConfigControls next to the sentinels they
belong to and read them from there in both callers. Row ids are
untouched, so picks still submit the harness's own spelling.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(claude-native): read the custom slot instead of guessing the model row

The terminal->web mirror mapped a concrete model id onto a picker row by
looking for a family name inside the id, so a routed Opus 4.9 landed on
the `opus` row that holds 4.8: the web showed the wrong model, and posting
that row back stepped the session off its launch pin. Resolve rows by
exact comparison against the launch pins instead, and read Claude Code's
one custom model slot to name its row rather than inferring it from the
model's spelling. A `[1m]` resolution stays a distinct row from its
non-bracket sibling.

The legacy `sonnet_5` row id and the substring spellings it used to be
matched by move into claude_model_vocabulary with a 0.10.0 removal note;
the substring leg now runs only when the exact comparison misses.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* refactor(web): drop the substring model-row match the picker never calls

`isModelImplicitlySelected` guessed which picker row a bound model belonged
to by searching for the row id inside the model name, which is why `sonnet`
matched `sonnet-5` and needed a special case per generation. Its only
caller sat in the branch taken when a session has no server-supplied model
list, and every native picker kind is on that list, so the branch ran with
an empty list and the call could not select anything.

Delete the function and collapse the caller to the server-list path. The
two suites that covered it go with it; nothing else exercised it.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* docs(claude-native): say plainly that gateway model discovery never fires

The launch env asks Claude Code to discover the gateway's model inventory,
and the comment claimed the only thing holding it back was a gateway that
did not serve `/v1/models` yet. The gateway serves it now, but the same env
sets CLAUDE_CODE_USE_GATEWAY, and the CLI fires that fetch only on its
first-party provider path — so the artifact is never written and the rows
read from it are always empty.

Name that in all three places a reader lands: the flag, the probe's
env-unset list (popping the nonessential-traffic switch is not enough), and
the artifact read itself.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* test(model-flows): add the harness-truth e2e suite, red-first

The model-flows design lands test-first: this suite encodes the target
behavior for every flow (pre-launch picker, default labels, create→launch
pane truth, gear parity, confirmed switching, terminal-side mirroring) and
is deliberately red today in the ways the analysis measured.

Two tiers. The hermetic tier drives the real SPA over the spawned server
with the session snapshot shaped at the browser edge and SSE frames pushed
through a captured stream controller; it runs in the normal e2e_ui lane.
The live tier (`live_model_flows` marker, opt-in via
OMNIGENT_E2E_MODEL_FLOWS=1) boots a real server + host from any checkout —
OMNIGENT_E2E_MODEL_FLOWS_REPO selects which, so the identical tests
produce the red-on-main matrix — flips provider shapes the way setup
writes default claims, launches real claude/codex TUIs, and asserts pane
truth over tmux.

Recorded pre-implementation: hermetic 4 red / 2 guard-green (the design's
predicted set exactly); live rows 1 and 5 red against unmodified main
(the frozen "Sonnet 4.6" static list; the empty/erroring codex pre-launch
answer).

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* feat(models): verbatim reported_model as the single display authority

Landing-order step 3 of the model-flows design. Sessions gain a
reported_model — the model the harness last said it is actually on, in
the harness's own spelling — stored as a new key in the existing
session_overrides blob (no DDL) and served on the snapshot's llm_model
field with precedence reported ?? spec model. external_model_change
writes and dedupes against it verbatim; the user's request
(model_override) is untouched, because requests and reports are separate
roles and only reports are ever displayed.

The claude forwarder now posts the status file's model byte-for-byte:
the alias-collapse mapper (_model_alias_for / _custom_slot_row_id) is
deleted — collapsing a routed Opus 4.9 onto the opus row holding 4.8 is
the bug class this kills — and the first-observation-silent-seed rule is
gone, so the launch's own model reports within seconds of spawn and the
composer is never blank-forever. The codex forwarder already posted raw
ids and needed no change.

The web renders and highlights models from the reported value alone:
exact id/model match against the catalog, with an off-catalog report
appended as its own raw row rather than relabeled onto a same-family
row. The sticky model becomes a pure preference — the silent bind-time
and delayed-catalog model_override PATCHes are removed (they wrote
requests the pane was never asked to honor), and session.model events
land on llmModel instead of the picker selection. Cost attribution
prefers the reported model too.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* feat(models): explicit launches from the shared catalog store

Landing-order step 4. Every native launch now pins its model explicitly,
resolved through the new on-disk catalog store
(omnigent/model_catalog_store.py — one probe result under a
launch-config fingerprint, read by every consumer), so nothing is left
to invisible CLI-private state and a stale config line can never govern
a session.

Claude: the enumeration probe runs in stream-json and captures its own
init-event model — the truthful Default — so claude_model_catalog marks
exactly one isDefault row (appending an off-list default, e.g. a
settings.json pin, as its own launchable row; never appending a bare
Anthropic spelling on an endpoint that rejects it). A Default
subscription launch passes --model with that default; an explicit
request is validated against the catalog and fails the launch loudly
when the list no longer carries it. The runner also records the launch
vocabulary onto the bridge after config resolution
(record_model_vocabulary), closing the model_env gap that made
mid-session /model conversion read the runner's ambient env.

Codex: the session-shaped probe home now links the account's real
auth.json (the catalog must answer for the account that will run — the
Sol-promised/Terra-offered mismatch dies here), a Default launch on
codex's own login resolves the account's real default instead of
inheriting the copied config line (the stale-gpt-5.4 400 class), and
build_codex_native_server emits -c model= alongside the config-copy pin
from one resolved value on every shape. A guard test pins the
argv/config-pin agreement across all provider shapes; on the profile
shape the file deliberately keeps codex's own spelling and the guard
asserts same-model rather than same-bytes (addendum).

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Serve every model surface from the shared harness catalog

One catalog file per (harness, launch-config fingerprint) now backs the
pre-launch picker, launch resolution, and the in-session gear:

- Host: catalog-backed model-options handlers with a concurrent, detached
  boot prewarm; probe failures answer ok+[] plus an error string the web
  new-session dialog displays. The in-memory ModelOptionsCache module is
  removed.
- Runner: unified GET /v1/sessions/{id}/model-options (harness-named
  routes stay as deprecated aliases until 0.11.0); the claude route waits
  briefly on the store's single-flight probe (503-pending past that) and
  the codex route writes live listings back to the store.
- Server: model-options loads go unified-first and fall back to the
  legacy route on 404; the hosts API forwards the host's error string.
- Deletions: static claude alias table, gateway-discovery artifact
  machinery, the configured-union composition, and the host's codex
  catalog reconstruction lanes.
- Tests isolate the catalog store per test so suites cannot touch the
  developer's real ~/.omnigent cache or boot real harness CLIs.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Confirm model switches through the harness before claiming them

Switching is now ask -> pending -> harness-confirmed on every lane:

- Runner (claude): after typing /model, verify against the statusLine
  snapshot the forwarder already polls (10s budget) — expected spellings
  come from the session's own catalog rows; a pane that never switches
  answers 503 so the server surfaces the swallowed-dialog case instead
  of the row silently claiming the pick. A shape with no snapshot stays
  unverifiable-but-successful.
- Runner (codex): the awaited thread/settings/update RPC is the
  confirmation; a missing Codex bridge now answers 503 instead of a
  silent 204, and plan-mode updates re-assert the reported model rather
  than a stale override.
- Server: the visible model_change_not_applied notice now carries the
  runner's own detail string.
- Web: a transient pendingModelChange marks the ask (spinner beside the
  composer chip); the chip keeps the reported model until session.model
  confirms, and the not-applied error (or a switch/bind) settles the
  indicator.

Also repairs tests/runner/conftest.py's REAL_CLAUDE_LAUNCH_CATALOG
export, which the previous commit's lint autofix stripped after its
consumers had been verified.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Name the true Default on every picker and pin gateway models

- The web's claude lane keeps the catalog's isDefault marker and all
  Default rows (new-chat select, its summary line, the session gear)
  label through the one shared defaultModelLabel — both harnesses now
  read "Default (X)" where X is the model a bare launch actually runs.
- Provider entries' models map (the existing flat tier keys — opus,
  sonnet, haiku, fable — beside default) now pins the claude alias
  vocabulary: the launch env derives ANTHROPIC_DEFAULT_*_MODEL from the
  declared tiers, models.default pins its own family when that family
  has no explicit key, and the declared ids become the config's
  routable set. Aliases on gateway endpoints resolve inside the
  gateway's own catalog instead of falling back to canonical Anthropic
  ids the gateway rejects.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Make the live model-flow rig trustworthy end to end

The step-8 before/after pass surfaced six defects in the live CUJ
scaffolding itself; with them fixed the suite is 16/16 green against the
implementation branch and red against main in the documented modes:

- Drop tests/conftest.py's inherited OMNIGENT_DISABLE_CATALOG_LOOKUP for
  the rig's spawned server/host — the databricks catalog was empty only
  inside the rig.
- Wait for the post-create navigation with page.wait_for_url: the sync
  Playwright API pumps events only inside playwright calls, so the old
  time.sleep poll read a page.url frozen at the landing route forever.
- Re-read a model dropdown opened during the host's boot-probe warm-up
  until rows (or the settled error) appear.
- Resolve a codex session's private CODEX_HOME through the bridge's own
  state.json (the dir is named by a runner-generated bridge id).
- Row 17: no Escape after a Radix select pick (it closes the whole gear
  modal), pick an effort that differs from the machine's global default,
  and poll for persistence while the browser is still open (the save's
  model leg holds until the pane confirms, so the effort PATCH is sent
  by the page seconds later).
- Snapshot and restore ~/.claude/settings.json around the suite: the
  real /model switches run under the real HOME and Claude persists every
  switch as the developer's global default.

Also: useHostModelOptions retries with backoff so a picker opened during
the boot-probe warm-up fills in when the single-flight probe completes
instead of pinning the transient error until reopened.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Wait out the boot-probe warm-up when reading Default labels

The codex launch pin now resolves through live Unity-Catalog discovery
(seconds on a cold host), so a landing model label read immediately
after opening the config renders the bare sentinel while the web's
retry loop is still filling the catalog. Give the label the same
warm-up wait the dropdown read already has.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Mark a model ask pending before its PATCH, not after

The PATCH is held open while the runner drives and confirms the switch,
so the harness's session.model report usually arrives before the PATCH
resolves. Setting pendingModelChange from the response overwrote the
report's clear and stranded the spinner until the hygiene timer. The ask
is now marked pending up front (and cleared if the PATCH throws); a
store test pins the report-beats-PATCH ordering.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Type full model ids verbatim on unpinned claude sessions

Picking a full-id catalog row (e.g. the appended default
'Opus 4.8 (1M context)') stepped down to '/model opus' — the family
alias resolves to claude's CURRENT generation, silently switching to
Opus 5 instead. The confirm layer caught and surfaced it; the
translation now passes claude-* full ids verbatim on envs with no alias
pins (claude's /model accepts full ids — the probe resolves them the
same way), while pinned envs keep exact-pin-or-fail-loud.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Defer mid-turn model switches instead of failing them

A /model typed during an active turn queues in Claude's composer and
applies when the turn settles — past the 10s confirm window — so the
runner surfaced a false 'was not switched' error for a switch still on
its way, and the injection's short dialog watch could leave the late
confirm dialog parked on the pane.

The confirm loop now answers the switch dialog whenever it renders
inside the window, and a timeout with the pane mid-turn answers success:
a detached watcher keeps answering the late dialog (hint-matched Enter
only, never blind; bounded budget) and the forwarder's verbatim report
settles the picker when the switch lands. An idle-pane timeout — the
genuine swallowed case — still fails loud.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Mark a provider launch pin as the claude catalog default

Default launches on provider-configured shapes pass --model
<config.model> explicitly, so the pin — not the enumeration run's own
model — is what a Default launch actually runs. The gateway-entry shape
(one pinned alias row) went unmarked when the enumeration reported no
default, leaving the picker on a bare 'Default'. Subscription shapes
keep the enumeration-derived marker, and the appended default row only
borrows the probe's printed label when it names the same model.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Merge origin/main and green the CI suite

The merge brought main's error-pill restyle, per-frame create refactor,
and codex live UC discovery alongside this branch's model-flow work.
Fixes to keep every suite green:

- Restore the SimpleNamespace import main added to tests/host/test_connect
  (the merge dropped it → ruff F821 + 2 NameErrors).
- Regenerate openapi.json for the reported_model wording (session.model
  event + llm_model field descriptions).
- Update the smart-routing-create catalog tests to expect the unified
  /model-options route the server now asks first (legacy alias is the
  404 fallback).
- Update the runner pending-catalog test: a provider shape's launch pin
  is appended as the marked default row.
- Adapt row15's e2e to main's collapsed error pill (expand to read the
  detail); move the routed-modal test's seed to llm_model (routed models
  arrive as the harness report now); pick codex landing options by their
  decorated display name (codex options now render display names like
  claude — the design's decorated-rows contract).
- Seed the in-session gear from the session's request only before any
  harness report exists, so a routed session names its model.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Make row15's error-pill expand retry-safe under suite load

The single headline click could land before the disclosure handler was
wired when the suite ran the pill under load, leaving the detail
collapsed and the assertion timing out. Retry the expand until the
detail shows — same as a person clicking again.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Null-guard the native model-id fold so cursor rows don't blank the page

findNativeModelOption's fold fallback (added when the codex catalog fold
moved client-side) called comparableModelId on option.model/id without a
null guard. Cursor picker rows arrive as { id, displayName } with
model === null on the wire (typed model?: string), and the
option.model !== undefined check let null through — comparableModelId(null)
then threw 'Cannot read properties of null (reading trim)' during render,
blanking the whole chat page for any cursor-native session.

comparableModelId is now null-safe (empty fold never matches a real
target) and the fallback rejects null ids/models. Regression test covers
a cursor-shaped options list with null models.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Add a per-harness e2e_ui render-smoke matrix (all native pickers)

One hermetic case per native model-picker harness (claude, codex, cursor,
kiro, opencode, pi): shape a seeded session as that harness with its
realistic model_options — including rows with an explicit model: null
(cursor/kiro/opencode's real wire shape, typed model?: string) and a
hostile null-id row — then render the session, open the gear, and assert
the composer renders, the model control lists the rows, and no uncaught
null-deref fires.

The null-model harness cases carry a non-matching model_override so the
model-id fold actually runs (an exact-id match would return before it),
which is precisely the path that once blanked the page. Validated red on
the pre-fix bundle (cursor/kiro/opencode crash) and green after — the
coverage the earlier per-harness tests missed by using model-omitted
(undefined) rows instead of the null shape.

Signed-off-by: dbczumar <corey.zumar@databricks.com>

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
2026-08-20 16:24:23 -07:00

411 lines
16 KiB
Python

"""Tests for native Codex bridge state helpers."""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from omnigent.codex_native_bridge import (
CodexNativeBridgeState,
cancel_pending_mcp_startup,
clear_active_turn_id_if_matches,
clear_bridge_state,
codex_home_for_bridge_dir,
codex_mcp_config_overrides,
mcp_startup_waiting_detail,
pending_mcp_servers,
prepare_bridge_dir,
read_bridge_startup_error,
read_bridge_state,
read_codex_config_model,
read_codex_home_config_model,
read_mcp_startup,
read_policy_hook_config,
settle_pending_mcp_startup,
update_mcp_server_startup,
write_bridge_startup_error,
write_bridge_state,
write_codex_config_model,
write_policy_hook_config,
)
def test_codex_mcp_config_overrides_isolate_the_bridge_interpreter(tmp_path: Path) -> None:
"""codex launches serve-mcp with ``-I`` so the workspace can't shadow omnigent.
The MCP server starts in the session workspace, and without ``-I`` python puts
that cwd on ``sys.path``, so a workspace that is an omnigent checkout supplies
the bridge's own package. Every other native bridge passes ``-I`` here.
:param tmp_path: Stands in for the per-session bridge dir.
"""
overrides = codex_mcp_config_overrides(tmp_path)
prefix = "mcp_servers.omnigent.args="
raw = next(o[len(prefix) :] for o in overrides if o.startswith(prefix))
assert json.loads(raw)[:4] == ["-I", "-m", "omnigent.claude_native_bridge", "serve-mcp"]
def _seed_active_turn(bridge_dir: Path, active_turn_id: str | None) -> None:
"""
Write bridge state with a given active turn id.
:param bridge_dir: Native Codex bridge directory.
:param active_turn_id: Active turn id to seed, e.g. ``"turn_1"``,
or ``None`` for no running turn.
:returns: None.
"""
write_bridge_state(
bridge_dir,
CodexNativeBridgeState(
session_id="conv_test",
socket_path=str(bridge_dir / "app-server.sock"),
thread_id="thread_test",
codex_home=str(bridge_dir / "codex-home"),
active_turn_id=active_turn_id,
cwd=str(bridge_dir),
),
)
def test_bridge_state_preserves_native_working_directory(tmp_path: Path) -> None:
"""Bridge state retains the cwd used for web-driven Codex turns."""
_seed_active_turn(tmp_path, "turn_1")
state = read_bridge_state(tmp_path)
assert state is not None
assert state.cwd == str(tmp_path)
clear_active_turn_id_if_matches(tmp_path, "turn_1")
updated = read_bridge_state(tmp_path)
assert updated is not None
assert updated.cwd == str(tmp_path)
@pytest.fixture
def bridge_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
"""
Create an isolated bridge directory rooted under ``tmp_path``.
:param tmp_path: pytest temp directory.
:param monkeypatch: pytest monkeypatch fixture.
:returns: Prepared bridge directory.
"""
monkeypatch.setattr("omnigent.codex_native_bridge._BRIDGE_ROOT", tmp_path / "codex-native")
return prepare_bridge_dir("bridge_test")
def _write_config(bridge_dir: Path, body: str) -> None:
"""
Write a ``config.toml`` into the bridge's per-session ``CODEX_HOME``.
:param bridge_dir: Bridge dir whose ``codex-home/config.toml`` is written.
:param body: Raw TOML body, e.g. ``'model = "gpt-5.4"\\n'``.
"""
home = codex_home_for_bridge_dir(bridge_dir)
home.mkdir(parents=True, exist_ok=True)
(home / "config.toml").write_text(body)
def test_read_codex_config_model_returns_top_level_model(bridge_dir: Path) -> None:
"""The top-level ``model`` key (what /model writes) is returned.
This is the cost gate's source of truth read at evaluation time; if it
returned the wrong key or None, a ``/model`` downgrade would never take
effect on the next tool call.
"""
_write_config(bridge_dir, 'model_provider = "databricks"\nmodel = "gpt-5.4"\n')
assert read_codex_config_model(bridge_dir) == "gpt-5.4"
def test_read_codex_home_config_model_reads_a_codex_home_directly(bridge_dir: Path) -> None:
"""A ``CODEX_HOME`` path yields the same model as the bridge-dir reader.
The runner's model-options endpoint holds the live bridge state's
``codex_home``, not the bridge dir, and needs the session's model to
mark which catalog row this session actually launched on.
"""
_write_config(bridge_dir, 'model = "gpt-5.6-luna"\n')
assert read_codex_home_config_model(codex_home_for_bridge_dir(bridge_dir)) == "gpt-5.6-luna"
def test_read_codex_config_model_none_when_missing(bridge_dir: Path) -> None:
"""No ``config.toml`` → ``None`` (fail-safe), so the caller falls back."""
assert read_codex_config_model(bridge_dir) is None
def test_read_codex_config_model_none_when_no_model_key(bridge_dir: Path) -> None:
"""A config without a top-level ``model`` key → ``None`` (no invented value)."""
_write_config(bridge_dir, 'model_reasoning_effort = "medium"\n')
assert read_codex_config_model(bridge_dir) is None
def test_read_codex_config_model_none_when_unparsable(bridge_dir: Path) -> None:
"""Malformed TOML → ``None``, not a crash (guards a partial write)."""
_write_config(bridge_dir, 'model = "gpt-5.4\n[broken')
assert read_codex_config_model(bridge_dir) is None
def test_write_codex_config_model_replaces_top_level_key(bridge_dir: Path) -> None:
"""The existing top-level ``model`` line is replaced, sections untouched.
An Omnigent-initiated switch (routing / web picker) must land on the same
key an in-TUI ``/model`` writes, or the forwarder's next config re-read
mirrors the stale launch model back and reverts the switch.
"""
_write_config(
bridge_dir,
'model = "databricks-gpt-5-5"\n'
'model_provider = "databricks"\n'
"[model_providers.databricks]\n"
'model = "section-model-not-touched"\n',
)
assert write_codex_config_model(bridge_dir, "gpt-5.6-luna") is True
assert read_codex_config_model(bridge_dir) == "gpt-5.6-luna"
body = (codex_home_for_bridge_dir(bridge_dir) / "config.toml").read_text()
assert 'model = "section-model-not-touched"' in body
assert 'model_provider = "databricks"' in body
def test_write_codex_config_model_inserts_when_absent(bridge_dir: Path) -> None:
"""A config with no top-level ``model`` gains one at the top."""
_write_config(bridge_dir, 'model_provider = "databricks"\n')
assert write_codex_config_model(bridge_dir, "gpt-5.6-luna") is True
assert read_codex_config_model(bridge_dir) == "gpt-5.6-luna"
def test_write_codex_config_model_creates_missing_file(bridge_dir: Path) -> None:
"""No codex-home/config.toml yet → the writer creates it (best-effort)."""
assert write_codex_config_model(bridge_dir, "gpt-5.6-luna") is True
assert read_codex_config_model(bridge_dir) == "gpt-5.6-luna"
def test_policy_hook_config_round_trips(bridge_dir: Path) -> None:
"""
Written Omnigent coordinates read back verbatim for the policy hook.
The codex hook subprocess depends on this exact payload to reach the
Omnigent server. A failure (dropped/renamed field) would leave the hook
unable to POST, silently disabling enforcement.
"""
write_policy_hook_config(
bridge_dir,
ap_server_url="http://127.0.0.1:8787",
ap_auth_headers={"Authorization": "Bearer abc"},
)
config = read_policy_hook_config(bridge_dir)
assert config == {
"ap_server_url": "http://127.0.0.1:8787",
"ap_auth_headers": {"Authorization": "Bearer abc"},
}
def test_policy_hook_config_absent_returns_none(bridge_dir: Path) -> None:
"""
Reading before any write returns None (no Omnigent server configured).
The hook treats None as "nothing to enforce" and no-ops. A failure
(e.g. raising, or returning a partial dict) would crash the hook or
make it POST to a missing URL.
"""
assert read_policy_hook_config(bridge_dir) is None
@pytest.mark.parametrize(
("active_turn_id", "completed_turn_id", "expected_return", "expected_active_after"),
[
# Matching terminal: the active turn really ended → clear + report
# cleared, so the forwarder posts idle.
("turn_1", "turn_1", True, None),
# Stale terminal for an older turn while a newer one is live → ignore,
# leaving the newer turn intact (no premature idle).
("turn_1", "turn_2", False, "turn_1"),
# No-id terminal while a turn is live is ambiguous → ignore. This is
# the fix: clearing here posted a premature idle that hid the
# "working" spinner mid-turn while Codex kept streaming.
("turn_1", None, False, "turn_1"),
# No-id terminal with no active turn: nothing to protect → clear is a
# no-op and reports cleared (the session is already idle).
(None, None, True, None),
# Id terminal with no active turn: it matches nothing → ignore.
(None, "turn_1", False, None),
],
)
def test_clear_active_turn_id_if_matches(
bridge_dir: Path,
active_turn_id: str | None,
completed_turn_id: str | None,
expected_return: bool,
expected_active_after: str | None,
) -> None:
"""
Terminal events only clear the active turn when they belong to it.
Guards the spinner/steering invariant: a terminal event clears the
active turn (and lets the forwarder post idle) only when it matches
the live turn. A stale id, or an ambiguous id-less event while a turn
is live, must be ignored so a still-running turn is not marked idle.
:param bridge_dir: Isolated bridge directory fixture.
:param active_turn_id: Active turn id seeded before the call, e.g.
``"turn_1"``, or ``None`` for no running turn.
:param completed_turn_id: Terminal event's turn id, e.g. ``"turn_1"``,
or ``None`` when Codex omitted it.
:param expected_return: Expected ``clear_active_turn_id_if_matches``
return — ``True`` means the forwarder will post idle.
:param expected_active_after: Expected ``active_turn_id`` afterward.
:returns: None.
"""
_seed_active_turn(bridge_dir, active_turn_id)
result = clear_active_turn_id_if_matches(bridge_dir, completed_turn_id)
# Return value drives whether the forwarder posts idle. A wrong True on
# the (active="turn_1", completed=None) row is the spinner bug: idle
# posted mid-turn. A wrong False on the matching row would leave the
# spinner stuck on after the turn really ended.
assert result is expected_return
state = read_bridge_state(bridge_dir)
assert state is not None
# The cleared/preserved active turn id also governs steering: a turn
# wrongly cleared here means later web messages stop steering it.
assert state.active_turn_id == expected_active_after
def test_clear_active_turn_id_if_matches_no_state_returns_true(bridge_dir: Path) -> None:
"""
With no bridge state on disk, clearing is a no-op that reports cleared.
A missing state file means there is no turn to protect, so the helper
returns True (nothing to ignore). A failure (returning False) would
make the forwarder treat a normal terminal as stale and never post
idle, hanging the spinner.
"""
# bridge_dir exists (fixture) but no state.json was written.
assert clear_active_turn_id_if_matches(bridge_dir, "turn_1") is True
def test_bridge_startup_error_round_trips_and_is_cleared(bridge_dir: Path) -> None:
"""
The startup-error breadcrumb round-trips, and ``clear_bridge_state``
drops it before each launch so stale failures don't linger (issue #59).
"""
assert read_bridge_startup_error(bridge_dir) is None
write_bridge_startup_error(bridge_dir, "thread never started (TimeoutError)")
assert read_bridge_startup_error(bridge_dir) == "thread never started (TimeoutError)"
clear_bridge_state(bridge_dir)
assert read_bridge_startup_error(bridge_dir) is None
def test_mcp_startup_updates_round_trip(bridge_dir: Path) -> None:
"""
Per-server MCP startup updates accumulate and read back (issue #2058).
The executor's first-turn gate and the runner's Stop handler both key
off this map; the ``pending``/``waiting`` views must name exactly the
servers whose latest status is ``starting``.
"""
assert read_mcp_startup(bridge_dir) == {}
assert pending_mcp_servers({}) == []
assert mcp_startup_waiting_detail({}) is None
update_mcp_server_startup(bridge_dir, "safe", "starting")
update_mcp_server_startup(bridge_dir, "storage-console", "starting")
servers = update_mcp_server_startup(bridge_dir, "safe", "failed", error="handshake failed")
assert servers == read_mcp_startup(bridge_dir)
assert read_mcp_startup(bridge_dir) == {
"safe": {"status": "failed", "error": "handshake failed"},
"storage-console": {"status": "starting", "error": None},
}
# Only still-starting servers are pending; the failed one settled.
assert pending_mcp_servers(read_mcp_startup(bridge_dir)) == ["storage-console"]
assert (
mcp_startup_waiting_detail(read_mcp_startup(bridge_dir))
== "MCP startup still waiting on storage-console"
)
def test_cancel_pending_mcp_startup_flips_only_starting(bridge_dir: Path) -> None:
"""
Stop's local cancel flips ``starting`` servers to ``cancelled`` only.
Settled servers (ready/failed) must keep their state — rewriting them
would misreport what actually happened; a second cancel is a no-op so
a repeated Stop doesn't claim it cancelled anything.
"""
update_mcp_server_startup(bridge_dir, "safe", "ready")
update_mcp_server_startup(bridge_dir, "testman", "failed", error="boom")
update_mcp_server_startup(bridge_dir, "storage-console", "starting")
assert cancel_pending_mcp_startup(bridge_dir) == ["storage-console"]
assert read_mcp_startup(bridge_dir) == {
"safe": {"status": "ready", "error": None},
"testman": {"status": "failed", "error": "boom"},
"storage-console": {"status": "cancelled", "error": None},
}
# Nothing pending anymore → repeat cancel reports nothing flipped.
assert cancel_pending_mcp_startup(bridge_dir) == []
def test_settle_pending_mcp_startup_drops_only_starting(bridge_dir: Path) -> None:
"""
Settling drops unresolved ``starting`` entries and keeps terminal ones.
Codex never delivers per-server outcomes to Omnigent's observer
connection, so at settle the unresolved entries are removed rather
than guessed; a locally-cancelled server must survive so the web band
can keep saying it was cancelled. A second settle is a no-op.
"""
update_mcp_server_startup(bridge_dir, "safe", "starting")
update_mcp_server_startup(bridge_dir, "storage-console", "cancelled")
servers, changed = settle_pending_mcp_startup(bridge_dir)
assert changed is True
assert servers == {"storage-console": {"status": "cancelled", "error": None}}
assert read_mcp_startup(bridge_dir) == servers
# Fully settled → nothing to drop, nothing rewritten.
assert settle_pending_mcp_startup(bridge_dir) == (servers, False)
def test_read_mcp_startup_ignores_malformed_entries(bridge_dir: Path) -> None:
"""
Malformed or unknown-status entries are dropped on read.
A corrupt file must degrade to "no state" rather than crash the
executor gate or feed a bogus status into the web UI.
"""
(bridge_dir / "mcp_startup.json").write_text("not json")
assert read_mcp_startup(bridge_dir) == {}
(bridge_dir / "mcp_startup.json").write_text(
'{"servers": {"ok": {"status": "ready"}, "bad": {"status": "exploded"},'
' "": {"status": "ready"}}}'
)
assert read_mcp_startup(bridge_dir) == {"ok": {"status": "ready", "error": None}}
def test_clear_bridge_state_removes_mcp_startup(bridge_dir: Path) -> None:
"""
``clear_bridge_state`` drops the MCP startup map with the other
runtime state, so a relaunch never gates its first turn on a prior
app-server's startup round.
"""
update_mcp_server_startup(bridge_dir, "safe", "starting")
clear_bridge_state(bridge_dir)
assert read_mcp_startup(bridge_dir) == {}