72b4469d723e28b9bcd7e2e236b84e110bcaf9c0
77 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Harry Yao
|
72b4469d72 |
perf(host): open daemon readiness polls tight, then back off (#5190)
The host-online, runner-online and Claude-terminal-ready waits all polled on a flat 0.5s cadence. Those waits gate every native-harness launch and usually resolve on the first probe or two — a warm host is already online, a fresh runner connects in a second or two — so a flat cadence spends up to a full interval doing nothing after the thing is already ready. Replace the fixed sleep with `daemon_poll_intervals()`: open at 0.1s, grow geometrically, hold at the existing 0.5s for the long tail. Fast launches notice readiness sooner without a long wait hammering the server. `DAEMON_POLL_INTERVAL_S` keeps its name and value as the steady-state cap, so `connect.py`'s runner-exit watcher still matches it; its comment now notes the client's opening probes are tighter. Signed-off-by: harry-yao_data <harry.yao@databricks.com> Co-authored-by: harry-yao_data <harry.yao@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com> |
||
|
Edwin He
|
4b6779febb |
feat(auth): refreshable credential for unattended host daemons (#4743)
* feat(auth): refreshable credential for unattended host daemons Implements login-issued refresh grants so unattended hosts can renew their session tokens instead of crashing when the initial JWT expires. - Server: OIDC callback persists refresh material and issues a login-scoped renewal grant (30-day TTL, reuses device-grant store/rotation machinery) - CLI: load_token() calls refresh_stored_token() on expiry, mints a fresh session JWT from the grant via POST /oauth/token - Host: treats post-connection 401/403 as retryable-with-reauth (attempts token refresh before failing); improves error text for expired tokens - Auth: login grants (no scope) bypass the delegated-token allowlist, keeping full authority; delegated tokens stay scope-restricted - Tests: new coverage for refresh cycles, OIDC mode token router, env override of grant lifetime Fixes OMNI-1127 / closes #1953. Co-authored-by: Isaac Signed-off-by: Edwin He <41037314+Edwinhe03@users.noreply.github.com> * fix(auth): address review findings for refreshable host credential Fixes P0-P3 security and robustness issues discovered in review: P0 (FEATURE-BREAKING): create_redeemed_grant called self._session() with no query_name, causing TypeError and breaking the entire login-grant feature. Now uses "insert_redeemed_device_grant" per CLAUDE.md conventions. P1 (SECURITY): LoginRequest.issue_refresh was a client-controllable bool, allowing XSS/form-hijack to obtain 30-day unattended credentials via browser login. Removed the field entirely; browser /auth/login now NEVER issues refresh material — only CLI/device flows do (server-side enforcement). P2.1 (ROBUSTNESS): _check_cookie dropped isinstance(grant_id, str) guard, allowing malformed grant_id claims to reach _grant_revoked(). Restored guard. P2.2 (ROBUSTNESS): _store_entry assumed token file was dict but _load_entry guards with isinstance(data, dict). Mirror the guard on write to prevent TypeError on corrupt files. Treat non-dict as empty (fail-safe). P3.1 (CLEANUP): load_token's expiry warning said "attempting automatic refresh" but load_token never refreshes. Reworded to reflect actual behavior. P3.2 (CLEANUP): _make_client_secret_gate was built twice (once in create_device_auth_router, again in included create_oauth_token_router), duplicating env reads and logs. Pass gate as parameter to avoid rebuild. P3.3 (CLEANUP): _grant_max_lifetime_seconds() re-parsed os.environ on every refresh/purge. Now called once at router mount, captured in closure. Added regression tests: - test_redeemed_grant_persistence_regression: grant row must be created (catches P0 TypeError) - test_browser_login_never_issues_refresh_token: browser login must NOT return refresh_token (catches P1 client-controllable flow) Follow-up (test reconciliation + lint, from running the suite): - Re-point the login-grant round-trip and session-authority tests to mint via issue_login_grant (the CLI/device path) now that browser /auth/login no longer returns refresh material. - Fix pre-existing runner-entry test lag: load_token mocks now accept the min_remaining_seconds kwarg the factory passes, and drop a stray OMNIGENT_RUNNER_DELEGATED_AUTH that contradicted a test's documented no-delegation scenario. - Fix a latent NameError: _grant_max_lifetime was referenced in create_device_auth_router but only bound in create_oauth_token_router; resolve it once at mount in the device router too. - Remove the now-dead device_grant_store wiring from the accounts auth router (its only use was the removed issue_refresh path). Co-authored-by: Isaac Signed-off-by: Edwin He <41037314+Edwinhe03@users.noreply.github.com> * fix(host): escalate to a re-auth prompt on sustained post-connect auth rejection A host rejected with 401/403 after it has already connected retries forever, so a transient VPN or proxy drop self-heals. Until now it only logged "check your VPN/network", so a permanently-rejected credential (a revoked or expired grant) looped silently and never told the operator to re-authenticate. After a sustained streak it now escalates with a louder warning plus a stderr line naming the omnigent login command, re-emitted periodically. It stays retryable and never fatal, so a recoverable daemon is not killed. Co-authored-by: Isaac Signed-off-by: Edwin He <41037314+Edwinhe03@users.noreply.github.com> * test(cli): use one import style in the refresh test Drop the mixed import omnigent.cli_auth + from-import inside test_refresh_survives_unwritable_state_dir; call store_token/refresh_stored_token via the ca alias. Resolves the code-quality bot finding on the PR. Co-authored-by: Isaac Signed-off-by: Edwin He <41037314+Edwinhe03@users.noreply.github.com> * fix(cli): drop dead accounts issue_refresh and cover the refresh-factory wiring Accounts /auth/login issues no refresh material (only the OIDC CLI-ticket flow does), so the accounts-login POST no longer sends the ignored issue_refresh field, and its misleading "older servers ignore it" comment is removed. Updates the CLI accounts-login test that asserted the field. Also adds a runner-entry test proving the load->refresh->fallback auth-token factory returns the refreshed token when the stored OIDC token has lapsed but a refresh grant is present — the integration that actually keeps an unattended host alive, previously only unit-tested at the refresh function itself. Co-authored-by: Isaac Signed-off-by: Edwin He <41037314+Edwinhe03@users.noreply.github.com> * fix(server): mount login-grant token router only for UnifiedAuthProvider The elif that mounts /oauth/token for login grants when the device flow is off only checked for a grant store, leaving auth_provider typed as the base AuthProvider (pyrefly bad-argument-type at the create_oauth_token_router call) and — for a non-Unified custom provider with a store — a latent runtime failure in _resolve_signing_config. Guard the branch on isinstance(auth_provider, UnifiedAuthProvider), matching the sibling device-flow branch. Co-authored-by: Isaac Signed-off-by: Edwin He <41037314+Edwinhe03@users.noreply.github.com> * test(e2e): accept min_remaining_seconds in managed-runner load_token mocks The auth-token factory now calls load_token(url, min_remaining_seconds=...); two managed-runner e2e tests monkeypatched load_token with a lambda that rejected the kwarg, raising TypeError. Accept **_kw, matching the runner-entry unit mocks. Co-authored-by: Isaac Signed-off-by: Edwin He <41037314+Edwinhe03@users.noreply.github.com> --------- Signed-off-by: Edwin He <41037314+Edwinhe03@users.noreply.github.com> |
||
|
Lee moon soo
|
5689ef33ee |
perf(host): shorten tunnel recovery (#4981)
* perf(host): shorten tunnel recovery Signed-off-by: Lee moon soo <moonsoo.lee@databricks.com> * Preserve loopback reconnect tolerance Signed-off-by: Lee moon soo <moonsoo.lee@databricks.com> * Trigger CI retry Signed-off-by: Lee moon soo <moonsoo.lee@databricks.com> --------- Signed-off-by: Lee moon soo <moonsoo.lee@databricks.com> |
||
|
Corey Zumar
|
1732faf3f3 |
feat: one harness-truth source for every model surface (listings, defaults, reports, confirmed switching) (#5022)
* feat(host): answer pre-launch model listings by probing the real harnesses
The pre-launch pickers were fed by catalog reconstruction — for a
Databricks-gateway codex host, serving-endpoint name enumeration: id
spellings the gateway's codex surface does not route, chat-only traps
(gpt-oss), no display names or effort ladders. The harness itself is
the only authority on what its /model picker would offer, so the host
now asks the harnesses:
- codex-native: probe_codex_model_options boots codex app-server with
the SAME Databricks materialization a session launch gets (shared
_databricks_launch_materialization, extracted from
build_codex_native_server so the two cannot drift), a persistent
probe CODEX_HOME (codex's own models_cache ETag makes refreshes
cheap), and passes model/list rows through verbatim with a single
default marker (launch pin first, else codex's own). Scoped to
Databricks-profile launches; everything else — and every probe
failure — falls open to the existing catalog path unchanged.
- claude-native: session launches (and the probe) now opt in to Claude
Code's gateway model discovery
(CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1 in the ucode env; the
fetch 404s harmlessly until the gateway serves /v1/models).
probe_claude_gateway_models runs claude -p "/model" with the launch
env so the harness executes its own discovery, then reads the
harness-written gateway-models.json artifact — no discovery
semantics replicated. Rows union with the configured tier rows,
exact-id deduped. The nonessential-traffic kill-switch is stripped
from the probe env (Claude treats it as covering discovery).
- claude-sdk: SDK-mode claude is a pass-through client with no catalog
of its own, so the endpoint listing is the harness truth — served
via the existing list_models_for_worker in the exact wire spelling
the SDK sends.
Serving stays off the probe path: a new host-side cache
(omnigent/host/model_options_cache.py) keys results by a resolved-
config fingerprint, serves stale-while-revalidating with single-flight
probes, and is prewarmed per tunnel connection — measured 65ms at the
REST route warm, ~1.3s joining the prewarm probe cold. The
model-options frame is now answered from a tracked task instead of
inline on the tunnel receive loop (a cold probe there stalled every
frame — same class as
|
||
|
Dhruv Gupta
|
741f2d29e4 |
fix(host,runner): reconnect host + all sessions promptly on laptop wake (#5054)
On macOS sleep the OS freezes every Omnigent process and drops the network, killing the host control-channel WebSocket and every runner (session) tunnel. On wake nothing reconnected promptly: the only liveness signal was the websockets keepalive ping (30s interval / 90s timeout), so a half-open post-sleep socket took up to ~120s to be noticed — and the server had already deregistered the host, so the desktop app showed "disconnected" that whole time while the terminal (a one-time startup banner) still read "connected". Add omnigent/suspend_watch.py: watch_for_resume() detects a resume by polling a short interval and comparing wall-clock vs monotonic-clock drift. The monotonic clock freezes during sleep on macOS/Linux while the realtime clock keeps counting, so a resume shows up as a large divergence; a merely-blocked event loop advances both equally, so this never false-fires on CPU stalls. Uses time.monotonic (never loop.time, which under uvloop includes sleep on macOS and would zero out the divergence). Wire it into both reconnect loops: - Host (connect.py): a watcher aborts the live tunnel (ws.transport.abort()) on wake and flags a prompt reconnect, so run() reattaches at the base backoff instead of the escalated one (required on a loopback server, where an abrupt close is not auto-classified as a benign recycle). - Runner (serve.py): a per-connection watcher aborts the tunnel and notes the resume so serve_tunnel reconnects promptly; each session self-heals on its own event loop, so no host orchestration is needed. Result: opening the laptop reconnects the host and all its sessions within ~5s instead of up to ~2 minutes. Windows degrades to todays keepalive behavior (its monotonic clock counts suspend) with no regression. Tests: unit tests for the detector (fires once on divergence, never on a blocked loop, survives a raising callback) plus host and runner integration tests (a simulated wake aborts the live tunnel and forces a prompt reconnect). Co-authored-by: Isaac Signed-off-by: Dhruv Gupta <dhruv.gupta@databricks.com> |
||
|
Corey Zumar
|
05eaa253d1 |
fix(host): retry Databricks auth refresh (#5014)
Signed-off-by: dbczumar <corey.zumar@databricks.com> |
||
|
Zeyi (Rice) Fan
|
03c7907966 |
fix(host): keep capability probes out of tunnel handshake (#4769)
## Related issue Closes [OMNI-2964](https://linear.app/omnigent/issue/OMNI-2964/fix-host-tunnel-connection-issue-when-it-fails-to-detect-hanress) ## Summary - Move harness and gateway capability discovery out of reconnect handshakes, bound startup discovery, and degrade probe failures to visible warnings with unknown metadata. - Add a backward-compatible `host.connection_error` frame so accepted tunnels can surface server-side setup failures with their stage and retryability. - Make background startup wait for the existing server-side host status before reporting success and retain reconnect regression coverage. ELI5: checking which agent CLIs are installed is optional setup information. A broken CLI should not prevent the host from introducing itself to the server, so the host now connects with that information marked unknown and refreshes it later. ```text host startup ── capability probe ──┬─ success → cached metadata └─ failure/timeout → warning + unknown │ ▼ WebSocket upgrade → host.hello → connected receive loop ▲ server setup failure → host.connection_error ``` ## Test Plan - `uv run pytest tests/host/test_frames.py tests/server/integration/test_host_tunnel_route.py tests/host/test_connect.py tests/host/test_cli_host.py -q` - `uv run pytest tests/host/test_connect.py::test_silent_connect_streak_escalates_and_slows_reconnects tests/host/test_connect.py::test_inbound_frame_resets_silent_connect_streak -q` - `uv run ruff check` on all changed Python and test files. - `uv run pyrefly check omnigent/host/connect.py omnigent/host/frames.py omnigent/server/routes/host_tunnel.py omnigent/cli.py` ## Demo N/A — backend/CLI reliability change with no visual UI. ## Type of change - [x] Bug fix - [ ] Feature - [ ] UI / frontend change - [ ] Refactor / chore - [ ] Docs - [ ] Test / CI - [ ] Breaking change ## Test coverage - [x] Unit tests added / updated - [x] Integration tests added / updated - [ ] E2E tests added / updated - [ ] Manual verification completed - [ ] Existing tests cover this change - [ ] Not applicable ## Coverage notes Automated coverage exercises capability exceptions and timeouts, server error propagation, background registration checks, retryability, and silent reconnect backoff. ## Changelog `omnigent host` now stays connected when optional harness detection fails and surfaces server-side tunnel setup errors. Signed-off-by: Zeyi (Rice) Fan <zeyi.f@databricks.com> |
||
|
Tomu Hirata
|
adcf83ccb6 |
feat(pi): Add searchable model picker for new sessions with Databricks Unity AI Gateway OAuth (#4961)
* feat(pi): add searchable start model picker Signed-off-by: Anthony Ivan <anthony.ivan@databricks.com> * fix(pi): harden model picker compatibility Signed-off-by: Anthony Ivan <anthony.ivan@databricks.com> * refactor(pi): simplify model picker filtering Signed-off-by: Anthony Ivan <anthony.ivan@databricks.com> --------- Signed-off-by: Anthony Ivan <anthony.ivan@databricks.com> Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> Co-authored-by: Anthony Ivan <anthony.ivan@databricks.com> |
||
|
Tomu Hirata
|
c2439c6fd7 |
fix(windows): pass Windows process essentials through harness env filters (#4886)
On native Windows, `agent_env.BASE_ALLOW_EXACT` (the shared deny-by-default env filter used by all harness executors) did not include SYSTEMROOT, COMSPEC, USERPROFILE, or the other Windows-mandatory constants. Any harness CLI spawned via `clean_agent_env` (codex, pi, claude-sdk, antigravity, …) died instantly on spawn because Winsock/crypto cannot initialise without SYSTEMROOT — the subprocess exited before reading stdin, causing the executor to await a JSON-RPC response that never arrived and silently idle to the 600s watchdog. The constant set already existed as `WINDOWS_ENV_PASSTHROUGH` in `_platform.py` and was already wired into `os_env._DEFAULT_ENV_PASSTHROUGH` and `connect._RUNNER_ENV_ALLOWLIST`. This commit adds it to `BASE_ALLOW_EXACT` so every harness executor inherits it automatically, matching the pattern used elsewhere. Also fixes three related Windows issues surfaced in omnigent-ai/omnigent#4851: - `PYTHONUTF8` was not forwarded through `_RUNNER_ENV_ALLOWLIST`, so the host daemon / runner subprocess printed Unicode status chars (✓ ↑) on the Windows ANSI code page (cp1252), raising `UnicodeEncodeError` and killing the host tunnel in an infinite reconnect loop. - `_session_create_validation.validate_existing_host_workspace` and `_workspace_validation.validate_workspace` required `workspace.startswith("/")`, rejecting every Windows drive-letter path (C:\…) from a connected Windows host. Windows absolute paths matching `^[A-Za-z]:[/\\]` are now accepted. - `harness_install._harness_cli_version_satisfies` returned `False` on `packaging.version.InvalidVersion`, so pre-release versions like `0.146.0-alpha.9.2` (newer than the declared floor) were reported as too-old and the harness was refused at the version gate. The fix extracts the leading X.Y.Z segment as a fallback for non-PEP-440 strings. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> |
||
|
Corey Zumar
|
f75c07ba56 |
perf(host): cut host-launched session-create latency (#4752)
Creating a claude-native session from the web UI paid three serial, avoidable costs between the create POST and the terminal appearing: - The host tunnel handled inbound frames strictly serially, so every create's launch frame queued behind that create's own background host.model_options CLI exec (650-794ms measured), and workspace validation's host.stat (2-9ms uncontended) queued behind landing-page prefetches for up to 1.3s. Frames now run on their own tasks; launch/stop keep arrival order via a lifecycle lock; a crashing handler is contained instead of tearing down the tunnel. - Terminal auto-create resolved ambient provider credentials (a ~0.7s `claude auth status` subprocess on macOS) inside the user-visible "Starting up..." window. The host now stamps the session's harness into the runner env, and claude-native runners prewarm the detection at boot, overlapping it with tunnel connect; the resolve consumes it one-shot. Other harnesses pay nothing. - The first launch of a daemon's life paid the runner zygote's one-time import (~1.5s) inline. run() now pre-starts the zygote at daemon boot via a helper shared with the launch path. Same rig, pristine main vs this change: workspace validation 1508-5003ms -> 2-5ms; launch-frame queueing 1185-1712ms -> 8-17ms; first-launch zygote import 1532ms -> 0ms; click->chat-page-open 1.6-2.0s -> 0.18-0.24s; click->"Starting up..." cleared 5.9-8.3s -> 3.6-4.9s. Signed-off-by: dbczumar <corey.zumar@databricks.com> |
||
|
Edwin He
|
9e5b8741d2 |
feat(cli): route host-scoped requests to the replica holding the host's tunnel (#4185)
Adds a per-host routing key (the ``X-Databricks-Omnigent-Slice-Key`` header) so that, on a horizontally-scaled multi-tenant deployment, every request scoped to a given host or session lands on the replica that holds that host's runner tunnel: the host's control tunnel, its runners' tunnels, and all of a session's turn/resource/stream traffic converge on one replica when they carry the same key (the host_id). On an unsharded / single-replica deployment the key is never emitted, so this is a no-op there. Client-side only. The key is built centrally in ``cli_auth.databricks_request_headers`` (gated on the workspace-hosted mount) and threaded through the one factory ``open_server_client`` plus ``_remote_headers`` / ``open_daemon_client``. Callers pass a host_id when they have one; runner-side callers (forwarders, permission checks) inherit it automatically from the ``OMNIGENT_RUNNER_SLICE_KEY`` env var the host stamps at runner launch, so no per-callsite change is needed there. The WebSocket attach handshake and its reconnects carry the same key. ``chat._remote_headers`` gains a ``host_id`` keyword (defaulting to ``None`` so probes and health checks are unaffected). ``_DatabricksTokenAuth`` resolves the session's host per request from the session→host map and can be repointed via ``pin_session`` when a client outlives its session (e.g. a ``--fork`` in the REPL lands under a new conversation id on a new host). Session-host state is always written on attach — clearing a stale mapping when the server reports no host matters as much as setting one. A ``tests/cli`` conftest fixture isolates the runner machine's own host identity so "no slice key on this call" assertions are hermetic regardless of whether the box running the suite is itself a host. Signed-off-by: Edwin He <41037314+Edwinhe03@users.noreply.github.com> |
||
|
Dhruv Gupta
|
59bedc1fac |
fix(host): keep the session workspace off the runner's sys.path (OMNI-2963) (#4688)
* fix(host): keep the session workspace off the runner's sys.path (OMNI-2963)
Opening a session inside an omnigent checkout ran a different omnigent than
the installed one. Runners are spawned with `python -m`, which prepends the
process cwd to sys.path, and since
|
||
|
Corey Zumar
|
5cd1f1c8cb |
fix(host): silent-endpoint backoff, slow-boot adoption, zygote respawn (#4563)
Four host-side fixes from a host-log forensics pass: - An endpoint that accepts the WS upgrade but never sends a frame no longer spins on the 0.5s recycle cadence forever (observed: ~6s cycles for 7 hours, silently): past 10 consecutive accepted-but- silent connections the host logs one ERROR, notifies the terminal once, and drops to normal backoff until a frame arrives. - ensure_local_omnigent_server no longer strands a slow-booting child: while the process is alive the readiness wait extends to a 120s boot ceiling (a ~39s first boot was observed failing the old 45s cutoff), and a final failure terminates and reaps the child before raising — previously it cleared the pidfile and left the server running, untracked. - A runner zygote that died mid-life is reaped and respawned on the next launch instead of latching _zygote_disabled for the daemon's life; start failures and alive-but-broken channels still disable it. - Self-allocated process logs that never received a record are swept at exit, and host shutdown awaits the reaper/watcher cancellations. Signed-off-by: dbczumar <corey.zumar@databricks.com> |
||
|
Harry Yao
|
a2de2b44ac |
fix(host): keep CLAUDE_CODE_USE_GATEWAY / ENABLE_TOOL_SEARCH in the runner env allowlist (#4553)
A launcher (e.g. Databricks' isaac) sets CLAUDE_CODE_USE_GATEWAY=1 and ENABLE_TOOL_SEARCH=true in its process env so the native-claude harness keeps MCP tool search on (schemas load on demand). But the host daemon env (`_build_host_daemon_env`) and the runner env (`_build_runner_env`) are both built from `_RUNNER_ENV_ALLOWLIST`, and neither var was on it — so they were stripped at daemon spawn and never reached the runner process. The native-claude provider path (`_provider_config_for_native_claude`, `_ucode_config_for_profile`, `_bedrock_config_for_native_claude`) reads CLAUDE_CODE_USE_GATEWAY from os.environ to decide whether to set CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1. With it stripped, the runner saw it absent, re-added the disable flag, and Claude Code turned tool search off — loading every MCP tool schema eagerly (~88k tokens for ~190 MCP tools at startup instead of on demand). Add both non-secret boolean flags to `_RUNNER_ENV_ALLOWLIST`, beside the existing CLAUDE_CODE_USE_BEDROCK / CLAUDE_CODE_SKIP_BEDROCK_AUTH flags (same category). The single allowlist is consulted by both gates, so the vars now survive daemon spawn and runner spawn and reach the guard. Tests: assert both vars survive `_build_host_daemon_env` (local + remote) and `_build_runner_env`. They fail before this change and pass after. Co-authored-by: harry-yao_data <harry.yao@databricks.com> |
||
|
Corey Zumar
|
b1e775e1c5 |
fix(host): exit after sustained connection-refused against a loopback server (#4544)
A host whose loopback server died reconnected forever at the 10s backoff cap — zombie 'omnigent host' processes looped for days against dead local ports. Connection-refused on loopback means nothing listens and no network path can recover, so after 30 consecutive refusals (~5 minutes at the cap) the host now logs one clear ERROR and exits through the same fail-loud path as permanent auth failures. Dual-stack refusals (asyncio's combined 'Multiple exceptions' OSError or exception groups) count only when every sub-error is refused; any successful connect or non-refused error resets the streak. Remote server URLs are unaffected and retry indefinitely so network outages recover. Signed-off-by: dbczumar <corey.zumar@databricks.com> |
||
|
Dhruv Gupta
|
43762a9892 |
fix(host): forward SSH_AUTH_SOCK to runners and harness CLIs (#4377)
Every runner-spawned context lost the ssh-agent socket, so any agent doing git-over-SSH or SSH-cert-authenticated tooling failed with "dial unix: missing address" (often surfacing as a confusing 401 from the endpoint, since such tools have no cached-token fallback). Two independent gates dropped it: - `_build_runner_env` filters the host env through `_RUNNER_ENV_ALLOWLIST`, which omitted SSH_AUTH_SOCK. This is also the list both host-daemon modes consult, so the one entry fixes the daemon hop too, including remote mode. - `clean_agent_env` is the shared deny-by-default filter for every vendor CLI, and its safe base omitted it. Fixing the shared base covers all seven harnesses rather than only the one whose report surfaced this. Classified as a path, not a bearer secret: it names a unix socket, and reaching the agent behind it still requires the user's own ssh-agent to be running and holding the key. Same footing as KUBECONFIG, already allowlisted. An ACTIVE OS sandbox deliberately keeps excluding it: that boundary exists to confine the agent, and signing with the user's keys is what it confines. `os_env.py` previously justified its exclusion by calling the variable "a credential surface masquerading as a path", which contradicts the classification above; that rationale is rewritten to rest on the sandbox boundary instead, so the codebase states one position. Downstream paths needed no change: `sys_os_shell` (sandbox inactive) and `sys_terminal_launch` both mirror the parent env, so they inherit the fix. Codex's `shell_environment_policy.inherit` was reported as a third gate requiring omnigent to force `inherit="all"`. It does not reproduce: on codex-cli 0.144.3 the default already passes SSH_AUTH_SOCK through (identical 72-var env), and only an explicit `inherit="core"` drops it. Forcing `all` would override that deliberate user choice, so no override is added. Co-authored-by: Isaac |
||
|
David O'Keeffe
|
3419de8da6 |
fix(host): run session runners in the workspace, not the daemon's cwd (#3974)
* fix(host): run session runners in the workspace, not the daemon's cwd A host daemon started from a directory that later disappears (a temp checkout, a removed worktree) passes that dead cwd to every runner it spawns. Path.cwd() then raises FileNotFoundError inside the runner and native sessions fail with "Native Pi terminal failed to start" — hit live while verifying the pi-native gateway fix. Spawn the runner with cwd=<session workspace>, which _build_runner_env already documents as the runner's cwd and which is verified to exist just above the spawn. Signed-off-by: David O'Keeffe <david.okeeffe@databricks.com> * chore: retrigger CI (flaky integration test) Signed-off-by: David O'Keeffe <david.okeeffe@databricks.com> * fix(host): require an explicit runner workspace on the zygote fork path fork_runner defaulted workspace to os.getcwd() — the daemon's cwd, the exact value the workspace fix exists to avoid. The forked child was already strict (it raises when the request carries no cwd), so the manager was the only lenient link: a call site that omitted the argument silently resurrected the deleted-cwd crash instead of failing loudly. Make the parameter required so both ends agree, and cover the zygote fork path's cwd, which had no test — only the direct Popen path did. --------- Signed-off-by: David O'Keeffe <david.okeeffe@databricks.com> Co-authored-by: dbczumar <corey.zumar@databricks.com> |
||
|
Corey Zumar
|
b624d47ef8 |
fix(runner): name the runner log file in "see runner logs" errors (#4295)
* fix(host): keep the tunnel receive loop responsive during readiness refresh The host->server tunnel disconnected with `4003 ping timeout`: the periodic harness-readiness refresh ran inline on the receive loop and could block it for ~60s — two 30s CLI probe subprocesses (`--version` and `auth status`) hanging on a wedged harness CLI. While blocked, the host never answered the server's application-level pings, so the server watchdog declared the host dead and closed the tunnel. Fix A (host/connect.py): move the readiness refresh into its own task, `_harness_readiness_loop`, so the receive loop only ever reads frames and answers pings — a slow probe can no longer stall the keepalive. Fix B (harness_install.py, harness_readiness.py, codex_native.py): bound readiness CLI probes to READINESS_CLI_PROBE_TIMEOUT_S (10s, matching goose's status-probe budget) instead of 30s, so a hung harness CLI fails fast on the refresh, the startup hello, and Ctrl-C. Setup and launch gating keep the lenient 30s default via behavior-preserving timeout parameters. Co-authored-by: Isaac Signed-off-by: dbczumar <corey.zumar@databricks.com> * test: accept the readiness probe timeout kwarg in harness CLI stubs Readiness now calls harness_cli_installed / harness_cli_logged_in with a timeout= kwarg (READINESS_CLI_PROBE_TIMEOUT_S); update the monkeypatch stubs in the affected suites to accept it so they exercise the same paths. Co-authored-by: Isaac Signed-off-by: dbczumar <corey.zumar@databricks.com> * test(host): cover off-loop readiness refresh and bounded CLI probe Fix A moved the harness-readiness refresh off the tunnel receive loop into _harness_readiness_loop. Rewrite the three live-host readiness tests to drive that loop directly: the old versions drove _serve_frames with a fake tunnel that blocks on recv, which under the pure recv loop never exits and hangs to the pytest timeout. Add a harness_install test asserting the readiness caller shortens the CLI probe subprocess timeout while setup/launch keep the 30s default. Co-authored-by: Isaac Signed-off-by: dbczumar <corey.zumar@databricks.com> * fix(runner): name the runner log file in "see runner logs" errors `omnigent codex` (and its siblings) surface the runner's message verbatim, so a failed native terminal start read: Codex terminal ensure failed (500): Native Codex terminal failed to start; see runner logs for details. which left the user hunting for a file whose name they could not know. The runner already knows its own log path — the host passes it as OMNIGENT_PROCESS_LOG_FILE when it spawns the subprocess — so name it: ... failed to start; see the runner log for details: ~/.omnigent/logs/runner/runner-<session>-<timestamp>.log Same treatment for the generic runner detail string (_client_safe_error_detail, ~40 call sites: harness spawn, spec resolve, model change, compact, MCP dispatch). The client-safe contract is unchanged: the raw cause still goes to the log only, and the path is home-relative so it points somewhere without leaking the account name. process_logging grows current_process_log_path() / process_log_reference() to publish the path, and display_log_path() is promoted out of host/connect.py (it was private there) so both sides format paths the same way. The daemon_launch "runner did not connect" message stops hardcoding ~/.omnigent/logs/runner/ and computes the real dir, so it is correct under OMNIGENT_DATA_DIR. Co-authored-by: Isaac Signed-off-by: dbczumar <corey.zumar@databricks.com> * test(runner): pin the runner log path instead of trusting test order The three tests asserting the new "see the runner log for details: <path>" messages set OMNIGENT_PROCESS_LOG_FILE and expected the message to name it. That holds only until some earlier test in the same xdist worker runs the real configure_process_logging: test_runner_entry's test_main_preserves_unexpected_runtime_errors calls main() without stubbing it, which allocates ~/.omnigent/logs/runner/runner-<timestamp>.log and publishes that path process-wide. The published path outranks the environment (it is what the process actually logs to), so the assertions saw the leaked path and the runner-app group failed in CI while passing when run alone. Pin both sources in one place: a pinned_runner_log fixture in tests/runner/conftest.py sets the published path and the env var, so the assertions hold whatever else the worker ran first. Co-authored-by: Isaac Signed-off-by: dbczumar <corey.zumar@databricks.com> --------- Signed-off-by: dbczumar <corey.zumar@databricks.com> |
||
|
Bryan Qiu
|
b268130340 |
Smart Routing MVP: per-task model and harness routing (#4074)
* feat(telemetry): routing decision and setting-change events Routing needs to be answerable after the fact: which arm the router picked, whether it was applied, and what the user changed. Adds ``RoutingDecisionEvent`` and ``RoutingSettingChangedEvent`` plus a ``model_labels`` helper that reduces a model id to a family/tier pair, so records stay useful without carrying raw model ids. Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * feat(sessions): persist routing decisions and session warnings A routing decision has to survive the turn that produced it, so the UI can show what the router chose and — crucially — whether it was actually applied. Adds ``RoutingDecisionData`` to the conversation entity with store support, and a ``session_warnings`` module for the non-fatal routing conditions a session needs to surface (router unreachable, verdict not applied) without failing the turn. Records are honest by construction: a decision that could not be applied is stored with ``applied=false`` and its reason rather than being dropped or reported as a success. Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * feat(routing): session-start smart routing core Adds the server-side routing core behind Smart Routing: an external ``task_v1`` route-options seam that offers the router the frozen arm menu its scenario requires, maps a pick back onto a servable catalog id via nearest-cost substitution, and derives the harness that can actually run it. Routing settings become one value object on ``RuntimeCaps`` so every consumer reads the same knobs instead of re-parsing config. Databricks model discovery resolves catalog spellings deterministically so the same endpoint is named the same way on every path. Reconciled against main's catalog-driven routing: - Main's ``_fetch_runner_catalog`` / ``_RunnerModel`` plumbing and its cost-tier ordering are the single source of live model availability; ``fetch_runner_models`` remains the id-only adapter over it. - Main's ``ModelIntent``-parameterized judge rubric replaces the family-specific tier hints. - Main's catalog wire-API check survives as ``_redirect_wire_incompatible_pick``, layered after the static ``_HARNESS_EXCLUDED_MODELS`` bar list. The two cover different things: the catalog knows what an endpoint advertises, the bar list knows the client-side rejections it does not. - ``model_family_token`` defers to ``is_codex_compatible_model`` so the GLM/Kimi delegate arms read as the codex family everywhere. The static ``MODEL_LISTS`` table is retained, unlike main, because the nearest-cost substitution needs a family cost ordering on paths with no catalog in reach (hook scripts, pre-session creates). Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * feat(server): route sessions at start and expose the decision Wires the routing core into session lifecycle. A session created in Smart Routing mode is routed once, at start, from the first user message: the verdict picks the harness and the model before the runner launches, and pre-launch host model options supply the candidate catalog when no runner exists yet. Later turns never re-route — a session's harness is settled once so a conversation cannot change identity underneath the user. The decision is exposed on the session snapshot and event stream with its applied state, so the UI can distinguish "the router picked X and we are running X" from "the router picked X and we could not apply it", rather than silently showing the request as the outcome. Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * feat(claude): apply a routed model to Claude Code A routed arm only matters if the harness actually runs it. Adds a Claude model vocabulary that maps between router arm ids, catalog spellings, and the ``/model`` names Claude Code accepts, and pins the CLI's family aliases to the frozen task_v1 Claude arms at launch so the first turn's switch can reach whatever the router picked. The vocabulary reads its catalog prefixes from one definition shared with the server seam, so the hook path — which cannot read server config — cannot drift from it. Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * feat(codex): apply a routed model to Codex The Codex side of the apply layer: the native app server and executor accept a routed model override and enforce it on the session they launch, so a verdict that names a GLM/Kimi delegate arm reaches the CLI instead of being dropped for the harness default. Codex spawns with no routable signal skip the router outright rather than routing on an empty prompt and recording a decision nobody asked for. Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * feat(routing): route sub-agent spawns from harness hooks Sub-agents spawned by a native CLI never pass through the server's session-create path, so they were unroutable. Adds hook scripts the Claude and Codex CLIs invoke at spawn time, plus a runner-side router that answers them, so a spawned child is routed on its own task text and launched on the chosen model. A child is only ever offered its parent's harness family: routing may change which model a sub-agent runs, never which vendor it belongs to. Hook commands run under ``python -I`` so a repo-local module on the CLI's cwd cannot shadow the interpreter's own imports. Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * feat(web): surface routing decisions and Smart Routing controls Adds the Smart Routing harness option to new-chat, a routing chip that shows the routed model on the session, a sub-agent routing row, and a warning banner for the non-fatal routing conditions the server reports. The chip reports what actually happened. When a decision could not be applied it says so and names the model in use, instead of showing the router's request as though it were the outcome. Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * test(routing): cover the routing apply layer end to end Adds the remaining routing coverage: the CLI's routing-client build, the native Smart Routing create path, an end-to-end routing integration test, and the discovery/override unit tests. Also updates the existing native bridge, forwarder, and launch-arg tests for the model-override plumbing. Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * docs(routing): record the routing design and verification state Captures the plan the implementation followed, the per-CUJ verification status, and the observed live-model state the harness bar list is derived from — the gateway rejections that catalog metadata does not advertise. Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * docs: registry stamps — rebased-tree battery green, session-start verified live Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * docs: re-sync CUJ walkthrough with the rebased tree Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * feat(routing): offer Smart Routing only where the apply layer can work Smart Routing rewrites a launch's model through the Databricks AI Gateway, so a host whose claude-native or codex inference resolves anywhere else (Bedrock, a plain API key, the vendor CLI's own login) got an option that could never take effect. Gate each surface on the fact that decides it. The host already resolves this at launch, so reuse those resolutions as a cheap config-only check — no process launch, no network — and report a `gateway_inference` map alongside `configured_harnesses` on registration and every readiness refresh. It rides the host frames into the store and out through GET /v1/hosts. A host that never reports it sends `null`, and `null` means unknown: nothing is gated away on older host builds. Web gates the three surfaces independently, classified in the single `smartRoutingAvailability` point as a new `not-gateway-backed` cause: Configure Claude Code's Model row needs the claude family, Configure Codex's needs the codex family, and the top-level Smart Routing harness row needs both (it drives the five-arm menu). Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * docs(routing): record the gateway-backed availability decision Plan §10 gains decision 9 (Smart Routing offered only where the apply layer can work, with the per-surface rule and the absent-means-unknown compatibility contract), and §8 gains the two follow-ups it defers: a liveness probe, and moving the routes:select call host-side so routing auth/workspace always matches the host's inference. CUJ_STATUS gains recipe R9 (point a host at a non-AIGW config and assert the option disappears) plus one pending check row per gated surface. Co-authored-by: Isaac Signed-off-by: Bryan Qiu <bryan.qiu@databricks.com> * docs: rewrite the CUJ walkthrough in simplified technical English Rewrite designs/CUJ_IMPLEMENTATION.md in ASD-STE100-inspired Simplified Technical English so every sentence parses one way only: active voice with a named actor, simple tenses, one statement per sentence, noun clusters of at most three words, and lists for any sequence of three or more steps. Add a six-term glossary (arm, seam, pane, rollout, canary, spelling) to the intro. Remove the hard 80-column wrapping so each paragraph is one soft-wrapped line. No facts change: every sha citation and every file:line reference is byte-identical to |
||
|
Corey Zumar
|
590b2b6376 |
fix(sandbox): supervise the in-sandbox host so a crash can't strand the box (#4155)
* fix(host): keep the tunnel receive loop responsive during readiness refresh The host->server tunnel disconnected with `4003 ping timeout`: the periodic harness-readiness refresh ran inline on the receive loop and could block it for ~60s — two 30s CLI probe subprocesses (`--version` and `auth status`) hanging on a wedged harness CLI. While blocked, the host never answered the server's application-level pings, so the server watchdog declared the host dead and closed the tunnel. Fix A (host/connect.py): move the readiness refresh into its own task, `_harness_readiness_loop`, so the receive loop only ever reads frames and answers pings — a slow probe can no longer stall the keepalive. Fix B (harness_install.py, harness_readiness.py, codex_native.py): bound readiness CLI probes to READINESS_CLI_PROBE_TIMEOUT_S (10s, matching goose's status-probe budget) instead of 30s, so a hung harness CLI fails fast on the refresh, the startup hello, and Ctrl-C. Setup and launch gating keep the lenient 30s default via behavior-preserving timeout parameters. Co-authored-by: Isaac Signed-off-by: dbczumar <corey.zumar@databricks.com> * fix(sandbox): supervise the in-sandbox host so a crash can't strand the box A sandbox container outlives the host process: PID 1 is `sleep infinity` or the provider's own init, never `omnigent host`. So when the host dies the container stays healthy and still billing, with nothing running in it. Nothing notices until the next message, and the only recovery is `relaunch_managed_host` re-provisioning a fresh sandbox — which discards the workspace: the clone, the installed dependencies, the harness state. Wrap every exec-model host launch in a restart loop at the one seam all providers funnel through (`run_background`), so a crashed host restarts in place and the workspace survives. No image changes, no init system, no new privileges — replacing PID 1 across seven provider images would mean booting systemd with cgroup mounts, which the Kubernetes Pod's "restricted" security posture forbids outright. To make restarting safe, give a permanent startup failure its own exit code instead of sharing 1 with generic crashes: without it, a revoked or expired launch token inside a remote sandbox becomes an invisible hot restart loop with nobody watching a terminal. The supervisor stands down on that code, on a clean exit, and on SIGTERM; anything else is a crash, retried with a doubling delay capped at 30s. OpenShell keeps its held exec stream — it reaps an exec's processes when the RPC returns, so `setsid nohup` genuinely cannot work there — but gains the same supervisor inside that stream. Kubernetes is untouched: it is entrypoint-as-host with a deliberate `restartPolicy: Never`, recovering by provisioning a replacement Pod rather than restarting in place. Signed-off-by: dbczumar <corey.zumar@databricks.com> * fix(sandbox): make the supervisor's stop contract and backoff cap explicit Review follow-ups on the in-sandbox host supervisor. A signal-kill of the host alone (SIGKILL -> 137) stays classified as a crash on purpose: that is what an OOM kill looks like, and restarting is the wanted response. The consequence is that a path meaning to STOP the host must signal the supervisor too, or the loop faithfully restarts it. Both in-sandbox stop paths already do — `foreground_kill_command` signals the pidfile's recorded pid (the supervisor, which the host `exec`s under), and islo's preserved-daemon stop matches "omnigent host" against full argv, which the supervisor's own `sh -c` argv contains. Documented so a future narrowing of either match doesn't silently turn a stop into a restart loop. The loop deliberately has no attempt ceiling — giving up would restore the stranded-empty-box failure it exists to prevent — so add an attempt counter to the restart log, making a persistently crashing host observable instead of an indistinguishable repeat. Cover the backoff clamp with a test asserting the full delay sequence (1, 2, 4, 8, 16, 30, 30, 30), and point the `_harness_cli_version_string` timeout example at READINESS_CLI_PROBE_TIMEOUT_S instead of a stale literal that disagreed with it. Signed-off-by: dbczumar <corey.zumar@databricks.com> --------- Signed-off-by: dbczumar <corey.zumar@databricks.com> |
||
|
Corey Zumar
|
046ee1bc59 |
fix(host): keep the tunnel receive loop responsive during readiness refresh (#4092)
* fix(host): keep the tunnel receive loop responsive during readiness refresh The host->server tunnel disconnected with `4003 ping timeout`: the periodic harness-readiness refresh ran inline on the receive loop and could block it for ~60s — two 30s CLI probe subprocesses (`--version` and `auth status`) hanging on a wedged harness CLI. While blocked, the host never answered the server's application-level pings, so the server watchdog declared the host dead and closed the tunnel. Fix A (host/connect.py): move the readiness refresh into its own task, `_harness_readiness_loop`, so the receive loop only ever reads frames and answers pings — a slow probe can no longer stall the keepalive. Fix B (harness_install.py, harness_readiness.py, codex_native.py): bound readiness CLI probes to READINESS_CLI_PROBE_TIMEOUT_S (10s, matching goose's status-probe budget) instead of 30s, so a hung harness CLI fails fast on the refresh, the startup hello, and Ctrl-C. Setup and launch gating keep the lenient 30s default via behavior-preserving timeout parameters. Co-authored-by: Isaac Signed-off-by: dbczumar <corey.zumar@databricks.com> * test: accept the readiness probe timeout kwarg in harness CLI stubs Readiness now calls harness_cli_installed / harness_cli_logged_in with a timeout= kwarg (READINESS_CLI_PROBE_TIMEOUT_S); update the monkeypatch stubs in the affected suites to accept it so they exercise the same paths. Co-authored-by: Isaac Signed-off-by: dbczumar <corey.zumar@databricks.com> * test(host): cover off-loop readiness refresh and bounded CLI probe Fix A moved the harness-readiness refresh off the tunnel receive loop into _harness_readiness_loop. Rewrite the three live-host readiness tests to drive that loop directly: the old versions drove _serve_frames with a fake tunnel that blocks on recv, which under the pure recv loop never exits and hangs to the pytest timeout. Add a harness_install test asserting the readiness caller shortens the CLI probe subprocess timeout while setup/launch keep the 30s default. Co-authored-by: Isaac Signed-off-by: dbczumar <corey.zumar@databricks.com> --------- Signed-off-by: dbczumar <corey.zumar@databricks.com> |
||
|
Tomu Hirata
|
566fc5bb5a |
perf(runner): bound per-runner memory via glibc arenas + threadpool cap (#3901)
* perf(runner): bound per-runner memory via glibc arenas + threadpool cap Each session spawns its own runner process, and each grows to ~200MB in prod, over-using host resources. Profiling shows ~123MB is the irreducible import floor; the growth on top is runtime bloat from threaded Python on glibc: the runner offloads heavily via asyncio.to_thread, the default executor sizes to min(32, cpu+4) threads, and glibc opens up to 8*ncpu malloc arenas that never return to the OS. Nothing tuned any of this. Three low-risk, env-gated levers (all no-ops or benign off Linux): - MALLOC_ARENA_MAX=2 + a 128 MiB trim threshold, injected into the runner child env at both spawn sites via a shared _proc.malloc_tuning_env() helper. Empty off Linux; OMNIGENT_RUNNER_MALLOC_ARENA_MAX=0 reverts. - Cap the asyncio default executor at 8 workers (runner.threadpool_max_workers config key, OMNIGENT_RUNNER_THREADPOOL_MAX_WORKERS env override), set before any to_thread use so the 20-thread default pool is never created. - gc.freeze() after app construction to drop the static import graph from GC's tracked set. This targets the runtime growth, not the import floor; collapsing the floor itself (a copy-on-write zygote) is tracked separately. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * fix(runner): apply the glibc arena cap at the zygote exec The zygote forkserver landed and is now the default runner spawn path, which silently defeated this branch's MALLOC_ARENA_MAX injection. glibc reads that variable once, when its allocator initializes at exec; a zygote-forked runner never execs, it just replaces os.environ, so the value arrived far too late to configure an allocator and the cap stopped applying to every runner. Move the injection to the zygote's own Popen -- the single real exec on this path -- so all forked runners and harnesses inherit an already-capped allocator. Two tests pin the contract at that boundary, including that an operator's explicit export still wins. The other two levers on this branch (the 8-worker threadpool cap and gc.freeze()) live inside _run_tunnel_from_env, which every runner reaches regardless of how it was started, so they were unaffected. Note in malloc_tuning_env why the arena cap is glibc-only: macOS libmalloc uses per-CPU magazines with madvise reclaim and ignores MALLOC_ARENA_MAX, so macOS hosts get their reduction from the threadpool cap (measured: 21 threads -> 9). Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> --------- Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> |
||
|
Tomu Hirata
|
4d4ddb2617 |
feat(runner): copy-on-write zygote forkserver for runner processes (#3921)
* spike(runner): measure copy-on-write savings from a warm-fork zygote Each session spawns its own runner process, and each pays a ~123MB import floor for omnigent's own graph plus pydantic/fastapi/httpx. Runtime tuning trims the growth on top but can't touch that floor; the only way to collapse it is to import the graph once in a warm parent and os.fork() a child per session, sharing the read-only import pages copy-on-write. This standalone script measures whether that COW sharing actually materializes before we commit to the full zygote architecture. It imports the runner graph once, forks N idle children, and reports aggregate memory against an N-process Popen baseline, optionally with gc.freeze(). Not wired into the daemon — this is a measurement gate, not a feature. On this macOS box (N=8) the fork path showed ~82% lower aggregate footprint than the Popen baseline, but macOS phys_footprint is only an indicative analog to Linux Pss and the children idle (no COW erosion from refcount page-dirtying), so a Linux-under-load measurement is still required before productionizing. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * feat(runner): add copy-on-write zygote forkserver for runner processes Every session spawns its own runner, and each pays the full ~120MB import floor (omnigent's graph + pydantic/fastapi/httpx). On a host running N sessions that floor is duplicated N times. This adds a zygote: a single long-lived process that imports the runner graph once and os.fork()s a child per session, so on Linux the read-only import pages are shared copy-on-write and each extra runner costs only the pages it dirties. Design (grounded in the daemon/runner lifecycle, not the naive sketch): - omnigent/runner/_zygote.py — the forkserver. Single-threaded, no event loop or network; imports the graph once, gc.freeze()s it, then blocks on an AF_UNIX control socket forking a child per request. The child reopens its log, replaces os.environ with the request env, and calls the unchanged _entry.main() — so it behaves exactly like `python -m omnigent.runner._entry`. It is Popen-exec'd by the daemon (never forked from it), so it inherits none of the daemon's asyncio loop / websocket / worker threads — the classic fork-in-multithreaded-async deadlock is avoided by construction. - omnigent/host/runner_zygote.py — the daemon-side client. ZygoteManager owns the control socket; ZygoteRunnerProc is a Popen-shaped shim so the existing _RunnerHandle / _watch_runner / _handle_stop paths are unchanged. The daemon is NOT the forked runner's parent, so poll()/returncode/wait() round-trip to the zygote (the real parent) for exit status while terminate()/kill() signal the pid directly. - connect.py — _handle_launch forks via the zygote when enabled, else the original Popen. RUNNER_PARENT_PID is set to the ZYGOTE's pid (not the daemon's) because the runner's orphan watchdog compares os.getppid(); daemon death -> control-socket EOF -> zygote exit -> runners reparent -> each tears itself down, preserving today's parent-death semantics through one hop. Gated behind OMNIGENT_RUNNER_ZYGOTE=1 and Linux-only; any zygote failure disables it for the daemon's life and falls back to a direct Popen, so it is never a hard dependency. Also removes the Phase-1 measurement spike script, which this supersedes. Verified on macOS: a real zygote subprocess forks children, reports pids and exit codes, isolates per-fork env, reaps cleanly, and tears down on stop (fork works on macOS even though the COW savings are Linux-only). The production memory win and a full session-through-the-tunnel run are unverified here — they need a Linux host under load, which this change is written to be turned on for. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * fix(runner): address zygote review feedback - connect.py: a failed fork no longer stops the running zygote. Stopping it would kill healthy runners already forked from it (their orphan watchdog sees the parent die), so one bad fork could take down unrelated live sessions. Latch a `_zygote_disabled` flag for future launches instead and retain the manager so the zygote is still reaped on daemon shutdown. - runner_zygote.py: wait() after kill() in stop() so a zygote that ignored SIGTERM is reaped rather than lingering as a zombie. - _zygote.py: unify the _entry/app/native import to a single `from ... import` (CodeQL flagged mixed import styles). - test: build the fresh-interpreter probe via an explicit newline join instead of implicit adjacent-string concatenation (CodeQL). Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * fix(runner): forward --log-to-stderr TTY fd through the zygote The direct-Popen launch path forwards OMNIGENT_LOG_TTY_FD via child_logging_popen_kwargs so a detached runner can still mirror logs to the daemon's terminal. The zygote path dropped it, so --log-to-stderr mirroring was lost for zygote-forked runners. Forward it across both hops: - daemon -> zygote: reuse child_logging_popen_kwargs to dup the TTY fd and add it to the zygote's pass_fds (the helper also rewrites env[LOG_TTY_FD] to the duped number). - zygote -> forked runner: the valid fd number inside the child is the one the zygote inherited, not the daemon-side number the payload carries, so the child restores LOG_TTY_FD from the zygote's own value (and clears a stale payload value when the zygote has no terminal mirror). Adds a test asserting a bogus payload LOG_TTY_FD is cleared in the child. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * fix(runner): address second round of zygote review feedback - _zygote.py: create the forked child's log file 0o600, not 0o644. Runner logs can carry secrets (tokens, prompts); matches create_process_log_path. - _zygote.py: the child guard now preserves SystemExit's code instead of flattening it to a traceback + exit 1, so a zygote-forked runner exits with the same code as `python -m omnigent.runner._entry` (main() raises SystemExit on a tunnel rejection). New test covers it via a raise seam. - runner_zygote.py: stop the partially-started zygote if the initial ping raises (timeout / EOF), so a failed start never leaks a process + socket. - runner_zygote.py: signal via signal.SIGTERM / signal.SIGKILL instead of the raw 15 / 9. - test: mark the suite posix_only (it uses os.fork / pass_fds) so cross- platform sweeps skip it on Windows. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * feat(runner): enable the zygote on all POSIX hosts, not just Linux The host daemon runs on the user's own machine — most often macOS — so a Linux-only gate denied the copy-on-write import-floor savings to the majority of hosts. Gate on IS_POSIX instead (the zygote needs os.fork + AF_UNIX fd-passing, both POSIX; Windows still takes the direct Popen path). macOS is the platform where fork-without-exec is riskiest (CoreFoundation/GCD abort a forked child that touches them), so this was verified rather than assumed. The abort is triggered by forking from a MULTI-threaded process, which the zygote already designs against: it forks from a single-threaded parent (asserted active_count()==1) and does create_app + all network work in the child. Evidence on this macOS box: - A faithful fork probe (fork from the single-threaded import state, child runs create_app + getaddrinfo + TLS ctx + asyncio + httpx) survived 5/5. The same work forked from a multi-threaded parent SIGSEGV'd 2/3 — confirming the single-threaded fork is what makes it safe. - test_host_launch_runner_and_session_round_trip passes with OMNIGENT_RUNNER_ZYGOTE=1: a real host daemon forks a runner through the zygote, the runner connects its tunnel, and a full mock-LLM session round-trip completes. The daemon log confirms the zygote path (distinct zygote/runner pids), not a Popen fallback. Also adds an info log on the successful zygote-fork path so operators can see the zygote is active and which pids are involved. Still opt-in behind OMNIGENT_RUNNER_ZYGOTE=1 with the full Popen fallback; the steady-state Pss win under load remains best measured on a Linux host. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * feat(runner): fork harness subprocesses from the runner zygote The harness subprocess (`python -m omnigent.runtime.harnesses._runner`) is a separate exec per conversation, so it re-pays its import floor — and that floor is ~54MB of the same common graph (fastapi/pydantic/omnigent-core) the runner zygote already holds resident. This extends the zygote to fork harness children too, sharing that graph copy-on-write instead of exec'ing a fresh interpreter. - _zygote.py: the serve loop becomes a single-threaded `selectors` multiplexer over the daemon socket PLUS one inherited control socket per forked runner. A new `fork_harness` command forks a child that reproduces `_runner.main(argv)` in-process. The runner-fork request/response bytes are unchanged; the new multiplexer wraps them rather than rewriting them. A forked child closes every inherited zygote-side socket (it never speaks the fork protocol). - _harness_zygote_client.py (new): the runner-side client. `HarnessZygoteClient` reads the inherited control-socket fd from OMNIGENT_RUNNER_ZYGOTE_HARNESS_FD; `ZygoteHarnessProc` is an asyncio.subprocess.Process-shaped shim (pid / returncode / wait / send_signal / kill) with a background poll task keeping returncode fresh for _wait_for_bind's synchronous reads. - process_manager.py: `_spawn_harness_process` forks via the zygote when the runner was itself zygote-forked, else the original create_subprocess_exec; disabled on first failure so it falls back for the process's life. - _runner.py: a zygote-forked harness has the zygote (not the runner) as OS parent, so its watchdog probes the runner pid explicitly instead of trusting os.getppid(), and skips PR_SET_PDEATHSIG (which would bind death to the zygote). Gated by OMNIGENT_HARNESS_ZYGOTE_FORKED. Present only when the runner itself was zygote-forked; any failure falls back to a direct exec, so the harness fork is never a hard dependency. The win is bounded to the ~54MB Python wrapper (the external claude/codex CLI is a separate exec no Python zygote can share) and materializes under multi-conversation fan-out. Verified on macOS: fork_harness forks, reports pid + exit code, round-trips argv, reaps, and leaves the daemon socket serving; existing process_manager tests unchanged. Linux Pss savings still unverified. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * fix(runner): clear pyrefly type errors in the zygote - ZygoteRunnerProc.wait: narrow on `timeout` (not just `deadline`) so TimeoutExpired(timeout=...) gets a `float`, not `float | None`. - _spawn_zygote_process: pass stdin/stdout/stderr explicitly with a typed `BinaryIO | None` log handle instead of a `dict[str, object]` splat that matched no Popen overload. - _ZygoteServer.serve: cast selector key.fileobj (HasFileno | int) to socket — only sockets are ever registered. - _ZygoteServer._on_readable: wrap the bytearray partition result in bytes() before dispatch, which expects bytes. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * fix(runner): harden zygote failure paths (crash recovery, exit-code leak) Review flagged three correctness bugs in the unhappy lifecycle paths; none are security issues but each is reachable in prod. 1. Unexpected zygote crash stranded the daemon's view of every child. The daemon isn't the runner's OS parent, so once the zygote died it had no channel to learn a runner exited — ZygoteManager.poll returned None ("still live") forever, so _watch_runner looped, _handle_runner_status reported gone sessions as alive, and _handle_stop's final wait() could hang. Now poll() probes the runner pid directly when the zygote is gone: a dead pid surfaces a non-zero sentinel (254) so the runner reads as dead-and- failed, not eternal alive. _handle_stop's post-kill wait() is now bounded. 2. _exit_codes leaked for a dropped runner's harness children. Exit codes were only popped via poll, but a dropped runner's harnesses have no remaining client to poll them — the entries accumulated (unbounded map growth + pid-reuse misattribution). _drop_runner now discards those descendants' codes and marks still-live ones orphaned: _reap waitpid's them (no zombies) but discards the code instead of storing it. 3. ZygoteHarnessProc.wait() masked a crashed harness as exit 0. If the zygote went away, wait() returned 0, so a harness that crashed on boot (bind failure, import error) read as a clean exit and the process manager could hang waiting for a bind that never comes. Now probes the harness pid and returns a non-zero sentinel when the code is unrecoverable. Also: tighten "Linux-only" docstrings to "POSIX; COW savings on Linux" (the gate is IS_POSIX and the path runs on macOS), and add a sleep test-seam so the new failure-path tests can hold a child genuinely alive. Tests: kill the zygote under a live runner and assert the daemon eventually sees it dead (not hanging); a dropped runner's harness code is not retained; a crashed harness with an unrecoverable code surfaces as failure, not 0. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * fix(runner): keep zygote poll/wait off the daemon event loop Review flagged a liveness regression on the enabled path: for a zygote-forked runner, poll()/wait() are blocking control-socket round-trips (with lock contention against a booting zygote that holds the lock across its ~120MB import), not the lock-free waitpid the direct-Popen path used. Calling them on the loop thread could freeze the whole daemon — all sessions, websocket traffic, heartbeats — until the import finishes or the 30s control timeout elapses. - _watch_runner: poll() now runs via asyncio.to_thread. - _handle_stop: now async; the poll/terminate/wait sequence runs off-loop in a _stop_runner_proc helper. Its dispatch site and three tests updated to await. - _tracked_runner_pids: include the zygote pid so the orphan reaper never waitpid's the zygote out from under ZygoteManager._proc on an unexpected crash (which would confuse is_running()/stop()). Also updates test_poll_after_stop to use a live child, since the crash-recovery sentinel (254) now correctly fires for an already-exited pid after stop(). Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * fix(runner): status query off-loop + enable zygote by default - _handle_runner_status did its poll() on the event loop, the one place the PR hadn't moved off it. For a zygote-forked runner poll() is a blocking control-socket round-trip (bounded only by the 30s control timeout, and contended against a booting zygote), so a slow zygote could stall the whole daemon for a single status query. Made it async and run the poll via asyncio.to_thread, matching _watch_runner / _handle_stop. Dispatch site and the three status tests updated to await. - Enable the zygote by default: OMNIGENT_RUNNER_ZYGOTE is now opt-OUT (=0/false/no/off), not opt-in. The host daemon runs on the user's own machine (most often macOS), so defaulting on lets most users share the ~120MB import floor. Still POSIX-gated with a full Popen fallback, so an unsupported platform or any zygote failure is transparent. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * fix: prevent mid-spawn launch leaks and harden zygote request handling Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> --------- Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> |
||
|
Serena Ruan
|
3cc3777413 |
fix(host): forward OMNIGENT_RUNNER_ENV_PASSTHROUGH through the remote daemon (#4050)
OMNIGENT_RUNNER_ENV_PASSTHROUGH lets an operator name extra env vars for the host to forward on to spawned runners (provider gateway wiring, config env: refs, etc.). It worked locally but was a silent no-op in --server mode: the remote daemon env is allowlisted by a prefix set of DATABRICKS_ + LC_/MLFLOW_/OTEL_/ OMNIGENT_OTEL_ — NOT plain OMNIGENT_ — so the control var itself was stripped at the CLI->daemon hop, and _build_runner_env never saw the names it listed. Any var forwarded through the passthrough (e.g. a Linear API key for the repro-agent) reached the runner locally but never remotely. Add OMNIGENT_RUNNER_ENV_PASSTHROUGH to _RUNNER_ENV_ALLOWLIST so it survives both hops. It carries only env var NAMES, not secrets, so allowlisting it leaks nothing on its own — each named var must still independently reach the daemon (here via the DATABRICKS_ prefix). Tests: a daemon-hop test (remote env keeps the control var) and an end-to-end two-hop test (a named var survives CLI->daemon->runner, an unnamed one doesn't). Both fail without the one-line allowlist change. Co-authored-by: Isaac |
||
|
Tomu Hirata
|
cd5bcd2d04 |
fix(host): recover from workspace-missing runner launch failures (#4023)
When a session's workspace directory no longer exists on the host
(e.g. a worktree was deleted), the host was returning a generic
failed status with no error_code, causing the server to silently
wait out the full connect timeout and then surface a generic
'runner_failed_to_start' banner.
Changes:
- Add WORKSPACE_MISSING_ERROR_CODE ('workspace_missing') to host/frames.py
- Host returns this code when workspace.is_dir() fails, alongside the
existing descriptive error message
- Server (routes_events.py post_event) handles workspace_missing the same
way as harness_not_configured: immediately consumes the user message and
persists an actionable runner_failed_to_start error item with the host's
'workspace path does not exist: ...' message instead of timing out into
a generic RUNNER_UNAVAILABLE
- orchestration.py _ensure_runner_relay_ready skips the connect-timeout
wait for workspace_missing (same as harness_not_configured), and records
the refusal in runner_exit_reports so snapshot-based renders also show
the actionable cause
Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com>
|
||
|
Tomu Hirata
|
21febb6cc8 |
fix(host): retry 401/403 on an already-connected host (#4025)
When the VPN drops, a corporate proxy answers the host tunnel's WebSocket upgrade with 401/403 before the request reaches the Omnigent server. `_classify_http_status` treated those as permanently fatal, so a live, already-registered host exited with code 1 and the user had to re-run `omnigent host` after reconnecting. A host that already completed an upgrade proved its credentials and authorization are valid, so a later 401/403 is almost always a transient network-path artifact. For a connected host, 401/403 now retries forever via the normal reconnect path (mirroring the existing login-redirect design), with a once-per-outage stderr notice so a foreground `omnigent host` isn't silent. A fresh, never-connected host still fails loud on the first 401/403. Fixes OMNI-2367. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> |
||
|
Pat Sukprasert
|
1262652a03 |
chore(lint): enforce pyrefly type checking (#3972)
Signed-off-by: Pat Sukprasert <pat.sukprasert@databricks.com> |
||
|
Pat Sukprasert
|
0e46accde4 |
narrow lazy import boundaries (#3941)
Signed-off-by: Pat Sukprasert <pat.sukprasert@databricks.com> |
||
|
Pat Sukprasert
|
bc12d9a881 |
fix typing for subprocess handles (#3935)
Signed-off-by: Pat Sukprasert <pat.sukprasert@databricks.com> |
||
|
Tomu Hirata
|
c74faadc1e |
fix(runner): forward provider api_key_ref env vars into runner subprocess (#3915)
* fix(pi): surface credential resolution error when gateway provider's env var is unset When a `kind: gateway` provider is configured as the pi harness default via `default: pi` and its `api_key_ref: env:VAR` cannot resolve (because VAR is not exported in the runner's environment), `_optional_provider_family` previously caught the OmnigentError from `resolve_secret` and returned None silently. The outer `_apply_provider_to_pi` then raised a generic "no family whose credentials resolve — set the api_key env var for its 'anthropic' or 'openai' family" message with no mention of which specific variable to export, making the error hard to act on. Change `_optional_provider_family` to return the captured error alongside None (as a tuple), and surface that error in the "no family resolves" message so the user sees exactly which env var (e.g. `$MY_TOKEN` from `api_key_ref: env:MY_TOKEN`) needs to be set. The design intent of the silent catch is preserved: a family whose key is unset is still treated as absent so pi can fall back to the other family when only one key is exported. The only change is that the fallback-failure error now carries the root cause. Closes #3788 Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * address review: fix return type annotation, correct keychain docstring, remove issue refs from tests Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * fix(runner): forward provider api_key_ref env vars into runner subprocess _build_runner_env filters the host environment before spawning the runner subprocess, passing only an allowlist of known credential vars (ANTHROPIC_API_KEY, OPENAI_API_KEY, etc.). A user who configures a gateway provider with a custom env var via api_key_ref: env:MY_TOKEN would find that MY_TOKEN is present in their shell and daemon process but stripped before reaching the runner — resolve_secret then fails, _optional_provider_family returns None for the family, and _apply_provider_to_pi raises the no-family- resolves error. Add provider_credential_env_vars(config) to provider_config.py, which scans all inline-family providers for api_key_ref: env:VAR and api_key: $VAR references and returns the set of env var names (plus OMNIGENT_-prefixed aliases). Wire this into _build_runner_env so those vars are automatically forwarded alongside the standard HARNESS_CREDENTIAL_ENV_VARS, without requiring users to list them in OMNIGENT_RUNNER_ENV_PASSTHROUGH by hand. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * fix(pi): add authHeader to generic openai provider entries in models.json Generic (non-Databricks) OpenAI-compatible gateways expect Authorization: Bearer <token>. The 'databricks' and 'databricks-completions' provider entries in the generated models.json were missing authHeader: True on the generic provider path, so Pi used the Databricks-native auth scheme instead — causing a 401 Missing Authentication header from the gateway. Add authHeader: True to both entries when is_generic_provider is true, matching the pattern already used by databricks-openai, databricks-anthropic, and databricks-mlflow. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> * fix(pi-native): qualify namespaced model ids in --model arg to prevent builtin routing When a gateway provider's model id contains a slash (e.g. an OpenRouter namespaced id like 'moonshotai/kimi-k2.5'), Pi's arg parser treats 'provider/model' in --model as a provider override, routing to the builtin 'moonshotai' provider instead of our custom 'omnigent' provider. The builtin has no API key, producing 'No API key for provider: openai-codex'. Pass the fully-qualified 'provider/model' form (e.g. 'omnigent/moonshotai/kimi-k2.5') when the model id contains a slash, so Pi's findExactModelReferenceMatch matches the canonical form under our provider first. Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> --------- Signed-off-by: Tomu Hirata <tomu.hirata@gmail.com> |
||
|
Pat Sukprasert
|
e711e907a8 |
refactor: tighten host process typing (#3773)
Signed-off-by: Pat Sukprasert <pat.sukprasert@databricks.com> |
||
|
Pat Sukprasert
|
c6cd36cad2 |
Filter Codex picker to compatible OpenAI models (#3668)
* fix codex launch model compatibility filtering Signed-off-by: Pat Sukprasert <pat.sukprasert@databricks.com> * fix(codex): tolerate model discovery failures Signed-off-by: Pat Sukprasert <pat.sukprasert@databricks.com> --------- Signed-off-by: Pat Sukprasert <pat.sukprasert@databricks.com> |
||
|
Pat Sukprasert
|
e662555092 |
feat(web): select Codex model before launch (#3556)
* feat(web): select Codex model before launch Signed-off-by: Pat Sukprasert <pat.sukprasert@databricks.com> * fix codex databricks default model label Signed-off-by: Pat Sukprasert <pat.sukprasert@databricks.com> --------- Signed-off-by: Pat Sukprasert <pat.sukprasert@databricks.com> |
||
|
Pat Sukprasert
|
f3d28c8e71 |
fix(host): guard orphan reaper against missing os.WNOHANG on Windows (#3627)
The host orphan reaper's waitpid fallback uses os.WNOHANG and os.waitpid(-1, ...), neither of which exists/works on native Windows. Windows also has no child reparenting to a subreaper, so there is nothing to reap. The periodic sweep swallowed the resulting AttributeError, but the final drain in run()'s finally block runs unguarded and would crash shutdown. Return early with 0 when os.WNOHANG is absent, matching the reaper's own "non-Linux is a no-op" contract. Co-authored-by: Isaac |
||
|
Manfred Calvo
|
cc39c4eac1 |
fix(tunnel): give websocket clients a verifying SSL context (certifi/OS-trust fallback) (#1731)
* fix(tunnel): give host/runner websocket tunnels a verifying SSL context On interpreters whose OpenSSL default cert path is uninitialized (python.org macOS framework builds before Install Certificates.command, and python-build-standalone interpreters used by uv), ssl.create_default_context() loads zero trust roots, so the host and runner wss:// tunnels failed with CERTIFICATE_VERIFY_FAILED and looped on reconnect. Add omnigent/tls.py (resolve_ca_file + cached client_ssl_context) that resolves a CA bundle OS-trust-store-first with a certifi fallback, and pass that context to both tunnel websockets.connect calls for wss:// (ws:// stays ssl=None). egress/ca.py:_system_ca_bundle now shares resolve_ca_file; certifi is promoted to an explicit dependency. Closes #1730 Co-authored-by: Isaac Signed-off-by: manffred-calvosanchez_data <manffred.calvosanchez@databricks.com> * fix(claude-native): pass a verifying SSL context to wss:// terminal-attach _websocket_connect opened wss:// terminal-attach connections (the scheme terminal_attach_url produces from an https workspace base_url) with a bare default SSL context, so claude-native attach to a remote workspace hit the same empty-trust-store failure fixed for the tunnels. Route it through client_ssl_context() for wss:// (ws:// stays ssl=None). Also realign a ws_tunnel test with the databricks_request_headers rename from main. Closes #1730 Co-authored-by: Isaac Signed-off-by: manffred-calvosanchez_data <manffred.calvosanchez@databricks.com> * chore(deps): record certifi in uv.lock pyproject.toml promoted certifi to an explicit dependency; add it to the omnigent package's dependencies and requires-dist in uv.lock so "uv sync --locked" passes in CI. certifi was already resolved transitively, so its package entry (with hashes) is unchanged — this only records the direct dependency edge. Co-authored-by: Isaac Signed-off-by: manffred-calvosanchez_data <manffred.calvosanchez@databricks.com> --------- Signed-off-by: manffred-calvosanchez_data <manffred.calvosanchez@databricks.com> Co-authored-by: Tomu Hirata <tomu.hirata@gmail.com> |
||
|
Zeyi (Rice) Fan
|
f2b2f80948 |
refactor(server)!: remove deprecated OMNIGENT_ACCOUNTS_ENABLED env alias (#3322)
## Related issue N/A ## Summary - Remove the long-deprecated `OMNIGENT_ACCOUNTS_ENABLED` environment-variable alias for the multi-user auth enable switch. The canonical name `OMNIGENT_AUTH_ENABLED` has existed since the repository was open-sourced. - Strip the alias logic from `omnigent/server/auth.py::_auth_enabled()`, the explicit-auth check in `omnigent/cli.py::_apply_bind_auth_defaults()`, and the runner env-propagation allowlist in `omnigent/host/connect.py`. - Delete the tests that exercised the alias and the obsolete comment in `tests/conftest.py`. ## Test Plan - `uv run ruff check omnigent/server/auth.py omnigent/cli.py omnigent/host/connect.py tests/conftest.py tests/cli/test_bind_auth_defaults.py tests/server/test_accounts.py tests/e2e/test_local_server_lifecycle_e2e.py` passed. - `uv run pytest tests/cli/test_bind_auth_defaults.py tests/server/test_accounts.py -q --no-header` passed (95 items). - `uv run pytest tests/server/test_accounts.py::test_resolve_auth_source_defaults_to_header tests/server/test_accounts.py::test_resolve_auth_source_opt_in_selects_accounts tests/server/test_accounts.py::test_factory_defaults_to_header_when_env_unset tests/cli/test_bind_auth_defaults.py -q --no-header` passed (15 items). - Verified no remaining references with `grep -R "OMNIGENT_ACCOUNTS_ENABLED" . --exclude-dir=.git --exclude-dir=.venv`. ## Demo N/A ## Type of change - [ ] Bug fix - [ ] Feature - [ ] UI / frontend change - [x] Refactor / chore - [ ] Docs - [ ] Test / CI - [x] Breaking change ## Test coverage - [x] Unit tests added / updated - [ ] Integration tests added / updated - [ ] E2E tests added / updated - [x] Manual verification completed - [x] Existing tests cover this change - [ ] Not applicable ## Coverage notes Removed the tests that specifically covered the deprecated alias; remaining tests continue to validate `OMNIGENT_AUTH_ENABLED` behavior. The refactor does not change the `OMNIGENT_AUTH_ENABLED=1 | =0` semantics. ## Changelog [Breaking] The deprecated `OMNIGENT_ACCOUNTS_ENABLED` environment variable has been removed; use `OMNIGENT_AUTH_ENABLED` instead. BREAKING CHANGE: Users and deploys still setting `OMNIGENT_ACCOUNTS_ENABLED` must rename the variable to `OMNIGENT_AUTH_ENABLED` before upgrading; the old name is no longer read or propagated. Signed-off-by: Zeyi (Rice) Fan <zeyi.f@databricks.com> |
||
|
Cathy Yin
|
76281b9438 |
feat(onboarding): write a harness provider credential from the UI (M3 backend) (#3088)
CI / gate (push) Failing after 6s
Doc sync / Classify and draft docs (push) Has been cancelled
CI / Pytest (databricks) (push) Has been cancelled
CI / Pytest (inner-rest) (push) Has been cancelled
CI / Pytest (integration-mock) (push) Has been cancelled
CI / Pytest (repl-sdk) (push) Has been cancelled
CI / Pytest (runtime-policies) (push) Has been cancelled
CI / Pytest (server-approvals) (push) Has been cancelled
CI / Pytest (server-rest) (push) Has been cancelled
CI / Pytest (slack) (push) Has been cancelled
CI / Pytest (spec-llms) (push) Has been cancelled
CI / Pytest (tools) (push) Has been cancelled
CI / Pytest (misc) (push) Has been cancelled
CI / Pytest (runner-app) (push) Has been cancelled
CI / Pytest (runtime-core) (push) Has been cancelled
CI / Pytest (runtime-harnesses) (push) Has been cancelled
CI / Pytest (server-integration) (push) Has been cancelled
CI / Pytest (stores) (push) Has been cancelled
CI / Pytest (stores-postgres) (push) Has been cancelled
CI / Pytest (stores-mysql) (push) Has been cancelled
CI / Pytest (codex-parity) (push) Has been cancelled
CI / Coverage report (push) Has been cancelled
* feat(onboarding): report the installed-but-unconfigured harness state
Add the credential axis to the picker-facing readiness map so Claude, Codex,
and Pi report the yellow "installed but no credential" state — the signal the
web setup dialog needs to know when to offer an "Add credential" action
(the M2 keystone of Setup-From-the-UI). Purely additive: the values
("needs-auth" / "binary-missing") already exist in HarnessAvailability, and
the launch gate (harness_is_configured) stays binary-only, so a not-yet-authed
harness is never blocked from launching.
- New _family_provider_configured(): whether an omnigent-managed provider
(API key / gateway) serves the harness's family, reading the same config
omni setup's overview does. Subscription-kind is excluded (that lives in the
CLI's own login, judged by harness_cli_logged_in). Local, side-effect free,
never raises (fails to "no credential").
- Claude: ready when a provider is configured OR the CLI subscription login is
present (was CLI-login only — an API-key-only user wrongly showed yellow).
Checks config first to avoid the status subprocess on the common path.
- Pi: gains the axis it lacked entirely (was binary-only → always green once
installed). No CLI login, so binary + provider: installed-but-no-provider is
now "needs-auth".
Codex already had this (unchanged, the template). Qwen/OpenCode env-auth
unchanged.
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* feat(onboarding): write a harness provider credential from the UI
Second PR of Setup-From-the-UI (M3, security-sensitive). Adds the path that
turns a yellow "installed but not configured" harness green from the browser
for Claude / Codex / Pi, host-agnostic (local or remote), reusing the
credential-write logic omni setup already uses.
Design: the server is an authz'd pass-through. It validates ownership + the
UI-auth allowlist and forwards the secret over the (TLS) tunnel; the host DAEMON
does the write on the runner. The server never persists the secret, and the
frame's secret_value field is redaction-named so it never lands on a telemetry
span. Gated behind OMNIGENT_HARNESS_INSTALL_ENABLED (default off) exactly like
the install route (404 when disabled).
- New non-interactive core omnigent/onboarding/harness_auth.py: store a key /
gateway (secret → keychain, else ~/.omnigent/secrets.json; a providers: entry
referencing keychain:<name>, never the raw key), adopt an existing host env
var by reference (env:<VAR>, value never read), and detect adoptable env
credentials (non-secret descriptors only). First provider on a family becomes
the default; unsupported families/kinds are refused.
- New host.store_secret / _result frame pair; host daemon handler resolves the
harness→family, calls the core, and re-reports readiness so the badge flips
without a reconnect. Pi maps to its preferred anthropic family.
- New route POST /v1/hosts/{id}/harnesses/{harness}/credential (owner-scoped,
allowlisted, flag-gated) + registry pending_secret_writes plumbing + tunnel
result resolution.
- Regenerated openapi.json.
Tests: core unit tests (incl. the no-raw-secret-in-config invariant), frame
round-trip + telemetry-redaction, host-handler unit tests, and a full route
integration test over a fake tunnel (ownership, flag-off, allowlist, failure
mapping). 243 pass across the affected suites.
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* feat(onboarding): detect adoptable credentials on the host (adopt flow)
Adds the read side of the adopt flow: a host.detect_credentials frame pair +
GET /v1/hosts/{id}/credentials/detected that reports the credentials already
present on the host as NON-secret descriptors (family + source label + env var
name), so the UI can offer a one-click "adopt" instead of asking the user to
paste a key they already have. The value is never read or sent — adopt writes
an env:<VAR> reference via the existing store_secret path.
Owner-scoped + flag-gated like the credential-write route. Decode drops
malformed entries so a garbled payload can't inject a non-string field the UI
would trust. Adds frame round-trip (+ malformed-drop), host-handler, and route
integration (+ flag-off) tests; regenerated openapi.json.
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* fix(onboarding): tighten the credential route + adopt guard (Polly review)
Two review fixes on the credential-write path:
- The route gated on ui_installable_harnesses(), which includes the env-auth
opencode/qwen — the host handler then rejected them, turning a client/allowlist
problem into a confusing 502. Add ui_credential_configurable_harnesses() (the
Claude/Codex/Pi families the host can actually write) and gate on it, so
opencode/qwen get a clean 400 with no frame forwarded.
- adopt_env_credential now refuses an env var that isn't set on the host —
adopting an unset var would persist a provider entry that resolves to nothing
at the first turn. (Runs on the runner, so os.environ is the host's env.)
Tests: opencode/qwen added to the 400-rejection parametrize; an unset-env-var
adopt rejection case.
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* fix(server): serialize concurrent credential writes to one host (Polly review)
Polly non-blocking note: unlike the install route (which coalesces via
inflight_installs), the credential route had no guard against overlapping
writes. The daemon's write is a non-atomic load→merge→save of config.yaml
(twice — entry, then default), so two writes to one host in quick succession
(a double-click, or key + gateway) could interleave and clobber a sibling
providers: entry.
Add a per-connection credential_write_lock held around the store-secret
round-trip so writes to one host serialize. A gateway/local host still
processes different hosts concurrently (the lock is per HostConnection).
Adds an integration test that holds the first reply and asserts the second
frame only reaches the host after the first completes.
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* fix(onboarding): make Pi's auth step UI-authable and trackable
Pi's setup_steps auth descriptor was still the M1 shape (action="setup",
command="omnigent setup", status_key=None). Two consequences surfaced in
manual testing: (1) status_key=None made the step "unknown", so the setup
dialog dropped it and wrongly showed "Pi is ready" with no action even though
readiness reported needs-auth; (2) even rendered it was a CLI signpost, not
the credential form.
Pi is UI-authable now (PR A gave it the needs-auth readiness axis; the UI has
the credential form), so its auth step becomes action="auth" (opens the inline
form, keyed on kind=="auth"), command=None (Pi has no subscription CLI login),
status_key="authed" (trackable, so it's not dropped and the dialog reflects
the real state). Qwen stays the untracked env-auth signpost (not UI-authable).
Updates the pi test and adds a qwen-stays-signpost test.
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* chore: use the `omni` CLI alias (omni setup) in setup guidance
Rename user-facing "omnigent setup" → "omni setup" in the harness setup-step
descriptors, the setup hint, and their doc-comments. `omni` is the installed
console entry point (pyproject: omni = omnigent.cli:main) and is already used
elsewhere in the codebase, so the shorter alias is correct and consistent.
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* test: fix CI drift on the M3 backend branch (omni setup + auth action)
Two "Pytest (misc)" failures on this branch were stale test expectations, not
product bugs:
- tests/host/test_connect.py asserted the unconfigured-launch error names
"omnigent setup", but the earlier `omni` CLI-alias rename made the runtime
message say "omni setup". Update the positive assertion and the cursor
test's negative assertion (which guards that Cursor points at its own
installer, not the generic setup command) to the new spelling.
- tests/test_harness_capabilities.py restricted setup-step actions to
("install", "command", "setup"), but Pi's UI-authable step uses action
"auth" (added when Pi's credential step became a form). Add "auth" to the
allowed set; codex's own two-step assertion is unchanged.
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* test: harden the install-flow e2e against a slow picker render
test_install_button_installs_missing_harness opened the agent picker and
immediately clicked the Codex row, but the picker mounts its rows only after
the /v1/agents fetch resolves. Under CI load that render lags, and a menu
opened before the data lands can render empty or re-close on the update — so
the bare open-then-click flaked with a 30s click timeout, the Codex row never
becoming actionable (seen across two different shards). Open the picker, wait
for the Codex row and re-open if the menu flapped, then click. No product
change; passes locally unchanged (the retry is a no-op on the fast path).
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* test: settle agent data before opening the picker in the install e2e
The install-flow e2e flaked (30s click timeout on the Codex row, then on the
picker trigger via an overlay pointer-interception when reopened). Root cause:
the picker opened before the /v1/agents fetch settled, racing the menu-open
against a re-render. Wait for the composer's "Set up Codex" notice (rendered
only once the Codex agent + its unconfigured host state load) BEFORE opening the
picker, then open once and click. Passes locally repeatedly.
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* test: stop driving the agent picker in the install e2e (kill the flake)
The picker interaction was redundant — the single seeded Codex agent is already
auto-selected, so the composer's "Set up Codex" notice is present without
opening the dropdown. Driving the picker only added a menu-open-vs-async-render
race that flaked under CI load. Wait for the notice directly (generous 60s) and
click it to open the setup dialog.
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* test: wait for network idle before asserting the setup notice (install e2e)
The "Set up Codex" notice depends on two async fetches re-rendering the
composer (/v1/agents auto-selecting the agent, /v1/hosts marking its harness
unconfigured). On loaded CI runners that chain lagged past the timeout and the
assertion raced the still-loading landing screen. Wait for network idle and the
host chip (readiness present) before asserting the notice.
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* test: drop networkidle wait in install e2e (WS keeps network busy)
wait_for_load_state("networkidle") never fires in this app — the shell holds a
long-lived sessions/updates WebSocket, so the network is never idle. That wait
just burned its timeout and then raced the still-loading landing screen (the
"Set up Codex" notice was intermittently absent on CI). Replace it with plain
element waits (host chip, then the notice) at a generous 60s, matching every
other e2e_ui test. Passes locally repeatedly.
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* fix(host): adopt an env credential under its own family, not the harness's
Review (isaac, MAJOR): pi consumes both the anthropic and openai families, so
the UI can offer an OpenAI env var (e.g. $OPENAI_API_KEY) as adoptable for pi.
`_handle_store_secret` derived the family solely from the harness (pi→anthropic)
and passed that to `adopt_env_credential`, so adopting that var wrote an
anthropic-family provider whose api_key_ref is env:OPENAI_API_KEY — mis-routed
to the anthropic endpoint, failing at run time. For the adopt kind, look the env
var up in the host's detected credentials and use its OWN detected family
(falling back to the harness family if absent). Adds a pi-adopts-OpenAI
regression test.
Also carry the install-flow e2e fix onto this branch: explicitly select Codex
in the picker and stub the /v1/sessions?kind=any agent scan so the seeded-DB
agents don't leak in and leave Claude Code selected (a CI-only flake).
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
* fix(host): harden the UI credential-write path (review feedback)
Addresses Polly's blocking finding + hardening notes and Pat's nits on the
store_secret/adopt path:
- BLOCKING (adopt boundary): the daemon's adopt handler fell back to the
harness-derived family when an env var wasn't detected, and adopt_env_credential
only checked the var was *set*. An owner hitting the raw API could name any set
env var (a DB password, an unrelated secret) and have it persisted as a provider
credential sent to the vendor endpoint. Now the handler refuses an env_var that
isn't in detect_adoptable_credentials() (no fallback) — enforcing server-side the
same "only adopt what was detected" restriction the UI presents.
- secrets.py: create the file-backend secrets.json 0600 atomically via
os.open(O_CREAT, 0o600) instead of open()+chmod-after, which briefly left a
freshly-created file group/world-readable. Now network-triggerable, so worth
closing. Fixes the stale "0600 from the start" comment.
- adopt_env_credential: presence-only env check (`in os.environ`, not `.get`) so
the "never reads the value" contract stays literally true.
- gateway base_url: reject a non-http(s) scheme at write time rather than writing
a malformed provider entry that fails opaquely at the first turn.
- connect.py: hoist the harness_auth / provider_config imports to module top
(no circular import) to match the sibling onboarding imports.
Adds regression tests: adopt refuses an undetected env var, gateway rejects a
non-http base_url, and secrets.json is 0600 even under a permissive umask.
Co-authored-by: Isaac
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
---------
Signed-off-by: xq-yin <xiaoqian.yin@databricks.com>
|
||
|
Sabhya Chhabria
|
82c25ffbec |
[claude] Load Databricks models dynamically (#2831)
* ✨ feat(claude): Load Databricks models live - Refresh the gateway catalog once per new native session and share the launch snapshot with the UI. - Keep provider-neutral aliases, cached fallback behavior, and authoritative model removals. Signed-off-by: sabhya-db <sabhya.chhabria@databricks.com> * 🐛 fix(claude): Handle delayed model catalogs - Retry sticky model handoff after live options arrive, including bind races - Map provider model ids and defaults to friendly active picker rows - Tighten model option contracts and cover backend/UI edge cases Signed-off-by: sabhya-db <sabhya.chhabria@databricks.com> * fix(api): regenerate OpenAPI schema Signed-off-by: sabhya-db <sabhya.chhabria@databricks.com> * 🐛 fix(claude): Mirror managed model catalog Signed-off-by: sabhya-db <sabhya.chhabria@databricks.com> * 🐛 fix(ui): Resolve launch models from host Signed-off-by: sabhya-db <sabhya.chhabria@databricks.com> * test: fix model discovery CI coverage Signed-off-by: sabhya-db <sabhya.chhabria@databricks.com> * test: stub host model discovery in e2e Signed-off-by: sabhya-db <sabhya.chhabria@databricks.com> * fix(claude): preserve live catalog routing Signed-off-by: sabhya-db <sabhya.chhabria@databricks.com> * fix(claude): don't treat a failed-primary empty catalog as authoritative Addresses the outstanding review round: - discover_databricks_claude_models: when the UC listing fails and the legacy gateway answers with no Claude routes, re-raise the primary error instead of returning {} — callers now fall back to cached ucode models rather than hard-failing the launch on a transient UC outage. - Warn when model-services pagination is truncated at the page budget. - Runner claude-model-options: answer ClickException config failures with 424 instead of the retryable 503, so the picker path stops conflating "no models configured" with "still booting". - chatStore bind race: a preserved raced-catalog selection must still exist in that catalog — a removed sticky alias no longer lingers visually selected. - Document that the pre-launch host catalog is an ambient-default preview; launch re-resolves with the session's agent spec. Co-authored-by: Isaac * test(e2e): pick the live catalog label in the model/effort scenario The config modal's Model rows now carry the host catalog's display names ("Opus 4.8"), not the static alias labels, so the exact-match click must use the mocked catalog's label. Co-authored-by: Isaac * chore: revert accidental uv.lock churn from the merge Co-authored-by: Isaac * fix(api): sync openapi.json with the host model-options docstring Co-authored-by: Isaac * fix(api): tolerate provider model rows without displayName Polly review: the shared NativeModelOption schema made displayName required and _model_options_from_wire validated all-or-nothing, so one Codex model/list or OpenCode /api/model row lacking displayName blanked the whole picker for the session. Restore displayName as optional (the UI already falls back to the id) and skip malformed rows individually instead of discarding the catalog. Co-authored-by: Isaac --------- Signed-off-by: sabhya-db <sabhya.chhabria@databricks.com> Co-authored-by: sabhya-db <sabhya.chhabria@databricks.com> |
||
|
Daniel Lok
|
23e498521f |
fix(runner): quiet idle-reaper shutdown instead of a scary error banner (#3060)
* fix(runner): quiet idle-reaper shutdown instead of a scary error banner When the runner idle monitor reaps an inactive runner after `runner.idle_timeout_s` (default 1h), the runner exits cleanly (code 0), but the UI rendered the same loud red `ErrorBanner` a genuine crash would — even though the session is fully reactivatable (host-bound sessions relaunch the runner on the next message). A clean idle shutdown tripped two banner-producing server paths: 1. Relay path (durable / reload banner): the runner's `GET /stream` dropped abruptly, so the SSE relay published `failed` + `runner_disconnected` and persisted it as a `last_task_error` label. 2. Host exit-report path (live): the host's `_watch_runner` reported `host.runner_exited`, which became `failed` + `runner_failed_to_start`. This treats a clean idle exit as benign (a genuine crash still shows the banner): - Runner drains its session streams before the idle shutdown: enqueues the `[DONE]` sentinel to each `GET /stream` so the relay returns cleanly (no `runner_disconnected`, no durable label). `serve_tunnel` now takes a `shutdown_event` + `on_graceful_shutdown` hook; on signal it waits for in-flight dispatch tasks to emit their end frames, then closes the socket with a normal close handshake (the handshake completing is the delivery confirmation — robust over a remote connection, not a timing nudge), and stops reconnecting. - Host suppresses the exit report for a clean (code-0) exit; a non-zero exit still reports its cause. Co-authored-by: Isaac * refactor(runner): address PR review nits on graceful-shutdown loop - Use asyncio.create_task instead of ensure_future in the graceful-shutdown read loop, matching the module convention (Copilot). - Make the graceful-shutdown serve test deterministic: pre-arm the shutdown event so the first recv() race resolves to it, dropping the real-time sleep(0.01) that could flake under load (Copilot). - Give the flagged bare `await task` an explicit effect via `assert task.result() is None` (CodeQL "statement has no effect"). Co-authored-by: Isaac * docs(runner): note the same-tick frame drop in graceful shutdown Polly/Copilot review flagged that if a frame and the shutdown signal complete in the same asyncio.wait tick, the shutdown branch wins and the frame is dropped. That is acceptable on the idle-reaper teardown path (a host-bound session replays/relaunches on the next message); document it so the trade-off is explicit for future readers. Co-authored-by: Isaac * refactor(runner): snapshot drain queues; create_task in tests Follow-up PR review nits (Copilot): - `_drain_session_streams` now iterates `list(_session_event_queues.values())`. The loop is synchronous (no await, so nothing interleaves on the event loop today), but snapshotting keeps the drain robust if a queue mutation ever moves off this atomic path — matching the `list(...)` idiom already used by the timer-cleanup / pane-reaper paths. - Switched the two remaining `asyncio.ensure_future(...)` test helpers to `asyncio.create_task(...)` for consistency with the module convention. Co-authored-by: Isaac * fix(runner): log recv failure while settling cancelled read on shutdown PR review (Copilot): the graceful-shutdown branch swallowed WebSocketException while awaiting the cancelled recv_task. If recv() had already failed with an abnormal close on the same tick the shutdown fired, the socket may be dead — so the drain's [DONE] frames won't reach the server and it will see a disconnect — yet there was no trace of why. Keep suppressing the exception (letting it propagate would skip _graceful_drain and reintroduce the abrupt drop this PR removes), but split the handling: silent on CancelledError (normal cancellation), debug-log on WebSocketException so the rare same-tick failure is diagnosable without disturbing the quiet UX. Co-authored-by: Isaac |
||
|
Cathy Yin
|
9b9d331964 |
feat(server): install a missing harness onto a connected host from the UI (backend, flag-gated) (#2912)
* feat(host): add install-harness tunnel frame pair + registry plumbing Adds the HostInstallHarnessFrame / HostInstallHarnessResultFrame pair to the host tunnel protocol, mirroring the existing HostCreateDirFrame request/result pattern, plus the pending_installs future map on HostConnection. This is the vocabulary the server and a connected host use to negotiate a UI-driven harness install (later PRs add the host handler, the route, and the frontend button). Additive only: no frame is sent or received yet, so behavior is unchanged. The result frame carries a freshly-recomputed readiness map (configured_harnesses, reusing _optional_str_availability_map) so the UI can flip the harness badge without waiting for a reconnect. Co-authored-by: Isaac Signed-off-by: xq-yin <xiaoqian.yin@databricks.com> * refactor(onboarding): surface install failure reason from install_harness_cli Extracts install_harness_cli_with_reason(key) -> tuple[bool, str | None] alongside the existing install_harness_cli(key) -> bool, which becomes a thin wrapper that discards the reason. Single implementation, no caller churn: the four setup-wizard call sites keep their boolean contract unchanged. The reason is derived from the existing failure branches (manual-only spec, missing installer, timeout, OS error, non-zero exit, post-install binary-not-found) without capturing installer output — so omni setup's live npm output UX is preserved. A later PR's UI-driven install returns this reason to the user instead of a bare failure. Co-authored-by: Isaac Signed-off-by: xq-yin <xiaoqian.yin@databricks.com> * feat(host): install harness on request + resolve the install result Adds the host daemon side of UI-driven install: - _handle_install_harness in host/connect.py runs install_harness_cli_with_reason off the event loop, recomputes configured_harness_map(), and returns a HostInstallHarnessResultFrame carrying either the fresh readiness map or a failure reason. - host_tunnel.py's receive loop resolves the pending_installs future. - A shared allowlist/resolver (ui_installable_harnesses / ui_install_key) in onboarding/harness_install.py is the single source of truth for which harnesses are UI-installable (claude, codex, pi, opencode, qwen) and their install-spec keys. Defence in depth: the handler re-checks ui_install_key, so a stray or spoofed frame can never drive the installer for a non-allowlisted harness (e.g. hermes, whose installer is a curl | bash). Inert until PR4 wires a sender: nothing emits HostInstallHarnessFrame yet. Co-authored-by: Isaac Signed-off-by: xq-yin <xiaoqian.yin@databricks.com> * feat(server): add UI harness-install route behind a default-off flag Adds POST /v1/hosts/{host_id}/harnesses/{harness}/install: the server endpoint the web UI's Install action calls. It validates in order — feature flag (404 when off) -> allowlist (400) -> auth/require_user -> owner (403) -> liveness (409) — then forwards a HostInstallHarnessFrame over the tunnel via _proxy_install_harness and returns the host's refreshed configured_harnesses map. - Reuses the _proxy_create_dir request/future/wait_for template; the install timeout (330s) sits above install_harness_cli's 300s subprocess ceiling so the result is received before the server gives up. - Concurrent installs of the same (host, harness) coalesce onto one in-flight task (conn.inflight_installs) so a double-click can't fire two non-race-safe global npm installs. - Gated by OMNIGENT_HARNESS_INSTALL_ENABLED, surfaced to the SPA via GET /v1/info (harness_install_enabled), mirroring smart_routing_enabled. Allowlist ordering (400 before 403) avoids leaking host ownership through error codes. Ships dark: with the flag off the route is 404, so merging this changes nothing in production. Co-authored-by: Isaac Signed-off-by: xq-yin <xiaoqian.yin@databricks.com> * fix(host): make UI install idempotent + widen the server wait End-to-end testing against a real host surfaced two issues the stubbed unit tests masked: - The host ran `npm install -g` even when the harness CLI was already on PATH; npm re-resolves over the network and took >60s for an already-present binary, so a repeat Install click hung. _handle_install_harness now short-circuits on harness_cli_installed(key) and just returns fresh readiness (reusing the existing check) — sub-second on the happy path. - The server's per-call wait (330s) sat only 30s above install_harness_cli's own 300s subprocess cap, so a genuine cold npm install could finish right as the server gave up — a "504 but actually installed" outcome. Widened to 420s (300s + 2min headroom for readiness recompute + tunnel latency). Verified end-to-end: happy path 200 in 0.8s (already-installed fast-path), a real cold opencode install completes route->tunnel->daemon->npm->readiness, hermes rejected 400, codex reports needs-auth post-install. Co-authored-by: Isaac Signed-off-by: xq-yin <xiaoqian.yin@databricks.com> * chore(openapi): regenerate spec for the harness-install route CI's openapi-drift guard flagged openapi.json as out of sync after the new POST /v1/hosts/{host_id}/harnesses/{harness}/install route. Regenerated via scripts/dump_openapi.py so the committed spec matches the app. Co-authored-by: Isaac Signed-off-by: xq-yin <xiaoqian.yin@databricks.com> * refactor(server): share the harness-install flag env-var name Extract OMNIGENT_HARNESS_INSTALL_ENABLED into a single HARNESS_INSTALL_ENABLED_ENV constant in hosts.py, read by both the install route and the /v1/info flag in app.py, so the flag the UI sees and the flag the route enforces can never drift on a typo. Also switch the install-task scheduling from asyncio.ensure_future to the more idiomatic asyncio.create_task. Co-authored-by: Isaac Signed-off-by: xq-yin <xiaoqian.yin@databricks.com> * feat(server): describe per-harness setup steps for the UI setup flow Extends the harness-install backend so the web UI can render a "set up this agent" checklist that mirrors omnigent setup, instead of a single Install button. - /v1/harnesses now carries an ordered setup_steps list per harness (install, then auth), derived from the existing HarnessInstallSpec so it can't drift from the real install/login commands. Claude/Codex/Pi/OpenCode/Qwen get a first-class two-step flow; other harnesses get a generic "run omnigent setup" step. - The host readiness map now reports a two-step signal (binary-missing / needs-auth) for Claude and OpenCode too, matching Codex, so the UI can show install-done vs sign-in-done. Pi/Qwen stay binary-only (their credential isn't locally determinable). - The launch gate (harness_is_configured) is unchanged and stays binary-only, so a not-signed-in harness is never blocked from launching. - /v1/info advertises installable_harnesses (bare + native spellings) so the UI offers setup only where the install route will accept it. Co-authored-by: Isaac Signed-off-by: xq-yin <xiaoqian.yin@databricks.com> * feat(server): key harness setup steps by every spelling for the UI The setup dialog looks up steps by the harness a session declares — often a native wrapper (codex-native) or an installable id that isn't a picker row (opencode/qwen), none of which appear in the harness catalog. Add harness_setup_steps_by_spelling() and return it from GET /v1/harnesses as a top-level setup_steps map so the dialog can resolve steps for whatever id it holds, without adding non-pickable rows to the catalog. Co-authored-by: Isaac Signed-off-by: xq-yin <xiaoqian.yin@databricks.com> * fix(server): use host.user_id in the install route's owner check The install route still compared host.owner, but the Host model's owner field was renamed to user_id (identity-columns unification on main). An authenticated install therefore 500'd with AttributeError. Switch to host.user_id (matching every other host route) and add an owner-mismatch test that exercises the ownership branch with a real user_id — the existing tests run unauthenticated, so the comparison was never hit. Co-authored-by: Isaac Signed-off-by: xq-yin <xiaoqian.yin@databricks.com> * docs(server): correct the setup-step "can't drift" comment The auth-step commands (codex login, etc.) are display-only literals, not derived from HarnessInstallSpec.login_args — only the install step's label is derived. Reword the comment/docstring so they don't overstate the guarantee. Co-authored-by: Isaac Signed-off-by: xq-yin <xiaoqian.yin@databricks.com> * Address review: family-keyed install coalescing + clearer naming - Coalesce concurrent UI installs on the resolved install *family* key (ui_install_key) rather than the raw spelling, so codex + codex-native (both the openai npm package) share one in-flight install. Cleanup is tied to task completion via add_done_callback and every caller awaits under asyncio.shield, so a cancelled request can't clear the map out from under a follow-up and start a second concurrent `npm install -g`. - Add an integration test that fires two overlapping same-family installs and asserts exactly one frame reaches the host. - Rename install_harness_cli_with_reason -> try_install_harness_cli and return a HarnessInstallResult NamedTuple instead of a bare tuple. - Trim the over-long install-handler docstring and UI-installable map comment to the essentials. Co-authored-by: Isaac Signed-off-by: xq-yin <xiaoqian.yin@databricks.com> --------- Signed-off-by: xq-yin <xiaoqian.yin@databricks.com> |
||
|
Daniel Lok
|
f70085da2f |
[auth] Reuse delegated credentials in host runners (#2762)
* ⚡ perf(auth): Reuse delegated runner credentials - Exchange host launch binding tokens for short-lived owner bearers before resolving user credentials.\n- Share runner auth with Claude and refresh hook snapshots without exposing the binding token. Signed-off-by: Daniel Lok <daniel.lok@databricks.com> * ♻️ refactor(auth): Address review feedback - Avoid logging bridge paths and collect cancelled refresh tasks explicitly.\n- Inject the refresh interval so tests use the existing direct import style. Signed-off-by: Daniel Lok <daniel.lok@databricks.com> * fix runner auth fallback behind Apps proxy Signed-off-by: Daniel Lok <daniel.lok@databricks.com> * perf(auth): bootstrap runners with host bearer Signed-off-by: Daniel Lok <daniel.lok@databricks.com> * docs(api): regenerate OpenAPI schema Signed-off-by: Daniel Lok <daniel.lok@databricks.com> --------- Signed-off-by: Daniel Lok <daniel.lok@databricks.com> |
||
|
Sabhya Chhabria
|
01f1db3df4 |
[host] Refresh harness readiness without reconnect (#2828)
* 🐛 fix(host): Refresh harness readiness live - Publish change-only readiness updates from connected hosts - Persist updates and notify web and desktop host queries * 🐛 fix(host): Harden readiness refresh contract - Cover full-refresh and unchanged-map timer paths - Centralize readiness states and reject partial or empty live maps --------- Co-authored-by: sabhya-db <sabhya.chhabria@databricks.com> |
||
|
Daniel Lok
|
ae8878ad70 |
fix(server): ask the host if a runner is coming before the connect grace (#2699)
* fix(server): ask the host if a runner is coming before the connect grace
A host-bound session's first message waits up to _HOST_BOUND_RUNNER_CONNECT_GRACE_S
for the pinned runner's tunnel to register before relaunching. That wait is
correct for a booting new-session runner but pure latency for one that will
never connect — a non-sticky Stop dropped it, or the host restarted and lost
it. Neither case writes a host.runner_exited report (Stop pops the runner
before terminating; a dead host never sends the frame), so the old blind wait
burned the full grace every time on restart.
The host is the authoritative owner of runner-process liveness — it holds the
Popen. Add a host.runner_status frame pair so the server can ask: alive
(booting/serving → wait), dead (tracked but exited → relaunch now), or unknown
(stopped, crashed, or lost to a host restart → relaunch now). The dispatch
path races this query against the connect grace: the runner connecting (or a
crash report) always wins if it lands first, and a dead/unknown verdict cuts
the wait short so the relaunch runs immediately. Running the query alongside
the wait — not before it — keeps it strictly a speed-up: a host that is
offline, too old to answer, or slow yields no verdict and the grace runs its
normal course with no added latency.
Host-absent at dispatch skips the grace entirely (there is no one to query)
and falls through to the existing relaunch/503, unchanged.
Co-authored-by: Isaac
* Address code-quality review on the runner-status query
- Drain cancelled tasks via asyncio.gather(..., return_exceptions=True)
instead of `await task` inside contextlib.suppress, in both the race
helper and the integration test. Functionally identical, but avoids the
bare-expression-statement the static analyzer flagged as "no effect"
(it doesn't model `await` as side-effecting).
- Harden _query_host_runner_status: map any unexpected exception (e.g. a
future resolved with an error) to None so the query can only ever speed
up the connect grace, never break the message POST. CancelledError stays
a BaseException and still propagates, so the race helper's cancel/drain
is unaffected. Covered by a new test that resolves the pending future
with an exception.
Co-authored-by: Isaac
* test(e2e): stub /health so the host-badge push test isolates useHosts status
test_hosts_changed_frame_updates_host_badge failed at its first
assertion (before any hosts_changed frame): the badge read "status
unknown" instead of the stubbed "online". The test intercepts the
WS /v1/sessions/updates stream to keep liveOnline undefined, but the
open-session GET /health poll is a second, independent source of
host_online — and the real endpoint emits host_online: null for a
session it finds without a host binding. That null reaches
useSessionHostOnline as a live signal, which HostBadge treats as
authoritative "unknown", overriding the useHosts status the test drives.
Patch /health to drop the seeded session from the batch sessions map so
useSessionHostOnline stays undefined ("not observed yet") and the badge
falls back to the useHosts status field — matching the test's stated
intent and the existing snapshot/list route patches. HostBadge behavior
is unchanged; this only repairs the test's mock world, which had the
snapshot claiming host-bound while /health said otherwise.
Co-authored-by: Isaac
|
||
|
Serena Ruan
|
57ec0e1db6 |
feat(files): serve session filesystem from host when runner is offline (#2676)
* feat(files): serve session filesystem from host when runner is offline When a session's runner process dies but its host is still connected, the file panel (browse / changed files / diffs / search / file content) used to go dark — every request 502/503'd and the user had to send a message to wake a new runner just to look at files. The server now falls back to reading the workspace over the existing host tunnel when the pinned runner is offline. A shared, read-only WorkspaceReader (confined to the workspace root) runs on the host and returns the same JSON shapes the runner's filesystem endpoints do, so the resolver (live runner -> host tunnel -> 503) and the frontend can't tell which side answered. The panel stays live with a passive "Asleep — files shown live from host" badge; no LLM, no wake-up. Built as a resolver chain so a future host-death snapshot source drops in as an additive third link without touching endpoints or the frontend. - omnigent/workspace_fs.py: read-only WorkspaceReader (list/read/search/ changes/diff), reusing the runner's path-validation, glob, pagination, and git change-registry helpers. - host tunnel: host.fs_request / host.fs_result frames + host handler + server-side proxy and pending-future routing. - server: _fs_get_with_host_fallback wraps the 5 FS GET endpoints; offline env-metadata is synthesized from the bound workspace. - web: useWorkspaceServeable gate (runner-online OR host-online, tri-state aware) replaces the runner-only gate across the FS hooks; host-served badge in FilesPanel. Test Plan: backend unit + integration (real host tunnel, offline runner, real git workspace), frontend hook unit tests, and e2e_ui (real browser) covering the file list + content viewer while the runner reads offline. Co-authored-by: Isaac * fix(files): address host-served FS review notes (bounded read, parity) Follow-up to the PR review on the host-served filesystem path: - WorkspaceReader now reads at most _MAX_READ_BYTES from disk (via a bounded open().read) in both _read_file and diff's `after`, instead of slurping the whole file — a multi-GB file opened while the runner is asleep can no longer OOM the host process. Matches the runner's cap. - _list_dir falls back to lstat for a broken symlink and lists it as type="file"/bytes=None instead of silently dropping it — restores the parity the docstring claims with the runner's list_dir. - Host FS failures now mirror the runner proxy's status mapping: a non-404/400 host error (e.g. git_status_failed) surfaces as 502 like _proxy_get_to_runner, and a 400 stays a 400. - Log a warning when a host fs op times out (the module's _logger was previously unused); drop a dead `text = ""` assignment. Adds tests for the oversize-read cap and the broken-symlink listing. Co-authored-by: Isaac * fix(files): keep oversize text as UTF-8 when truncation splits a codepoint Follow-up to the PR review: WorkspaceReader._file_content_payload sliced the read at _MAX_READ_BYTES on a raw byte boundary, so a text file larger than the cap whose cut fell inside a multi-byte UTF-8 codepoint raised UnicodeDecodeError and was served base64 — diverging from the runner, which truncates on a valid boundary and keeps encoding="utf-8". Now, when we truncated and the only invalid bytes are a partial trailing codepoint (error within the last 3 bytes), drop them and re-decode as text. A genuinely binary file has invalid bytes earlier in the buffer, so it still falls through to base64. Adds tests for both. Co-authored-by: Isaac |
||
|
Tomu Hirata
|
13da60cf32 |
feat(telemetry): propagate host installation ID to SessionCreatedEvent (#2667)
* feat(telemetry): propagate host installation ID to SessionCreatedEvent Adds `installation_id` to `HostHelloFrame` so the host daemon advertises its local installation ID on connect. The server stores it in the `HostRegistry` via a new `get_host_installation_id` helper, then passes it as `host_installation_id` on `SessionCreatedEvent` so hosted sessions can be correlated back to a specific host machine in telemetry. * test(telemetry): add tests for host_installation_id telemetry feature Cover HostHelloFrame encode/decode roundtrip with and without installation_id, HostRegistry.get_host_installation_id with and without a registered host, and _build_record promoting host_installation_id to top-level data rather than params. |
||
|
Tomu Hirata
|
2b3b54a48e |
feat(telemetry): add usage telemetry system for session lifecycle events (#2457)
* feat(telemetry): add usage telemetry system for session lifecycle events
Adds a new omnigent/telemetry package with fire-and-forget product
analytics for session created, stopped, and deleted events. Telemetry
is completely opt-out (OMNIGENT_TELEMETRY=0, DO_NOT_TRACK=1, or any CI
env var suppresses all instrumentation) and never raises exceptions into
application code.
Key pieces:
- omnigent/telemetry/: new package with installation_id, client,
events, and surface modules
- HelloFrame.installation_id: runner propagates its installation ID
through the WS tunnel handshake so the server can correlate
runner-side and server-side identities
- TunnelRegistry.get_runner_installation_id(): convenience accessor
- sessions.py: stamps omnigent.client surface label at create time,
emits SessionStoppedEvent and SessionDeletedEvent at the right hooks
- app.py: initialises the telemetry client at lifespan startup and
emits SessionCreatedEvent inside _on_runner_connect
* fix(telemetry): emit session.created at create time, not on runner reconnect
Move SessionCreatedEvent emission from _on_runner_connect (which fires on
every reconnect for all bound sessions) to create_session, so the event
fires exactly once per session at creation time. Remove runner_installation_id
from the event schema since it is no longer available at emit time. Prime
the installation-id cache in init_client() to avoid synchronous file I/O
on the event loop in stop/delete handlers. Add unit tests for classify_surface,
is_disabled, and get_installation_id.
* fix(telemetry): address Copilot review comments
- Replace bare except pass blocks with _logger.debug() calls or
explanatory comments so intent is explicit
- Rename _INSTALLATION_ID_CACHE/_CACHE_INITIALIZED to _cache/_cache_initialized
to resolve unused-global-variable warnings
* fix(telemetry): consolidate imports, defense-in-depth opt-out, hash only user_id
- Move all telemetry imports to top-level in sessions.py; alias the three
event classes (_TelSession*Event) to avoid name clash with the existing
SessionCreatedEvent SSE schema class
- Add is_disabled() check inside TelemetryClient.emit() so opt-out is
enforced even if a call site skips the module-level guard
- Hash only user_id (not installation_id:user_id) since user_id is the
only PII; installation_id is already a random UUID with no PII value
- Add omnigent/telemetry/*.py to BLE001/SIM105 ruff ignore list — broad
exception catches are intentional at every telemetry boundary
* fix(telemetry): remove unused surface label stamp and _tel_disabled import
The omnigent.client label was written but never read anywhere. Surface
is already captured directly in SessionCreatedEvent from the User-Agent
header, so the extra label write was redundant. _tel_disabled is now
handled internally by emit().
* fix(telemetry): align wire format with API Gateway / Kinesis schema
- Wrap batches in {"records": [{"data": {...}, "partition-key": "..."}]}
instead of {"events": [...]}
- Add required envelope fields to each record: event_name, session_id
(per-process UUID), omnigent_version, schema_version, python_version,
operating_system, timestamp_ns, status, duration_ms, environment
- Serialize event-specific fields into data.params as a JSON string to
satisfy additionalProperties: false on the gateway schema
- installation_id remains a top-level data field (explicitly in schema)
- Add _detect_environment() for docker/cloud environment tagging
- Reorder events.py fields to put installation_id first (top-level field)
* feat(telemetry): support DISABLE_TELEMETRY env var and config.yaml opt-out
- Add DISABLE_TELEMETRY as an alias for OMNIGENT_DISABLE_TELEMETRY
- Read telemetry: false / telemetry:\n enabled: false from
~/.omnigent/config.yaml (honouring OMNIGENT_CONFIG_HOME)
- Config check is last in precedence so env vars always win
* fix(telemetry): only support telemetry: false in config.yaml
* feat(telemetry): hardcode staging/prod endpoints based on version
- Dev/pre-release versions (*.dev*, *a*, *b*, *rc*) route to staging
- Final releases route to production
- OMNIGENT_TELEMETRY_ENDPOINT env var still overrides for local testing
- Remove the 'no endpoint = silent no-op' behaviour; endpoint is always set
* feat(telemetry): add explicit runner-side opt-out via HelloFrame.telemetry_opt_out
- Replace installation_id in HelloFrame with telemetry_opt_out bool
- Runner sets telemetry_opt_out=True when its local is_disabled() is True
(honours OMNIGENT_TELEMETRY=0, DISABLE_TELEMETRY, DO_NOT_TRACK, CI vars,
and telemetry: false in config.yaml on the host machine)
- Replace get_runner_installation_id() with is_runner_telemetry_opted_out()
on TunnelRegistry
- Server skips session.created emit (best-effort) when runner signals opt-out
* feat(telemetry): link opt-out to host instead of runner
- Add telemetry_opt_out to HostHelloFrame (encode/decode in host/frames.py)
- Host sets telemetry_opt_out=True in connect.py when its is_disabled() is True
- Add HostRegistry.is_host_telemetry_opted_out(host_id)
- sessions.py checks host_id opt-out instead of runner_id — host is stable
and persistent; runner is ephemeral (one per session)
- Runner-side telemetry_opt_out in HelloFrame retained for CLI sessions
(omnigent claude/pi) which have no host
* fix(telemetry): address remaining Copilot empty-except comments
- _resolve_endpoint: log debug on version parse failure
- init_client: log debug on TelemetryClient init failure
* feat(telemetry): add remote config fetch (MLflow pattern)
- Fetch {config_url}/{version}.json at startup in a daemon thread
- Config fields: ingestion_url (required), disable_telemetry (kill-switch),
disable_events (per-event list), disable_os, rollout_percentage
- Consumer waits for config before sending; discards buffered events if
config fetch fails or kill-switch is set
- Per-event disable_events checked at emit time AND at send time
- OMNIGENT_TELEMETRY_CONFIG_URL env var overrides config URL for testing
- Staging config URL for dev/pre-release; production for final releases
- Remove hardcoded _ENDPOINT_PROD/_ENDPOINT_STAGING — ingestion_url comes
from config now
* style(telemetry): fix test formatting (pre-commit ruff format)
* fix(telemetry): update tests to use renamed cache vars (_cache/_cache_initialized)
* fix(telemetry): update config URLs to omnigent-telemetry.io domain
* fix(telemetry): use actual Omnigent session_id instead of per-process UUID
Pop session_id from event fields to the top-level data.session_id so
the gateway receives the real conversation ID. The per-process UUID was
confusing and didn't match the schema description 'Omnigent session
identifier'.
* fix(telemetry): start threads eagerly and reduce batch interval to 10s
- Start config fetch + consumer threads in init_client() rather than
lazily on first emit(), so config is pre-fetched before the first event
- Reduce _BATCH_INTERVAL_S from 30s to 10s so events are flushed promptly
in low-volume usage (waiting 30s explains why endpoint wasn't being hit)
* fix(telemetry): format anon_user_id as installation_id_hash(user_id)
* fix(telemetry): promote anon_user_id to top-level data field; revert to sha256(user_id)
- Pop anon_user_id from event fields into data envelope alongside
installation_id (requires infra schema update to allow the field)
- Revert anon_user_id format back to plain sha256(user_id)[:16]
* fix(telemetry): salt anon_user_id with installation_id to prevent rainbow table attacks
* fix(telemetry): remove params truncation that produced invalid JSON
* fix(telemetry): respect telemetry: false in -c config.yaml for server
- Add server_config param to init_client() — checks config.get('telemetry') is False
- Thread cfg from CLI server command into create_app(server_config=cfg)
- create_app passes it into the lifespan which calls init_client(config=server_config)
* fix(telemetry): remove OMNIGENT_TELEMETRY_DISABLE env var
* fix(telemetry): fix config.yaml opt-out and add missing tests
- Replace yaml.safe_load with regex match in _config_telemetry_disabled
to avoid spec/parser.py corrupting SafeLoader.yaml_implicit_resolvers
which caused 'false' to parse as a string instead of a boolean
- Add tests: DISABLE_TELEMETRY, OMNIGENT_DISABLE_TELEMETRY, config.yaml
telemetry:false, config.yaml telemetry:true, init_client server_config
|
||
|
Zeyi (Rice) Fan
|
4face30b9d |
✨ feat(logging): Add process log routing (#2468)
* ✨ feat(logging): Add process log routing
Related issue: N/A
Summary:
- Route server, host, runner, and CLI logs through shared process logging under $OMNIGENT_DATA_DIR/logs/<destination>/.
- Add global --debug and --log-to-stderr controls, including fd-based terminal mirroring for omnidev.
- Update omnidev to pass --log-to-stderr to Omnigent server and host processes.
Test Plan:
- cargo fmt --check
- cargo test (dev/omnidev)
- .venv/bin/python -m pytest tests/test_process_logging.py tests/cli/test_cli_diagnostics.py tests/cli/test_cli.py tests/cli/test_server_lifecycle.py tests/host/test_local_server.py tests/host/test_connect.py tests/runner/test_runner_entry.py
- .venv/bin/pre-commit run --all-files
Demo:
N/A
Type of change:
- [x] Feature
- [x] Refactor / chore
- [x] Test / CI
Test coverage:
- [x] Unit tests added / updated
- [x] Existing tests cover this change
Coverage notes:
Automated tests cover process logging helpers, CLI flags/log discovery, server lifecycle, host-spawned runner logging, runner entrypoint logging, and omnidev command construction.
Changelog:
Omnigent writes process logs to per-destination files and can mirror them to the terminal with --log-to-stderr.
* Fix process log routing checks
|
||
|
Pat Sukprasert
|
3864413eb1 |
fix(harnesses): flow Anthropic gateway creds host→runner→Claude Code launch (#2371)
* fix(harnesses): flow Anthropic gateway creds host→runner→Claude Code launch A browser-created managed sandbox running claude-native against an Anthropic-compatible gateway (e.g. LiteLLM) needs ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL, and ANTHROPIC_MODEL to survive three hops. Each hop dropped or ignored the model / gateway wiring, so sessions failed with invalid-model or auth errors, or hung on Claude Code's custom-key menu. - Host→runner env: forward ANTHROPIC_MODEL through the harness credential allowlist next to ANTHROPIC_API_KEY / ANTHROPIC_BASE_URL, so the runner no longer resolves model=None. - Ambient provider synthesis: an ambient ANTHROPIC_API_KEY now honors companion ANTHROPIC_BASE_URL and ANTHROPIC_MODEL, mirroring the OpenAI branch, so a gateway key routes to the gateway with the served model pinned instead of api.anthropic.com with no model. - Native launch + tmux delivery: when an apiKeyHelper delivers the credential, strip the raw ANTHROPIC_API_KEY (and CLAUDECODE) from the Claude terminal child so Claude Code doesn't open its custom-API-key menu, and teach the prompt-readiness scan to ignore selected numbered menu rows so the first web message isn't typed into that menu. Co-authored-by: omnigent <noreply@omnigent.ai> * test(harnesses): pin apiKeyHelper no-raw-key invariant, fail loud The helper-path key strip in the Claude terminal env relies on build_native_claude_terminal_env never emitting a raw ANTHROPIC_API_KEY when an apiKeyHelper is configured. If a future change starts injecting the raw key on that path, it would silently reintroduce Claude Code's custom-API-key menu hang. Raise at the env-build seam when the invariant breaks, and pin it with a focused unit test. Co-authored-by: omnigent <noreply@omnigent.ai> * test(harnesses): pin Databricks-gateway helper-path env shape Existing helper-path coverage is generic gateway-shaped; add a test for the Databricks ucode/profile case real users run. Through _claude_terminal_env_unset and the terminal-env build, assert the child drops DATABRICKS_CONFIG_PROFILE and the raw key / nested-session marker while apiKeyHelper, ANTHROPIC_BASE_URL, and the gateway model survive, so Claude Code still authenticates against Databricks. Co-authored-by: omnigent <noreply@omnigent.ai> * docs(harnesses): trim comments on the Anthropic gateway cred path Tighten the comments and docstrings introduced by this branch to match the repo's comment guidance: keep them short and focused on the scenario, drop redundant restatement, and remove paragraphs that duplicate a nearby docstring. Preserve the load-bearing "why" — the Databricks profile drop at the terminal-child hop, the apiKeyHelper raw-key guard, and the readiness-scan menu-glyph rationale. Comment-only; no executable code changed. Co-authored-by: Isaac * 🐛 fix(harnesses): Strip nested Claude marker * 🐛 fix(harnesses): Recognize numbered Claude drafts --------- Co-authored-by: omnigent <noreply@omnigent.ai> |
||
|
Edwin He
|
76bb9002d9 |
Route out-of-process native posters through databricks_request_headers (#2328)
The pi JS extension and the opencode policy plugin run OUT of the runner process and POST to the omnigent server with a hand-rolled `Authorization: Bearer` header, bypassing databricks_request_headers -- the single chokepoint that folds in the server-routing selectors (X-Databricks-Org-Id and the opaque OMNIGENT_DATABRICKS_EXTRA_HEADERS map that some Databricks deployments use to pin a request to a specific server instance). Without those selectors their POSTs can land on a different server instance than the one the runner and the web UI are bound to, so on a multi-instance deployment pi's streamed items never reach the browser's in-process event stream (they only appear on reload) and opencode's policy evaluation hits a different instance. - cli_auth: fold OMNIGENT_DATABRICKS_EXTRA_HEADERS into databricks_request_headers (opaque JSON header map; no-op when unset). - pi: build the extension config.authHeaders (launch + per-turn refresh) via databricks_request_headers. - opencode: bake the full routing header map as OMNIGENT_POLICY_HEADERS and merge it in the policy plugin, replacing the bearer-only OMNIGENT_POLICY_AUTH. - host: allowlist OMNIGENT_DATABRICKS_EXTRA_HEADERS in the host->runner env builder so a host forwards the routing selectors to the runners it spawns. Without it the host tunnel lands on the selected instance while its runners fall back to the default one (their tunnel + callbacks register elsewhere), so the session's runner is unreachable from the instance serving the UI and the session reports runner_failed_to_start. In-runner Python clients already route via _RunnerDatabricksAuth / _remote_headers; the gaps were the two out-of-process posters and the host->runner env handoff. Co-authored-by: Isaac Signed-off-by: Edwinhe03 <41037314+Edwinhe03@users.noreply.github.com> |
||
|
Tomu Hirata
|
91d6746b44 |
feat(cli): add omnigent debug logs command (#2273)
* feat(cli): add `omnigent debug logs` command Exposes runner, server, and CLI diagnostic log files via the debug subgroup so operators can inspect them without navigating the ~/.omnigent/logs/ directory manually. --type [runner|server|cli] which log category (default: runner) --list list files with sizes and timestamps -n / --lines N tail last N lines (0 = whole file) -f / --follow stream in real-time (tail -f) * feat(cli): filter runner logs by session id Embeds the session id in each runner log filename (runner-conv_abc123-<random>.log) so all relaunches for a session are discoverable. Adds --session SESSION_ID to `omnigent debug logs` to show all log files for a session oldest-first. * fix(cli): address Polly review on debug logs command - Separate runner into two types: runner (logs/runner/, local CLI) and host-runner (logs/host-runner/, host daemon) — fixes the blocking bug where the default type pointed at the wrong directory - Broaden server glob to *server*.log to cover both server-*.log (omnigent run) and local-server-*.log (background daemon) - Scope --session to --type host-runner only (where session ids are embedded in filenames) - Guard --follow on Windows with IS_WINDOWS check - Add min=0 bound to --lines to reject negative values |