feat: server-side streaming dictation for the composer mic button (#2093)
* feat(server): streaming dictation endpoint (local speech-to-text) Adds WS /v1/dictation/stream + GET /v1/dictation availability probe, backed by a lazily-loaded sherpa-onnx streaming transducer (new optional extra: omnigent[dictation]) with optional online re-punctuation. Fills the gap documented in web/electron/README.md: dictation where the browser Web Speech API has no backend, with audio never leaving the operator's infrastructure. A deterministic fake engine (OMNIGENT_DICTATION_ENGINE=fake) keeps CI hermetic and will drive the Playwright e2e test. See designs/server-dictation.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(web): stream server dictation into the composer mic button When the browser has no Web Speech backend (Electron, Firefox, Chromium), the mic button now falls back to the server recognizer: GET /v1/info advertises dictation_available, an AudioWorklet downsamples the mic to 16 kHz PCM over WS /v1/dictation/stream, and partial transcripts form live in the composer via a replaceable interim region (useDictationInsert) shared by ChatPage and NewChatDialog. Web Speech behavior is unchanged where it works. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(e2e-ui): dictation loop against the fake engine Fake mic (Chromium fake media device) -> AudioWorklet -> dictation WS -> OMNIGENT_DICTATION_ENGINE=fake -> transcript lands in the composer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: ruff format + regenerated openapi.json for dictation routes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: prettier formatting Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(e2e-ui): honor plugin context args in the dictation test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: drop the caller-less GET /v1/dictation probe ponytail review: the web UI only reads dictation_available from GET /v1/info, so the dedicated probe endpoint had no caller. Also simplify the engine singleton (config never changes mid-process; tests inject engine_provider) — a failed load still caches nothing, so gaining models doesn't require a restart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: hardware sizing table for dictation models Measured on Apple M-series and an Intel N95 mini-PC: the default Nemotron 0.6B is too slow for N95-class servers (0.6-0.7x realtime); the mid-size streaming zipformer decodes 1.4-2.3x realtime there in ~190 MB and held accuracy in spot checks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(server): remote dictation worker relay with local fallback OMNIGENT_DICTATION_REMOTE_URL relays takes to a dictation worker on a beefier LAN box over the existing wire protocol; local models (when installed) serve as a lazy fallback when the worker is down. Ships a standalone single-route worker entrypoint (python -m omnigent.server.dictation_worker). Motivated by real hardware: an N95 main server decodes the default 0.6B model at only 0.6x realtime, but a workstation on the same LAN runs it at 9x. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deps): pin sherpa-onnx-core + numpy explicitly in the dictation extra sherpa-onnx's wheel metadata declares its native payload package (sherpa-onnx-core, which carries libonnxruntime) inconsistently across platforms, so it was missing from uv.lock — failing the hashed OSV audit in CI and breaking aarch64 installs. Pinning it explicitly fixes both and removes the fetch script's aarch64 fixup. numpy is imported directly by the engine, so declare it instead of riding transitives. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: harden dictation take lifecycle (adversarial review findings) Server: the route now closes the engine stream handle on every exit path — an abandoned take (browser vanished mid-dictation) previously leaked the remote relay's worker WebSocket and reader thread, holding a worker capacity slot forever and eventually starving dictation for everyone. Web client, all confirmed by review: - useDictationInsert strips the interim region only when the draft still ends with the exact text it inserted, so dictation can never delete user-typed text; ref bookkeeping moved out of the setState updater (StrictMode double-invokes updaters). - The worklet flushes its partial chunk before stop() tears the graph down — trailing speech under the 100 ms boundary was being clipped from every take. - Client ready/stop budgets now exceed the server's cold-load and worker-flush budgets (40 s / 15 s), so slow first takes and slow tail flushes no longer fail or drop text spuriously. - The 1013 at-capacity close surfaces as "busy — try again" instead of "unavailable", and engine-init error frames surface their message. - A socket close during audio-graph setup now fails the start instead of resolving a dead session that silently drops all audio. - Web Speech network-error fallback is per take, not sticky: a transient blip in real Chrome no longer permanently downgrades the page to the server model, and stale events from the dead recognizer can no longer clobber the live server take's state (which could leave the mic recording while the button showed idle). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: dictation model choices for other languages Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(web): format dictation files * fix(server): close dictation takes even when the task is cancelled An ASGI server cancels the websocket handler task on shutdown. The cleanup awaited asyncio.to_thread(handle.close) inside finally, so the CancelledError could arrive before the worker thread ran close() -- about half the time, measured. contextlib.suppress(Exception) never caught it: CancelledError is a BaseException. Create the close task before the first await point and shield it, so it runs to completion while cancellation propagates. Hold a strong ref (asyncio keeps only a weak one) and retrieve the result so a failing close logs instead of warning. Also corrects the comments: an abandoned take is reaped by the ASGI server's ping timeout (~20s), not held forever. Verified against a live worker with OMNIGENT_DICTATION_MAX_STREAMS=1. * refactor(dictation): split out remote, add engine registry, fold beautify Keep this PR focused on local dictation and make future model swaps cheap: - Defer the remote worker (RemoteDictationEngine, dictation_worker.py, and the close-on-cancel machinery that existed to release a worker slot) to a follow-up PR. Remote only helps a narrow deployment; local sherpa runs at many-times realtime on any normal machine, so this does not block testing. - Select engines by name from a registry (register_engine); get_engine and engine_availability resolve from it instead of an if/elif ladder. Adding an engine is one call with a factory + availability probe. - Fold punctuation into the sherpa engine and drop beautify from the DictationStreamHandle protocol. Emitted text is display-ready, so the seam is PCM-in -> text-out -> close; models that punctuate themselves (Whisper, Parakeet) implement nothing extra. Co-authored-by: Isaac * chore: re-trigger CI checks Empty commit to re-run the security scan and CI on this PR. Co-authored-by: Isaac * build(deps): minimize dictation lock diff to sherpa-only, public index The merge re-lock rewrote every uv.lock URL to the Databricks internal index proxy and would fail the public-registry lint. Restore public pypi.org / files.pythonhosted.org URLs so the lockfile diff versus main is only the two dictation packages (sherpa-onnx, sherpa-onnx-core), with no unrelated churn. Co-authored-by: Isaac * fix(web): sync ServerInfo test fixtures with merged capability fields The main merge made single_user/sharing_mode/public_sharing_enabled required on ServerInfo while dictation_available became required from this PR, but four test fixtures each construct a ServerInfo literal missing the other side's fields, failing tsc (and the web build via Docker/E2E-UI). Add the missing fields so every fixture is a complete ServerInfo. Co-authored-by: Isaac --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Ubuntu <kerry.chang@your.hostname.com>
This commit is contained in:
John Bonewitz
@@ -0,0 +1,239 @@
|
||||
# Server-side streaming dictation
|
||||
|
||||
## Problem
|
||||
|
||||
The composer mic button (`web/src/components/ComposerMicButton.tsx`) relies on
|
||||
the browser Web Speech API. That API is only backed by a real recognizer in
|
||||
official Chrome/Safari builds (Google/Apple cloud speech); it is unavailable
|
||||
in Electron, Firefox, Chromium, and most self-hosted contexts. Today the
|
||||
button renders nothing (or "Dictation unavailable") in those environments —
|
||||
`web/electron/README.md` documents the gap and prescribes the fix: capture
|
||||
audio in the client and transcribe it on the Omnigent server.
|
||||
|
||||
This design adds that path: a streaming speech-to-text WebSocket on the
|
||||
server, backed by a local [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)
|
||||
model (CPU, no cloud, no per-request cost), with the mic button falling back
|
||||
to it whenever Web Speech is unavailable.
|
||||
|
||||
## Goals
|
||||
|
||||
- Dictation works in Electron, Firefox/Chromium, and the iOS/Android wrappers
|
||||
(mic permissions are already wired in all three).
|
||||
- Audio never leaves the operator's infrastructure.
|
||||
- Live partial transcripts stream into the composer while the user speaks
|
||||
(the Web Speech path today only inserts final utterances).
|
||||
- Zero new required dependencies: the STT engine ships as an optional extra
|
||||
(`omnigent[dictation]`), imported lazily, mirroring the `s3`/`modal`/
|
||||
`daytona` extras' posture. Servers without the extra (or without models)
|
||||
report `available: false` and the web UI silently keeps its current
|
||||
behavior.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Voice *conversations* (TTS replies, wake words, hands-free turn taking).
|
||||
- Replacing the Web Speech path where it works today.
|
||||
- Terminal REPL dictation (possible follow-up; shares the engine).
|
||||
- Speaker diarization, translation, non-English models beyond whatever
|
||||
sherpa-onnx model the operator installs.
|
||||
|
||||
## Server
|
||||
|
||||
### Engine — `omnigent/server/dictation.py`
|
||||
|
||||
A small engine layer isolates the recognizer behind a protocol so tests
|
||||
(and alternate backends, e.g. Whisper or an OpenAI-compatible
|
||||
transcription API) don't need the native dependency:
|
||||
|
||||
```python
|
||||
class DictationStreamHandle(Protocol):
|
||||
def feed_pcm16(self, data: bytes) -> DictationUpdate: ... # decode a chunk
|
||||
def finish(self) -> str: ... # flush tail, final text
|
||||
def close(self) -> None: ... # release (client vanished)
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DictationUpdate:
|
||||
partial: str # current in-progress utterance, display-ready (revisable)
|
||||
finalized: str | None # utterance completed by endpointing, if any
|
||||
```
|
||||
|
||||
Emitted text is **display-ready** — an engine that needs punctuation/casing
|
||||
applies it internally before returning, so the route and protocol stay
|
||||
engine-agnostic. Most modern models (Whisper, Parakeet) punctuate
|
||||
themselves; sherpa is the exception (see below).
|
||||
|
||||
**Engine registry.** Engines are registered by name and selected via
|
||||
`OMNIGENT_DICTATION_ENGINE`:
|
||||
|
||||
```python
|
||||
register_engine("sherpa", lambda: SherpaDictationEngine(...), available=_sherpa_available)
|
||||
register_engine("fake", FakeDictationEngine)
|
||||
```
|
||||
|
||||
Adding an engine (Whisper, Parakeet, a hosted API) is one `register_engine`
|
||||
call with a factory and an optional availability probe — no edits to
|
||||
`get_engine` or `engine_availability`. Third-party engines register
|
||||
themselves on import. The default (unset env var) is `sherpa`.
|
||||
|
||||
`SherpaDictationEngine` implements the protocol with a process-wide
|
||||
`OnlineRecognizer` (streaming transducer: `encoder/decoder/joiner + tokens`)
|
||||
shared across connections — the ~650 MB weights load once — plus one
|
||||
recognizer *stream* per WebSocket. Endpointing folds completed utterances
|
||||
into `finalized` and resets the stream, exactly the loop proven in pi-voice.
|
||||
An optional `OnlinePunctuation` model re-punctuates partials/finals
|
||||
(lowercase + strip punctuation before re-adding, throttled) so the live
|
||||
preview reads like a sentence. This punctuation is **internal** to the
|
||||
sherpa engine — the raw transducer emits lowercase, unpunctuated text, so
|
||||
the streams beautify before returning; it is not part of the protocol.
|
||||
|
||||
Decode calls are CPU-bound → they run via `asyncio.to_thread`, serialized by
|
||||
a per-engine `threading.Lock` (sherpa recognizer streams are not documented
|
||||
thread-safe), with a module-level semaphore capping concurrent dictation
|
||||
connections (default 2, `OMNIGENT_DICTATION_MAX_STREAMS`).
|
||||
|
||||
### Configuration
|
||||
|
||||
| Env var | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `OMNIGENT_DICTATION_MODEL_DIR` | `~/.omnigent/models/dictation/asr` | dir containing `encoder*.onnx`, `decoder*.onnx`, `joiner*.onnx`, `tokens.txt` |
|
||||
| `OMNIGENT_DICTATION_PUNCT_DIR` | `~/.omnigent/models/dictation/punct` | optional online-punctuation model dir (`model*.onnx` + `bpe.vocab`) |
|
||||
| `OMNIGENT_DICTATION_MAX_STREAMS` | `2` | concurrent dictation WebSockets |
|
||||
| `OMNIGENT_DICTATION_ENGINE` | unset (`sherpa`) | engine to use by registered name; `fake` for tests |
|
||||
|
||||
`scripts/fetch-dictation-models.sh` downloads a known-good pair (streaming
|
||||
Nemotron 0.6 B int8 + English online punctuation, both Apache-2.0 upstream)
|
||||
into the default locations. Availability is computed lazily and cached:
|
||||
extra installed **and** ASR model dir populated.
|
||||
|
||||
**Hardware sizing.** Any sherpa-onnx streaming transducer directory works —
|
||||
point `OMNIGENT_DICTATION_MODEL_DIR` at it. Streaming dictation needs ≥1×
|
||||
realtime decode; measured with this engine loop (int8, 4 threads, 100 ms
|
||||
chunks):
|
||||
|
||||
| Model | Apple M-series | Intel N95 (4 E-cores, loaded box) | RAM |
|
||||
|---|---|---|---|
|
||||
| Nemotron 0.6 B (fetch-script default) | ~9× realtime | 0.6–0.7× — **too slow** | ~1.0 GB |
|
||||
| `streaming-zipformer-en-2023-06-26` | — | 1.4–2.3× realtime | ~190 MB |
|
||||
| `streaming-zipformer-en-20M` | — | 3.6–4.9× realtime | ~130 MB |
|
||||
|
||||
On N100/N95-class mini-PC servers, use the mid-size zipformer (accuracy held
|
||||
up in spot checks; the 20 M model audibly degrades) and consider
|
||||
`OMNIGENT_DICTATION_MAX_STREAMS=1`.
|
||||
|
||||
**Other languages.** The engine is language-agnostic — dictation speaks
|
||||
whatever language the installed model was trained on. The
|
||||
[sherpa-onnx streaming-model catalog](https://k2-fsa.github.io/sherpa/onnx/pretrained_models/online-transducer/index.html)
|
||||
includes Chinese, Chinese/English bilingual
|
||||
(`sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20`), French
|
||||
(`sherpa-onnx-streaming-zipformer-fr-2023-04-14`), Korean, and more; point
|
||||
`OMNIGENT_DICTATION_MODEL_DIR` at any of them. Two caveats: the fetch
|
||||
script's punctuation model is English-only, so leave
|
||||
`OMNIGENT_DICTATION_PUNCT_DIR` unpopulated for other languages (raw
|
||||
recognizer output is emitted as-is), and the mic button's `lang` prop only
|
||||
affects the Web Speech path — the server path's language is decided by the
|
||||
operator's model choice.
|
||||
|
||||
Where a mini-PC server can't run the model an operator wants at realtime, a
|
||||
follow-up adds a **remote worker**: a `RemoteDictationEngine` (registered as
|
||||
`OMNIGENT_DICTATION_ENGINE=remote`) that relays takes over this same wire
|
||||
protocol to a beefier LAN box. It slots into the registry without changing
|
||||
the route or the protocol, so it ships separately from this core PR.
|
||||
|
||||
### Routes — `omnigent/server/routes/dictation.py`
|
||||
|
||||
`create_dictation_router(*, auth_provider=None, engine_provider=None)`,
|
||||
registered in `create_app` under `/v1` like every other router. Dictation is
|
||||
not session-scoped (the new-chat composer has no session yet), so auth is
|
||||
identity-level only: authenticated user required when an auth provider is
|
||||
configured, open in single-user/dev mode — the same posture as
|
||||
`GET /v1/harnesses`.
|
||||
|
||||
Availability rides the existing boot-time capability probe —
|
||||
`dictation_available` on **`GET /v1/info`** — rather than a dedicated
|
||||
endpoint; the UI needs one boolean, once per page load.
|
||||
|
||||
- **`WS /v1/dictation/stream`** — wire protocol (documented in the module
|
||||
docstring, mirroring `terminal_attach.py`):
|
||||
- **Client → server, binary frames**: raw 16 kHz mono s16le PCM.
|
||||
- **Client → server, text frames**: JSON control messages.
|
||||
`{"type": "stop"}` requests a flush; unknown shapes are ignored for
|
||||
forward compatibility.
|
||||
- **Server → client, text frames**: JSON events.
|
||||
- `{"type": "ready"}` — sent once after accept; the client may start
|
||||
streaming audio.
|
||||
- `{"type": "partial", "text": ...}` — revisable in-progress utterance,
|
||||
throttled to ~6 Hz.
|
||||
- `{"type": "final", "text": ...}` — an utterance completed by
|
||||
endpointing; the client appends it and clears the partial region.
|
||||
- `{"type": "stopped", "text": ...}` — response to `stop`: the flushed
|
||||
tail utterance (possibly empty). The server closes after sending it.
|
||||
- `{"type": "error", "message": ...}` — fatal; server closes.
|
||||
|
||||
The route holds no session state; a connection is one dictation take.
|
||||
|
||||
## Web
|
||||
|
||||
### Capture — `web/src/lib/dictation.ts`
|
||||
|
||||
`DictationSession` owns the full client pipeline:
|
||||
`getUserMedia({audio})` → `AudioContext` → `AudioWorkletNode` (the worklet,
|
||||
inlined as a Blob module, downsamples from the context rate to 16 kHz and
|
||||
converts Float32 → Int16, posting 100 ms chunks) → binary WS frames via
|
||||
`resolveWebSocketUrl("/v1/dictation/stream")` (the same host seam the
|
||||
terminal-attach and session-updates sockets ride, so embed hosts and the
|
||||
Vite dev proxy keep working). Callbacks: `onPartial`, `onFinal`, `onError`;
|
||||
`stop()` sends `{"type":"stop"}`, resolves with the flushed tail, and
|
||||
releases the mic tracks and audio context.
|
||||
|
||||
Availability comes from the existing `/v1/info` capability context
|
||||
(`useServerInfo().dictation_available`) — no extra request.
|
||||
|
||||
### Mic button — `ComposerMicButton.tsx`
|
||||
|
||||
Mode selection: **Web Speech when the browser has a working one, server
|
||||
dictation otherwise** — no behavior change for Chrome/Safari users;
|
||||
Electron, Firefox, and Chromium gain a working button. "Working" cannot be
|
||||
detected statically: Electron and plain Chromium expose the
|
||||
`SpeechRecognition` constructor but its cloud backend rejects them at
|
||||
runtime with a `network` error. So Web Speech stays primary whenever the
|
||||
constructor exists, and a take that dies with `network` falls back to the
|
||||
server **for that take** (retried immediately, so the user's click still
|
||||
lands); the next take tries Web Speech again, so a transient blip in real
|
||||
Chrome never permanently downgrades the page. With no constructor at all
|
||||
(Firefox), takes go to the server directly.
|
||||
|
||||
New optional prop `onInterim?: (text: string) => void`. In server mode the
|
||||
button emits `onInterim` for partial frames and the existing
|
||||
`onTranscript` for finals. Both composers (`ChatPage`, `NewChatDialog`)
|
||||
share a small hook, `useDictationInsert(setValue)`, that appends finals and
|
||||
maintains a replaceable trailing interim region in the textarea value, so
|
||||
text forms live while speaking. When `onInterim` is absent (Web Speech
|
||||
mode), behavior is exactly today's.
|
||||
|
||||
## Testing
|
||||
|
||||
- **Server (pytest, `tests/server/routes/test_dictation.py`)**: drive the
|
||||
real route with `TestClient.websocket_connect` and a fake engine injected
|
||||
through `engine_provider` — no sherpa dependency in CI. Cases:
|
||||
`/v1/info` availability (with and without an engine), ready→partial→final
|
||||
→stopped flow, stop-flush, auth rejection with a no-identity provider,
|
||||
stream-cap rejection.
|
||||
- **Engine unit tests** skip unless sherpa-onnx and models are present
|
||||
(developer machines), keeping CI hermetic.
|
||||
- **Web (Vitest, `ComposerMicButton.test.tsx` + `dictation.test.ts`)**:
|
||||
mode selection, partial/final callback flow against a mocked WebSocket and
|
||||
mocked AudioWorklet capture.
|
||||
- **e2e (Playwright, `tests/e2e_ui/`)**: a fake engine selected via env
|
||||
(`OMNIGENT_DICTATION_ENGINE=fake`, emits a scripted transcript) lets the
|
||||
full browser→WS→server→composer loop run headless without a mic:
|
||||
the test grants fake mic permissions, clicks the mic button, and asserts
|
||||
the scripted text lands in the composer.
|
||||
|
||||
## Rollout / compatibility
|
||||
|
||||
- No schema changes, no migrations, no new required deps.
|
||||
- Servers without the extra: `/v1/info` reports `dictation_available: false`;
|
||||
the web UI behaves exactly as today.
|
||||
- Old web clients against new servers: unaffected (new route + one new
|
||||
`/v1/info` field only).
|
||||
- New web clients against old servers: `/v1/info` lacks the field → treated
|
||||
as unavailable → today's behavior.
|
||||
+21
-3
@@ -65,6 +65,7 @@ from omnigent.server.performance_metrics import (
|
||||
from omnigent.server.routes.builtin_agents import create_builtin_agents_router
|
||||
from omnigent.server.routes.comments import create_comments_router
|
||||
from omnigent.server.routes.default_policies import create_default_policies_router
|
||||
from omnigent.server.routes.dictation import create_dictation_router
|
||||
from omnigent.server.routes.harnesses import create_harnesses_router
|
||||
from omnigent.server.routes.imports import create_imports_router
|
||||
from omnigent.server.routes.policy_registry import create_policy_registry_router
|
||||
@@ -1999,9 +2000,10 @@ def create_app(
|
||||
source, the login URL, whether first-run admin setup is
|
||||
still pending (``needs_setup``), coarse capability
|
||||
booleans (``databricks_features``,
|
||||
``managed_sandboxes_enabled``, ``single_user``), the short
|
||||
sandbox provider name (``sandbox_provider``) the web UI labels
|
||||
the new-session sandbox option with, and the installed
|
||||
``managed_sandboxes_enabled``, ``dictation_available``,
|
||||
``single_user``), the short sandbox provider name
|
||||
(``sandbox_provider``) the web UI labels the new-session
|
||||
sandbox option with, and the installed
|
||||
``server_version`` (already public via ``/api/version``).
|
||||
"""
|
||||
from omnigent.server.auth import UnifiedAuthProvider, local_single_user_enabled
|
||||
@@ -2077,6 +2079,13 @@ def create_app(
|
||||
)
|
||||
except ImportError:
|
||||
smart_routing_enabled = False
|
||||
# dictation_available gates the composer mic button's server
|
||||
# speech-to-text fallback (designs/server-dictation.md). Checks
|
||||
# config presence only (extra installed + models on disk) — no
|
||||
# model is loaded here.
|
||||
from omnigent.server.dictation import engine_availability
|
||||
|
||||
dictation_available, _ = engine_availability()
|
||||
return {
|
||||
"accounts_enabled": accounts_enabled,
|
||||
"single_user": single_user,
|
||||
@@ -2089,6 +2098,7 @@ def create_app(
|
||||
"public_sharing_enabled": public_sharing_enabled,
|
||||
"server_version": _server_version(),
|
||||
"smart_routing_enabled": smart_routing_enabled,
|
||||
"dictation_available": dictation_available,
|
||||
}
|
||||
|
||||
@app.get("/v1/me", response_model=None) # Union return type (dict | JSONResponse)
|
||||
@@ -2195,6 +2205,14 @@ def create_app(
|
||||
prefix="/v1",
|
||||
tags=["harnesses"],
|
||||
)
|
||||
# Server-side speech-to-text behind the composer mic button
|
||||
# (designs/server-dictation.md). Availability is probed lazily, so
|
||||
# registering unconditionally is free for servers without the extra.
|
||||
app.include_router(
|
||||
create_dictation_router(auth_provider=auth_provider),
|
||||
prefix="/v1",
|
||||
tags=["dictation"],
|
||||
)
|
||||
app.include_router(
|
||||
create_terminal_attach_router(
|
||||
auth_provider=auth_provider,
|
||||
|
||||
@@ -0,0 +1,525 @@
|
||||
"""Local streaming speech-to-text engine for composer dictation.
|
||||
|
||||
Backs the ``WS /v1/dictation/stream`` route
|
||||
(:mod:`omnigent.server.routes.dictation`) with an on-server recognizer
|
||||
so dictation works where the browser Web Speech API does not (Electron,
|
||||
Firefox/Chromium, self-hosted deployments) and audio never leaves the
|
||||
operator's infrastructure. See ``designs/server-dictation.md``.
|
||||
|
||||
Engine selection
|
||||
----------------
|
||||
|
||||
Engines are looked up by name in a small registry
|
||||
(:func:`register_engine`), selected via ``OMNIGENT_DICTATION_ENGINE``:
|
||||
|
||||
- unset (default) — the sherpa-onnx engine. Requires the ``dictation``
|
||||
extra (``pip install omnigent[dictation]``) and a streaming transducer
|
||||
model on disk; both are checked lazily so the base install carries no
|
||||
new dependencies.
|
||||
- ``sherpa`` — the same engine, named explicitly.
|
||||
- ``fake`` — a deterministic scripted engine used by tests and the
|
||||
Playwright e2e suite; no native dependency, no models, no microphone.
|
||||
|
||||
Adding an engine (e.g. Whisper) is one :func:`register_engine` call with
|
||||
a factory and an availability probe — no edits to :func:`get_engine` or
|
||||
:func:`engine_availability`. Third-party engines register themselves on
|
||||
import.
|
||||
|
||||
sherpa-onnx engine
|
||||
------------------
|
||||
|
||||
A process-wide ``OnlineRecognizer`` (streaming transducer:
|
||||
``encoder/decoder/joiner + tokens.txt``) is shared across connections so
|
||||
the model weights load once; each WebSocket gets its own recognizer
|
||||
*stream*. Endpoint detection folds completed utterances into
|
||||
``DictationUpdate.finalized`` and resets the stream. An optional online
|
||||
punctuation model re-punctuates emitted text (the raw transducer output
|
||||
is lowercased and stripped of punctuation first — the model wants clean
|
||||
input) so live partials read like sentences. The recognizer returns
|
||||
display-ready text directly; punctuation is an internal detail, not part
|
||||
of the engine protocol (most models — Whisper, Parakeet — punctuate
|
||||
themselves).
|
||||
|
||||
Recognizer calls are CPU-bound and sherpa streams are not documented
|
||||
thread-safe, so every recognizer/punctuation call holds the engine's
|
||||
``threading.Lock``; callers run them via ``asyncio.to_thread`` to keep
|
||||
the event loop responsive.
|
||||
|
||||
Model layout
|
||||
------------
|
||||
|
||||
====================================== ==========================================
|
||||
Env var Default
|
||||
====================================== ==========================================
|
||||
``OMNIGENT_DICTATION_MODEL_DIR`` ``~/.omnigent/models/dictation/asr``
|
||||
``OMNIGENT_DICTATION_PUNCT_DIR`` ``~/.omnigent/models/dictation/punct``
|
||||
====================================== ==========================================
|
||||
|
||||
The ASR dir must contain ``encoder*.onnx``, ``decoder*.onnx``,
|
||||
``joiner*.onnx`` and ``tokens.txt`` (int8 variants preferred when both
|
||||
are present). The punctuation dir (``model*.onnx`` + ``bpe.vocab``) is
|
||||
optional — without it, raw recognizer output is emitted as-is.
|
||||
``scripts/fetch-dictation-models.sh`` downloads a known-good pair into
|
||||
the default locations.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
_logger = logging.getLogger(__name__)
|
||||
|
||||
ENGINE_ENV = "OMNIGENT_DICTATION_ENGINE"
|
||||
MODEL_DIR_ENV = "OMNIGENT_DICTATION_MODEL_DIR"
|
||||
PUNCT_DIR_ENV = "OMNIGENT_DICTATION_PUNCT_DIR"
|
||||
MAX_STREAMS_ENV = "OMNIGENT_DICTATION_MAX_STREAMS"
|
||||
|
||||
#: Built-in engine names. The default (empty ``OMNIGENT_DICTATION_ENGINE``)
|
||||
#: resolves to the sherpa engine.
|
||||
ENGINE_SHERPA = "sherpa"
|
||||
ENGINE_FAKE = "fake"
|
||||
_DEFAULT_ENGINE = ENGINE_SHERPA
|
||||
|
||||
#: The one PCM format the stream route accepts: 16 kHz mono s16le.
|
||||
SAMPLE_RATE = 16000
|
||||
_BYTES_PER_SECOND = SAMPLE_RATE * 2
|
||||
|
||||
#: Stable machine-readable unavailability reasons.
|
||||
REASON_EXTRA_NOT_INSTALLED = "extra_not_installed"
|
||||
REASON_MODELS_MISSING = "models_missing"
|
||||
REASON_UNKNOWN_ENGINE = "unknown_engine"
|
||||
|
||||
DEFAULT_MAX_STREAMS = 2
|
||||
|
||||
# Endpoint rules mirror sherpa-onnx defaults tuned for dictation: a long
|
||||
# hard stop (rule1, silence with no text yet), a shorter pause once
|
||||
# something was said (rule2), and a max utterance length (rule3).
|
||||
_RULE1_MIN_TRAILING_SILENCE_S = 3.5
|
||||
_RULE2_MIN_TRAILING_SILENCE_S = 1.6
|
||||
_RULE3_MIN_UTTERANCE_LENGTH_S = 30.0
|
||||
|
||||
_PUNCT_STRIP_RE = re.compile(r"[.,?!:;…]+")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DictationUpdate:
|
||||
"""Result of feeding one audio chunk to a dictation stream.
|
||||
|
||||
:param partial: The current in-progress utterance, display-ready
|
||||
(punctuated/cased by the engine if it does that). Revisable —
|
||||
later updates may rewrite earlier words as more context arrives.
|
||||
:param finalized: An utterance completed by endpoint detection (a
|
||||
pause), if one closed on this chunk, display-ready. The partial
|
||||
restarts empty after a finalized utterance.
|
||||
"""
|
||||
|
||||
partial: str
|
||||
finalized: str | None = None
|
||||
|
||||
|
||||
class DictationStreamHandle(Protocol):
|
||||
"""One dictation take: a stateful recognizer stream.
|
||||
|
||||
All methods are synchronous and CPU-bound; call them via
|
||||
``asyncio.to_thread`` from async code. Emitted text is display-ready:
|
||||
engines that need punctuation/casing apply it internally before
|
||||
returning (see the sherpa engine), so the route just forwards text.
|
||||
"""
|
||||
|
||||
def feed_pcm16(self, data: bytes) -> DictationUpdate:
|
||||
"""Feed a chunk of 16 kHz mono s16le PCM and decode it."""
|
||||
...
|
||||
|
||||
def finish(self) -> str:
|
||||
"""Flush trailing audio and return the final tail utterance."""
|
||||
...
|
||||
|
||||
def close(self) -> None:
|
||||
"""Release the take's resources without flushing (client vanished).
|
||||
|
||||
Idempotent, and safe after :meth:`finish`. A no-op for the
|
||||
in-process engines (the stream frees with the handle); the hook
|
||||
exists for engines holding an external resource.
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
class DictationEngine(Protocol):
|
||||
"""Factory for dictation streams; one engine is shared per process."""
|
||||
|
||||
def create_stream(self) -> DictationStreamHandle:
|
||||
"""Open a fresh recognizer stream for one connection."""
|
||||
...
|
||||
|
||||
|
||||
#: An engine's availability probe: ``() -> (available, reason)`` where
|
||||
#: *reason* is ``None`` when available, else a machine-readable
|
||||
#: ``REASON_*`` string. Called without loading any model.
|
||||
AvailabilityProbe = Callable[[], "tuple[bool, str | None]"]
|
||||
EngineFactory = Callable[[], DictationEngine]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _EngineEntry:
|
||||
factory: EngineFactory
|
||||
available: AvailabilityProbe
|
||||
|
||||
|
||||
_ENGINE_REGISTRY: dict[str, _EngineEntry] = {}
|
||||
|
||||
|
||||
def register_engine(
|
||||
name: str,
|
||||
factory: EngineFactory,
|
||||
*,
|
||||
available: AvailabilityProbe | None = None,
|
||||
) -> None:
|
||||
"""Register a dictation engine under *name*.
|
||||
|
||||
Selected via ``OMNIGENT_DICTATION_ENGINE=<name>``. This is the whole
|
||||
swap-in surface: a new engine (Whisper, Parakeet, …) is one call with
|
||||
a factory and an optional availability probe — no edits to
|
||||
:func:`get_engine` or :func:`engine_availability`.
|
||||
|
||||
:param name: Selector value, e.g. ``"whisper"``.
|
||||
:param factory: Builds the engine on first use (weights load here —
|
||||
keep it lazy).
|
||||
:param available: Probe returning ``(available, reason)`` without
|
||||
loading a model. Defaults to always-available (``(True, None)``)
|
||||
— right for engines with no optional dependency or model on disk.
|
||||
"""
|
||||
_ENGINE_REGISTRY[name] = _EngineEntry(
|
||||
factory=factory,
|
||||
available=available or (lambda: (True, None)),
|
||||
)
|
||||
|
||||
|
||||
def _asr_dir() -> Path:
|
||||
default = Path.home() / ".omnigent" / "models" / "dictation" / "asr"
|
||||
return Path(os.environ.get(MODEL_DIR_ENV) or default).expanduser()
|
||||
|
||||
|
||||
def _punct_dir() -> Path:
|
||||
default = Path.home() / ".omnigent" / "models" / "dictation" / "punct"
|
||||
return Path(os.environ.get(PUNCT_DIR_ENV) or default).expanduser()
|
||||
|
||||
|
||||
def max_streams() -> int:
|
||||
"""Concurrent dictation connections allowed (decode is CPU-bound)."""
|
||||
raw = os.environ.get(MAX_STREAMS_ENV, "")
|
||||
try:
|
||||
value = int(raw)
|
||||
except ValueError:
|
||||
return DEFAULT_MAX_STREAMS
|
||||
return value if value > 0 else DEFAULT_MAX_STREAMS
|
||||
|
||||
|
||||
def _pick_model_file(model_dir: Path, stem: str) -> Path | None:
|
||||
"""Find ``<stem>*.onnx`` in *model_dir*, preferring int8 variants.
|
||||
|
||||
Quantized files decode fastest on CPU and are what the fetch script
|
||||
installs; float fallbacks let operators drop in any upstream export.
|
||||
"""
|
||||
candidates = sorted(model_dir.glob(f"{stem}*.onnx"))
|
||||
if not candidates:
|
||||
return None
|
||||
for candidate in candidates:
|
||||
if "int8" in candidate.name:
|
||||
return candidate
|
||||
return candidates[0]
|
||||
|
||||
|
||||
def _asr_files(model_dir: Path) -> dict[str, Path] | None:
|
||||
"""Resolve the transducer file set, or ``None`` if incomplete."""
|
||||
tokens = model_dir / "tokens.txt"
|
||||
encoder = _pick_model_file(model_dir, "encoder")
|
||||
decoder = _pick_model_file(model_dir, "decoder")
|
||||
joiner = _pick_model_file(model_dir, "joiner")
|
||||
if not tokens.is_file() or encoder is None or decoder is None or joiner is None:
|
||||
return None
|
||||
return {"tokens": tokens, "encoder": encoder, "decoder": decoder, "joiner": joiner}
|
||||
|
||||
|
||||
def _punct_files(punct_dir: Path) -> dict[str, Path] | None:
|
||||
"""Resolve the optional punctuation file set, or ``None``."""
|
||||
model = _pick_model_file(punct_dir, "model")
|
||||
vocab = punct_dir / "bpe.vocab"
|
||||
if model is None or not vocab.is_file():
|
||||
return None
|
||||
return {"model": model, "vocab": vocab}
|
||||
|
||||
|
||||
def _sherpa_available() -> tuple[bool, str | None]:
|
||||
"""Availability probe for the sherpa engine (loads nothing)."""
|
||||
if importlib.util.find_spec("sherpa_onnx") is None:
|
||||
return False, REASON_EXTRA_NOT_INSTALLED
|
||||
if _asr_files(_asr_dir()) is None:
|
||||
return False, REASON_MODELS_MISSING
|
||||
return True, None
|
||||
|
||||
|
||||
def _selected_engine_name() -> str:
|
||||
"""Resolve the configured engine name (default: sherpa)."""
|
||||
return os.environ.get(ENGINE_ENV, "").strip() or _DEFAULT_ENGINE
|
||||
|
||||
|
||||
def engine_availability() -> tuple[bool, str | None]:
|
||||
"""Report whether dictation can serve, without loading any model.
|
||||
|
||||
Resolves the configured engine and calls its registered availability
|
||||
probe. Unknown engine names report unavailable.
|
||||
|
||||
:returns: ``(available, reason)`` where *reason* is ``None`` when
|
||||
available, else a machine-readable ``REASON_*`` string.
|
||||
"""
|
||||
entry = _ENGINE_REGISTRY.get(_selected_engine_name())
|
||||
if entry is None:
|
||||
return False, REASON_UNKNOWN_ENGINE
|
||||
return entry.available()
|
||||
|
||||
|
||||
_engine_lock = threading.Lock()
|
||||
_engine: DictationEngine | None = None
|
||||
|
||||
|
||||
def get_engine() -> DictationEngine:
|
||||
"""Return the process-wide engine, loading models on first use.
|
||||
|
||||
The configured engine name is resolved once, on the first successful
|
||||
load — a failed load caches nothing, so a server that gains models
|
||||
later serves the next take without a restart. Tests never hit this:
|
||||
they inject an engine through the router's ``engine_provider``.
|
||||
|
||||
:raises RuntimeError: When the configured engine is unknown or
|
||||
unavailable (check :func:`engine_availability` first), or the
|
||||
model fails to load.
|
||||
"""
|
||||
global _engine
|
||||
with _engine_lock:
|
||||
if _engine is not None:
|
||||
return _engine
|
||||
name = _selected_engine_name()
|
||||
entry = _ENGINE_REGISTRY.get(name)
|
||||
if entry is None:
|
||||
raise RuntimeError(f"unknown dictation engine: {name!r}")
|
||||
available, reason = entry.available()
|
||||
if not available:
|
||||
raise RuntimeError(f"dictation unavailable: {reason}")
|
||||
_engine = entry.factory()
|
||||
return _engine
|
||||
|
||||
|
||||
class SherpaDictationEngine:
|
||||
"""Streaming sherpa-onnx transducer + optional online punctuation."""
|
||||
|
||||
def __init__(self, asr_dir: Path, punct_dir: Path) -> None:
|
||||
"""Load models eagerly; construction is slow (seconds).
|
||||
|
||||
:param asr_dir: Directory holding the streaming transducer.
|
||||
:param punct_dir: Directory holding the optional punctuation
|
||||
model; silently skipped when absent or incomplete.
|
||||
:raises RuntimeError: If the ASR file set is incomplete.
|
||||
"""
|
||||
import sherpa_onnx
|
||||
|
||||
files = _asr_files(asr_dir)
|
||||
if files is None:
|
||||
raise RuntimeError(f"dictation ASR model incomplete in {asr_dir}")
|
||||
_logger.info("Loading dictation ASR model from %s", asr_dir)
|
||||
self._recognizer = sherpa_onnx.OnlineRecognizer.from_transducer(
|
||||
tokens=str(files["tokens"]),
|
||||
encoder=str(files["encoder"]),
|
||||
decoder=str(files["decoder"]),
|
||||
joiner=str(files["joiner"]),
|
||||
num_threads=4,
|
||||
sample_rate=SAMPLE_RATE,
|
||||
feature_dim=80,
|
||||
enable_endpoint_detection=True,
|
||||
rule1_min_trailing_silence=_RULE1_MIN_TRAILING_SILENCE_S,
|
||||
rule2_min_trailing_silence=_RULE2_MIN_TRAILING_SILENCE_S,
|
||||
rule3_min_utterance_length=_RULE3_MIN_UTTERANCE_LENGTH_S,
|
||||
decoding_method="greedy_search",
|
||||
provider="cpu",
|
||||
)
|
||||
self._punct: Any = None
|
||||
punct_files = _punct_files(punct_dir)
|
||||
if punct_files is not None:
|
||||
try:
|
||||
self._punct = sherpa_onnx.OnlinePunctuation(
|
||||
sherpa_onnx.OnlinePunctuationConfig(
|
||||
model_config=sherpa_onnx.OnlinePunctuationModelConfig(
|
||||
cnn_bilstm=str(punct_files["model"]),
|
||||
bpe_vocab=str(punct_files["vocab"]),
|
||||
num_threads=1,
|
||||
provider="cpu",
|
||||
)
|
||||
)
|
||||
)
|
||||
except Exception: # noqa: BLE001 - punctuation is best-effort
|
||||
_logger.warning(
|
||||
"dictation punctuation model failed to load from %s; "
|
||||
"emitting raw recognizer output",
|
||||
punct_dir,
|
||||
exc_info=True,
|
||||
)
|
||||
# Serializes all recognizer/punctuation calls: sherpa streams are
|
||||
# not documented thread-safe, and decode is CPU-bound anyway.
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def _beautify(self, text: str) -> str:
|
||||
"""Re-punctuate and re-case *text* for display.
|
||||
|
||||
Internal: the raw transducer emits lowercase, punctuation-free
|
||||
text, so the streams call this before returning so partials/finals
|
||||
read like sentences. Identity when no punctuation model loaded.
|
||||
"""
|
||||
if self._punct is None or not text:
|
||||
return text
|
||||
# The model expects lowercase, punctuation-free input.
|
||||
cleaned = _PUNCT_STRIP_RE.sub("", text.lower())
|
||||
try:
|
||||
with self._lock:
|
||||
return self._punct.add_punctuation_with_case(cleaned)
|
||||
except Exception: # noqa: BLE001 - never fail a take over cosmetics
|
||||
return text
|
||||
|
||||
def create_stream(self) -> _SherpaStream:
|
||||
"""Open a recognizer stream for one connection."""
|
||||
with self._lock:
|
||||
return _SherpaStream(self, self._recognizer.create_stream())
|
||||
|
||||
|
||||
class _SherpaStream:
|
||||
"""Per-connection recognizer stream (see :class:`DictationStreamHandle`)."""
|
||||
|
||||
def __init__(self, engine: SherpaDictationEngine, stream: Any) -> None:
|
||||
self._engine = engine
|
||||
self._stream = stream
|
||||
|
||||
def feed_pcm16(self, data: bytes) -> DictationUpdate:
|
||||
"""Decode one PCM chunk; fold an endpoint into ``finalized``."""
|
||||
import numpy as np
|
||||
|
||||
# Drop a trailing odd byte rather than crash the take; the next
|
||||
# frame realigns (client frames are always whole samples).
|
||||
usable = len(data) - (len(data) % 2)
|
||||
if usable <= 0:
|
||||
return DictationUpdate(partial="")
|
||||
samples = np.frombuffer(data[:usable], dtype=np.int16).astype(np.float32) / 32768.0
|
||||
engine = self._engine
|
||||
recognizer = engine._recognizer
|
||||
with engine._lock:
|
||||
self._stream.accept_waveform(SAMPLE_RATE, samples)
|
||||
while recognizer.is_ready(self._stream):
|
||||
recognizer.decode_stream(self._stream)
|
||||
partial = recognizer.get_result(self._stream).strip()
|
||||
finalized: str | None = None
|
||||
if recognizer.is_endpoint(self._stream):
|
||||
if partial:
|
||||
finalized = partial
|
||||
partial = ""
|
||||
recognizer.reset(self._stream)
|
||||
# Punctuate outside the recognizer lock's decode section (beautify
|
||||
# takes the lock itself). Emit display-ready text so the route and
|
||||
# protocol stay engine-agnostic.
|
||||
return DictationUpdate(
|
||||
partial=engine._beautify(partial),
|
||||
finalized=engine._beautify(finalized) if finalized else None,
|
||||
)
|
||||
|
||||
def finish(self) -> str:
|
||||
"""Flush the tail: pad with silence, drain, return final text."""
|
||||
import numpy as np
|
||||
|
||||
engine = self._engine
|
||||
recognizer = engine._recognizer
|
||||
with engine._lock:
|
||||
# One second of silence pushes trailing speech past the
|
||||
# feature window so the last words decode.
|
||||
self._stream.accept_waveform(SAMPLE_RATE, np.zeros(SAMPLE_RATE, dtype=np.float32))
|
||||
self._stream.input_finished()
|
||||
while recognizer.is_ready(self._stream):
|
||||
recognizer.decode_stream(self._stream)
|
||||
tail = recognizer.get_result(self._stream).strip()
|
||||
return engine._beautify(tail)
|
||||
|
||||
def close(self) -> None:
|
||||
"""No-op: the recognizer stream frees with the handle."""
|
||||
|
||||
|
||||
#: Scripted transcript the fake engine reveals; asserted verbatim by the
|
||||
#: server route tests and the Playwright e2e test.
|
||||
FAKE_SCRIPT = "server dictation smoke test transcript"
|
||||
|
||||
# The fake reveals one word per this much audio, so tests control the
|
||||
# transcript by the number of bytes they send.
|
||||
_FAKE_BYTES_PER_WORD = _BYTES_PER_SECOND // 10
|
||||
|
||||
|
||||
class FakeDictationEngine:
|
||||
"""Deterministic engine for tests: audio bytes in, script words out.
|
||||
|
||||
Reveals one word of :data:`FAKE_SCRIPT` per 100 ms of audio fed
|
||||
(regardless of content), finalizing the sentence when it completes.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
#: The most recently opened stream, for cleanup assertions.
|
||||
self.last_stream: _FakeStream | None = None
|
||||
|
||||
def create_stream(self) -> _FakeStream:
|
||||
"""Open a scripted stream."""
|
||||
self.last_stream = _FakeStream()
|
||||
return self.last_stream
|
||||
|
||||
|
||||
class _FakeStream:
|
||||
"""Per-connection scripted stream (see :class:`FakeDictationEngine`)."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._words = FAKE_SCRIPT.split()
|
||||
self._bytes_seen = 0
|
||||
self._done = False
|
||||
self.closed = False
|
||||
|
||||
def feed_pcm16(self, data: bytes) -> DictationUpdate:
|
||||
"""Reveal script words proportional to audio fed."""
|
||||
if self._done:
|
||||
return DictationUpdate(partial="")
|
||||
self._bytes_seen += len(data)
|
||||
revealed = self._bytes_seen // _FAKE_BYTES_PER_WORD
|
||||
if revealed >= len(self._words):
|
||||
self._done = True
|
||||
return DictationUpdate(partial="", finalized=" ".join(self._words))
|
||||
return DictationUpdate(partial=" ".join(self._words[:revealed]))
|
||||
|
||||
def finish(self) -> str:
|
||||
"""Return the words revealed so far as the tail utterance."""
|
||||
if self._done:
|
||||
return ""
|
||||
revealed = min(self._bytes_seen // _FAKE_BYTES_PER_WORD, len(self._words))
|
||||
self._done = True
|
||||
return " ".join(self._words[:revealed])
|
||||
|
||||
def close(self) -> None:
|
||||
"""Record the close so tests can assert take cleanup."""
|
||||
self.closed = True
|
||||
|
||||
|
||||
# Built-in engines register themselves at import. The sherpa factory is
|
||||
# lazy (weights load on first take), so importing this module costs no
|
||||
# model RAM.
|
||||
register_engine(
|
||||
ENGINE_SHERPA,
|
||||
lambda: SherpaDictationEngine(_asr_dir(), _punct_dir()),
|
||||
available=_sherpa_available,
|
||||
)
|
||||
register_engine(ENGINE_FAKE, FakeDictationEngine)
|
||||
@@ -0,0 +1,206 @@
|
||||
"""Streaming dictation route: the transcription WebSocket.
|
||||
|
||||
This module hosts the server-side speech-to-text surface behind the
|
||||
composer mic button (``designs/server-dictation.md``):
|
||||
|
||||
- ``WS /v1/dictation/stream`` — one connection per dictation take.
|
||||
|
||||
Availability is advertised as ``dictation_available`` on ``GET /v1/info``
|
||||
(the web UI's boot-time capability probe); there is no separate probe
|
||||
endpoint.
|
||||
|
||||
Wire protocol on the WebSocket
|
||||
------------------------------
|
||||
|
||||
- **Client → server, binary frames**: raw 16 kHz mono s16le PCM. The
|
||||
browser worklet downsamples from the capture rate before sending.
|
||||
- **Client → server, text frames**: JSON control messages.
|
||||
``{"type": "stop"}`` asks the server to flush trailing audio and
|
||||
finish the take. Unknown shapes are ignored so future control
|
||||
messages don't break older servers.
|
||||
- **Server → client, text frames**: JSON events.
|
||||
- ``{"type": "ready"}`` — sent once after the engine is ready;
|
||||
the client may start streaming audio.
|
||||
- ``{"type": "partial", "text": ...}`` — revisable in-progress
|
||||
utterance, throttled server-side.
|
||||
- ``{"type": "final", "text": ...}`` — an utterance completed by
|
||||
endpoint detection (a pause). The client appends it and clears
|
||||
its partial region.
|
||||
- ``{"type": "stopped", "text": ...}`` — reply to ``stop``: the
|
||||
flushed tail utterance (possibly empty). The server closes the
|
||||
socket after sending it.
|
||||
- ``{"type": "error", "message": ...}`` — fatal; the server closes.
|
||||
|
||||
Auth
|
||||
----
|
||||
|
||||
Dictation is not session-scoped — the new-chat composer dictates before
|
||||
any session exists — so the check is identity-level only, matching
|
||||
``GET /v1/harnesses``: when an auth provider is configured the caller
|
||||
must be authenticated (the WebSocket handshake carries identity via the
|
||||
ingress/dev proxy exactly like the terminal-attach socket); in
|
||||
single-user/dev mode the route is open.
|
||||
|
||||
Capacity
|
||||
--------
|
||||
|
||||
Decoding is CPU-bound, so concurrent takes are capped (default 2,
|
||||
``OMNIGENT_DICTATION_MAX_STREAMS``). Over-cap connections are accepted
|
||||
and immediately closed with code 1013 (try again later) so the client
|
||||
can distinguish "busy" from "broken".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from typing import Final
|
||||
|
||||
from fastapi import APIRouter, WebSocket, WebSocketDisconnect, WebSocketException
|
||||
from starlette import status
|
||||
|
||||
from omnigent.server.auth import AuthProvider
|
||||
from omnigent.server.dictation import (
|
||||
DictationEngine,
|
||||
DictationStreamHandle,
|
||||
get_engine,
|
||||
max_streams,
|
||||
)
|
||||
|
||||
_logger = logging.getLogger(__name__)
|
||||
|
||||
_WS_CLOSE_TRY_AGAIN_LATER: Final[int] = 1013
|
||||
_WS_CLOSE_INTERNAL_ERROR: Final[int] = 1011
|
||||
|
||||
#: Minimum interval between partial-transcript pushes. Keeps the socket
|
||||
#: chatty enough for live text without a frame per audio chunk.
|
||||
_PARTIAL_INTERVAL_S: Final[float] = 0.15
|
||||
|
||||
|
||||
def create_dictation_router(
|
||||
*,
|
||||
auth_provider: AuthProvider | None = None,
|
||||
engine_provider: Callable[[], DictationEngine] | None = None,
|
||||
) -> APIRouter:
|
||||
"""Build the router carrying the dictation stream route.
|
||||
|
||||
Wired into the FastAPI app under the ``/v1`` prefix in
|
||||
:func:`omnigent.server.app.create_app`.
|
||||
|
||||
:param auth_provider: Optional provider used to authenticate the
|
||||
WebSocket handshake. ``None`` preserves single-user/dev
|
||||
behavior (open).
|
||||
:param engine_provider: Engine factory override for tests. Defaults
|
||||
to :func:`omnigent.server.dictation.get_engine`, which resolves
|
||||
the configured engine and loads models on first use.
|
||||
:returns: An :class:`APIRouter` carrying the stream route.
|
||||
"""
|
||||
router = APIRouter()
|
||||
resolve_engine = engine_provider or get_engine
|
||||
# Router-scoped so each app (and each test app) gets its own cap.
|
||||
slots = asyncio.Semaphore(max_streams())
|
||||
|
||||
@router.websocket("/dictation/stream")
|
||||
async def dictation_stream(websocket: WebSocket) -> None:
|
||||
"""Transcribe one dictation take (see module docstring)."""
|
||||
if auth_provider is not None and auth_provider.get_user_id(websocket) is None:
|
||||
raise WebSocketException(
|
||||
code=status.WS_1008_POLICY_VIOLATION,
|
||||
reason="authentication required",
|
||||
)
|
||||
await websocket.accept()
|
||||
|
||||
if slots.locked():
|
||||
await websocket.close(
|
||||
code=_WS_CLOSE_TRY_AGAIN_LATER,
|
||||
reason="dictation is at capacity; try again shortly",
|
||||
)
|
||||
return
|
||||
|
||||
async with slots:
|
||||
# Engine construction loads model weights — seconds on first
|
||||
# use. Run it off-loop; later takes reuse the shared engine.
|
||||
try:
|
||||
engine = await asyncio.to_thread(resolve_engine)
|
||||
handle: DictationStreamHandle = await asyncio.to_thread(engine.create_stream)
|
||||
except Exception:
|
||||
_logger.exception("dictation engine failed to initialize")
|
||||
with contextlib.suppress(RuntimeError):
|
||||
await websocket.send_text(
|
||||
json.dumps({"type": "error", "message": "dictation engine unavailable"})
|
||||
)
|
||||
await websocket.close(code=_WS_CLOSE_INTERNAL_ERROR)
|
||||
return
|
||||
# Release the take on every exit — normal stop, abrupt browser
|
||||
# disconnect, or a crash mid-send. For the in-process engines
|
||||
# close() just frees the recognizer stream, so a best-effort
|
||||
# close on the way out is enough.
|
||||
try:
|
||||
await websocket.send_text(json.dumps({"type": "ready"}))
|
||||
await _pump_dictation(websocket, handle)
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
await asyncio.to_thread(handle.close)
|
||||
|
||||
return router
|
||||
|
||||
|
||||
async def _pump_dictation(websocket: WebSocket, handle: DictationStreamHandle) -> None:
|
||||
"""Shuttle audio in and transcript events out until stop/disconnect.
|
||||
|
||||
:param websocket: The accepted browser-facing WebSocket.
|
||||
:param handle: The per-connection recognizer stream.
|
||||
"""
|
||||
last_partial_sent = ""
|
||||
last_partial_at = 0.0
|
||||
try:
|
||||
while True:
|
||||
message = await websocket.receive()
|
||||
if message.get("type") == "websocket.disconnect":
|
||||
return
|
||||
|
||||
data = message.get("bytes")
|
||||
if data is not None:
|
||||
update = await asyncio.to_thread(handle.feed_pcm16, data)
|
||||
if update.finalized:
|
||||
await websocket.send_text(
|
||||
json.dumps({"type": "final", "text": update.finalized})
|
||||
)
|
||||
last_partial_sent = ""
|
||||
last_partial_at = 0.0
|
||||
now = time.monotonic()
|
||||
if (
|
||||
update.partial != last_partial_sent
|
||||
and now - last_partial_at >= _PARTIAL_INTERVAL_S
|
||||
):
|
||||
await websocket.send_text(
|
||||
json.dumps({"type": "partial", "text": update.partial})
|
||||
)
|
||||
last_partial_sent = update.partial
|
||||
last_partial_at = now
|
||||
continue
|
||||
|
||||
text_frame = message.get("text")
|
||||
if text_frame is None:
|
||||
continue
|
||||
try:
|
||||
control = json.loads(text_frame)
|
||||
except ValueError:
|
||||
continue
|
||||
if isinstance(control, dict) and control.get("type") == "stop":
|
||||
tail = await asyncio.to_thread(handle.finish)
|
||||
await websocket.send_text(json.dumps({"type": "stopped", "text": tail}))
|
||||
await websocket.close()
|
||||
return
|
||||
# Unknown control messages are ignored for forward compat.
|
||||
except WebSocketDisconnect:
|
||||
return
|
||||
except Exception:
|
||||
_logger.exception("dictation stream failed")
|
||||
with contextlib.suppress(RuntimeError):
|
||||
await websocket.send_text(json.dumps({"type": "error", "message": "dictation failed"}))
|
||||
await websocket.close(code=_WS_CLOSE_INTERNAL_ERROR)
|
||||
+1
-1
@@ -7135,7 +7135,7 @@
|
||||
},
|
||||
"/v1/info": {
|
||||
"get": {
|
||||
"description": "Runtime capabilities probe for the SPA + CLI.\n\nReturned at app boot by the frontend (and by `omnigent login` when it needs to choose between flows). Drives\nconditional route registration and chrome on the SPA side\n\u2014 when `accounts_enabled` is false, the SPA never\nregisters `/login`, `/register`, `/members` and\nnever renders the AccountMenu, so the bundle behaves\nidentically to a pre-PR-2008 build for header / OIDC\ndeploys (in particular, the internal hosted product that\nsyncs from this repo).\n\nAuthentication: this endpoint is intentionally UNAUTHED\nso the SPA can probe it before holding a session cookie.\nIt exposes no sensitive state \u2014 only the active auth\nsource, the login URL, whether first-run admin setup is\nstill pending (`needs_setup`), coarse capability\nbooleans (`databricks_features`,\n`managed_sandboxes_enabled`, `single_user`), the short\nsandbox provider name (`sandbox_provider`) the web UI labels\nthe new-session sandbox option with, and the installed\n`server_version` (already public via `/api/version`).",
|
||||
"description": "Runtime capabilities probe for the SPA + CLI.\n\nReturned at app boot by the frontend (and by `omnigent login` when it needs to choose between flows). Drives\nconditional route registration and chrome on the SPA side\n\u2014 when `accounts_enabled` is false, the SPA never\nregisters `/login`, `/register`, `/members` and\nnever renders the AccountMenu, so the bundle behaves\nidentically to a pre-PR-2008 build for header / OIDC\ndeploys (in particular, the internal hosted product that\nsyncs from this repo).\n\nAuthentication: this endpoint is intentionally UNAUTHED\nso the SPA can probe it before holding a session cookie.\nIt exposes no sensitive state \u2014 only the active auth\nsource, the login URL, whether first-run admin setup is\nstill pending (`needs_setup`), coarse capability\nbooleans (`databricks_features`,\n`managed_sandboxes_enabled`, `dictation_available`,\n`single_user`), the short sandbox provider name\n(`sandbox_provider`) the web UI labels the new-session\nsandbox option with, and the installed\n`server_version` (already public via `/api/version`).",
|
||||
"operationId": "info_v1_info_get",
|
||||
"responses": {
|
||||
"200": {
|
||||
|
||||
@@ -181,6 +181,22 @@ antigravity = ["google-antigravity>=0.1,<1"]
|
||||
# extra. The wheel bundles the Copilot CLI binary it drives (~90MB), which is
|
||||
# why it is not part of the baseline install. Pin major.
|
||||
copilot = ["github-copilot-sdk>=1,<2"]
|
||||
# Server-side streaming dictation (`WS /v1/dictation/stream`,
|
||||
# designs/server-dictation.md). Local CPU speech-to-text via sherpa-onnx;
|
||||
# the engine module imports it lazily, so only servers offering dictation
|
||||
# need the extra. Models are fetched separately
|
||||
# (scripts/fetch-dictation-models.sh). sherpa-onnx-core (the native
|
||||
# onnxruntime payload) is pinned explicitly because sherpa-onnx's wheel
|
||||
# metadata declares it inconsistently across platforms — without the
|
||||
# explicit pin it is missing from uv.lock, which breaks both the hashed
|
||||
# OSV audit and aarch64 installs.
|
||||
dictation = [
|
||||
"sherpa-onnx>=1.13,<2",
|
||||
"sherpa-onnx-core>=1.13,<2",
|
||||
# The engine converts PCM frames with numpy directly; declare it
|
||||
# rather than riding whichever extra happens to pull it in.
|
||||
"numpy>=1.24,<3",
|
||||
]
|
||||
# Cursor SDK harness (`harness: cursor`). Optional like antigravity: the
|
||||
# harness imports the cursor-sdk lazily on first turn, so only `--harness
|
||||
# cursor` users need this extra (`omnigent[cursor]`). Was a baseline dependency;
|
||||
|
||||
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
# Downloads the sherpa-onnx models the server dictation engine expects
|
||||
# (designs/server-dictation.md) into ~/.omnigent/models/dictation/:
|
||||
# asr/ streaming Nemotron transducer (int8, ~650 MB) — the recognizer
|
||||
# punct/ online CNN-BiLSTM punctuation (int8, ~38 MB) — live re-punctuation
|
||||
#
|
||||
# Both are Apache-2.0 upstream releases packaged by k2-fsa. If these exact
|
||||
# URLs move, the catalogs are:
|
||||
# https://k2-fsa.github.io/sherpa/onnx/pretrained_models/index.html
|
||||
# https://k2-fsa.github.io/sherpa/onnx/punctuation/pretrained_models.html
|
||||
# Any streaming transducer dir (encoder/decoder/joiner + tokens.txt) works;
|
||||
# point OMNIGENT_DICTATION_MODEL_DIR / OMNIGENT_DICTATION_PUNCT_DIR at
|
||||
# alternates.
|
||||
set -euo pipefail
|
||||
|
||||
DEST="${OMNIGENT_DICTATION_MODEL_ROOT:-$HOME/.omnigent/models/dictation}"
|
||||
ASR_TARBALL="sherpa-onnx-nemotron-speech-streaming-en-0.6b-560ms-int8-2026-04-25"
|
||||
PUNCT_TARBALL="sherpa-onnx-online-punct-en-2024-08-06"
|
||||
ASR_GH="https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models"
|
||||
PUNCT_GH="https://github.com/k2-fsa/sherpa-onnx/releases/download/punctuation-models"
|
||||
|
||||
mkdir -p "$DEST"
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
dl() { # dl <url> <out>
|
||||
if command -v wget >/dev/null 2>&1; then wget -O "$2" "$1"
|
||||
else curl -fL -o "$2" "$1"; fi
|
||||
}
|
||||
|
||||
fetch() { # fetch <tarball-stem> <base-url> <dest-subdir> <label>
|
||||
local stem="$1" base="$2" sub="$3" label="$4"
|
||||
if [ -n "$(ls -A "$DEST/$sub" 2>/dev/null)" ]; then
|
||||
echo ">> $sub/ already populated, skipping $label"
|
||||
return
|
||||
fi
|
||||
echo ">> downloading $label ($stem)"
|
||||
dl "$base/$stem.tar.bz2" "$TMP/$stem.tar.bz2"
|
||||
tar -xjf "$TMP/$stem.tar.bz2" -C "$TMP"
|
||||
rm -rf "$DEST/$sub"
|
||||
mv "$TMP/$stem" "$DEST/$sub"
|
||||
}
|
||||
|
||||
fetch "$ASR_TARBALL" "$ASR_GH" "asr" "streaming ASR model (~650 MB)"
|
||||
fetch "$PUNCT_TARBALL" "$PUNCT_GH" "punct" "punctuation model (~38 MB)"
|
||||
|
||||
echo ">> dictation models ready under $DEST"
|
||||
ls -d "$DEST"/*/
|
||||
@@ -0,0 +1,87 @@
|
||||
"""e2e: server-side dictation streams transcripts into the composer.
|
||||
|
||||
Drives the full loop the unit tests can't: mic capture (Chromium's fake
|
||||
media device) → AudioWorklet 16 kHz PCM frames → ``WS /v1/dictation/stream``
|
||||
→ the server's fake engine (``OMNIGENT_DICTATION_ENGINE=fake``, set by the
|
||||
``live_server`` fixture) → transcript events → live text in the composer
|
||||
textarea. The fake engine reveals one word of its script per 100 ms of
|
||||
audio received, so a second of fake-mic streaming produces the full
|
||||
sentence, finalized by the engine, without any ASR model.
|
||||
|
||||
The test pins the *no-Web-Speech* entry into server mode by stripping the
|
||||
SpeechRecognition constructors before the app boots (Playwright's Chromium
|
||||
exposes them, but its cloud backend is dead in automation). The other
|
||||
entry — Web Speech present but failing at runtime with a ``network`` error
|
||||
— is pinned in ``web/src/components/ComposerMicButton.test.tsx``.
|
||||
|
||||
A failure here means one of:
|
||||
|
||||
- ``/v1/info`` stopped advertising ``dictation_available`` (capability
|
||||
plumbing in ``omnigent/server/app.py`` or ``web/src/lib/capabilities.ts``).
|
||||
- The WebSocket route broke (``omnigent/server/routes/dictation.py``).
|
||||
- The capture pipeline broke (``web/src/lib/dictation.ts`` worklet/socket).
|
||||
- The composer stopped applying interim/final updates
|
||||
(``ComposerMicButton.tsx`` / ``useDictationInsert.ts`` / ``ChatPage.tsx``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from playwright.sync_api import Browser, expect
|
||||
|
||||
from omnigent.server.dictation import FAKE_SCRIPT as _FAKE_SCRIPT
|
||||
|
||||
# The capability probe caches per page load; the worklet chunks audio at
|
||||
# 100 ms; CI machines are slow — a generous ceiling keeps this deflaked.
|
||||
_TRANSCRIPT_TIMEOUT_MS = 20_000
|
||||
|
||||
|
||||
def test_dictation_streams_transcript_into_composer(
|
||||
browser: Browser,
|
||||
browser_context_args: dict[str, Any],
|
||||
seeded_session: tuple[str, str],
|
||||
) -> None:
|
||||
"""Click the mic, speak (fake device), watch the transcript form."""
|
||||
base_url, session_id = seeded_session
|
||||
# Spread the plugin's context args so --video/--tracing keep working
|
||||
# even though this test builds its own context for the mic permission.
|
||||
context = browser.new_context(**browser_context_args, permissions=["microphone"])
|
||||
try:
|
||||
page = context.new_page()
|
||||
# Force server mode deterministically: without Web Speech
|
||||
# constructors the button picks the server path directly instead
|
||||
# of relying on Chromium's runtime "network" failure timing.
|
||||
page.add_init_script(
|
||||
"Object.defineProperty(window, 'SpeechRecognition',"
|
||||
" { value: undefined, configurable: true });"
|
||||
"Object.defineProperty(window, 'webkitSpeechRecognition',"
|
||||
" { value: undefined, configurable: true });"
|
||||
)
|
||||
page.goto(f"{base_url}/c/{session_id}")
|
||||
|
||||
composer = page.get_by_placeholder("Ask the agent anything…")
|
||||
expect(composer).to_be_visible()
|
||||
|
||||
# The button only renders once /v1/info reports dictation_available,
|
||||
# so its visibility already asserts the capability plumbing.
|
||||
mic = page.get_by_role("button", name="Voice dictation")
|
||||
expect(mic).to_be_visible()
|
||||
|
||||
mic.click()
|
||||
expect(mic).to_have_attribute("aria-pressed", "true")
|
||||
|
||||
# The fake engine finalizes its script after ~0.5 s of audio; the
|
||||
# finalized sentence must land in the composer verbatim.
|
||||
expect(composer).to_have_value(
|
||||
re.compile(re.escape(_FAKE_SCRIPT)),
|
||||
timeout=_TRANSCRIPT_TIMEOUT_MS,
|
||||
)
|
||||
|
||||
mic.click()
|
||||
expect(mic).to_have_attribute("aria-pressed", "false")
|
||||
# Stopping must not clobber the finalized text.
|
||||
expect(composer).to_have_value(re.compile(re.escape(_FAKE_SCRIPT)))
|
||||
finally:
|
||||
context.close()
|
||||
@@ -291,6 +291,12 @@ def browser_type_launch_args(
|
||||
launch_args["args"] = [
|
||||
*launch_args.get("args", []),
|
||||
f"--host-resolver-rules=MAP {_PUBLIC_LOOPBACK_HOST} 127.0.0.1",
|
||||
# Headless Chromium has no microphone; the dictation test
|
||||
# (chat/test_dictation.py) needs getUserMedia to yield a fake
|
||||
# input stream without a permission prompt. No effect on tests
|
||||
# that never touch media capture.
|
||||
"--use-fake-device-for-media-stream",
|
||||
"--use-fake-ui-for-media-stream",
|
||||
]
|
||||
# The pinned Playwright Docker image (the visual-snapshot renderer, both in
|
||||
# ui-snapshot.yml and the local regen script) runs as root, where Chromium
|
||||
@@ -883,6 +889,10 @@ def live_server(
|
||||
"OPENAI_API_KEY": "mock-key",
|
||||
# Strip any ambient Anthropic credentials so they don't leak in.
|
||||
"ANTHROPIC_API_KEY": "",
|
||||
# Deterministic dictation engine: /v1/info advertises dictation and
|
||||
# WS /v1/dictation/stream transcribes any audio into FAKE_SCRIPT,
|
||||
# so chat/test_dictation.py needs no sherpa models or real ASR.
|
||||
"OMNIGENT_DICTATION_ENGINE": os.environ.get("OMNIGENT_DICTATION_ENGINE", "fake"),
|
||||
}
|
||||
log_handle = open(log_path, "w") # noqa: SIM115 — handle lives for Popen lifetime; closed in finally
|
||||
proc = subprocess.Popen(
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
"""Tests for the dictation stream WS and its ``/v1/info`` capability bit.
|
||||
|
||||
The WebSocket tests drive the real route against the deterministic
|
||||
:class:`FakeDictationEngine` injected through ``engine_provider`` — no
|
||||
sherpa-onnx dependency, no models, no microphone. The engine reveals one
|
||||
word of ``FAKE_SCRIPT`` per 100 ms of audio fed, so tests control the
|
||||
transcript by the number of PCM bytes they send.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
from starlette.websockets import WebSocketDisconnect
|
||||
|
||||
from omnigent.server import dictation as dictation_engine
|
||||
from omnigent.server.dictation import FAKE_SCRIPT, MAX_STREAMS_ENV, FakeDictationEngine
|
||||
from omnigent.server.routes.dictation import create_dictation_router
|
||||
|
||||
# One fake-engine "word" of audio: 100 ms of 16 kHz mono s16le.
|
||||
_WORD_BYTES = b"\x00" * (16000 * 2 // 10)
|
||||
_SCRIPT_WORDS = FAKE_SCRIPT.split()
|
||||
|
||||
|
||||
class _NoIdentityAuthProvider:
|
||||
"""Auth provider whose handshake yields no identity."""
|
||||
|
||||
def get_user_id(self, request: object) -> None:
|
||||
"""Always return ``None`` (unauthenticated)."""
|
||||
del request
|
||||
return
|
||||
|
||||
|
||||
def _fake_app(**router_kwargs: object) -> FastAPI:
|
||||
"""Bare app carrying only the dictation router with a fake engine."""
|
||||
app = FastAPI()
|
||||
router_kwargs.setdefault("engine_provider", FakeDictationEngine)
|
||||
app.include_router(create_dictation_router(**router_kwargs), prefix="/v1")
|
||||
return app
|
||||
|
||||
|
||||
async def test_info_carries_dictation_capability(
|
||||
client: httpx.AsyncClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""GET /v1/info advertises dictation for the web UI capability probe."""
|
||||
monkeypatch.setenv(dictation_engine.ENGINE_ENV, dictation_engine.ENGINE_FAKE)
|
||||
resp = await client.get("/v1/info")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["dictation_available"] is True
|
||||
|
||||
|
||||
async def test_info_reports_dictation_unavailable(
|
||||
client: httpx.AsyncClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: object,
|
||||
) -> None:
|
||||
"""Without an engine (no extra or no models) /v1/info advertises false."""
|
||||
monkeypatch.setenv(dictation_engine.MODEL_DIR_ENV, str(tmp_path))
|
||||
monkeypatch.delenv(dictation_engine.ENGINE_ENV, raising=False)
|
||||
resp = await client.get("/v1/info")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["dictation_available"] is False
|
||||
|
||||
|
||||
def test_stream_partial_final_stop_flow() -> None:
|
||||
"""Audio in → ready, partial, final, stopped events out."""
|
||||
with TestClient(_fake_app()) as tc, tc.websocket_connect("/v1/dictation/stream") as ws:
|
||||
assert json.loads(ws.receive_text()) == {"type": "ready"}
|
||||
|
||||
# Two words of audio → a partial with the first two script words.
|
||||
ws.send_bytes(_WORD_BYTES * 2)
|
||||
partial = json.loads(ws.receive_text())
|
||||
assert partial == {"type": "partial", "text": " ".join(_SCRIPT_WORDS[:2])}
|
||||
|
||||
# The rest of the script → the fake finalizes the sentence.
|
||||
ws.send_bytes(_WORD_BYTES * (len(_SCRIPT_WORDS) - 2))
|
||||
final = json.loads(ws.receive_text())
|
||||
assert final == {"type": "final", "text": FAKE_SCRIPT}
|
||||
|
||||
ws.send_text(json.dumps({"type": "stop"}))
|
||||
stopped = json.loads(ws.receive_text())
|
||||
assert stopped == {"type": "stopped", "text": ""}
|
||||
|
||||
|
||||
def test_stream_stop_flushes_tail() -> None:
|
||||
"""stop mid-utterance returns the un-finalized words as the tail."""
|
||||
with TestClient(_fake_app()) as tc, tc.websocket_connect("/v1/dictation/stream") as ws:
|
||||
assert json.loads(ws.receive_text())["type"] == "ready"
|
||||
ws.send_bytes(_WORD_BYTES * 3)
|
||||
assert json.loads(ws.receive_text())["type"] == "partial"
|
||||
ws.send_text(json.dumps({"type": "stop"}))
|
||||
stopped = json.loads(ws.receive_text())
|
||||
assert stopped == {"type": "stopped", "text": " ".join(_SCRIPT_WORDS[:3])}
|
||||
|
||||
|
||||
def test_stream_ignores_unknown_control_messages() -> None:
|
||||
"""Unknown text frames are ignored for forward compatibility."""
|
||||
with TestClient(_fake_app()) as tc, tc.websocket_connect("/v1/dictation/stream") as ws:
|
||||
assert json.loads(ws.receive_text())["type"] == "ready"
|
||||
ws.send_text(json.dumps({"type": "does-not-exist"}))
|
||||
ws.send_text("not json at all")
|
||||
# The stream is still alive and transcribing after both.
|
||||
ws.send_bytes(_WORD_BYTES)
|
||||
assert json.loads(ws.receive_text()) == {
|
||||
"type": "partial",
|
||||
"text": _SCRIPT_WORDS[0],
|
||||
}
|
||||
|
||||
|
||||
def test_stream_closes_take_on_abrupt_disconnect() -> None:
|
||||
"""A vanished client still releases the take (worker-slot safety).
|
||||
|
||||
The remote relay engine holds a worker capacity slot until its
|
||||
handle is closed; the route must close handles on the disconnect
|
||||
path, not just on a clean stop.
|
||||
"""
|
||||
engine = FakeDictationEngine()
|
||||
app = _fake_app(engine_provider=lambda: engine)
|
||||
with TestClient(app) as tc:
|
||||
with tc.websocket_connect("/v1/dictation/stream") as ws:
|
||||
assert json.loads(ws.receive_text())["type"] == "ready"
|
||||
ws.send_bytes(_WORD_BYTES)
|
||||
# Exit without stop: an abrupt browser disconnect.
|
||||
assert engine.last_stream is not None
|
||||
deadline = time.monotonic() + 5
|
||||
while not engine.last_stream.closed and time.monotonic() < deadline:
|
||||
time.sleep(0.02)
|
||||
assert engine.last_stream.closed
|
||||
|
||||
|
||||
def test_stream_rejects_unauthenticated_handshake() -> None:
|
||||
"""With an auth provider and no identity, the handshake is refused."""
|
||||
app = _fake_app(auth_provider=_NoIdentityAuthProvider())
|
||||
with TestClient(app) as tc:
|
||||
with pytest.raises(WebSocketDisconnect):
|
||||
with tc.websocket_connect("/v1/dictation/stream") as ws:
|
||||
ws.receive_text()
|
||||
|
||||
|
||||
def test_stream_capacity_cap(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Connections beyond the stream cap close with 1013 (try later)."""
|
||||
monkeypatch.setenv(MAX_STREAMS_ENV, "1")
|
||||
with TestClient(_fake_app()) as tc:
|
||||
with tc.websocket_connect("/v1/dictation/stream") as first:
|
||||
assert json.loads(first.receive_text())["type"] == "ready"
|
||||
with tc.websocket_connect("/v1/dictation/stream") as second:
|
||||
with pytest.raises(WebSocketDisconnect) as excinfo:
|
||||
second.receive_text()
|
||||
assert excinfo.value.code == 1013
|
||||
@@ -0,0 +1,191 @@
|
||||
"""Unit tests for the dictation engine layer (no route, no WebSocket).
|
||||
|
||||
Everything here runs without the ``dictation`` extra except the last
|
||||
test, which exercises the real sherpa-onnx engine end-to-end and skips
|
||||
itself unless the extra and a model are installed (developer machines).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from omnigent.server import dictation
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_engine_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Isolate each test from ambient dictation env configuration."""
|
||||
monkeypatch.delenv(dictation.ENGINE_ENV, raising=False)
|
||||
monkeypatch.delenv(dictation.MODEL_DIR_ENV, raising=False)
|
||||
monkeypatch.delenv(dictation.PUNCT_DIR_ENV, raising=False)
|
||||
monkeypatch.delenv(dictation.MAX_STREAMS_ENV, raising=False)
|
||||
|
||||
|
||||
def _touch_asr_files(model_dir: Path) -> None:
|
||||
for name in ("encoder.int8.onnx", "decoder.int8.onnx", "joiner.int8.onnx", "tokens.txt"):
|
||||
(model_dir / name).touch()
|
||||
|
||||
|
||||
def test_availability_fake_engine(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The fake engine is always available, extra or not."""
|
||||
monkeypatch.setenv(dictation.ENGINE_ENV, dictation.ENGINE_FAKE)
|
||||
assert dictation.engine_availability() == (True, None)
|
||||
|
||||
|
||||
def test_availability_extra_not_installed(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Without the sherpa-onnx package the probe says extra_not_installed."""
|
||||
monkeypatch.setattr(dictation.importlib.util, "find_spec", lambda name: None)
|
||||
assert dictation.engine_availability() == (
|
||||
False,
|
||||
dictation.REASON_EXTRA_NOT_INSTALLED,
|
||||
)
|
||||
|
||||
|
||||
def test_availability_models_missing(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
|
||||
"""With the package but an empty model dir the probe says models_missing."""
|
||||
monkeypatch.setattr(dictation.importlib.util, "find_spec", lambda name: object())
|
||||
monkeypatch.setenv(dictation.MODEL_DIR_ENV, str(tmp_path))
|
||||
assert dictation.engine_availability() == (
|
||||
False,
|
||||
dictation.REASON_MODELS_MISSING,
|
||||
)
|
||||
|
||||
|
||||
def test_availability_with_models(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
|
||||
"""A populated model dir plus the package reports available."""
|
||||
monkeypatch.setattr(dictation.importlib.util, "find_spec", lambda name: object())
|
||||
_touch_asr_files(tmp_path)
|
||||
monkeypatch.setenv(dictation.MODEL_DIR_ENV, str(tmp_path))
|
||||
assert dictation.engine_availability() == (True, None)
|
||||
|
||||
|
||||
def test_pick_model_file_prefers_int8(tmp_path: Path) -> None:
|
||||
"""int8 quantizations win over float exports of the same stem."""
|
||||
(tmp_path / "encoder.onnx").touch()
|
||||
(tmp_path / "encoder.int8.onnx").touch()
|
||||
picked = dictation._pick_model_file(tmp_path, "encoder")
|
||||
assert picked is not None and picked.name == "encoder.int8.onnx"
|
||||
|
||||
|
||||
def test_max_streams_parsing(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Bad or non-positive values fall back to the default."""
|
||||
assert dictation.max_streams() == dictation.DEFAULT_MAX_STREAMS
|
||||
monkeypatch.setenv(dictation.MAX_STREAMS_ENV, "5")
|
||||
assert dictation.max_streams() == 5
|
||||
monkeypatch.setenv(dictation.MAX_STREAMS_ENV, "0")
|
||||
assert dictation.max_streams() == dictation.DEFAULT_MAX_STREAMS
|
||||
monkeypatch.setenv(dictation.MAX_STREAMS_ENV, "lots")
|
||||
assert dictation.max_streams() == dictation.DEFAULT_MAX_STREAMS
|
||||
|
||||
|
||||
def test_get_engine_is_a_singleton_and_failure_caches_nothing(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""One engine per process; a failed load leaves the slot empty for retry."""
|
||||
monkeypatch.setattr(dictation, "_engine", None)
|
||||
# Unavailable (empty model dir) → raises and caches nothing.
|
||||
monkeypatch.setattr(dictation.importlib.util, "find_spec", lambda name: object())
|
||||
monkeypatch.setenv(dictation.MODEL_DIR_ENV, str(tmp_path))
|
||||
with pytest.raises(RuntimeError):
|
||||
dictation.get_engine()
|
||||
assert dictation._engine is None
|
||||
# Becomes available (fake engine) → loads once, then reuses.
|
||||
monkeypatch.setenv(dictation.ENGINE_ENV, dictation.ENGINE_FAKE)
|
||||
first = dictation.get_engine()
|
||||
assert isinstance(first, dictation.FakeDictationEngine)
|
||||
assert dictation.get_engine() is first
|
||||
|
||||
|
||||
def test_get_engine_rejects_unknown_name(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""An unregistered engine name is unavailable and raises on load."""
|
||||
monkeypatch.setattr(dictation, "_engine", None)
|
||||
monkeypatch.setenv(dictation.ENGINE_ENV, "does-not-exist")
|
||||
assert dictation.engine_availability() == (False, dictation.REASON_UNKNOWN_ENGINE)
|
||||
with pytest.raises(RuntimeError):
|
||||
dictation.get_engine()
|
||||
|
||||
|
||||
def test_register_engine_is_selectable(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A registered engine is selected by name with no core edits.
|
||||
|
||||
Mirrors what adding Whisper looks like: one register_engine call, then
|
||||
OMNIGENT_DICTATION_ENGINE picks it up.
|
||||
"""
|
||||
monkeypatch.setattr(dictation, "_engine", None)
|
||||
monkeypatch.setitem(
|
||||
dictation._ENGINE_REGISTRY,
|
||||
"probe-engine",
|
||||
dictation._EngineEntry(
|
||||
factory=dictation.FakeDictationEngine,
|
||||
available=lambda: (True, None),
|
||||
),
|
||||
)
|
||||
monkeypatch.setenv(dictation.ENGINE_ENV, "probe-engine")
|
||||
assert dictation.engine_availability() == (True, None)
|
||||
assert isinstance(dictation.get_engine(), dictation.FakeDictationEngine)
|
||||
|
||||
|
||||
def test_fake_stream_reveals_script_by_bytes() -> None:
|
||||
"""One script word per 100 ms of audio; sentence finalizes when done."""
|
||||
word = b"\x00" * (dictation.SAMPLE_RATE * 2 // 10)
|
||||
words = dictation.FAKE_SCRIPT.split()
|
||||
stream = dictation.FakeDictationEngine().create_stream()
|
||||
|
||||
update = stream.feed_pcm16(word * 2)
|
||||
assert update.partial == " ".join(words[:2])
|
||||
assert update.finalized is None
|
||||
|
||||
update = stream.feed_pcm16(word * (len(words) - 2))
|
||||
assert update.partial == ""
|
||||
assert update.finalized == dictation.FAKE_SCRIPT
|
||||
|
||||
# After the script completes, the stream stays quiet.
|
||||
assert stream.feed_pcm16(word).partial == ""
|
||||
assert stream.finish() == ""
|
||||
|
||||
|
||||
def test_fake_stream_finish_returns_tail() -> None:
|
||||
"""finish() mid-script returns the revealed words."""
|
||||
word = b"\x00" * (dictation.SAMPLE_RATE * 2 // 10)
|
||||
words = dictation.FAKE_SCRIPT.split()
|
||||
stream = dictation.FakeDictationEngine().create_stream()
|
||||
stream.feed_pcm16(word * 3)
|
||||
assert stream.finish() == " ".join(words[:3])
|
||||
|
||||
|
||||
def test_sherpa_engine_transcribes_test_wav() -> None:
|
||||
"""Real-model smoke test; skips unless the extra + models are installed.
|
||||
|
||||
Hermetic on CI (always skipped there); on a developer machine with
|
||||
models fetched via ``scripts/fetch-dictation-models.sh`` it exercises
|
||||
the true engine: PCM in → partial/finalized text out.
|
||||
"""
|
||||
pytest.importorskip("sherpa_onnx")
|
||||
asr_dir = dictation._asr_dir()
|
||||
if dictation._asr_files(asr_dir) is None:
|
||||
pytest.skip(f"no dictation ASR model in {asr_dir}")
|
||||
wavs = sorted(asr_dir.glob("test_wavs/*.wav"))
|
||||
if not wavs:
|
||||
pytest.skip("model dir has no test_wavs to decode")
|
||||
|
||||
import wave
|
||||
|
||||
engine = dictation.SherpaDictationEngine(asr_dir, dictation._punct_dir())
|
||||
stream = engine.create_stream()
|
||||
with wave.open(str(wavs[0])) as wav:
|
||||
assert wav.getframerate() == dictation.SAMPLE_RATE
|
||||
pcm = wav.readframes(wav.getnframes())
|
||||
|
||||
texts: list[str] = []
|
||||
chunk = dictation.SAMPLE_RATE * 2 // 10 # 100 ms
|
||||
for i in range(0, len(pcm), chunk):
|
||||
update = stream.feed_pcm16(pcm[i : i + chunk])
|
||||
if update.finalized:
|
||||
texts.append(update.finalized)
|
||||
tail = stream.finish()
|
||||
if tail:
|
||||
texts.append(tail)
|
||||
transcript = " ".join(texts)
|
||||
assert len(transcript.split()) >= 3, transcript
|
||||
@@ -3014,6 +3014,11 @@ dev = [
|
||||
{ name = "sqlalchemy-cloudflare-d1" },
|
||||
{ name = "types-pyyaml" },
|
||||
]
|
||||
dictation = [
|
||||
{ name = "numpy" },
|
||||
{ name = "sherpa-onnx" },
|
||||
{ name = "sherpa-onnx-core" },
|
||||
]
|
||||
e2b = [
|
||||
{ name = "e2b" },
|
||||
]
|
||||
@@ -3087,6 +3092,7 @@ requires-dist = [
|
||||
{ name = "modal", marker = "extra == 'modal'", specifier = ">=1.0,<2" },
|
||||
{ name = "moto", extras = ["s3"], marker = "extra == 'dev'", specifier = ">=5,<6" },
|
||||
{ name = "mypy", marker = "extra == 'dev'", specifier = ">=1.10" },
|
||||
{ name = "numpy", marker = "extra == 'dictation'", specifier = ">=1.24,<3" },
|
||||
{ name = "omnigent-client", editable = "sdks/python-client" },
|
||||
{ name = "omnigent-slack", marker = "extra == 'slack'", editable = "integrations/slack" },
|
||||
{ name = "omnigent-ui-sdk", editable = "sdks/ui" },
|
||||
@@ -3127,6 +3133,8 @@ requires-dist = [
|
||||
{ name = "respx", marker = "extra == 'dev'", specifier = ">=0.21,<1" },
|
||||
{ name = "rich", specifier = ">=14,<15" },
|
||||
{ name = "ruff", marker = "extra == 'dev'", specifier = ">=0.6" },
|
||||
{ name = "sherpa-onnx", marker = "extra == 'dictation'", specifier = ">=1.13,<2" },
|
||||
{ name = "sherpa-onnx-core", marker = "extra == 'dictation'", specifier = ">=1.13,<2" },
|
||||
{ name = "sqlalchemy", specifier = ">=2.0,<3" },
|
||||
{ name = "sqlalchemy-cloudflare-d1", marker = "extra == 'dev'", specifier = "==0.3.10" },
|
||||
{ name = "starlette", specifier = ">=1.0.1,<2" },
|
||||
@@ -3137,7 +3145,7 @@ requires-dist = [
|
||||
{ name = "websockets", specifier = ">=10.4,<15" },
|
||||
{ name = "zstandard", specifier = ">=0.22,<1" },
|
||||
]
|
||||
provides-extras = ["claude-sdk", "openai-agents", "all", "bedrock", "s3", "vertex", "modal", "daytona", "boxlite", "cwsandbox", "e2b", "islo", "openshell", "kubernetes", "tracing", "agents-sdk", "antigravity", "copilot", "cursor", "hindsight", "memory", "slack", "databricks", "dev"]
|
||||
provides-extras = ["claude-sdk", "openai-agents", "all", "bedrock", "s3", "vertex", "modal", "daytona", "boxlite", "cwsandbox", "e2b", "islo", "openshell", "kubernetes", "tracing", "agents-sdk", "antigravity", "copilot", "dictation", "cursor", "hindsight", "memory", "slack", "databricks", "dev"]
|
||||
|
||||
[[package]]
|
||||
name = "omnigent-client"
|
||||
@@ -5005,6 +5013,53 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/5d/40/e1e72872c6354b306daef1703549e8e83b4d43cfea356311bf722a043752/setuptools-83.0.0-py3-none-any.whl", hash = "sha256:29b23c360f22f414dc7336bb39178cc7bcbf6021ed2733cde173f09dba19abb3", upload-time = "2026-07-04T15:31:20.885Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sherpa-onnx"
|
||||
version = "1.13.4"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/ca/f8/735244770b4bc63f85fabdad0e46d6ec1f4cc24e64f6e082c2e0fea92b8c/sherpa_onnx-1.13.4.tar.gz", hash = "sha256:29547692418513ad88034c2b5f98985e33042b2351e4ab375469f19a8de18c5f", upload-time = "2026-07-07T13:04:55.145Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/8e/57/179e3a6c1fec33aa6535051feddd5da36e5622d35630b12a67a2805b76b3/sherpa_onnx-1.13.4-cp312-cp312-linux_armv7l.whl", hash = "sha256:bcf64f2d853a1afe236e9e220df62f2f53ef6ad792ca7e406d6173ec003319b8", upload-time = "2026-07-07T13:50:35.989Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e8/34/b6d3483b08ec8a4a141e978c4b92530fd0a61dd571a575c1fe24bee300d7/sherpa_onnx-1.13.4-cp312-cp312-macosx_10_15_universal2.whl", hash = "sha256:2257545ea170f58b7977309793979d6d078761b7fdb0528561285f8ead4169db", upload-time = "2026-07-07T12:29:40.234Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/82/37/07f03e97f157b206f6e62d722ec7c5ff41c7e9dc6aa2dc7de69b57e39b5a/sherpa_onnx-1.13.4-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:02b57dc2c829976eb842e6aee6a0e4ac3b9991aeb5afa89fd44eb71d848a4ecd", upload-time = "2026-07-07T12:10:19.957Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/fc/79/ee999f0c3b7789077d0939716a38234573d139f851a31409aa028fe2c610/sherpa_onnx-1.13.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:84e58b5a074b97c5307c9b6221d1d20fbf412a1a5dff4960ca9c32bb5184219f", upload-time = "2026-07-07T11:48:22.518Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f4/90/9b67ed3e7adc79daf0ba49c4936a691521488125b04fe469b64a8b5398ff/sherpa_onnx-1.13.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f709e6dd02ebf7d37dcb02d5eadc5fb66c9922dd5809df770c1ef5d625ae7a44", upload-time = "2026-07-07T11:58:30.98Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/cc/b1/8dfe5d1d72c92ea1c95db999a95b61bfbb9769f1c569f06e572eda095c52/sherpa_onnx-1.13.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5f0158f3513d3adab1ebba0c26f0c815e53ba13b96846d92ef095ae25d648860", upload-time = "2026-07-07T12:58:59.654Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d4/04/cfd543933ae24430d124e533847c73a84a5f0efd60f07bbc6403032f9624/sherpa_onnx-1.13.4-cp312-cp312-win32.whl", hash = "sha256:d49928a3455bae1dd4e93f6b013cfbd2c3ccb5cde74aabae3710b656e7d79b6b", upload-time = "2026-07-07T12:02:06.689Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/bb/bb/1e723ab703a1e354f390de19981ec0c347576f87be01915d826dc6fc9f41/sherpa_onnx-1.13.4-cp312-cp312-win_amd64.whl", hash = "sha256:b8436ffe2763b3fd522fbac8fe53f47d611721c84819c241acfb65d122403d7d", upload-time = "2026-07-07T13:01:42.293Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/4b/36/45b17335f041f1383f6fd142ab57c2d8a337ba2386b7547b125ec9d780af/sherpa_onnx-1.13.4-cp313-cp313-linux_armv7l.whl", hash = "sha256:9e98dc5e0559ad953f227fc884958c71b10c65a93667331405e7d4441ed5f76d", upload-time = "2026-07-07T14:18:59.791Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/68/d7/1e9a7dedab2da8af1a8417b4f4d5f496bd7700a71b59c5e085de5e10761b/sherpa_onnx-1.13.4-cp313-cp313-macosx_10_15_universal2.whl", hash = "sha256:083747c2d0362ead0501cc773a618be19862a800b3f8f259d3bd3486f1494af4", upload-time = "2026-07-07T13:02:05.512Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/62/28/09aa9461e8bdf894ba8466e047e40fb5da8aaa6d68c19cf1e2aabe01e706/sherpa_onnx-1.13.4-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:b4f54363b264b16148a724b4442f00cda97fcd4e9beeda3d75637753910e8557", upload-time = "2026-07-07T12:28:22.056Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/21/ed/d07787dd4be4119e6587c840f6b417c2d57c14d694d334af609d68cb5a41/sherpa_onnx-1.13.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8ec5394b4ea73bf01e6883cf078348f87350f4eb3567d51d92cae77ea2582403", upload-time = "2026-07-07T12:47:31.367Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/6d/c2/281d84dc9e448ea99d7fb77708cbe1cc7cfd8c7d669727dc94385a9e4ca5/sherpa_onnx-1.13.4-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a39352ceb2ec6671a1f252fb768fff75bb2f0bc849cca5f66f490e89910a860d", upload-time = "2026-07-07T12:10:09.762Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/db/47/da3ea14ab647a4f6580227853fe29353e1173ff77064d42c0bb31d01b453/sherpa_onnx-1.13.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:88af596be24eac32982dd64fcac30af99d9130ca498bfa1a0064189c8498195b", upload-time = "2026-07-07T13:10:10.265Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/44/90/84205ff383ba9335c3821c2cd6d514350f52199cb640ec094517f1f911a0/sherpa_onnx-1.13.4-cp313-cp313-win32.whl", hash = "sha256:0cabb508a15be22138f9fb7695d7ec5f3893ecd088ee419b9df559fed7e8f649", upload-time = "2026-07-07T12:51:53.196Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/82/40/ee8a0a8c83fc6d7f5245a5a031e471d3b115e20cce867e7abb2f9d4185c9/sherpa_onnx-1.13.4-cp313-cp313-win_amd64.whl", hash = "sha256:17050fdfb48d37ae996364f697c554a1399740d18e5a56b143c011d00cfed3e0", upload-time = "2026-07-07T12:32:59.882Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/8e/9e/cb97cf04c0c4de0a1d43463952e387a938eea8f6ab94e2eb95af21162f57/sherpa_onnx-1.13.4-cp314-cp314-linux_armv7l.whl", hash = "sha256:7ddb3d46fe0d6cc745d222e055844dbb24fd4e66935d4ead67d1315911ea7a46", upload-time = "2026-07-07T14:37:28.79Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/cc/0f/7fbed45ef8437d20967f4577514e8903b7f4a87a8cea9b9fed9e2b18fa45/sherpa_onnx-1.13.4-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:ff9237b98c173dabf8f5f6317c4ead54afd4b35179b03be652f65e1e121bee74", upload-time = "2026-07-07T13:01:45.126Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/9a/90/191b3b73af1b54584f9f96e256d9ca9f6f3781cacdb4287c7efea6b25094/sherpa_onnx-1.13.4-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:972137cbf19d501ea6a51857528b340ce1c3d204571e3d56dcc72eded5827418", upload-time = "2026-07-07T11:46:13.903Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/1f/b5/29997d7de29cae8e3f54e8a993fcf14d48d5a7960deab10df479fcbc7d64/sherpa_onnx-1.13.4-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:ff38ac153527fda0f6158e7c097b66e980a6840455d908e037ce09a4a4c1ce14", upload-time = "2026-07-07T12:26:40.457Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/8c/e5/8c601626448358ac8571100db051f35af5ef02c1d12ab1cb026436474a22/sherpa_onnx-1.13.4-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7a6bf37e45727b0e94661a29b34927e9edc4b3849411cc38036e30d81df48d31", upload-time = "2026-07-07T12:03:40.699Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5a/4b/49b4be95af2e12bfa8a92a7eba720cc68e4da178847ffa4ddb66479d4e9c/sherpa_onnx-1.13.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:16b046f1c7ceecf666947d652c928278c377847db7f697c9e94af9feaf20ac2c", upload-time = "2026-07-07T13:05:36.871Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/96/1c/a80bfad89846ccb1da4d18ef46bf364c83cff9514a0e298110c8de480e14/sherpa_onnx-1.13.4-cp314-cp314-win32.whl", hash = "sha256:39af016b9fb7c053da2270e44182fb8932eeab991dcdc963bbc4366324e9ec51", upload-time = "2026-07-07T12:13:06.663Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/89/02/1e10f71be635a3f9ef793b07ab88ddb7afa82ab722f3ea275a4877da1bcb/sherpa_onnx-1.13.4-cp314-cp314-win_amd64.whl", hash = "sha256:cb1834182c4047b8edb1dceeed8d5cf7d6e10295a4079e5e0fea674b4314db06", upload-time = "2026-07-07T13:03:09.057Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sherpa-onnx-core"
|
||||
version = "1.13.4"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/ad/69/ae33a8cb1ecc30e0c4638d76772e56161162d57bef748380790e1257841f/sherpa_onnx_core-1.13.4-py3-none-macosx_10_15_universal2.whl", hash = "sha256:737099a817998e4d74379dcd44d8d1b332a3fa1822780be174334c3d1d1e2451", upload-time = "2026-07-07T11:41:37.506Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/2a/8c/a1336beab226d228f62bdbe1cacf1439a2f6cf8714baac98f1f031dd2f60/sherpa_onnx_core-1.13.4-py3-none-macosx_10_15_x86_64.whl", hash = "sha256:674ea57eb6458002dab4e39749d21a7364b2f962f4a4958a3ee4c351b093cafd", upload-time = "2026-07-07T11:31:03.394Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a2/26/d0d6bebea4ef8b7de6eed1a335235d1acce9197425c08eecb57ef107904d/sherpa_onnx_core-1.13.4-py3-none-macosx_11_0_arm64.whl", hash = "sha256:7820183581b711a68e30281a4fb36c0af8ef5615bfc789234fb259439824a014", upload-time = "2026-07-07T11:42:28.755Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/81/29/15859e896574230d5377738ef27e8647cbd19fa325d75b1a00074791798d/sherpa_onnx_core-1.13.4-py3-none-manylinux2014_aarch64.whl", hash = "sha256:b4e4d17eb0d5c569bf4c9effcbc3daef57cf7e2b8418e07ae90f90c9b60b35d5", upload-time = "2026-07-07T11:41:30.935Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/41/be/38c57721d71ee74d984b1ca21720a8ca8477d6d341026af24ff658866ef9/sherpa_onnx_core-1.13.4-py3-none-manylinux2014_x86_64.whl", hash = "sha256:367aa06cee90b3fd7959d4e071d6fc821710b859af399b4987e5c3119ee6ae2a", upload-time = "2026-07-07T12:21:33.627Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/7e/17/949ee6d5cff4ec9e2dcda014276a798b814e2eb80d35d820884e6e0614ff/sherpa_onnx_core-1.13.4-py3-none-manylinux_2_35_armv7l.whl", hash = "sha256:3b9cc7da5ce4a2333a9ae211f39c34dee981b13f2c9c8680fbf2d9636e87d617", upload-time = "2026-07-07T14:25:04.94Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/3b/71/e2fb4b965b86bd3ee6ca35d81999e9edad47c20d2c8fc4e03db8083494bc/sherpa_onnx_core-1.13.4-py3-none-win32.whl", hash = "sha256:ba9de2f463ae67ee2947f1bd9b981a1e39e6b831e281bf5088068251b8ec4dde", upload-time = "2026-07-07T11:49:11.781Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/95/b0/c3d59ac76f3db873e41bd0cb4fc30b352a278da3289217985aaae3650211/sherpa_onnx_core-1.13.4-py3-none-win_amd64.whl", hash = "sha256:0a6949cf0fd83adb9fbcfdf5c27b8907a57f7b48626db703c7f6037be9b61764", upload-time = "2026-07-07T12:03:05.196Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "six"
|
||||
version = "1.17.0"
|
||||
|
||||
+10
-7
@@ -67,15 +67,18 @@ adds native niceties:
|
||||
OS-level mic gate is open too (packaged builds ship
|
||||
`NSMicrophoneUsageDescription`).
|
||||
|
||||
> **Caveat — Web Speech may still not transcribe in Electron.** Granting the
|
||||
> **Caveat — Web Speech does not transcribe in Electron.** Granting the
|
||||
> mic clears the _permission_ gate, but `SpeechRecognition` also depends on
|
||||
> Google's cloud speech backend keyed to official Google Chrome builds, which
|
||||
> Electron's bundled Chromium does **not** ship. So recognition can still
|
||||
> fail (typically a `network` error) even with the mic allowed. The web app
|
||||
> degrades gracefully (the button shows "Dictation unavailable" rather than
|
||||
> crashing). Fully reliable in-app dictation would require a MediaRecorder
|
||||
> capture + a server-side transcription endpoint (e.g. Whisper) wired to the
|
||||
> composer's existing `onAudioRecorded` fallback — not yet implemented.
|
||||
> Electron's bundled Chromium does **not** ship. Electron therefore uses the
|
||||
> **server-side dictation fallback** instead: when the connected server has
|
||||
> the `dictation` extra and models installed (`GET /v1/info` reports
|
||||
> `dictation_available`), a take that fails with Web Speech's `network`
|
||||
> error falls back to streaming audio to `WS /v1/dictation/stream` and
|
||||
> transcribing on the server — no cloud, no Chrome dependency. See
|
||||
> `designs/server-dictation.md`. Without the server extra, the button still
|
||||
> renders (the constructor exists) but shows "Dictation unavailable" when
|
||||
> clicked, as before.
|
||||
|
||||
## How it works (zero UI duplication)
|
||||
|
||||
|
||||
@@ -1,17 +1,57 @@
|
||||
// Tests for ComposerMicButton — Web Speech API voice dictation.
|
||||
// Tests for ComposerMicButton — Web Speech API voice dictation plus the
|
||||
// server-dictation fallback.
|
||||
//
|
||||
// The button toggles a SpeechRecognition session; final transcripts are
|
||||
// emitted via onTranscript. It renders nothing when the browser has no
|
||||
// SpeechRecognition constructor. None of this is e2e-testable (CI has no real
|
||||
// mic / Web Speech engine), so it's pinned here by stubbing the global
|
||||
// SpeechRecognition constructor with a fake whose addEventListener captures the
|
||||
// handlers the test then fires. getUserMedia (used only for the visualizer) is
|
||||
// stubbed to reject so no AudioContext is constructed in jsdom.
|
||||
// Web Speech mode: the button toggles a SpeechRecognition session; final
|
||||
// transcripts are emitted via onTranscript. It renders nothing when the
|
||||
// browser has no SpeechRecognition constructor AND the server offers no
|
||||
// dictation. None of this is e2e-testable (CI has no real mic / Web Speech
|
||||
// engine), so it's pinned here by stubbing the global SpeechRecognition
|
||||
// constructor with a fake whose addEventListener captures the handlers the
|
||||
// test then fires. getUserMedia (used only for the visualizer) is stubbed to
|
||||
// reject so no AudioContext is constructed in jsdom.
|
||||
//
|
||||
// Server mode: when there is no SpeechRecognition constructor but the
|
||||
// /v1/info capability probe reports dictation_available, the button drives a
|
||||
// DictationSession instead (mocked here — the real transport needs a mic,
|
||||
// an AudioWorklet, and a WebSocket; the full loop runs in the Playwright
|
||||
// e2e test against the server's fake engine).
|
||||
|
||||
import { act, cleanup, fireEvent, render, screen } from "@testing-library/react";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi, type Mock } from "vitest";
|
||||
import { CapabilitiesContext } from "@/lib/CapabilitiesContext";
|
||||
import type { ServerInfo } from "@/lib/capabilities";
|
||||
import type { DictationSessionEvents } from "@/lib/dictation";
|
||||
import { ComposerMicButton } from "./ComposerMicButton";
|
||||
|
||||
// Controllable DictationSession stand-in for the server-mode tests. The
|
||||
// factory reads the mutable spies at call time, so each test installs its
|
||||
// own behavior in beforeEach.
|
||||
type SessionStub = { stop: () => Promise<string>; cancel: () => void };
|
||||
let sessionStartMock: Mock<(events: DictationSessionEvents) => Promise<SessionStub>>;
|
||||
let sessionStopMock: Mock<() => Promise<string>>;
|
||||
let sessionCancelMock: Mock<() => void>;
|
||||
let sessionEvents: DictationSessionEvents | null;
|
||||
|
||||
vi.mock("@/lib/dictation", () => {
|
||||
class DictationBusyError extends Error {}
|
||||
return {
|
||||
DictationBusyError,
|
||||
DictationSession: {
|
||||
start: (events: DictationSessionEvents) => sessionStartMock(events),
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
function installDictationSession() {
|
||||
sessionEvents = null;
|
||||
sessionStopMock = vi.fn(async () => "");
|
||||
sessionCancelMock = vi.fn();
|
||||
sessionStartMock = vi.fn(async (events: DictationSessionEvents) => {
|
||||
sessionEvents = events;
|
||||
return { stop: sessionStopMock, cancel: sessionCancelMock };
|
||||
});
|
||||
}
|
||||
|
||||
/** Captured event handlers keyed by event type, fed by the fake recognition. */
|
||||
let handlers: Record<string, (event: unknown) => void>;
|
||||
let startSpy: ReturnType<typeof vi.fn>;
|
||||
@@ -49,6 +89,7 @@ function resultEvent(transcript: string) {
|
||||
|
||||
beforeEach(() => {
|
||||
installSpeechRecognition();
|
||||
installDictationSession();
|
||||
// The visualizer's getUserMedia is best-effort; reject so no AudioContext
|
||||
// (unavailable in jsdom) is ever constructed. Capture the original descriptor
|
||||
// first so afterEach can restore it — otherwise this navigator stub leaks.
|
||||
@@ -142,3 +183,203 @@ describe("ComposerMicButton", () => {
|
||||
expect(button).toHaveAttribute("title", "Voice dictation");
|
||||
});
|
||||
});
|
||||
|
||||
/** ServerInfo with dictation on; the other capabilities are irrelevant here. */
|
||||
const DICTATION_INFO: ServerInfo = {
|
||||
accounts_enabled: false,
|
||||
single_user: false,
|
||||
login_url: null,
|
||||
needs_setup: false,
|
||||
databricks_features: false,
|
||||
managed_sandboxes_enabled: false,
|
||||
sandbox_provider: null,
|
||||
sharing_mode: "on",
|
||||
public_sharing_enabled: true,
|
||||
server_version: "test",
|
||||
smart_routing_enabled: false,
|
||||
dictation_available: true,
|
||||
};
|
||||
|
||||
function renderServerMode(
|
||||
props: Partial<React.ComponentProps<typeof ComposerMicButton>> = {},
|
||||
info: ServerInfo = DICTATION_INFO,
|
||||
) {
|
||||
// No SpeechRecognition constructor → the component must pick server mode.
|
||||
vi.stubGlobal("SpeechRecognition", undefined);
|
||||
vi.stubGlobal("webkitSpeechRecognition", undefined);
|
||||
return render(
|
||||
<CapabilitiesContext.Provider value={info}>
|
||||
<ComposerMicButton onTranscript={vi.fn()} {...props} />
|
||||
</CapabilitiesContext.Provider>,
|
||||
);
|
||||
}
|
||||
|
||||
async function clickMic() {
|
||||
// toggle() kicks off the async DictationSession.start; flush it inside act.
|
||||
await act(async () => {
|
||||
fireEvent.click(screen.getByRole("button", { name: "Voice dictation" }));
|
||||
});
|
||||
}
|
||||
|
||||
describe("ComposerMicButton (server dictation)", () => {
|
||||
it("renders the button when the server advertises dictation", () => {
|
||||
renderServerMode();
|
||||
expect(screen.getByRole("button", { name: "Voice dictation" })).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("renders nothing when neither Web Speech nor the server can help", () => {
|
||||
const { container } = renderServerMode({}, { ...DICTATION_INFO, dictation_available: false });
|
||||
expect(container).toBeEmptyDOMElement();
|
||||
});
|
||||
|
||||
it("starts a session on click and reflects the recording state", async () => {
|
||||
renderServerMode();
|
||||
await clickMic();
|
||||
expect(sessionStartMock).toHaveBeenCalledTimes(1);
|
||||
expect(screen.getByRole("button", { name: "Voice dictation" })).toHaveAttribute(
|
||||
"aria-pressed",
|
||||
"true",
|
||||
);
|
||||
});
|
||||
|
||||
it("routes partials to onInterim and finals to onTranscript", async () => {
|
||||
const onTranscript = vi.fn();
|
||||
const onInterim = vi.fn();
|
||||
renderServerMode({ onTranscript, onInterim });
|
||||
await clickMic();
|
||||
|
||||
act(() => sessionEvents?.onPartial("hello wor"));
|
||||
expect(onInterim).toHaveBeenCalledWith("hello wor");
|
||||
expect(onTranscript).not.toHaveBeenCalled();
|
||||
|
||||
act(() => sessionEvents?.onFinal("Hello, world."));
|
||||
expect(onTranscript).toHaveBeenCalledWith("Hello, world.");
|
||||
});
|
||||
|
||||
it("stop click flushes the tail into onTranscript", async () => {
|
||||
const onTranscript = vi.fn();
|
||||
sessionStopMock = vi.fn(async () => "tail words");
|
||||
renderServerMode({ onTranscript });
|
||||
await clickMic();
|
||||
await clickMic();
|
||||
|
||||
expect(sessionStopMock).toHaveBeenCalledTimes(1);
|
||||
expect(onTranscript).toHaveBeenCalledWith("tail words");
|
||||
expect(screen.getByRole("button", { name: "Voice dictation" })).toHaveAttribute(
|
||||
"aria-pressed",
|
||||
"false",
|
||||
);
|
||||
});
|
||||
|
||||
it("stop with an empty tail clears the interim region instead", async () => {
|
||||
const onTranscript = vi.fn();
|
||||
const onInterim = vi.fn();
|
||||
renderServerMode({ onTranscript, onInterim });
|
||||
await clickMic();
|
||||
await clickMic();
|
||||
|
||||
expect(onTranscript).not.toHaveBeenCalled();
|
||||
expect(onInterim).toHaveBeenCalledWith("");
|
||||
});
|
||||
|
||||
it("surfaces mic permission denial in the tooltip", async () => {
|
||||
sessionStartMock = vi.fn(async () => {
|
||||
throw new DOMException("denied", "NotAllowedError");
|
||||
});
|
||||
renderServerMode();
|
||||
await clickMic();
|
||||
expect(screen.getByRole("button", { name: "Voice dictation" })).toHaveAttribute(
|
||||
"title",
|
||||
"Microphone permission denied",
|
||||
);
|
||||
});
|
||||
|
||||
it("a mid-take transport error resets state and reports unavailable", async () => {
|
||||
const onInterim = vi.fn();
|
||||
renderServerMode({ onInterim });
|
||||
await clickMic();
|
||||
|
||||
act(() => sessionEvents?.onError("dictation failed"));
|
||||
const button = screen.getByRole("button", { name: "Voice dictation" });
|
||||
expect(button).toHaveAttribute("aria-pressed", "false");
|
||||
expect(button).toHaveAttribute("title", "Dictation unavailable");
|
||||
expect(onInterim).toHaveBeenCalledWith("");
|
||||
});
|
||||
|
||||
it("falls back to server dictation when Web Speech dies with a network error", async () => {
|
||||
// Electron / plain Chromium: the SpeechRecognition constructor exists
|
||||
// (so Web Speech is picked first) but its cloud backend rejects the
|
||||
// build at runtime with "network". The take must fall back to server
|
||||
// dictation so the user's click still lands.
|
||||
const onInterim = vi.fn();
|
||||
render(
|
||||
<CapabilitiesContext.Provider value={DICTATION_INFO}>
|
||||
<ComposerMicButton onTranscript={vi.fn()} onInterim={onInterim} />
|
||||
</CapabilitiesContext.Provider>,
|
||||
);
|
||||
const button = screen.getByRole("button", { name: "Voice dictation" });
|
||||
|
||||
fireEvent.click(button);
|
||||
expect(startSpy).toHaveBeenCalledTimes(1);
|
||||
await act(async () => handlers.error?.({ error: "network" }));
|
||||
|
||||
// The take restarted on the server path, with no error tooltip for
|
||||
// the silent switch, and partials flow.
|
||||
expect(sessionStartMock).toHaveBeenCalledTimes(1);
|
||||
expect(button).toHaveAttribute("aria-pressed", "true");
|
||||
expect(button).toHaveAttribute("title", "Voice dictation");
|
||||
act(() => sessionEvents?.onPartial("via server"));
|
||||
expect(onInterim).toHaveBeenCalledWith("via server");
|
||||
|
||||
// Stale events from the dead recognizer must not clobber the live
|
||||
// server take's state (Chrome fires "end" after a failed start).
|
||||
act(() => handlers.end?.({}));
|
||||
expect(button).toHaveAttribute("aria-pressed", "true");
|
||||
|
||||
// The fallback is per take, not sticky: after stopping, the next
|
||||
// take tries Web Speech again (a transient Chrome blip must not
|
||||
// permanently downgrade the page to the server model).
|
||||
await clickMic(); // stop the server take
|
||||
await clickMic(); // next take
|
||||
expect(startSpy).toHaveBeenCalledTimes(2);
|
||||
expect(sessionStartMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("reports a busy server distinctly from a broken one", async () => {
|
||||
const { DictationBusyError } = await import("@/lib/dictation");
|
||||
sessionStartMock = vi.fn(async () => {
|
||||
throw new DictationBusyError("at capacity");
|
||||
});
|
||||
renderServerMode();
|
||||
await clickMic();
|
||||
expect(screen.getByRole("button", { name: "Voice dictation" })).toHaveAttribute(
|
||||
"title",
|
||||
"Dictation is busy — try again shortly",
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps the plain error path when the server offers no dictation", async () => {
|
||||
render(
|
||||
<CapabilitiesContext.Provider value={{ ...DICTATION_INFO, dictation_available: false }}>
|
||||
<ComposerMicButton onTranscript={vi.fn()} />
|
||||
</CapabilitiesContext.Provider>,
|
||||
);
|
||||
const button = screen.getByRole("button", { name: "Voice dictation" });
|
||||
fireEvent.click(button);
|
||||
await act(async () => handlers.error?.({ error: "network" }));
|
||||
expect(sessionStartMock).not.toHaveBeenCalled();
|
||||
expect(button).toHaveAttribute("title", "Dictation unavailable");
|
||||
});
|
||||
|
||||
it("cancels the session when the composer goes disabled mid-take", async () => {
|
||||
const { rerender } = renderServerMode();
|
||||
await clickMic();
|
||||
|
||||
rerender(
|
||||
<CapabilitiesContext.Provider value={DICTATION_INFO}>
|
||||
<ComposerMicButton onTranscript={vi.fn()} disabled />
|
||||
</CapabilitiesContext.Provider>,
|
||||
);
|
||||
expect(sessionCancelMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
"use client";
|
||||
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { useServerInfo } from "@/lib/CapabilitiesContext";
|
||||
import { DictationBusyError, DictationSession } from "@/lib/dictation";
|
||||
import { cn } from "@/lib/utils";
|
||||
import { MicIcon, SquareIcon } from "lucide-react";
|
||||
import { useCallback, useEffect, useRef, useState } from "react";
|
||||
@@ -55,23 +57,54 @@ const BAR_BASELINE = 0.2;
|
||||
|
||||
export type ComposerMicButtonProps = {
|
||||
onTranscript: (text: string) => void;
|
||||
/**
|
||||
* Streaming partial transcripts (server dictation only): called with the
|
||||
* revisable in-progress utterance as it forms, and with "" when the take
|
||||
* ends without finalizing it. Utterances that do finalize arrive via
|
||||
* onTranscript, which supersedes the pending interim. When absent, the
|
||||
* server path still works but only finals are inserted — the same
|
||||
* behavior the Web Speech path has always had.
|
||||
*/
|
||||
onInterim?: (text: string) => void;
|
||||
disabled?: boolean;
|
||||
lang?: string;
|
||||
};
|
||||
|
||||
/** getUserMedia permission failures, distinct from transport failures. */
|
||||
const isPermissionError = (error: unknown): boolean =>
|
||||
error instanceof DOMException &&
|
||||
(error.name === "NotAllowedError" || error.name === "SecurityError");
|
||||
|
||||
export const ComposerMicButton = ({
|
||||
onTranscript,
|
||||
onInterim,
|
||||
disabled,
|
||||
lang = "en-US",
|
||||
}: ComposerMicButtonProps) => {
|
||||
// null Ctor → no Web Speech support → render nothing (no server fallback).
|
||||
// Web Speech is primary whenever the browser has the constructor
|
||||
// (Chrome/Safari, unchanged behavior); with no constructor at all
|
||||
// (Firefox) takes use server dictation when GET /v1/info advertises it.
|
||||
// A constructor is no guarantee of a backend — Electron and plain
|
||||
// Chromium error at runtime with "network" — so a failed Web Speech
|
||||
// take falls back to the server per take (see handleError). Per-take,
|
||||
// not sticky: a transient blip in real Chrome must not permanently
|
||||
// downgrade the page to the local model.
|
||||
const [Ctor] = useState(getRecognitionCtor);
|
||||
const serverInfo = useServerInfo();
|
||||
const serverAvailable = serverInfo !== "loading" && serverInfo.dictation_available;
|
||||
// Mirrored into a ref so the mount-time recognition handlers (closed
|
||||
// over [Ctor, lang]) see the current probe result.
|
||||
const serverAvailableRef = useRef(serverAvailable);
|
||||
serverAvailableRef.current = serverAvailable;
|
||||
const [isListening, setIsListening] = useState(false);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const recognitionRef = useRef<SpeechRecognitionLike | null>(null);
|
||||
// Ref so the result handler isn't re-attached on every parent re-render.
|
||||
const sessionRef = useRef<DictationSession | null>(null);
|
||||
// Refs so handlers aren't re-attached on every parent re-render.
|
||||
const onTranscriptRef = useRef(onTranscript);
|
||||
onTranscriptRef.current = onTranscript;
|
||||
const onInterimRef = useRef(onInterim);
|
||||
onInterimRef.current = onInterim;
|
||||
// Synced prop ref so the recognition result handler (closure over the
|
||||
// mount-time effect) can drop late events when the composer goes
|
||||
// disabled mid-utterance.
|
||||
@@ -81,6 +114,13 @@ export const ComposerMicButton = ({
|
||||
// Prevents rapid double-clicks from calling recognition.start() twice,
|
||||
// which throws InvalidStateError in Chrome.
|
||||
const transitionRef = useRef(false);
|
||||
// Server-take guard, deliberately separate from transitionRef: a failed
|
||||
// Web Speech attempt fires a late "end" event that resets transitionRef,
|
||||
// which must not unlock a second server take mid-handshake.
|
||||
const serverBusyRef = useRef(false);
|
||||
// Lets the mount-time Web Speech error handler start the fallback take
|
||||
// without closing over toggleServer's identity.
|
||||
const toggleServerRef = useRef<() => Promise<void>>(async () => {});
|
||||
|
||||
// Written via .style.transform from rAF — avoids 60Hz React re-renders.
|
||||
const barRefs = useRef<(HTMLSpanElement | null)[]>(BAR_BINS.map(() => null));
|
||||
@@ -94,18 +134,35 @@ export const ComposerMicButton = ({
|
||||
recognition.interimResults = false;
|
||||
recognition.lang = lang;
|
||||
|
||||
// A dead recognizer keeps firing start/end/error after the take has
|
||||
// fallen back to the server; those stale events must not clobber the
|
||||
// server session's isListening/transition state.
|
||||
const serverTakeOwnsState = () => sessionRef.current !== null || serverBusyRef.current;
|
||||
|
||||
const handleStart = () => {
|
||||
if (serverTakeOwnsState()) return;
|
||||
transitionRef.current = false;
|
||||
setError(null);
|
||||
setIsListening(true);
|
||||
};
|
||||
const handleEnd = () => {
|
||||
if (serverTakeOwnsState()) return;
|
||||
transitionRef.current = false;
|
||||
setIsListening(false);
|
||||
};
|
||||
const handleError = (event: Event) => {
|
||||
if (serverTakeOwnsState()) return;
|
||||
transitionRef.current = false;
|
||||
const err = (event as SpeechRecognitionErrorEventLike).error;
|
||||
// "network" means the recognizer's cloud backend refused us —
|
||||
// always the case in Electron/plain Chromium, occasionally a
|
||||
// transient blip in real Chrome. Serve THIS take from the server
|
||||
// instead; the next take tries Web Speech again.
|
||||
if (err === "network" && serverAvailableRef.current && !disabledRef.current) {
|
||||
setIsListening(false);
|
||||
void toggleServerRef.current();
|
||||
return;
|
||||
}
|
||||
// "no-speech" / "aborted" are routine (silence timeout, user stop).
|
||||
if (err === "not-allowed" || err === "service-not-allowed") {
|
||||
setError("Microphone permission denied");
|
||||
@@ -147,9 +204,18 @@ export const ComposerMicButton = ({
|
||||
|
||||
// Auto-stop if the composer goes disabled mid-dictation. Stops the
|
||||
// recognizer; the disabledRef guard in handleResult catches any final
|
||||
// events still queued before the end event fires.
|
||||
// events still queued before the end event fires. A server session is
|
||||
// cancelled outright (no tail flush) — the take is moot once the
|
||||
// composer can't accept text.
|
||||
useEffect(() => {
|
||||
if (!(disabled && isListening)) return;
|
||||
if (sessionRef.current) {
|
||||
sessionRef.current.cancel();
|
||||
sessionRef.current = null;
|
||||
setIsListening(false);
|
||||
onInterimRef.current?.("");
|
||||
return;
|
||||
}
|
||||
try {
|
||||
recognitionRef.current?.stop();
|
||||
} catch {
|
||||
@@ -158,6 +224,16 @@ export const ComposerMicButton = ({
|
||||
}
|
||||
}, [disabled, isListening]);
|
||||
|
||||
// Release the mic if the component unmounts mid-take (e.g. the
|
||||
// new-chat dialog closes while dictating).
|
||||
useEffect(
|
||||
() => () => {
|
||||
sessionRef.current?.cancel();
|
||||
sessionRef.current = null;
|
||||
},
|
||||
[],
|
||||
);
|
||||
|
||||
// Second getUserMedia stream just for visualization — Web Speech API
|
||||
// hides its audio buffer. Chrome batches the permission to one prompt.
|
||||
useEffect(() => {
|
||||
@@ -226,7 +302,69 @@ export const ComposerMicButton = ({
|
||||
};
|
||||
}, [isListening]);
|
||||
|
||||
// Server-dictation toggle. Start resolves only once the mic + socket
|
||||
// handshake are up, so isListening flips exactly when audio flows.
|
||||
const toggleServer = useCallback(async () => {
|
||||
if (serverBusyRef.current) return;
|
||||
serverBusyRef.current = true;
|
||||
const session = sessionRef.current;
|
||||
if (session) {
|
||||
sessionRef.current = null;
|
||||
const tail = (await session.stop()).trim();
|
||||
if (!disabledRef.current) {
|
||||
// A non-empty tail supersedes the pending interim via
|
||||
// onTranscript; an empty one just clears the interim region.
|
||||
if (tail) onTranscriptRef.current(tail);
|
||||
else onInterimRef.current?.("");
|
||||
}
|
||||
setIsListening(false);
|
||||
serverBusyRef.current = false;
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const next = await DictationSession.start({
|
||||
onPartial: (text) => {
|
||||
if (!disabledRef.current) onInterimRef.current?.(text);
|
||||
},
|
||||
onFinal: (text) => {
|
||||
const trimmed = text.trim();
|
||||
if (trimmed && !disabledRef.current) onTranscriptRef.current(trimmed);
|
||||
},
|
||||
onError: () => {
|
||||
sessionRef.current = null;
|
||||
setError("Dictation unavailable");
|
||||
setIsListening(false);
|
||||
onInterimRef.current?.("");
|
||||
},
|
||||
});
|
||||
sessionRef.current = next;
|
||||
setError(null);
|
||||
setIsListening(true);
|
||||
} catch (startError) {
|
||||
setError(
|
||||
startError instanceof DictationBusyError
|
||||
? "Dictation is busy — try again shortly"
|
||||
: isPermissionError(startError)
|
||||
? "Microphone permission denied"
|
||||
: "Dictation unavailable",
|
||||
);
|
||||
setIsListening(false);
|
||||
}
|
||||
serverBusyRef.current = false;
|
||||
}, []);
|
||||
toggleServerRef.current = toggleServer;
|
||||
|
||||
const toggle = useCallback(() => {
|
||||
// An active (or starting) server take is owned by the server path,
|
||||
// whichever mode started it.
|
||||
if (sessionRef.current || serverBusyRef.current) {
|
||||
void toggleServer();
|
||||
return;
|
||||
}
|
||||
if (!Ctor) {
|
||||
if (serverAvailable) void toggleServer();
|
||||
return;
|
||||
}
|
||||
// Guard against rapid clicks landing before start/end event fires.
|
||||
if (transitionRef.current) return;
|
||||
const recognition = recognitionRef.current;
|
||||
@@ -240,9 +378,9 @@ export const ComposerMicButton = ({
|
||||
// user can try again, and let the next event reconcile state.
|
||||
transitionRef.current = false;
|
||||
}
|
||||
}, [isListening]);
|
||||
}, [isListening, Ctor, serverAvailable, toggleServer]);
|
||||
|
||||
if (!Ctor) return null;
|
||||
if (!Ctor && !serverAvailable) return null;
|
||||
|
||||
// Stable accessible name with aria-pressed signals toggle state to
|
||||
// screen readers. Error text takes over the tooltip when set.
|
||||
|
||||
@@ -67,6 +67,7 @@ function serverInfo(overrides: Partial<ServerInfo> = {}): ServerInfo {
|
||||
public_sharing_enabled: true,
|
||||
server_version: null,
|
||||
smart_routing_enabled: false,
|
||||
dictation_available: false,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -112,6 +112,7 @@ const SERVER_INFO_OFFLINE_FALLBACK: ServerInfo = {
|
||||
public_sharing_enabled: true,
|
||||
server_version: null,
|
||||
smart_routing_enabled: false,
|
||||
dictation_available: false,
|
||||
};
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
// Tests for useDictationInsert — the replaceable trailing interim region
|
||||
// that lets server dictation stream live text into a plain-string draft.
|
||||
//
|
||||
// Invariant under test throughout: dictation must never delete text it
|
||||
// didn't write. The interim region is stripped only when the draft still
|
||||
// ends with the exact text the hook inserted.
|
||||
|
||||
import { act, renderHook } from "@testing-library/react";
|
||||
import { StrictMode, useState, type ReactNode } from "react";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { useDictationInsert } from "./useDictationInsert";
|
||||
|
||||
/** Harness pairing the hook with the same useState shape the composers use. */
|
||||
function renderDictation(
|
||||
initial = "",
|
||||
wrapper?: ({ children }: { children: ReactNode }) => ReactNode,
|
||||
) {
|
||||
return renderHook(
|
||||
() => {
|
||||
const [value, setValue] = useState(initial);
|
||||
const dictation = useDictationInsert(setValue);
|
||||
return { value, setRaw: (next: string) => setValue(() => next), ...dictation };
|
||||
},
|
||||
wrapper ? { wrapper } : undefined,
|
||||
);
|
||||
}
|
||||
|
||||
describe("useDictationInsert", () => {
|
||||
it("streams interim text as a replaceable trailing region", () => {
|
||||
const { result } = renderDictation();
|
||||
act(() => result.current.replaceInterim("hello"));
|
||||
expect(result.current.value).toBe("hello");
|
||||
act(() => result.current.replaceInterim("hello world"));
|
||||
expect(result.current.value).toBe("hello world");
|
||||
// Partials are revisable — a shorter rewrite replaces, never appends.
|
||||
act(() => result.current.replaceInterim("help"));
|
||||
expect(result.current.value).toBe("help");
|
||||
});
|
||||
|
||||
it("finalizing replaces the interim and pins the text", () => {
|
||||
const { result } = renderDictation();
|
||||
act(() => result.current.replaceInterim("hello wor"));
|
||||
act(() => result.current.appendFinal("Hello, world."));
|
||||
expect(result.current.value).toBe("Hello, world.");
|
||||
// The finalized text is no longer part of any interim region.
|
||||
act(() => result.current.replaceInterim("next"));
|
||||
expect(result.current.value).toBe("Hello, world. next");
|
||||
});
|
||||
|
||||
it("space-separates from an existing draft without doubling spaces", () => {
|
||||
const { result } = renderDictation("draft");
|
||||
act(() => result.current.replaceInterim("spoken"));
|
||||
expect(result.current.value).toBe("draft spoken");
|
||||
act(() => result.current.appendFinal("Spoken."));
|
||||
expect(result.current.value).toBe("draft Spoken.");
|
||||
|
||||
const trailing = renderDictation("draft ");
|
||||
act(() => trailing.result.current.appendFinal("Spoken."));
|
||||
expect(trailing.result.current.value).toBe("draft Spoken.");
|
||||
});
|
||||
|
||||
it("clearing the interim restores the base draft", () => {
|
||||
const { result } = renderDictation("draft");
|
||||
act(() => result.current.replaceInterim("partial words"));
|
||||
act(() => result.current.replaceInterim(""));
|
||||
expect(result.current.value).toBe("draft");
|
||||
});
|
||||
|
||||
it("survives the draft shrinking underneath a pending interim", () => {
|
||||
const { result } = renderDictation();
|
||||
act(() => result.current.replaceInterim("some long partial"));
|
||||
// Send clears the draft out from under the pending interim region.
|
||||
act(() => result.current.setRaw(""));
|
||||
// A late update must not slice into (or resurrect) stale text.
|
||||
act(() => result.current.replaceInterim("after"));
|
||||
expect(result.current.value).toBe("after");
|
||||
act(() => result.current.appendFinal("After."));
|
||||
expect(result.current.value).toBe("After.");
|
||||
});
|
||||
|
||||
it("never deletes text the user typed after the interim", () => {
|
||||
const { result } = renderDictation();
|
||||
act(() => result.current.replaceInterim("hello wor"));
|
||||
// The user clicks into the composer and types after the interim.
|
||||
act(() => result.current.setRaw("hello wor, urgent"));
|
||||
// The draft no longer ends with the tracked interim → nothing is
|
||||
// stripped; the update appends instead of slicing typed text away.
|
||||
act(() => result.current.appendFinal("Hello world."));
|
||||
expect(result.current.value).toBe("hello wor, urgent Hello world.");
|
||||
});
|
||||
|
||||
it("is StrictMode-safe (updaters are pure; double-invoke is a no-op)", () => {
|
||||
const strict = ({ children }: { children: ReactNode }) => <StrictMode>{children}</StrictMode>;
|
||||
const { result } = renderDictation("draft", strict);
|
||||
act(() => result.current.replaceInterim("one"));
|
||||
act(() => result.current.replaceInterim("one two"));
|
||||
expect(result.current.value).toBe("draft one two");
|
||||
act(() => result.current.appendFinal("One, two."));
|
||||
expect(result.current.value).toBe("draft One, two.");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,69 @@
|
||||
// Shared composer glue for dictation transcripts.
|
||||
//
|
||||
// The mic button emits two kinds of text (see ComposerMicButton):
|
||||
// - final utterances (onTranscript) — append permanently, and
|
||||
// - interim partials (onInterim, server dictation only) — a revisable
|
||||
// trailing region that forms live while the user speaks and is
|
||||
// rewritten on every update until an utterance finalizes.
|
||||
//
|
||||
// Both composers (ChatPage's Composer and NewChatDialog) hold their draft in
|
||||
// a plain useState string, so the revisable region is implemented as a value
|
||||
// transform: the hook remembers the exact interim text it last inserted and
|
||||
// strips it only when the draft still ends with it verbatim. If it doesn't —
|
||||
// the user typed after it, sent the message, or edited the draft — the
|
||||
// marker is simply dropped and nothing is removed: dictation must never
|
||||
// delete text it didn't write.
|
||||
//
|
||||
// The marker ref is read before and written after each setDraft call, never
|
||||
// inside the updater — updaters must stay pure because React StrictMode
|
||||
// double-invokes them.
|
||||
|
||||
import { useCallback, useRef } from "react";
|
||||
|
||||
type SetDraft = (updater: (prev: string) => string) => void;
|
||||
|
||||
/** Separator so dictated text never fuses with existing draft words. */
|
||||
function joined(base: string, text: string): string {
|
||||
if (!text) return base;
|
||||
if (!base || base.endsWith(" ") || base.endsWith("\n")) return base + text;
|
||||
return `${base} ${text}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove the tracked interim region, but only if the draft still ends
|
||||
* with it verbatim; also drop the single space separator we added.
|
||||
*/
|
||||
function stripMarker(prev: string, marker: string): string {
|
||||
if (!marker || !prev.endsWith(marker)) return prev;
|
||||
const base = prev.slice(0, prev.length - marker.length);
|
||||
return base.endsWith(" ") ? base.slice(0, -1) : base;
|
||||
}
|
||||
|
||||
export function useDictationInsert(setDraft: SetDraft): {
|
||||
/** Append a final utterance, replacing any pending interim region. */
|
||||
appendFinal: (text: string) => void;
|
||||
/** Replace the pending interim region ("" clears it). */
|
||||
replaceInterim: (text: string) => void;
|
||||
} {
|
||||
const interimRef = useRef("");
|
||||
|
||||
const replaceInterim = useCallback(
|
||||
(text: string) => {
|
||||
const marker = interimRef.current;
|
||||
interimRef.current = text;
|
||||
setDraft((prev) => joined(stripMarker(prev, marker), text));
|
||||
},
|
||||
[setDraft],
|
||||
);
|
||||
|
||||
const appendFinal = useCallback(
|
||||
(text: string) => {
|
||||
const marker = interimRef.current;
|
||||
interimRef.current = "";
|
||||
setDraft((prev) => joined(stripMarker(prev, marker), text));
|
||||
},
|
||||
[setDraft],
|
||||
);
|
||||
|
||||
return { appendFinal, replaceInterim };
|
||||
}
|
||||
@@ -103,6 +103,14 @@ export interface ServerInfo {
|
||||
* (``OMNIGENT_SMART_ROUTING=1`` + ``llm:`` config). Hidden by default.
|
||||
*/
|
||||
smart_routing_enabled: boolean;
|
||||
/**
|
||||
* True when the server can transcribe dictation audio
|
||||
* (``WS /v1/dictation/stream``; the ``dictation`` extra plus models
|
||||
* are installed). Gates the composer mic button's server
|
||||
* speech-to-text fallback where the browser Web Speech API has no
|
||||
* backend (Electron, Firefox/Chromium).
|
||||
*/
|
||||
dictation_available: boolean;
|
||||
}
|
||||
|
||||
/** Sentinel used when the probe fails — accounts is off, no login URL. */
|
||||
@@ -121,6 +129,7 @@ const _OFF: ServerInfo = {
|
||||
public_sharing_enabled: true,
|
||||
server_version: null,
|
||||
smart_routing_enabled: false,
|
||||
dictation_available: false,
|
||||
};
|
||||
|
||||
let _cached: ServerInfo | null = null;
|
||||
@@ -161,6 +170,7 @@ export async function resolveServerInfo(): Promise<ServerInfo> {
|
||||
public_sharing_enabled: data.public_sharing_enabled !== false,
|
||||
server_version: typeof data.server_version === "string" ? data.server_version : null,
|
||||
smart_routing_enabled: data.smart_routing_enabled === true,
|
||||
dictation_available: data.dictation_available === true,
|
||||
};
|
||||
return _cached;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
// Tests for the dictation socket protocol parsing. The DictationSession
|
||||
// transport itself (mic + AudioWorklet + WebSocket) can't run in jsdom;
|
||||
// its behavior against the component is pinned in ComposerMicButton.test.tsx
|
||||
// with a mocked session, and the full loop runs in the Playwright e2e test
|
||||
// against the server's fake engine.
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { parseDictationEvent } from "./dictation";
|
||||
|
||||
describe("parseDictationEvent", () => {
|
||||
it("parses the transcript event shapes", () => {
|
||||
expect(parseDictationEvent('{"type":"ready"}')).toEqual({ type: "ready" });
|
||||
expect(parseDictationEvent('{"type":"partial","text":"hel"}')).toEqual({
|
||||
type: "partial",
|
||||
text: "hel",
|
||||
});
|
||||
expect(parseDictationEvent('{"type":"final","text":"hello."}')).toEqual({
|
||||
type: "final",
|
||||
text: "hello.",
|
||||
});
|
||||
expect(parseDictationEvent('{"type":"stopped","text":""}')).toEqual({
|
||||
type: "stopped",
|
||||
text: "",
|
||||
});
|
||||
expect(parseDictationEvent('{"type":"error","message":"boom"}')).toEqual({
|
||||
type: "error",
|
||||
message: "boom",
|
||||
});
|
||||
});
|
||||
|
||||
it("returns null for malformed or unknown frames", () => {
|
||||
expect(parseDictationEvent("not json")).toBeNull();
|
||||
expect(parseDictationEvent("42")).toBeNull();
|
||||
expect(parseDictationEvent("null")).toBeNull();
|
||||
expect(parseDictationEvent('{"type":"future-thing"}')).toBeNull();
|
||||
// Known types with a missing/mistyped payload are dropped, not crashed on.
|
||||
expect(parseDictationEvent('{"type":"partial"}')).toBeNull();
|
||||
expect(parseDictationEvent('{"type":"partial","text":7}')).toBeNull();
|
||||
expect(parseDictationEvent('{"type":"error"}')).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,367 @@
|
||||
// Server-side dictation transport: mic → 16 kHz PCM → WS /v1/dictation/stream.
|
||||
//
|
||||
// ComposerMicButton uses this as the fallback when the browser Web Speech API
|
||||
// has no working backend (Electron, Firefox/Chromium — see
|
||||
// web/electron/README.md). One DictationSession is one dictation take: it
|
||||
// owns the microphone stream, an AudioWorklet that downsamples the capture
|
||||
// rate to 16 kHz mono s16le, and the WebSocket that streams those frames to
|
||||
// the server and receives transcript events back. The wire protocol is
|
||||
// documented in omnigent/server/routes/dictation.py; availability is gated
|
||||
// by the `dictation_available` capability from GET /v1/info.
|
||||
//
|
||||
// The WebSocket URL rides the host seam (`resolveWebSocketUrl`) exactly like
|
||||
// the terminal-attach and session-updates sockets, so embed hosts and the
|
||||
// Vite dev proxy keep working. Identity rides the ingress/dev proxy on the
|
||||
// handshake, as with those sockets.
|
||||
|
||||
import { resolveWebSocketUrl } from "@/lib/host";
|
||||
|
||||
/** A transcript event pushed by the server over the dictation stream. */
|
||||
export type DictationEvent =
|
||||
| { type: "ready" }
|
||||
| { type: "partial"; text: string }
|
||||
| { type: "final"; text: string }
|
||||
| { type: "stopped"; text: string }
|
||||
| { type: "error"; message: string };
|
||||
|
||||
export type DictationSessionEvents = {
|
||||
/** Revisable in-progress utterance (server-throttled to ~6 Hz). */
|
||||
onPartial: (text: string) => void;
|
||||
/** An utterance completed by a pause; append it and clear the partial. */
|
||||
onFinal: (text: string) => void;
|
||||
/** Fatal error after start. The session has already cleaned itself up. */
|
||||
onError: (message: string) => void;
|
||||
};
|
||||
|
||||
/**
|
||||
* The server was reachable but at its concurrent-take cap (WS close 1013).
|
||||
* Transient by definition — callers should message "busy, try again",
|
||||
* not "unavailable".
|
||||
*/
|
||||
export class DictationBusyError extends Error {}
|
||||
|
||||
/**
|
||||
* Parse one text frame from the dictation socket into a typed event.
|
||||
* Returns null for frames that don't match the protocol (ignored for
|
||||
* forward compatibility, mirroring the server's posture on control
|
||||
* messages it doesn't know).
|
||||
*/
|
||||
export function parseDictationEvent(raw: string): DictationEvent | null {
|
||||
let data: unknown;
|
||||
try {
|
||||
data = JSON.parse(raw);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (typeof data !== "object" || data === null) return null;
|
||||
const frame = data as { type?: unknown; text?: unknown; message?: unknown };
|
||||
switch (frame.type) {
|
||||
case "ready":
|
||||
return { type: "ready" };
|
||||
case "partial":
|
||||
case "final":
|
||||
case "stopped":
|
||||
return typeof frame.text === "string" ? { type: frame.type, text: frame.text } : null;
|
||||
case "error":
|
||||
return typeof frame.message === "string" ? { type: "error", message: frame.message } : null;
|
||||
default:
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// Client budgets must exceed the server's own worst cases or takes fail
|
||||
// spuriously right when they'd have succeeded. The dominant cost is the
|
||||
// first take's engine construction, which loads the model weights (seconds
|
||||
// on a cold server); the stop budget just needs to outlast the tail flush.
|
||||
const READY_TIMEOUT_MS = 20_000;
|
||||
const STOP_TIMEOUT_MS = 5_000;
|
||||
|
||||
// How long stop() waits for the worklet to post its final partial chunk
|
||||
// before tearing the audio graph down. Message-port turnaround is
|
||||
// milliseconds; this is only a stuck-worklet backstop.
|
||||
const FLUSH_TIMEOUT_MS = 250;
|
||||
|
||||
/** WS close code the server sends when at its concurrent-take cap. */
|
||||
const WS_CLOSE_TRY_AGAIN_LATER = 1013;
|
||||
|
||||
const TARGET_RATE = 16_000;
|
||||
|
||||
// AudioWorklet processor, inlined as a Blob module so no separate asset has
|
||||
// to survive the Vite build. Linear-interpolation downsample from the
|
||||
// context capture rate to 16 kHz, Float32 → Int16, posted in 100 ms chunks.
|
||||
// (When the context already runs at 16 kHz — we ask for that — the step is
|
||||
// 1 and the loop degenerates to a plain format conversion.)
|
||||
//
|
||||
// Any message on the port means "flush": the partially-filled chunk is
|
||||
// posted, then a null marker — so stop() can capture trailing speech that
|
||||
// hasn't crossed the 100 ms boundary before tearing the graph down.
|
||||
const WORKLET_SOURCE = `
|
||||
const TARGET_RATE = ${TARGET_RATE};
|
||||
const CHUNK_SAMPLES = TARGET_RATE / 10; // 100 ms per posted chunk
|
||||
class Pcm16Downsampler extends AudioWorkletProcessor {
|
||||
constructor() {
|
||||
super();
|
||||
this.step = sampleRate / TARGET_RATE;
|
||||
this.pos = 0; // fractional read position, carried across blocks
|
||||
this.pending = new Int16Array(CHUNK_SAMPLES);
|
||||
this.filled = 0;
|
||||
this.port.onmessage = () => {
|
||||
if (this.filled > 0) {
|
||||
const out = this.pending.slice(0, this.filled);
|
||||
this.filled = 0;
|
||||
this.port.postMessage(out, [out.buffer]);
|
||||
}
|
||||
this.port.postMessage(null);
|
||||
};
|
||||
}
|
||||
process(inputs) {
|
||||
const channel = inputs[0] && inputs[0][0];
|
||||
if (!channel || channel.length === 0) return true;
|
||||
let pos = this.pos;
|
||||
while (pos < channel.length) {
|
||||
const i = Math.floor(pos);
|
||||
const s0 = channel[i];
|
||||
const s1 = i + 1 < channel.length ? channel[i + 1] : s0;
|
||||
const sample = s0 + (s1 - s0) * (pos - i);
|
||||
const clamped = Math.max(-1, Math.min(1, sample));
|
||||
this.pending[this.filled++] = clamped < 0 ? clamped * 0x8000 : clamped * 0x7fff;
|
||||
if (this.filled === CHUNK_SAMPLES) {
|
||||
const out = this.pending;
|
||||
this.pending = new Int16Array(CHUNK_SAMPLES);
|
||||
this.filled = 0;
|
||||
this.port.postMessage(out, [out.buffer]);
|
||||
}
|
||||
pos += this.step;
|
||||
}
|
||||
this.pos = pos - channel.length;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
registerProcessor("omnigent-pcm16-downsampler", Pcm16Downsampler);
|
||||
`;
|
||||
|
||||
let _workletUrl: string | null = null;
|
||||
|
||||
function workletUrl(): string {
|
||||
if (_workletUrl === null) {
|
||||
_workletUrl = URL.createObjectURL(
|
||||
new Blob([WORKLET_SOURCE], { type: "application/javascript" }),
|
||||
);
|
||||
}
|
||||
return _workletUrl;
|
||||
}
|
||||
|
||||
/**
|
||||
* One live dictation take against the server recognizer.
|
||||
*
|
||||
* Construct via {@link DictationSession.start}, which resolves once the
|
||||
* mic, audio graph, and socket handshake are all up — so a resolved
|
||||
* session is guaranteed to be streaming. End it with {@link stop} (flushes
|
||||
* the tail utterance) or {@link cancel} (immediate teardown, e.g. unmount).
|
||||
*/
|
||||
export class DictationSession {
|
||||
private readonly events: DictationSessionEvents;
|
||||
private readonly ws: WebSocket;
|
||||
private readonly mediaStream: MediaStream;
|
||||
private readonly audioContext: AudioContext;
|
||||
private readonly workletNode: AudioWorkletNode;
|
||||
private stopResolve: ((tail: string) => void) | null = null;
|
||||
private flushResolve: (() => void) | null = null;
|
||||
private closed = false;
|
||||
|
||||
private constructor(
|
||||
events: DictationSessionEvents,
|
||||
ws: WebSocket,
|
||||
mediaStream: MediaStream,
|
||||
audioContext: AudioContext,
|
||||
workletNode: AudioWorkletNode,
|
||||
) {
|
||||
this.events = events;
|
||||
this.ws = ws;
|
||||
this.mediaStream = mediaStream;
|
||||
this.audioContext = audioContext;
|
||||
this.workletNode = workletNode;
|
||||
|
||||
workletNode.port.onmessage = (msg: MessageEvent<Int16Array<ArrayBuffer> | null>) => {
|
||||
if (msg.data === null) {
|
||||
// Flush marker: the worklet has posted everything it had.
|
||||
this.flushResolve?.();
|
||||
this.flushResolve = null;
|
||||
return;
|
||||
}
|
||||
if (ws.readyState === WebSocket.OPEN) ws.send(msg.data.buffer);
|
||||
};
|
||||
ws.onmessage = (msg) => {
|
||||
if (typeof msg.data !== "string") return;
|
||||
const event = parseDictationEvent(msg.data);
|
||||
if (event === null) return;
|
||||
if (event.type === "partial") this.events.onPartial(event.text);
|
||||
else if (event.type === "final") this.events.onFinal(event.text);
|
||||
else if (event.type === "stopped") this.resolveStop(event.text);
|
||||
else if (event.type === "error") this.fail(event.message);
|
||||
};
|
||||
ws.onclose = () => {
|
||||
// A close during stop() is the normal end of a take; any other
|
||||
// close means the server went away mid-dictation.
|
||||
if (this.stopResolve !== null) this.resolveStop("");
|
||||
else if (!this.closed) this.fail("Dictation connection closed");
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Acquire the mic, open the socket, and wait for the server's ready
|
||||
* handshake. Rejects (with everything torn down) when the mic is
|
||||
* denied, the socket fails, the server is at capacity
|
||||
* ({@link DictationBusyError}), or the engine never comes up.
|
||||
*/
|
||||
static async start(events: DictationSessionEvents): Promise<DictationSession> {
|
||||
const mediaStream = await navigator.mediaDevices.getUserMedia({
|
||||
audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true },
|
||||
});
|
||||
|
||||
let ws: WebSocket | null = null;
|
||||
let audioContext: AudioContext | null = null;
|
||||
try {
|
||||
ws = new WebSocket(resolveWebSocketUrl("/v1/dictation/stream"));
|
||||
ws.binaryType = "arraybuffer";
|
||||
await waitForReady(ws);
|
||||
// Detect a close during the async audio-graph setup below: the
|
||||
// handler-swap in the constructor would otherwise never see it and
|
||||
// start() would resolve a dead session that silently drops audio.
|
||||
let closedDuringSetup = false;
|
||||
const markClosed = () => {
|
||||
closedDuringSetup = true;
|
||||
};
|
||||
ws.addEventListener("close", markClosed);
|
||||
|
||||
// Ask for the target rate directly — Chrome/Firefox resample the
|
||||
// capture for us and the worklet's downsampler becomes a no-op.
|
||||
// Some platforms reject the hint; the worklet handles any rate.
|
||||
try {
|
||||
audioContext = new AudioContext({ sampleRate: TARGET_RATE });
|
||||
} catch {
|
||||
audioContext = new AudioContext();
|
||||
}
|
||||
await audioContext.audioWorklet.addModule(workletUrl());
|
||||
const source = audioContext.createMediaStreamSource(mediaStream);
|
||||
const node = new AudioWorkletNode(audioContext, "omnigent-pcm16-downsampler");
|
||||
// The worklet only renders while it reaches the destination; route
|
||||
// it through a muted gain so nothing is audible.
|
||||
const mute = audioContext.createGain();
|
||||
mute.gain.value = 0;
|
||||
source.connect(node);
|
||||
node.connect(mute);
|
||||
mute.connect(audioContext.destination);
|
||||
|
||||
ws.removeEventListener("close", markClosed);
|
||||
if (closedDuringSetup || ws.readyState !== WebSocket.OPEN) {
|
||||
throw new Error("dictation connection closed during setup");
|
||||
}
|
||||
return new DictationSession(events, ws, mediaStream, audioContext, node);
|
||||
} catch (error) {
|
||||
for (const track of mediaStream.getTracks()) track.stop();
|
||||
if (audioContext && audioContext.state !== "closed") void audioContext.close();
|
||||
ws?.close();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* End the take: flush the worklet's trailing samples (so the last words
|
||||
* make it to the recognizer), release the mic, ask the server to flush,
|
||||
* and resolve with the flushed tail utterance ("" on timeout/close).
|
||||
*/
|
||||
async stop(): Promise<string> {
|
||||
if (this.closed) return "";
|
||||
await this.flushWorklet();
|
||||
this.teardownAudio();
|
||||
if (this.ws.readyState !== WebSocket.OPEN) {
|
||||
this.closed = true;
|
||||
return "";
|
||||
}
|
||||
return new Promise<string>((resolve) => {
|
||||
this.stopResolve = resolve;
|
||||
this.ws.send(JSON.stringify({ type: "stop" }));
|
||||
setTimeout(() => this.resolveStop(""), STOP_TIMEOUT_MS);
|
||||
});
|
||||
}
|
||||
|
||||
/** Immediate teardown without waiting for the tail (unmount, disable). */
|
||||
cancel(): void {
|
||||
this.teardownAudio();
|
||||
this.closed = true;
|
||||
this.ws.close();
|
||||
}
|
||||
|
||||
/** Ask the worklet to post its partial chunk; wait for its marker. */
|
||||
private flushWorklet(): Promise<void> {
|
||||
return new Promise<void>((resolve) => {
|
||||
this.flushResolve = resolve;
|
||||
try {
|
||||
this.workletNode.port.postMessage("flush");
|
||||
} catch {
|
||||
resolve();
|
||||
return;
|
||||
}
|
||||
setTimeout(resolve, FLUSH_TIMEOUT_MS);
|
||||
});
|
||||
}
|
||||
|
||||
private resolveStop(tail: string): void {
|
||||
this.closed = true;
|
||||
const resolve = this.stopResolve;
|
||||
this.stopResolve = null;
|
||||
this.ws.close();
|
||||
resolve?.(tail);
|
||||
}
|
||||
|
||||
private fail(message: string): void {
|
||||
this.closed = true;
|
||||
this.teardownAudio();
|
||||
this.ws.close();
|
||||
this.events.onError(message);
|
||||
}
|
||||
|
||||
private teardownAudio(): void {
|
||||
for (const track of this.mediaStream.getTracks()) track.stop();
|
||||
if (this.audioContext.state !== "closed") void this.audioContext.close();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve when the server sends its ready frame; reject on error frame,
|
||||
* close (typed {@link DictationBusyError} for the 1013 at-capacity close),
|
||||
* or timeout.
|
||||
*/
|
||||
function waitForReady(ws: WebSocket): Promise<void> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const timer = setTimeout(() => {
|
||||
reject(new Error("dictation server did not become ready"));
|
||||
}, READY_TIMEOUT_MS);
|
||||
ws.onmessage = (msg) => {
|
||||
if (typeof msg.data !== "string") return;
|
||||
const event = parseDictationEvent(msg.data);
|
||||
if (event?.type === "ready") {
|
||||
clearTimeout(timer);
|
||||
resolve();
|
||||
} else if (event?.type === "error") {
|
||||
// The engine failed to initialize; surface its message rather
|
||||
// than the generic close that follows.
|
||||
clearTimeout(timer);
|
||||
reject(new Error(event.message));
|
||||
}
|
||||
};
|
||||
ws.onerror = () => {
|
||||
clearTimeout(timer);
|
||||
reject(new Error("dictation connection failed"));
|
||||
};
|
||||
ws.onclose = (event) => {
|
||||
clearTimeout(timer);
|
||||
reject(
|
||||
event.code === WS_CLOSE_TRY_AGAIN_LATER
|
||||
? new DictationBusyError("dictation is at capacity")
|
||||
: new Error("dictation connection closed"),
|
||||
);
|
||||
};
|
||||
});
|
||||
}
|
||||
@@ -89,6 +89,7 @@ const _bootProbe: Promise<ServerInfo> = Promise.race([
|
||||
public_sharing_enabled: true,
|
||||
server_version: null,
|
||||
smart_routing_enabled: false,
|
||||
dictation_available: false,
|
||||
}),
|
||||
1500,
|
||||
),
|
||||
|
||||
@@ -86,6 +86,7 @@ import { usePermissions } from "@/hooks/usePermissions";
|
||||
import type { CodexModelOption, SandboxStatus, Session, SessionStatus } from "@/lib/types";
|
||||
import { usePromptHistory } from "@/hooks/usePromptHistory";
|
||||
import { useAutoGrowTextarea } from "@/hooks/useAutoGrowTextarea";
|
||||
import { useDictationInsert } from "@/hooks/useDictationInsert";
|
||||
import { useIOSNativeKeyboardVisible } from "@/hooks/useIOSNativeKeyboardInset";
|
||||
import type { MessageContentBlock } from "@/lib/blocks";
|
||||
import {
|
||||
@@ -3798,6 +3799,7 @@ export function Composer({
|
||||
subAgentLabel = null,
|
||||
}: ComposerProps) {
|
||||
const [value, setValue] = useState("");
|
||||
const dictation = useDictationInsert(setValue);
|
||||
const [files, setFiles] = useState<File[]>([]);
|
||||
const [attachmentError, setAttachmentError] = useState<string | null>(null);
|
||||
const [commandError, setCommandError] = useState<string | null>(null);
|
||||
@@ -4880,13 +4882,18 @@ export function Composer({
|
||||
<ComposerMicButton
|
||||
disabled={disabled || isReadOnly || hasPendingElicitation}
|
||||
onTranscript={(text) => {
|
||||
setValue((prev) => (prev ? `${prev} ${text}` : text));
|
||||
dictation.appendFinal(text);
|
||||
dirtyRef.current = true;
|
||||
// Dictation is a user-driven edit — exit prompt-recall mode
|
||||
// so ArrowUp/ArrowDown don't clobber the dictated text.
|
||||
resetCursor();
|
||||
if (commandError !== null) setCommandError(null);
|
||||
}}
|
||||
onInterim={(text) => {
|
||||
dictation.replaceInterim(text);
|
||||
dirtyRef.current = true;
|
||||
resetCursor();
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
{/* Cost toggle + agent picker + Send — right side */}
|
||||
|
||||
@@ -320,6 +320,7 @@ function serverInfo(overrides: Partial<ServerInfo> = {}): ServerInfo {
|
||||
public_sharing_enabled: true,
|
||||
server_version: null,
|
||||
smart_routing_enabled: false,
|
||||
dictation_available: false,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -653,6 +653,7 @@ function renderLanding(infoOverrides: Partial<ServerInfo> = {}, route = "/") {
|
||||
public_sharing_enabled: true,
|
||||
server_version: null,
|
||||
smart_routing_enabled: false,
|
||||
dictation_available: false,
|
||||
...infoOverrides,
|
||||
};
|
||||
return render(
|
||||
|
||||
@@ -102,6 +102,7 @@ import {
|
||||
type AvailableAgent,
|
||||
} from "@/hooks/useAvailableAgents";
|
||||
import { useAutoGrowTextarea } from "@/hooks/useAutoGrowTextarea";
|
||||
import { useDictationInsert } from "@/hooks/useDictationInsert";
|
||||
import { useRecentWorkspaces } from "@/hooks/useRecentWorkspaces";
|
||||
import { useDirectorySessions } from "@/hooks/useDirectorySessions";
|
||||
import { useRunnerHealthRegistration } from "@/hooks/RunnerHealthProvider";
|
||||
@@ -1827,6 +1828,7 @@ export function NewChatLandingScreen() {
|
||||
useNativeServerSwitcherForMainSurface(landingSurface, true);
|
||||
|
||||
const [message, setMessage] = useState<string>(() => landingDraft?.message ?? "");
|
||||
const dictation = useDictationInsert(setMessage);
|
||||
const textareaRef = useRef<HTMLTextAreaElement>(null);
|
||||
const isComposingRef = useRef(false);
|
||||
// maxRows 9 = 180px of 20px lines, matching the composer's 200px
|
||||
@@ -3285,7 +3287,8 @@ export function NewChatLandingScreen() {
|
||||
</Button>
|
||||
<ComposerMicButton
|
||||
disabled={creating}
|
||||
onTranscript={(text) => setMessage((prev) => (prev ? `${prev} ${text}` : text))}
|
||||
onTranscript={dictation.appendFinal}
|
||||
onInterim={dictation.replaceInterim}
|
||||
/>
|
||||
</div>
|
||||
<div className="flex items-center gap-0.5">
|
||||
|
||||
@@ -136,6 +136,7 @@ function serverInfo(overrides: Partial<ServerInfo> = {}): ServerInfo {
|
||||
public_sharing_enabled: true,
|
||||
server_version: null,
|
||||
smart_routing_enabled: false,
|
||||
dictation_available: false,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user