github-featured/livekit--agents
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity

feat(eot): add audio models AGT-2520 (#4722)

Browse Source 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Brian Yin <brian.yin@livekit.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Long Chen <longch1024@gmail.com>
Co-authored-by: Aditya <147301021+adityajha2005@users.noreply.github.com>
Co-authored-by: Théo Monnom <theo.8bits@gmail.com>
This commit is contained in:
Chenghao Mou
2026-06-16 19:28:59 +01:00
committed by GitHub
parent 887ced38b1
commit 61b14fdcd9
112 changed files with 5231 additions and 699 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+1
View File
@@ -35,6 +35,7 @@ it.
| Marker | Flag | Meaning |
|--------|------|---------|
| `pytest.mark.unit` | `--unit` | fast, hermetic, no external providers/credentials/network |
| `pytest.mark.audio_eot` | `--audio_eot` | hermetic audio end-of-turn / turn-detection suite |
| `pytest.mark.plugin("name")` | `--plugin [name]` | provider integration test (needs that provider's deps/keys) |
| `pytest.mark.stt` | `--stt` | cross-provider speech-to-text suite (`tests/test_stt.py`) |
| `pytest.mark.tts` | `--tts` | cross-provider text-to-speech suite (`tests/test_tts.py`) |
MODEL_LICENSE
+113
View File
@@ -0,0 +1,113 @@
LIVEKIT MODEL LICENSE AGREEMENT
1. Introduction
LiveKit Incorporated ("LiveKit") is making available its proprietary models for
use pursuant to the terms and conditions of this Agreement. As further
described below, you may use these LiveKit models freely but can only use them
together with the LiveKit Agents framework. You cannot use the LiveKit models
on a standalone basis or with any other frameworks.
BY CLICKING "I ACCEPT," OR BY DOWNLOADING, INSTALLING, OR OTHERWISE ACCESSING
OR USING THE LIVEKIT MATERIALS, YOU AGREE THAT YOU HAVE READ AND UNDERSTOOD,
AND, AS A CONDITION TO YOUR USE OF THE LIVEKIT MATERIALS, YOU AGREE TO BE
BOUND BY, THE FOLLOWING TERMS AND CONDITIONS.
2. Definitions
"Agreement" means this LiveKit Model License Agreement.
"Documentation" means the specifications, manuals, and documentation
accompanying any LiveKit Model and distributed by LiveKit.
"Licensee" or "you" means the individual or entity agreeing to be bound by
this Agreement.
"LiveKit Agents" means the proprietary LiveKit software framework for building
real-time multimodal AI applications with programmable backend participants.
"LiveKit Materials" means, collectively, the LiveKit Models and Documentation.
"LiveKit Model" means any of LiveKit's proprietary software models or
algorithms, including machine-learning software code, model weights,
inference-enabling software code, training-enabling software code, and
fine-tuning enabling software code. Any derivative works of a LiveKit Model,
whether developed by LiveKit, you, or any third party, will be deemed the
"LiveKit Model" for the purposes of this Agreement.
3. License Rights
Right to Use LiveKit Materials. Subject to the terms and conditions of this
Agreement, including the requirements of Section 3.b, LiveKit grants you a
nonexclusive, nontransferable, worldwide, royalty-free license under LiveKit's
intellectual property rights to use, reproduce, distribute, copy, and create
derivative works of the LiveKit Materials.
Limitation on Use. As a condition to your use of the LiveKit Materials, you
agree: (i) not to use any LiveKit Models on a standalone basis or with any
frameworks other than LiveKit Agents; (ii) not to use any LiveKit Materials or
any output from, or results of using, LiveKit Models (including any derivative
works thereof) to improve or otherwise develop any other models that are not
LiveKit Models; or (iii) distribute or otherwise make available the LiveKit
Materials (including any derivative works thereof) except (x) pursuant to the
terms of this Agreement, and (y) you reproduce the above copyright notice.
4. Intellectual Property
The LiveKit Materials are owned by LiveKit and its licensors. Except for the
rights granted to you under this Agreement, all rights are reserved and no
other express or implied rights are granted.
You will own any derivative works that you created from the LiveKit Materials,
subject to the terms of this Agreement.
5. Disclaimer
UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, LIVEKIT PROVIDES
THE LIVEKIT MATERIALS, AND ANY OUTPUT OR RESULTS THEREFROM, ON AN "AS IS"
BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED,
INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE,
NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU
ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR
REDISTRIBUTING THE LIVEKIT MATERIALS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR
USE OF THE LIVEKIT MATERIALS AND ANY OUTPUT AND RESULTS.
6. Limitation of Liability
IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE),
CONTRACT, OR OTHERWISE, UNLESS REQUIRED BY APPLICABLE LAW (SUCH AS DELIBERATE
AND GROSSLY NEGLIGENT ACTS) OR AGREED TO IN WRITING, WILL LIVEKIT BE LIABLE TO
YOU FOR INDIRECT DAMAGES, INCLUDING ANY SPECIAL, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES OF ANY CHARACTER ARISING AS A RESULT OF THIS AGREEMENT OR OUT OF THE
USE OR INABILITY TO USE THE LIVEKIT MATERIALS OR ANY OUTPUT OR RESULTS
THEREFROM (INCLUDING BUT NOT LIMITED TO DAMAGES FOR LOSS OF GOODWILL, WORK
STOPPAGE, COMPUTER FAILURE OR MALFUNCTION, OR ANY AND ALL OTHER COMMERCIAL
DAMAGES OR LOSSES), EVEN IF LIVEKIT HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
7. Trademarks
This Agreement does not grant permission to use the trade names, trademarks,
service marks, or product names of LiveKit, except as required for reasonable
and customary use in describing the origin of the LiveKit Materials.
8. Term and Termination
The term of this Agreement commences upon your acceptance of this Agreement
and continues in effect until you cease using the LiveKit Materials or it is
terminated by either party (on immediate written notice to the other party).
This Agreement will automatically terminate if you breach any of its terms.
Upon termination, you must immediately cease all use of the LiveKit Materials.
Sections 4, 5, 6, and 9 will survive termination.
9. Governing Law and Venue
This Agreement is subject to the laws of the State of California, without
regard to its conflict of laws principles. The UN Convention on Contracts for
the International Sale of Goods does not apply to this Agreement. The courts
located in San Francisco, California, have exclusive jurisdiction for any
dispute arising out of this Agreement.
+ + + +
Last Updated: November 25, 2024
README.md
+7 -4
View File
@@ -47,7 +47,7 @@ agents that can see, hear, and understand.
To install the core Agents library, along with plugins for popular model providers:
```bash
pip install "livekit-agents[openai,silero,deepgram,cartesia,turn-detector]"
pip install "livekit-agents[openai,deepgram,cartesia]"
```
## Docs and guides
@@ -92,7 +92,6 @@ from livekit.agents import (
function_tool,
inference,
)
from livekit.plugins import silero
@function_tool
@@ -111,7 +110,7 @@ server = AgentServer()
@server.rtc_session()
async def entrypoint(ctx: JobContext):
session = AgentSession(
vad=silero.VAD.load(),
vad=inference.VAD(),
# any combination of STT, LLM, TTS, or realtime API can be used
# this example shows LiveKit Inference, a unified API to access different models via LiveKit Cloud
# to use model provider keys directly, replace with the following:
@@ -200,7 +199,7 @@ class StoryAgent(Agent):
async def entrypoint(ctx: JobContext):
userdata = StoryData()
session = AgentSession[StoryData](
vad=silero.VAD.load(),
vad=inference.VAD(),
stt="deepgram/nova-3",
llm="openai/gpt-4.1-mini",
tts="cartesia/sonic-3:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
@@ -382,6 +381,10 @@ python myagent.py start
Runs the agent with production-ready optimizations.
## License
The Agents framework is licensed under [Apache-2.0](LICENSE). The LiveKit turn detection models are licensed under the [LiveKit Model License](MODEL_LICENSE).
## Contributing
The Agents framework is under active development in a rapidly evolving field. We welcome and appreciate contributions of any kind, be it feedback, bugfixes, features, new plugins and tools, or better documentation. You can file issues under this repo, open a PR, or chat with us in the [LiveKit community](https://docs.livekit.io/intro/community/).
examples/README.md
+1 -1
View File
@@ -54,7 +54,7 @@ To run the examples, you'll need:
- A [LiveKit Cloud](https://cloud.livekit.io) account or a local [LiveKit server](https://github.com/livekit/livekit)
- API keys for the model providers you want to use in a `.env` file
- Python 3.9 or higher
- Python 3.10 or higher
- [uv](https://docs.astral.sh/uv/)
### Environment file
examples/avatar_agents/audio_wave/agent_worker.py
-2
View File
@@ -20,7 +20,6 @@ from livekit.agents import (
from livekit.agents.voice.avatar import AvatarSession, DataStreamAudioOutput
from livekit.agents.voice.io import PlaybackFinishedEvent, PlaybackStartedEvent
from livekit.agents.voice.room_io import ATTRIBUTE_PUBLISH_ON_BEHALF
from livekit.plugins import silero
load_dotenv()
@@ -114,7 +113,6 @@ async def entrypoint(ctx: JobContext):
stt=inference.STT("deepgram/nova-3"),
llm=inference.LLM("google/gemini-2.5-flash"),
tts=inference.TTS("cartesia/sonic-3"),
vad=silero.VAD.load(),
resume_false_interruption=False,
)
examples/avatar_agents/keyframe/agent_worker.py
+1 -4
View File
@@ -14,9 +14,8 @@ from livekit.agents import (
function_tool,
inference,
)
from livekit.plugins import keyframe, silero
from livekit.plugins import keyframe
from livekit.plugins.keyframe import Emotion
from livekit.plugins.turn_detector.multilingual import MultilingualModel
load_dotenv()
@@ -51,8 +50,6 @@ async def entrypoint(ctx: JobContext):
llm=inference.LLM("google/gemini-2.5-flash"),
tts=inference.TTS("cartesia/sonic-3"),
resume_false_interruption=False,
vad=silero.VAD.load(),
turn_detection=MultilingualModel(),
)
avatar = keyframe.AvatarSession(persona_slug="public:cosmo_persona-1.5-live")
examples/drive-thru/agent.py
-4
View File
@@ -34,8 +34,6 @@ from livekit.agents import (
function_tool,
inference,
)
from livekit.plugins import silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
load_dotenv()
@@ -487,8 +485,6 @@ async def drive_thru_agent(ctx: JobContext) -> None:
),
llm=inference.LLM("openai/gpt-4.1-mini"),
tts=inference.TTS("cartesia/sonic-3", voice="f786b574-daa5-4673-aa0c-cbe3e8534c02"),
turn_detection=MultilingualModel(),
vad=silero.VAD.load(),
max_tool_steps=10,
)
examples/frontdesk/agent.py
-4
View File
@@ -38,8 +38,6 @@ from livekit.agents.evals import (
task_completion_judge,
tool_use_judge,
)
from livekit.plugins import silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
load_dotenv()
@@ -266,8 +264,6 @@ async def frontdesk_agent(ctx: JobContext):
stt=inference.STT("deepgram/nova-3"),
llm=inference.LLM("google/gemini-2.5-flash"),
tts=inference.TTS("cartesia/sonic-3", voice="39b376fc-488e-4d0c-8b37-e00b72059fdd"),
turn_detection=MultilingualModel(),
vad=silero.VAD.load(),
max_tool_steps=1,
)
examples/healthcare/agent.py
-2
View File
@@ -32,7 +32,6 @@ from livekit.agents.beta.workflows import (
WarmTransferTask,
)
from livekit.agents.llm import ToolError, function_tool
from livekit.plugins import silero
logger = logging.getLogger("HealthcareAgent")
@@ -754,7 +753,6 @@ async def entrypoint(ctx: JobContext):
stt=inference.STT("deepgram/nova-3", language="multi"),
llm=inference.LLM("openai/gpt-4.1-mini"),
tts=inference.TTS("inworld/inworld-tts-1"),
vad=silero.VAD.load(),
preemptive_generation=True,
)
examples/inference/agent.py
-2
View File
@@ -12,7 +12,6 @@ from livekit.agents import (
cli,
inference,
)
from livekit.plugins import silero
from livekit.rtc import RpcInvocationData
logger = logging.getLogger("inference")
@@ -59,7 +58,6 @@ async def entrypoint(ctx: JobContext) -> None:
stt=inference.STT(model=DEFAULT_STT),
llm=inference.LLM(model=DEFAULT_LLM),
tts=inference.TTS(model=DEFAULT_TTS),
vad=silero.VAD.load(),
)
def parse_value(payload: str, fallback: str) -> str:
examples/other/elevenlab_scribe_v2.py
+2 -10
View File
@@ -2,8 +2,8 @@ import logging
from dotenv import load_dotenv
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, JobProcess, cli, inference
from livekit.plugins import elevenlabs, silero
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, cli, inference
from livekit.plugins import elevenlabs
logger = logging.getLogger("realtime-scribe-v2")
logger.setLevel(logging.INFO)
@@ -14,13 +14,6 @@ load_dotenv()
server = AgentServer()
def prewarm(proc: JobProcess) -> None:
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
@server.rtc_session()
async def entrypoint(ctx: JobContext) -> None:
# Using ElevenLabs STT plugin directly for realtime mode support
@@ -37,7 +30,6 @@ async def entrypoint(ctx: JobContext) -> None:
session: AgentSession = AgentSession(
allow_interruptions=True,
vad=ctx.proc.userdata["vad"],
stt=stt,
llm=inference.LLM("openai/gpt-4.1-mini"),
tts=inference.TTS("cartesia/sonic-3"),
examples/other/kokoro_tts.py
+9 -10
View File
@@ -2,9 +2,16 @@ import logging
from dotenv import load_dotenv
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, JobProcess, cli, metrics
from livekit.agents import (
Agent,
AgentServer,
AgentSession,
JobContext,
cli,
metrics,
)
from livekit.agents.voice import MetricsCollectedEvent
from livekit.plugins import deepgram, openai, silero
from livekit.plugins import deepgram, openai
logger = logging.getLogger("kokoro-tts-agent")
@@ -27,13 +34,6 @@ class MyAgent(Agent):
server = AgentServer()
def prewarm(proc: JobProcess):
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
@server.rtc_session()
async def entrypoint(ctx: JobContext):
# each log entry will include these fields
@@ -42,7 +42,6 @@ async def entrypoint(ctx: JobContext):
"user_id": "your user_id",
}
session = AgentSession(
vad=ctx.proc.userdata["vad"],
# any combination of STT, LLM, TTS, or realtime API can be used
llm=openai.LLM(model="gpt-4.1-mini"),
stt=deepgram.STT(model="nova-3", language="multi"),
examples/other/transcription/multi-user-transcriber.py
+1 -11
View File
@@ -10,7 +10,6 @@ from livekit.agents import (
AgentSession,
AutoSubscribe,
JobContext,
JobProcess,
StopResponse,
cli,
inference,
@@ -18,7 +17,6 @@ from livekit.agents import (
room_io,
utils,
)
from livekit.plugins import silero
load_dotenv()
@@ -91,9 +89,7 @@ class MultiUserTranscriber:
if participant.identity in self._sessions:
return self._sessions[participant.identity]
session = AgentSession(
vad=self.ctx.proc.userdata["vad"],
)
session = AgentSession()
await session.start(
agent=Transcriber(
participant_identity=participant.identity,
@@ -136,11 +132,5 @@ async def entrypoint(ctx: JobContext):
ctx.add_shutdown_callback(cleanup)
def prewarm(proc: JobProcess):
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
if __name__ == "__main__":
cli.run_app(server)
examples/other/transcription/transcriber.py
+3 -2
View File
@@ -11,11 +11,12 @@ from livekit.agents import (
MetricsCollectedEvent,
StopResponse,
cli,
inference,
llm,
metrics,
room_io,
)
from livekit.plugins import openai, silero
from livekit.plugins import openai
load_dotenv()
@@ -50,7 +51,7 @@ async def entrypoint(ctx: JobContext):
session = AgentSession(
# vad is needed for non-streaming STT implementations
vad=silero.VAD.load(min_silence_duration=0.3),
vad=inference.VAD(model="silero", min_silence_duration=0.3),
)
@session.on("metrics_collected")
examples/other/transcription/translator.py
+1 -2
View File
@@ -17,7 +17,7 @@ from livekit.agents import (
room_io,
utils,
)
from livekit.plugins import openai, silero
from livekit.plugins import openai
load_dotenv()
@@ -76,7 +76,6 @@ async def entrypoint(ctx: JobContext):
session = AgentSession(
# vad is only needed for non-streaming STT implementations
vad=silero.VAD.load(),
)
@session.on("metrics_collected")
examples/primitives/echo-agent.py
+3 -2
View File
@@ -9,9 +9,9 @@ from livekit.agents import (
AutoSubscribe,
JobContext,
cli,
inference,
)
from livekit.agents.vad import VADEventType
from livekit.plugins import silero
load_dotenv()
logger = logging.getLogger("echo-agent")
@@ -35,7 +35,8 @@ async def entrypoint(ctx: JobContext):
participant=participant,
track_source=rtc.TrackSource.SOURCE_MICROPHONE,
)
vad = silero.VAD.load(
vad = inference.VAD(
model="silero",
min_speech_duration=0.2,
min_silence_duration=0.6,
)
examples/survey/agent.py
-4
View File
@@ -21,8 +21,6 @@ from livekit.agents import (
)
from livekit.agents.beta.workflows import GetEmailTask, TaskGroup
from livekit.agents.llm import function_tool
from livekit.plugins import silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
logger = logging.getLogger("SurveyAgent")
@@ -354,8 +352,6 @@ async def entrypoint(ctx: JobContext):
llm=inference.LLM("google/gemini-2.5-flash"),
stt=inference.STT("deepgram/nova-3", language="multi"),
tts=inference.TTS("inworld/inworld-tts-1"),
vad=silero.VAD.load(),
turn_detection=MultilingualModel(),
preemptive_generation=True,
)
examples/telephony/amd.py
-12
View File
@@ -11,12 +11,9 @@ from livekit.agents import (
AgentServer,
AgentSession,
JobContext,
JobProcess,
cli,
inference,
)
from livekit.plugins import silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
logger = logging.getLogger("basic-agent")
@@ -35,13 +32,6 @@ class MyAgent(Agent):
server = AgentServer()
def prewarm(proc: JobProcess):
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
@server.rtc_session()
async def entrypoint(ctx: JobContext):
ctx.log_context_fields = {
@@ -51,8 +41,6 @@ async def entrypoint(ctx: JobContext):
stt=inference.STT("deepgram/nova-3", language="multi"),
llm=inference.LLM("openai/gpt-4.1-mini"),
tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
turn_detection=MultilingualModel(),
vad=ctx.proc.userdata["vad"],
preemptive_generation=True,
)
examples/telephony/bank-ivr/ivr_navigator_agent.py
-10
View File
@@ -9,7 +9,6 @@ from livekit.agents import (
AgentServer,
AgentSession,
JobContext,
JobProcess,
MetricsCollectedEvent,
RunContext,
cli,
@@ -17,7 +16,6 @@ from livekit.agents import (
metrics,
)
from livekit.agents.llm.tool_context import function_tool
from livekit.plugins import silero
logger = logging.getLogger("phone-tree-agent")
@@ -76,13 +74,6 @@ class DtmfAgent(Agent):
context.session.shutdown(drain=True)
def prewarm(proc: JobProcess) -> None:
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
@server.rtc_session(agent_name=PHONE_TREE_AGENT_DISPATCH_NAME)
async def dtmf_session(ctx: JobContext) -> None:
await ctx.connect()
@@ -91,7 +82,6 @@ async def dtmf_session(ctx: JobContext) -> None:
}
session: AgentSession = AgentSession(
vad=ctx.proc.userdata["vad"],
llm=inference.LLM("openai/gpt-4.1"),
stt=inference.STT("deepgram/nova-3"),
tts=inference.TTS("rime/arcana"),
examples/telephony/bank-ivr/ivr_system_agent.py
-12
View File
@@ -22,7 +22,6 @@ from livekit.agents import (
AgentSession,
AgentTask,
JobContext,
JobProcess,
MetricsCollectedEvent,
cli,
inference,
@@ -30,8 +29,6 @@ from livekit.agents import (
)
from livekit.agents.beta.workflows.dtmf_inputs import GetDtmfTask
from livekit.agents.llm.tool_context import ToolError
from livekit.plugins import silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
load_dotenv()
@@ -624,13 +621,6 @@ class RewardsTask(BaseBankTask):
SubmenuTaskType = DepositAccountsTask | CreditCardsTask | LoansTask | RewardsTask
def prewarm(proc: JobProcess) -> None:
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
@server.rtc_session(agent_name=BANK_IVR_DISPATCH_NAME)
async def bank_ivr_session(ctx: JobContext) -> None:
ctx.log_context_fields = {"room": ctx.room.name}
@@ -639,11 +629,9 @@ async def bank_ivr_session(ctx: JobContext) -> None:
state = SessionState()
session: AgentSession[SessionState] = AgentSession(
vad=ctx.proc.userdata["vad"],
llm=inference.LLM("openai/gpt-4.1"),
stt=inference.STT("deepgram/nova-3"),
tts=inference.TTS("cartesia/sonic-3"),
turn_detection=MultilingualModel(),
userdata=state,
)
examples/telephony/basic_dtmf_agent.py
-4
View File
@@ -18,8 +18,6 @@ from livekit.agents.beta.workflows.dtmf_inputs import (
from livekit.agents.llm.tool_context import ToolError, function_tool
from livekit.agents.voice.events import RunContext
from livekit.agents.worker import AgentServer
from livekit.plugins import silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
logger = logging.getLogger("dtmf-agent")
@@ -136,11 +134,9 @@ async def entrypoint(ctx: JobContext) -> None:
}
session: AgentSession = AgentSession(
vad=silero.VAD.load(),
llm=inference.LLM("openai/gpt-4.1-mini"),
stt=inference.STT("deepgram/nova-3"),
tts=inference.TTS("inworld/inworld-tts-1"),
turn_detection=MultilingualModel(),
)
@session.on("metrics_collected")
examples/voice_agents/annotated_tool_args.py
-2
View File
@@ -8,7 +8,6 @@ from pydantic import Field # noqa: F401
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, cli, inference
from livekit.agents.llm import function_tool
from livekit.plugins import silero
logger = logging.getLogger("annotated-tool-args")
logger.setLevel(logging.INFO)
@@ -92,7 +91,6 @@ server = AgentServer()
@server.rtc_session()
async def entrypoint(ctx: JobContext):
agent = AgentSession(
vad=silero.VAD.load(),
stt=inference.STT("deepgram/nova-3"),
llm=inference.LLM("google/gemini-2.5-flash"),
tts=inference.TTS("rime/arcana"),
examples/voice_agents/async_tool_agent.py
-2
View File
@@ -24,7 +24,6 @@ from livekit.agents import (
inference,
llm,
)
from livekit.plugins import silero
logger = logging.getLogger("async-travel-helper")
@@ -219,7 +218,6 @@ async def entrypoint(ctx: JobContext):
llm=inference.LLM("google/gemini-3.1-flash-lite"),
tts=inference.TTS("cartesia/sonic-3", voice="e07c00bc-4134-4eae-9ea4-1a55fb45746b"),
# llm=google.realtime.RealtimeModel(),
vad=silero.VAD.load(),
turn_handling={"interruption": {"mode": "vad"}},
)
examples/voice_agents/basic_agent.py
+1 -15
View File
@@ -7,7 +7,6 @@ from livekit.agents import (
AgentServer,
AgentSession,
JobContext,
JobProcess,
MetricsCollectedEvent,
RunContext,
TurnHandlingOptions,
@@ -19,8 +18,6 @@ from livekit.agents import (
)
from livekit.agents.beta import EndCallTool
from livekit.agents.llm import function_tool
from livekit.plugins import silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
# uncomment to enable Krisp background voice/noise cancellation
# from livekit.plugins import noise_cancellation
@@ -71,13 +68,6 @@ class MyAgent(Agent):
server = AgentServer()
def prewarm(proc: JobProcess) -> None:
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
@server.rtc_session()
async def entrypoint(ctx: JobContext) -> None:
# each log entry will include these fields
@@ -94,11 +84,7 @@ async def entrypoint(ctx: JobContext) -> None:
# Text-to-speech (TTS) is your agent's voice, turning the LLM's text into speech that the user can hear
# See all available models as well as voice selections at https://docs.livekit.io/agents/models/tts/
tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
vad=ctx.proc.userdata["vad"],
turn_handling=TurnHandlingOptions(
# VAD and turn detection are used to determine when the user is speaking and when the agent should respond
# See more at https://docs.livekit.io/agents/build/turns
turn_detection=MultilingualModel(),
interruption={
# sometimes background noise could interrupt the agent session, these are considered false positive interruptions
# when it's detected, you may resume the agent's speech
@@ -114,7 +100,7 @@ async def entrypoint(ctx: JobContext) -> None:
tts_text_transforms=[
"filter_emoji",
"filter_markdown",
text_transforms.replace({"LiveKit": "<<ˈ|l|aɪ|v>> <<ˈ|k|ɪ|t>>"}),
text_transforms.replace({"LiveKit": "<<ˈ|l|aɪ|v|k|ɪ|t>>"}),
],
)
examples/voice_agents/dynamic_tool_creation.py
-2
View File
@@ -18,7 +18,6 @@ from livekit.agents import (
function_tool,
inference,
)
from livekit.plugins import silero
logger = logging.getLogger("grok-agent")
logger.setLevel(logging.INFO)
@@ -115,7 +114,6 @@ async def entrypoint(ctx: JobContext):
)
session = AgentSession(
vad=silero.VAD.load(),
stt=inference.STT("deepgram/nova-3"),
llm=inference.LLM("openai/gpt-4.1-mini"),
tts=inference.TTS("cartesia/sonic-3"),
examples/voice_agents/email_example.py
-2
View File
@@ -13,7 +13,6 @@ from livekit.agents import (
inference,
)
from livekit.agents.llm import function_tool
from livekit.plugins import silero
logger = logging.getLogger("get-email-agent")
@@ -61,7 +60,6 @@ server = AgentServer()
@server.rtc_session()
async def entrypoint(ctx: JobContext):
session = AgentSession(
vad=silero.VAD.load(),
llm=inference.LLM("openai/gpt-4.1-mini"),
stt=inference.STT("deepgram/nova-3"),
tts=inference.TTS("cartesia/sonic-3"),
examples/voice_agents/error_callback.py
-2
View File
@@ -9,7 +9,6 @@ from livekit.agents import AgentServer, JobContext, cli, inference
from livekit.agents.utils.audio import audio_frames_from_file
from livekit.agents.voice import Agent, AgentSession
from livekit.agents.voice.events import CloseEvent, ErrorEvent
from livekit.plugins import silero
from livekit.rtc import ParticipantKind
logger = logging.getLogger("my-worker")
@@ -30,7 +29,6 @@ async def entrypoint(ctx: JobContext):
stt=inference.STT("deepgram/nova-3"),
llm=inference.LLM("openai/gpt-4.1-mini"),
tts=inference.TTS("cartesia/sonic-3"),
vad=silero.VAD.load(),
)
custom_error_audio = os.path.join(pathlib.Path(__file__).parent.absolute(), "error_message.ogg")
examples/voice_agents/fast-preresponse.py
+1 -2
View File
@@ -13,7 +13,7 @@ from livekit.agents import (
llm,
)
from livekit.agents.llm.chat_context import ChatContext, ChatMessage
from livekit.plugins import deepgram, groq, openai, silero
from livekit.plugins import deepgram, groq, openai
logger = logging.getLogger("pre-reseponse-agent")
@@ -82,7 +82,6 @@ async def entrypoint(ctx: JobContext):
session = AgentSession(
stt=deepgram.STT(),
tts=openai.TTS(),
vad=silero.VAD.load(),
)
await session.start(PreResponseAgent(), room=ctx.room)
examples/voice_agents/flush_llm_node.py
-2
View File
@@ -18,7 +18,6 @@ from livekit.agents import (
llm,
metrics,
)
from livekit.plugins import silero
logger = logging.getLogger("flush-llm-node")
@@ -111,7 +110,6 @@ server = AgentServer()
@server.rtc_session()
async def entrypoint(ctx: JobContext):
session = AgentSession(
vad=silero.VAD.load(),
llm="openai/gpt-4.1-mini",
stt="deepgram/nova-3:en",
tts="cartesia/sonic-3:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
examples/voice_agents/gnani_agent.py
+3 -10
View File
@@ -15,16 +15,16 @@ from livekit.agents import (
AgentServer,
AgentSession,
JobContext,
JobProcess,
MetricsCollectedEvent,
RunContext,
TurnHandlingOptions,
cli,
inference,
metrics,
room_io,
)
from livekit.agents.llm import function_tool
from livekit.plugins import gnani, groq, silero
from livekit.plugins import gnani, groq
logger = logging.getLogger("gnani-agent")
@@ -68,13 +68,6 @@ class GnaniTestAgent(Agent):
server = AgentServer()
def prewarm(proc: JobProcess) -> None:
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
@server.rtc_session()
async def entrypoint(ctx: JobContext) -> None:
ctx.log_context_fields = {
@@ -85,7 +78,7 @@ async def entrypoint(ctx: JobContext) -> None:
stt=gnani.STT(language="en-IN"),
llm=groq.LLM(model="llama-3.1-8b-instant"),
tts=gnani.TTS(voice="Karan"),
vad=ctx.proc.userdata["vad"],
vad=inference.VAD(),
turn_handling=TurnHandlingOptions(
interruption={
"resume_false_interruption": True,
examples/voice_agents/grok/README.md
+1 -1
View File
@@ -21,7 +21,7 @@ export XAI_API_KEY=<your-xai-api-key>
### 2. Install dependencies
```bash
uv add "livekit-agents[xai,silero,turn-detector]" livekit-plugins-noise-cancellation
uv add "livekit-agents[xai,silero]" livekit-plugins-noise-cancellation
```
### 3. Run the agent
examples/voice_agents/grok/grok_voice_agent_api.py
+1 -12
View File
@@ -7,12 +7,10 @@ from livekit.agents import (
AgentServer,
AgentSession,
JobContext,
JobProcess,
cli,
room_io,
)
from livekit.plugins import silero, xai
from livekit.plugins.turn_detector.multilingual import MultilingualModel
from livekit.plugins import xai
# uncomment lines 18 and 66-68 to enable Krisp background voice/noise cancellation
# from livekit.plugins import noise_cancellation
@@ -35,13 +33,6 @@ class Assistant(Agent):
server = AgentServer()
def prewarm(proc: JobProcess):
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
@server.rtc_session()
async def my_agent(ctx: JobContext):
ctx.log_context_fields = {
@@ -50,9 +41,7 @@ async def my_agent(ctx: JobContext):
session = AgentSession(
llm=xai.realtime.RealtimeModel(voice="ara"),
turn_detection=MultilingualModel(),
tools=[xai.realtime.XSearch(), xai.realtime.WebSearch()],
vad=ctx.proc.userdata["vad"],
preemptive_generation=True,
)
examples/voice_agents/inactive_user.py
-2
View File
@@ -12,7 +12,6 @@ from livekit.agents import (
cli,
inference,
)
from livekit.plugins import silero
logger = logging.getLogger("get-email-agent")
@@ -24,7 +23,6 @@ server = AgentServer()
@server.rtc_session()
async def entrypoint(ctx: JobContext):
session = AgentSession(
vad=silero.VAD.load(),
llm=inference.LLM("openai/gpt-4.1-mini"),
stt=inference.STT("deepgram/nova-3"),
tts=inference.TTS("cartesia/sonic-3"),
examples/voice_agents/instructions_per_modality.py
-10
View File
@@ -8,13 +8,11 @@ from livekit.agents import (
AgentServer,
AgentSession,
JobContext,
JobProcess,
cli,
function_tool,
inference,
)
from livekit.agents.beta import Instructions
from livekit.plugins import silero
logger = logging.getLogger("instructions-per-modality")
@@ -82,20 +80,12 @@ class SchedulingAgent(Agent):
server = AgentServer()
def prewarm(proc: JobProcess) -> None:
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
@server.rtc_session()
async def entrypoint(ctx: JobContext) -> None:
session = AgentSession(
stt=inference.STT("deepgram/nova-3"),
llm=inference.LLM("openai/gpt-4.1-mini"),
tts=inference.TTS("cartesia/sonic-3"),
vad=ctx.proc.userdata["vad"],
)
await session.start(agent=SchedulingAgent(), room=ctx.room)
examples/voice_agents/langfuse_trace.py
+2 -4
View File
@@ -21,8 +21,7 @@ from livekit.agents.stt import FallbackAdapter as FallbackSTTAdapter
from livekit.agents.telemetry import set_tracer_provider
from livekit.agents.tts import FallbackAdapter as FallbackTTSAdapter
from livekit.agents.voice import MetricsCollectedEvent
from livekit.plugins import openai, silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
from livekit.plugins import openai
logger = logging.getLogger("langfuse-trace-example")
@@ -95,7 +94,6 @@ class Kelly(Agent):
inference.TTS("rime/arcana"),
]
),
turn_detection=MultilingualModel(),
tools=[lookup_weather],
)
@@ -149,7 +147,7 @@ async def entrypoint(ctx: JobContext):
ctx.add_shutdown_callback(flush_trace)
session = AgentSession(vad=silero.VAD.load())
session = AgentSession()
@session.on("metrics_collected")
def _on_metrics_collected(ev: MetricsCollectedEvent):
examples/voice_agents/langgraph_agent.py
+2 -14
View File
@@ -12,12 +12,10 @@ from livekit.agents import (
AgentServer,
AgentSession,
JobContext,
JobProcess,
cli,
inference,
)
from livekit.plugins import langchain, silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
from livekit.plugins import langchain
logger = logging.getLogger("basic-agent")
@@ -30,18 +28,11 @@ load_dotenv()
# In order to run this example, you need the following dependencies
# - langchain[openai]
# - langgraph
# - livekit-agents[openai,silero,langchain,deepgram,turn_detector]
# - livekit-agents[openai,silero,langchain,deepgram]
server = AgentServer()
def prewarm(proc: JobProcess):
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
class State(TypedDict):
messages: Annotated[list[BaseMessage], add_messages]
@@ -71,12 +62,9 @@ async def entrypoint(ctx: JobContext):
)
session = AgentSession(
vad=ctx.proc.userdata["vad"],
# any combination of STT, LLM, TTS, or realtime API can be used
stt=inference.STT("deepgram/nova-3", language="multi"),
tts=inference.TTS("cartesia/sonic-3"),
# use LiveKit's turn detection model
turn_detection=MultilingualModel(),
)
await session.start(
examples/voice_agents/llamaindex-rag/chat_engine.py
-2
View File
@@ -22,7 +22,6 @@ from livekit.agents import (
llm,
)
from livekit.agents.voice.agent import ModelSettings
from livekit.plugins import silero
load_dotenv()
@@ -54,7 +53,6 @@ class ChatEngineAgent(Agent):
"with users will be voice. You should use short and concise "
"responses, and avoiding usage of unpronouncable punctuation."
),
vad=silero.VAD.load(),
stt=inference.STT("deepgram/nova-3"),
llm=DummyLLM(), # use a dummy LLM to enable the pipeline reply
tts=inference.TTS("cartesia/sonic-3"),
examples/voice_agents/llamaindex-rag/query_engine.py
-2
View File
@@ -18,7 +18,6 @@ from livekit.agents import (
inference,
llm,
)
from livekit.plugins import silero
load_dotenv()
@@ -59,7 +58,6 @@ async def entrypoint(ctx: JobContext):
"with users will be voice. You should use short and concise "
"responses, and avoiding usage of unpronouncable punctuation."
),
vad=silero.VAD.load(),
stt=inference.STT("deepgram/nova-3"),
llm=inference.LLM("openai/gpt-4.1-mini"),
tts=inference.TTS("cartesia/sonic-3"),
examples/voice_agents/llamaindex-rag/retrieval.py
+1 -2
View File
@@ -19,7 +19,7 @@ from livekit.agents import (
llm,
)
from livekit.agents.voice.agent import ModelSettings
from livekit.plugins import deepgram, openai, silero
from livekit.plugins import deepgram, openai
load_dotenv()
@@ -46,7 +46,6 @@ class RetrievalAgent(Agent):
"with users will be voice. You should use short and concise "
"responses, and avoiding usage of unpronouncable punctuation."
),
vad=silero.VAD.load(),
stt=deepgram.STT(),
llm=openai.LLM(),
tts=openai.TTS(),
examples/voice_agents/long_running_function.py
-2
View File
@@ -13,7 +13,6 @@ from livekit.agents import (
inference,
)
from livekit.agents.llm import function_tool
from livekit.plugins import silero
logger = logging.getLogger("long-running-function")
logger.setLevel(logging.INFO)
@@ -69,7 +68,6 @@ async def entrypoint(ctx: JobContext):
stt=inference.STT("deepgram/nova-3"),
llm=inference.LLM("openai/gpt-4.1-mini"),
tts=inference.TTS("cartesia/sonic-3"),
vad=silero.VAD.load(),
)
await session.start(agent=MyAgent(), room=ctx.room)
examples/voice_agents/mcp/mcp-agent.py
-4
View File
@@ -3,8 +3,6 @@ import logging
from dotenv import load_dotenv
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, cli, inference, mcp
from livekit.plugins import silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
logger = logging.getLogger("mcp-agent")
@@ -32,11 +30,9 @@ server = AgentServer()
@server.rtc_session()
async def entrypoint(ctx: JobContext):
session = AgentSession(
vad=silero.VAD.load(),
stt=inference.STT("deepgram/nova-3", language="multi"),
llm=inference.LLM("openai/gpt-4.1-mini"),
tts=inference.TTS("cartesia/sonic-3"),
turn_detection=MultilingualModel(),
tools=[
mcp.MCPToolset(
id="mcp_toolset_1", mcp_server=mcp.MCPServerHTTP(url="http://localhost:8000/sse")
examples/voice_agents/multi_agent.py
+1 -10
View File
@@ -10,7 +10,6 @@ from livekit.agents import (
AgentSession,
ChatContext,
JobContext,
JobProcess,
RunContext,
cli,
metrics,
@@ -18,7 +17,7 @@ from livekit.agents import (
from livekit.agents.job import get_job_context
from livekit.agents.llm import function_tool
from livekit.agents.voice import MetricsCollectedEvent
from livekit.plugins import deepgram, openai, silero
from livekit.plugins import deepgram, openai
# uncomment to enable Krisp BVC noise cancellation, currently supported on Linux and MacOS
# from livekit.plugins import noise_cancellation
@@ -132,17 +131,9 @@ class StoryAgent(Agent):
server = AgentServer()
def prewarm(proc: JobProcess):
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
@server.rtc_session()
async def entrypoint(ctx: JobContext):
session = AgentSession[StoryData](
vad=ctx.proc.userdata["vad"],
# any combination of STT, LLM, TTS, or realtime API can be used
llm=openai.LLM(model="gpt-4.1-mini"),
stt=deepgram.STT(model="nova-3"),
examples/voice_agents/nvidia_test.py
+2 -10
View File
@@ -6,25 +6,18 @@ from livekit.agents import (
Agent,
AgentSession,
JobContext,
JobProcess,
WorkerOptions,
cli,
)
from livekit.plugins import nvidia, openai, silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
from livekit.plugins import nvidia, openai
logger = logging.getLogger("basic-agent")
load_dotenv()
def prewarm(proc: JobProcess):
proc.userdata["vad"] = silero.VAD.load()
async def entrypoint(ctx: JobContext):
session = AgentSession(
vad=ctx.proc.userdata["vad"],
llm=openai.LLM(model="gpt-4.1-mini"),
stt=nvidia.STT(),
tts=nvidia.TTS(),
@@ -32,7 +25,6 @@ async def entrypoint(ctx: JobContext):
resume_false_interruption=True,
false_interruption_timeout=1.0,
min_interruption_duration=0.2,
turn_detection=MultilingualModel(),
)
await session.start(
@@ -42,4 +34,4 @@ async def entrypoint(ctx: JobContext):
if __name__ == "__main__":
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm))
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
examples/voice_agents/raw_function_description.py
+2 -2
View File
@@ -11,7 +11,7 @@ from livekit.agents import (
cli,
function_tool,
)
from livekit.plugins import openai, silero # noqa: F401
from livekit.plugins import openai # noqa: F401
# This demo defines an agent using a raw function tool to open predefined gates via enum input.
# When using raw function tools, compatibility across LLM providers is not guaranteed,
@@ -72,7 +72,7 @@ async def entrypoint(ctx: JobContext):
# stt=openai.STT(),
# llm=openai.LLM(),
# tts=openai.TTS(),
# vad=silero.VAD.load(),
# vad=inference.VAD(model="silero"),
llm=openai.realtime.RealtimeModel()
)
await session.start(RawFunctionAgent(), room=ctx.room)
examples/voice_agents/realtime_joke_teller.py
+1 -2
View File
@@ -56,7 +56,7 @@ from livekit.agents import (
room_io,
)
from livekit.agents.llm import function_tool
from livekit.plugins import aws, silero
from livekit.plugins import aws
load_dotenv()
@@ -221,7 +221,6 @@ async def entrypoint(ctx: agents.JobContext):
stt=aws.STT(),
llm=aws.LLM(),
tts=aws.TTS(),
vad=silero.VAD.load(),
)
else:
print("⚡ Using REALTIME mode: Nova Sonic 2.0")
examples/voice_agents/realtime_turn_detector.py
+4 -11
View File
@@ -3,9 +3,9 @@ import logging
from dotenv import load_dotenv
from google.genai import types # noqa: F401
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, JobProcess, cli
from livekit.plugins import deepgram, google, openai, silero # noqa: F401
from livekit.plugins.turn_detector.multilingual import MultilingualModel
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, cli
from livekit.agents.inference import TurnDetector
from livekit.plugins import deepgram, google, openai # noqa: F401
logger = logging.getLogger("realtime-turn-detector")
logger.setLevel(logging.INFO)
@@ -24,8 +24,7 @@ server = AgentServer()
async def entrypoint(ctx: JobContext):
session = AgentSession(
allow_interruptions=True,
turn_detection=MultilingualModel(),
vad=ctx.proc.userdata["vad"],
turn_detection=TurnDetector(),
stt=deepgram.STT(),
# To use OpenAI Realtime API
llm=openai.realtime.RealtimeModel(
@@ -48,11 +47,5 @@ async def entrypoint(ctx: JobContext):
await session.start(agent=Agent(instructions="You are a helpful assistant."), room=ctx.room)
def prewarm(proc: JobProcess):
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
if __name__ == "__main__":
cli.run_app(server)
examples/voice_agents/realtime_video_agent.py
+1 -2
View File
@@ -11,7 +11,7 @@ from livekit.agents import (
room_io,
voice, # noqa: F401
)
from livekit.plugins import google, silero
from livekit.plugins import google
logger = logging.getLogger("realtime-video-agent")
@@ -23,7 +23,6 @@ server = AgentServer()
@server.rtc_session()
async def entrypoint(ctx: JobContext):
session = AgentSession(
vad=silero.VAD.load(),
# both Gemini and OpenAI Realtime API support streaming video input
llm=google.realtime.RealtimeModel(),
# customize how video frames are sampled
examples/voice_agents/restaurant_agent.py
-4
View File
@@ -8,9 +8,6 @@ from pydantic import Field
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, RunContext, cli, inference
from livekit.agents.llm import function_tool
from livekit.plugins import silero
# from livekit.plugins import noise_cancellation
# This example demonstrates a multi-agent system where tasks are delegated to sub-agents
# based on the user's request.
@@ -330,7 +327,6 @@ async def entrypoint(ctx: JobContext):
stt=inference.STT(model="deepgram/nova-3"),
llm=inference.LLM(model="openai/gpt-4.1-mini"),
tts=inference.TTS(model="cartesia/sonic-3"),
vad=silero.VAD.load(),
max_tool_steps=5,
# to use realtime model, replace the stt, llm, tts and vad with the following
# llm=openai.realtime.RealtimeModel(voice="alloy"),
examples/voice_agents/resume_interrupted_agent.py
-2
View File
@@ -3,7 +3,6 @@ import logging
from dotenv import load_dotenv
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, cli, inference
from livekit.plugins import silero
logger = logging.getLogger("resume-agent")
@@ -21,7 +20,6 @@ server = AgentServer()
@server.rtc_session()
async def entrypoint(ctx: JobContext):
session = AgentSession(
vad=silero.VAD.load(),
llm=inference.LLM("openai/gpt-4.1-mini"),
stt=inference.STT("deepgram/nova-3"),
tts=inference.TTS("cartesia/sonic-3"),
examples/voice_agents/session_close_callback.py
+2 -4
View File
@@ -4,7 +4,7 @@ from dotenv import load_dotenv
from livekit.agents import Agent, AgentServer, AgentSession, CloseEvent, JobContext, cli
from livekit.agents.beta.tools import EndCallTool
from livekit.plugins import google, silero # noqa: F401
from livekit.plugins import google # noqa: F401
logger = logging.getLogger("my-worker")
logger.setLevel(logging.INFO)
@@ -42,9 +42,7 @@ server = AgentServer()
@server.rtc_session()
async def entrypoint(ctx: JobContext):
session = AgentSession(
vad=silero.VAD.load(),
)
session = AgentSession()
await session.start(agent=MyAgent(), room=ctx.room)
examples/voice_agents/silent_function_call.py
-2
View File
@@ -12,7 +12,6 @@ from livekit.agents import (
inference,
)
from livekit.agents.llm import function_tool
from livekit.plugins import silero
logger = logging.getLogger("silent-function-call")
logger.setLevel(logging.INFO)
@@ -60,7 +59,6 @@ async def entrypoint(ctx: JobContext):
stt=inference.STT("deepgram/nova-3"),
llm=inference.LLM("openai/gpt-4.1-mini"),
tts=inference.TTS("cartesia/sonic-3"),
vad=silero.VAD.load(),
)
@session.on("function_tools_executed")
examples/voice_agents/speaker_id_multi_speaker.py
+1 -2
View File
@@ -8,7 +8,7 @@ from dotenv import load_dotenv
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, cli
from livekit.agents.stt import MultiSpeakerAdapter
from livekit.plugins import deepgram, openai, silero, speechmatics # noqa: F401
from livekit.plugins import deepgram, openai, speechmatics # noqa: F401
# Load environment variables from .env file
# Required: SPEECHMATICS_API_KEY, OPENAI_API_KEY
@@ -54,7 +54,6 @@ server = AgentServer()
@server.rtc_session()
async def entrypoint(ctx: JobContext) -> None:
session = AgentSession(
vad=silero.VAD.load(),
llm=openai.LLM(),
tts=openai.TTS(),
stt=MultiSpeakerAdapter(
examples/voice_agents/speedup_output_audio.py
-4
View File
@@ -16,7 +16,6 @@ from livekit.agents import (
inference,
utils,
)
from livekit.plugins import silero
try:
import librosa
@@ -92,8 +91,6 @@ server = AgentServer()
def prewarm(proc: JobProcess):
proc.userdata["vad"] = silero.VAD.load()
# warmup the librosa JIT
librosa.effects.time_stretch(np.random.randn(16000).astype(np.float32), rate=1.2)
@@ -109,7 +106,6 @@ async def entrypoint(ctx: JobContext):
"user_id": "your user_id",
}
session = AgentSession(
vad=ctx.proc.userdata["vad"],
llm=inference.LLM("openai/gpt-4.1-mini"),
stt=inference.STT("deepgram/nova-3"),
tts=inference.TTS("cartesia/sonic-3"),
examples/voice_agents/structured_output.py
+2 -6
View File
@@ -18,8 +18,7 @@ from livekit.agents import (
ModelSettings,
cli,
)
from livekit.plugins import openai, silero
from livekit.plugins.turn_detector.english import EnglishModel
from livekit.plugins import openai
logger = logging.getLogger("structured-output")
load_dotenv()
@@ -128,10 +127,7 @@ server = AgentServer()
@server.rtc_session()
async def entrypoint(ctx: JobContext):
session = AgentSession(
vad=silero.VAD.load(),
turn_detection=EnglishModel(),
)
session = AgentSession()
await session.start(agent=MyAgent(), room=ctx.room)
examples/voice_agents/timed_agent_transcript.py
+1 -2
View File
@@ -7,7 +7,7 @@ from dotenv import load_dotenv
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, cli, inference, room_io
from livekit.agents.types import TimedString
from livekit.agents.voice.agent import ModelSettings
from livekit.plugins import cartesia, silero
from livekit.plugins import cartesia
logger = logging.getLogger("my-worker")
logger.setLevel(logging.INFO)
@@ -44,7 +44,6 @@ async def entrypoint(ctx: JobContext):
stt=inference.STT("deepgram/nova-3"),
llm=inference.LLM("google/gemini-2.5-flash"),
tts=cartesia.TTS(),
vad=silero.VAD.load(),
# enable TTS-aligned transcript, can be configured at the Agent level as well
use_tts_aligned_transcript=True,
)
examples/voice_agents/tool_search_agent.py
-2
View File
@@ -31,7 +31,6 @@ from livekit.agents import (
)
from livekit.agents.beta.toolsets import ToolProxyToolset, ToolSearchToolset
from livekit.agents.metrics.base import LLMMetrics
from livekit.plugins import silero
logger = logging.getLogger("tool-search-example")
logger.setLevel(logging.INFO)
@@ -172,7 +171,6 @@ async def entrypoint(ctx: JobContext):
llm=inference.LLM("openai/gpt-4.1-mini"),
tts=inference.TTS("cartesia/sonic-3"),
stt=inference.STT("deepgram/nova-3"),
vad=silero.VAD.load(),
)
# Track token usage to observe prompt caching behavior.
examples/voice_agents/ultravox_realtime_api.py
-2
View File
@@ -11,7 +11,6 @@ from livekit.agents import (
function_tool,
room_io,
)
from livekit.plugins import silero
from livekit.plugins.ultravox.realtime import RealtimeModel
logger = logging.getLogger("ultravox-agent")
@@ -28,7 +27,6 @@ class MyAgent(Agent):
voice="Jessica",
language_hint="en",
),
vad=silero.VAD.load(),
)
async def on_enter(self):
examples/voice_agents/zapier_mcp_integration.py
-13
View File
@@ -9,14 +9,11 @@ from livekit.agents import (
AgentSession,
AutoSubscribe,
JobContext,
JobProcess,
cli,
inference,
mcp,
metrics,
)
from livekit.plugins import silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
load_dotenv(dotenv_path=".env.local")
logger = logging.getLogger("voice-agent")
@@ -32,8 +29,6 @@ class Assistant(Agent):
stt=inference.STT("deepgram/nova-3"),
llm=inference.LLM("google/gemini-2.5-flash"),
tts=inference.TTS("rime/arcana"),
# use LiveKit's transformer-based turn detector
turn_detection=MultilingualModel(),
)
async def on_enter(self):
@@ -45,13 +40,6 @@ class Assistant(Agent):
server = AgentServer()
def prewarm(proc: JobProcess):
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
@server.rtc_session()
async def entrypoint(ctx: JobContext):
logger.info(f"connecting to room {ctx.room.name}")
@@ -75,7 +63,6 @@ async def entrypoint(ctx: JobContext):
logger.warning("ZAPIER_MCP_SERVER environment variable not set. MCP integration disabled.")
session = AgentSession(
vad=ctx.proc.userdata["vad"],
# minimum delay for endpointing, used when turn detector believes the user is done with their turn # noqa: E501
min_endpointing_delay=0.5,
# maximum delay for endpointing, used when turn detector does not believe the user is done with their turn # noqa: E501
examples/warm-transfer/warm_transfer.py
+1 -4
View File
@@ -13,8 +13,7 @@ from livekit.agents import (
)
from livekit.agents.beta.workflows import WarmTransferTask
from livekit.agents.llm import ToolError, function_tool
from livekit.plugins import noise_cancellation, silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
from livekit.plugins import noise_cancellation
logger = logging.getLogger("warm-transfer")
@@ -96,11 +95,9 @@ server = AgentServer()
@server.rtc_session(agent_name="sip-inbound")
async def entrypoint(ctx: JobContext):
session = AgentSession(
vad=silero.VAD.load(),
llm="openai/gpt-4.1-mini",
stt="deepgram/nova-3:en",
tts="cartesia/sonic-3:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
turn_detection=MultilingualModel(),
)
support_agent = SupportAgent()
livekit-agents/livekit/agents/inference/__init__.py
+7
View File
@@ -1,3 +1,4 @@
from .eot import TurnDetector, TurnDetectorModels, TurnDetectorVersions
from .interruption import (
AdaptiveInterruptionDetector,
InterruptionDataFrameType,
@@ -7,17 +8,23 @@ from .interruption import (
from .llm import LLM, LLMModels, LLMStream
from .stt import STT, STTModels
from .tts import TTS, TTSModels
from .vad import VAD, VADModels
__all__ = [
"STT",
"TTS",
"LLM",
"VAD",
"LLMStream",
"STTModels",
"TTSModels",
"LLMModels",
"VADModels",
"AdaptiveInterruptionDetector",
"InterruptionDetectionError",
"OverlappingSpeechEvent",
"InterruptionDataFrameType",
"TurnDetector",
"TurnDetectorModels",
"TurnDetectorVersions",
]
livekit-agents/livekit/agents/inference/_warmup.py
+10
View File
@@ -0,0 +1,10 @@
"""Side-effect module: imported in the forkserver preload list so the
native ``livekit-local-inference`` model singletons are loaded inside the
forkserver process. Child job processes then inherit the resident weight
pages via COW.
"""
import livekit.local_inference as _li
_li.init_vad()
_li.init_eot()
livekit-agents/livekit/agents/inference/eot/__init__.py
+4
View File
@@ -0,0 +1,4 @@
from .detector import TurnDetector
from .languages import TurnDetectorModels, TurnDetectorVersions
__all__ = ["TurnDetector", "TurnDetectorModels", "TurnDetectorVersions"]
livekit-agents/livekit/agents/inference/eot/base.py
+389
View File
@@ -0,0 +1,389 @@
"""Audio EOT detector base, the per-window inference stream (with built-in
cloud→local fallback), and the transport Protocol concrete backends satisfy.
Lives next to its transports rather than in ``voice/turn`` so the
fallback logic is a peer of the transports it switches between rather than a
template-method-across-packages.
"""
from __future__ import annotations
import asyncio
import time
from dataclasses import dataclass
from typing import Literal, Protocol, runtime_checkable
from livekit import rtc
from ... import utils
from ..._exceptions import APITimeoutError
from ...language import LanguageCode
from ...log import logger
from ...types import (
DEFAULT_API_CONNECT_OPTIONS,
APIConnectOptions,
)
from ...utils import aio
from ...voice.turn import TurnDetectionEvent
from .languages import ThresholdOptions, TurnDetectorModels
DEFAULT_SAMPLE_RATE: int = 16000
MIN_SILENCE_DURATION_MS = 200
"""Minimum VAD silence the audio EOT detector needs before it sends
an inference request. Enforced against the caller-supplied VAD's
``min_silence_duration`` in ``AudioRecognition``."""
@dataclass
class TurnDetectorOptions:
sample_rate: int
thresholds: ThresholdOptions
@runtime_checkable
class _StreamingTurnDetectionTransport(Protocol):
async def run(self) -> None: ...
def run_inference(self, request_id: str) -> None: ...
def push_frame(self, frame: rtc.AudioFrame) -> None: ...
def flush(self) -> None: ...
def attach(self, stream: _BaseStreamingTurnDetectorStream) -> None: ...
def detach(self) -> None: ...
class _BaseStreamingTurnDetector(rtc.EventEmitter[Literal["metrics_collected"]]):
def __init__(self, *, opts: TurnDetectorOptions) -> None:
super().__init__()
self._opts = opts
@property
def model(self) -> TurnDetectorModels:
raise NotImplementedError
@property
def provider(self) -> str:
return "livekit"
def stream(
self,
*,
conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS,
) -> _BaseStreamingTurnDetectorStream:
raise NotImplementedError
async def unlikely_threshold(self, language: LanguageCode | None) -> float | None:
return self._opts.thresholds.lookup(language)
async def backchannel_threshold(self, language: LanguageCode | None) -> float | None:
return self._opts.thresholds.lookup_backchannel(language)
async def supports_language(self, language: LanguageCode | None) -> bool:
return self._opts.thresholds.supports(language)
class _BaseStreamingTurnDetectorStream:
@dataclass
class _FlushSentinel:
reason: str | None = None
def __init__(
self,
*,
detector: _BaseStreamingTurnDetector,
opts: TurnDetectorOptions,
transport: _StreamingTurnDetectionTransport,
model: TurnDetectorModels = "turn-detector-v1-mini",
) -> None:
self._detector = detector
self._opts = opts
self._transport = transport
self._transport.attach(self)
self._model: TurnDetectorModels = model
self._is_fallback = False
self._warned_cloud_failure = False
self._warned_local_failure = False
self._transport_task: asyncio.Task[None] | None = None
self._fallback_requested = False
self._audio_input_sample_rate: int | None = None
self._audio_input_num_channels: int | None = None
self._audio_resampler: rtc.AudioResampler | None = None
self._audio_ch = aio.Chan[
rtc.AudioFrame | _BaseStreamingTurnDetectorStream._FlushSentinel
]()
self._request_id: str | None = None
self._request_fut: asyncio.Future[TurnDetectionEvent] | None = None
self._tasks: set[asyncio.Task[None]] = set()
self._task = asyncio.create_task(self._main_task())
# region: detector proxies
@property
def model(self) -> TurnDetectorModels:
# The stream owns its active model, so after a fallback this reports
# "turn-detector-v1-mini". The detector and stream share one mutable
# ``ThresholdOptions``, and the cloud→local fallback it performs is
# one-way and sticky: once degraded it never returns to cloud, so the
# detector view stays consistent for the rest of its lifetime.
return self._model
@property
def provider(self) -> str:
return self._detector.provider
@property
def is_fallback(self) -> bool:
return self._is_fallback
async def unlikely_threshold(self, language: LanguageCode | None) -> float | None:
return self._opts.thresholds.lookup(language)
async def backchannel_threshold(self, language: LanguageCode | None) -> float | None:
return self._opts.thresholds.lookup_backchannel(language)
async def supports_language(self, language: LanguageCode | None) -> bool:
return self._opts.thresholds.supports(language)
# endregion
# region: inference requests
def predict(self) -> asyncio.Future[TurnDetectionEvent]:
"""Start a new inference request and return its future."""
if self._audio_ch.closed:
fut: asyncio.Future[TurnDetectionEvent] = asyncio.get_running_loop().create_future()
fut.set_result(self._default_event(1.0))
return fut
self.cancel_inference() # supersede any previous request
self._request_id = utils.shortuuid("turn_request_")
self._request_fut = asyncio.get_running_loop().create_future()
self._transport.run_inference(self._request_id)
return self._request_fut
def cancel_inference(self, *, timed_out: bool = False) -> None:
"""Close the current inference request (new speech, turn boundary,
prediction timeout, mode change) and fall back if needed.
"""
if self._request_id is not None:
fut = self._request_fut
self._request_id = None
self._request_fut = None
if fut is not None and not fut.done():
fut.set_result(self._default_event(0.0))
# trigger fallback immediately
if timed_out and self._model == "turn-detector-v1":
self._fall_back_to_local(reason=APITimeoutError("eot prediction timed out"))
def flush(self, reason: str | None = None) -> None:
# Idempotent: a second call sends another sentinel that transports
# treat as a no-op (cloud: redundant session_flush; local: empty trim).
if self._audio_ch.closed:
return
for resampled_frame in self._flush_audio_resampler():
self._audio_ch.send_nowait(resampled_frame)
self._audio_ch.send_nowait(_BaseStreamingTurnDetectorStream._FlushSentinel(reason=reason))
self.cancel_inference()
@staticmethod
def _default_event(probability: float) -> TurnDetectionEvent:
return TurnDetectionEvent(
type="eot_prediction",
last_speaking_time=time.time(),
end_of_turn_probability=probability,
)
# endregion
# region: audio ingress
def push_audio(self, frame: rtc.AudioFrame) -> None:
if self._audio_ch.closed:
return
for resampled_frame in self._resample_audio_frame(frame):
self._audio_ch.send_nowait(resampled_frame)
def end_input(self) -> None:
self.flush()
self._audio_ch.close()
def _resample_audio_frame(self, frame: rtc.AudioFrame) -> list[rtc.AudioFrame]:
if self._audio_input_sample_rate is None or self._audio_input_num_channels is None:
self._audio_input_sample_rate = frame.sample_rate
self._audio_input_num_channels = frame.num_channels
if self._audio_input_sample_rate != self._opts.sample_rate:
self._audio_resampler = rtc.AudioResampler(
input_rate=self._audio_input_sample_rate,
output_rate=self._opts.sample_rate,
num_channels=self._audio_input_num_channels,
quality=rtc.AudioResamplerQuality.QUICK,
)
elif (
frame.sample_rate != self._audio_input_sample_rate
or frame.num_channels != self._audio_input_num_channels
):
logger.error(
"a frame with different audio format was already pushed",
extra={
"sample_rate": frame.sample_rate,
"expected_sample_rate": self._audio_input_sample_rate,
"num_channels": frame.num_channels,
"expected_num_channels": self._audio_input_num_channels,
},
)
return []
if self._audio_resampler is None:
return [frame]
return self._audio_resampler.push(frame)
def _flush_audio_resampler(self) -> list[rtc.AudioFrame]:
frames = self._audio_resampler.flush() if self._audio_resampler is not None else []
self._reset_audio_resampler()
return frames
def _reset_audio_resampler(self) -> None:
self._audio_resampler = None
self._audio_input_sample_rate = None
self._audio_input_num_channels = None
# endregion
# region: results
def _resolve_prediction(
self,
request_id: str,
probability: float,
*,
inference_duration: float | None = None,
detection_delay: float | None = None,
backchannel_probability: float | None = None,
) -> None:
"""Accept a prediction from a transport. Stale response is ignored."""
if request_id != self._request_id:
return
fut = self._request_fut
self._request_id = None
self._request_fut = None
if fut is not None and not fut.done():
fut.set_result(
TurnDetectionEvent(
type="eot_prediction",
last_speaking_time=time.time(),
end_of_turn_probability=probability,
detection_delay=detection_delay,
inference_duration=inference_duration,
backchannel_probability=backchannel_probability,
)
)
# endregion
# region: teardown
async def aclose(self) -> None:
self._transport.detach()
self.end_input() # the flush inside closes the in-flight request
await aio.cancel_and_wait(self._task)
await aio.cancel_and_wait(*self._tasks)
self.cancel_inference() # defensive, normally a no-op
# endregion
# region: main task + fallback
async def _main_task(self) -> None:
await self._run()
async def _drain_audio_channel(self) -> None:
async for item in self._audio_ch:
if isinstance(item, _BaseStreamingTurnDetectorStream._FlushSentinel):
self._transport.flush()
else:
self._transport.push_frame(item)
async def _run(self) -> None:
"""Run the active transport, retrying on cloud failure by swapping in
a local transport in-place. ``turn-detector-v1-mini`` just runs the
transport once and surfaces failures to the caller via
``_resolve_prediction`` (default 1.0)."""
while True:
task = asyncio.create_task(self._transport.run())
self._transport_task = task
try:
await task
return
except asyncio.CancelledError:
# _fall_back_to_local sets _fallback_requested before cancelling
# this child task; any other cancellation (e.g. aclose cancelling
# the parent) leaves the flag unset and propagates.
if self._fallback_requested:
self._fallback_requested = False
continue
if not task.done():
await aio.cancel_and_wait(task)
raise
except Exception as e: # noqa: BLE001 — any cloud error degrades to local
if self._model == "turn-detector-v1":
self._fall_back_to_local(reason=e)
continue
self._on_local_failure(reason=e)
return
def _fall_back_to_local(self, *, reason: BaseException) -> None:
# Lazy import: transports.py imports this module for the Protocol and
# constants, so importing it at module load would cycle.
from .transports import _LocalTransport
if not self._warned_cloud_failure:
logger.warning(
"cloud turn detector failed (%s); falling back to local mini model",
reason,
)
self._warned_cloud_failure = True
self._emit_default_for_inflight()
self._transport.detach()
self._opts.thresholds._to_local_fallback()
from .detector import TurnDetector
if isinstance(self._detector, TurnDetector):
self._detector._model = "turn-detector-v1-mini"
self._transport = _LocalTransport(opts=self._opts)
self._transport.attach(self)
self._model = "turn-detector-v1-mini"
self._is_fallback = True
# If transport.run() is still in flight (e.g. predict timeout while
# the cloud session was otherwise idle), signal+cancel so _run loops
# onto the local transport. Without this the orphaned WS lingers until
# the gateway closes it for inactivity and the ensuing error surfaces
# as a misleading log.
task = self._transport_task
if task is not None and not task.done():
self._fallback_requested = True
task.cancel()
def _on_local_failure(self, *, reason: BaseException) -> None:
if not self._warned_local_failure:
logger.warning(
"local audio turn detector failed (%s); defaulting to 1.0 and retrying on next turn",
reason,
)
self._warned_local_failure = True
self._emit_default_for_inflight()
def _emit_default_for_inflight(self) -> None:
# Positive default so any waiter commits after min_endpointing_delay.
request_id = self._request_id
if request_id is not None:
self._resolve_prediction(request_id, 1.0)
# endregion
livekit-agents/livekit/agents/inference/eot/detector.py
+175
View File
@@ -0,0 +1,175 @@
"""Audio end-of-turn detector with cloud → local fallback."""
from __future__ import annotations
from dataclasses import replace
import aiohttp
from ... import utils
from ...language import LanguageCode
from ...log import logger
from ...types import (
DEFAULT_API_CONNECT_OPTIONS,
NOT_GIVEN,
APIConnectOptions,
NotGivenOr,
)
from ...utils import is_given
from .._utils import get_default_inference_url
from .base import (
DEFAULT_SAMPLE_RATE,
TurnDetectorOptions,
_BaseStreamingTurnDetector,
_BaseStreamingTurnDetectorStream,
_StreamingTurnDetectionTransport,
)
from .languages import ThresholdOptions, TurnDetectorModels, TurnDetectorVersions
from .transports import _CloudTransport, _CloudTransportOptions, _LocalTransport
__all__ = ["TurnDetector"]
class TurnDetector(_BaseStreamingTurnDetector):
def __init__(
self,
*,
version: NotGivenOr[TurnDetectorVersions] = NOT_GIVEN,
unlikely_threshold: NotGivenOr[float | dict[LanguageCode | str, float]] = NOT_GIVEN,
backchannel_threshold: NotGivenOr[float | dict[LanguageCode | str, float]] = NOT_GIVEN,
base_url: NotGivenOr[str] = NOT_GIVEN,
api_key: NotGivenOr[str] = NOT_GIVEN,
api_secret: NotGivenOr[str] = NOT_GIVEN,
sample_rate: int = DEFAULT_SAMPLE_RATE,
http_session: aiohttp.ClientSession | None = None,
conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS,
) -> None:
auto = not is_given(version)
resolved_version: TurnDetectorVersions = (
version
if is_given(version)
else ("v1" if (utils.is_hosted() or utils.is_dev_mode()) else "v1-mini")
)
resolved_model: TurnDetectorModels = (
"turn-detector-v1" if resolved_version == "v1" else "turn-detector-v1-mini"
)
cloud_opts: _CloudTransportOptions | None = None
if resolved_version == "v1":
lk_base_url = utils.resolve_env_var(
base_url,
"LIVEKIT_INFERENCE_URL",
default=get_default_inference_url(),
)
lk_api_key = utils.resolve_env_var(
api_key, "LIVEKIT_INFERENCE_API_KEY", "LIVEKIT_API_KEY", default=""
)
lk_api_secret = utils.resolve_env_var(
api_secret,
"LIVEKIT_INFERENCE_API_SECRET",
"LIVEKIT_API_SECRET",
default="",
)
missing: list[str] = []
if not lk_base_url:
missing.append("LIVEKIT_INFERENCE_URL")
if not lk_api_key:
missing.append("LIVEKIT_API_KEY")
if not lk_api_secret:
missing.append("LIVEKIT_API_SECRET")
if missing:
if auto:
logger.warning(
"LIVEKIT_INFERENCE_URL is set but %s missing; "
"falling back to the turn-detector-v1-mini model",
", ".join(missing),
)
resolved_model = "turn-detector-v1-mini"
else:
raise ValueError(
f"TurnDetector(version='v1') requires "
f"{', '.join(missing)} (env or constructor argument)."
)
else:
cloud_opts = _CloudTransportOptions(
base_url=lk_base_url,
api_key=lk_api_key,
api_secret=lk_api_secret,
conn_options=conn_options,
)
opts = TurnDetectorOptions(
sample_rate=sample_rate,
thresholds=ThresholdOptions(resolved_model, unlikely_threshold, backchannel_threshold),
)
super().__init__(opts=opts)
self._model: TurnDetectorModels = resolved_model
self._cloud_opts = cloud_opts
self._http_session = http_session
self._warn_threshold_override()
@property
def model(self) -> TurnDetectorModels:
return self._model
def _warn_threshold_override(self) -> None:
thresholds = self._opts.thresholds
if is_given(overrides := thresholds.overrides):
logger.warning(
"a non-default turn detection threshold was provided "
"(unlikely_threshold=%s); the server provides calibrated defaults and "
"overriding them may be suboptimal",
overrides,
)
if is_given(bc_overrides := thresholds.backchannel_overrides):
logger.warning(
"a non-default backchannel threshold was provided "
"(backchannel_threshold=%s); the server provides calibrated defaults and "
"overriding them may be suboptimal",
bc_overrides,
)
def update_options(
self,
*,
unlikely_threshold: NotGivenOr[float | dict[LanguageCode | str, float]] = NOT_GIVEN,
backchannel_threshold: NotGivenOr[float | dict[LanguageCode | str, float]] = NOT_GIVEN,
) -> None:
if is_given(unlikely_threshold):
self._opts.thresholds.update_overrides(unlikely_threshold)
if is_given(backchannel_threshold):
self._opts.thresholds.update_backchannel_overrides(backchannel_threshold)
self._warn_threshold_override()
def stream(
self,
*,
conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS,
) -> _BaseStreamingTurnDetectorStream:
cloud_opts = (
replace(self._cloud_opts, conn_options=conn_options)
if self._cloud_opts is not None
else None
)
transport: _StreamingTurnDetectionTransport
if self._model == "turn-detector-v1":
assert cloud_opts is not None, "turn-detector-v1 requires cloud_opts"
transport = _CloudTransport(
detector=self,
opts=self._opts,
cloud_opts=cloud_opts,
http_session=self._http_session,
)
else:
transport = _LocalTransport(opts=self._opts)
return _BaseStreamingTurnDetectorStream(
detector=self,
opts=self._opts,
transport=transport,
model=self._model,
)
livekit-agents/livekit/agents/inference/eot/languages.py
+207
View File
@@ -0,0 +1,207 @@
"""Per-language ``unlikely`` thresholds for the mini detector."""
from __future__ import annotations
from typing import Literal, cast
from ..._exceptions import APIError
from ...language import LanguageCode
from ...types import NOT_GIVEN, NotGivenOr
from ...utils.misc import is_given
LOCAL_LANGUAGES: dict[str, float] = {
"ar": 0.3500,
"de": 0.2450,
"en": 0.3600,
"es": 0.3500,
"fr": 0.2850,
"hi": 0.3050,
"id": 0.3450,
"it": 0.2300,
"ja": 0.2950,
"ko": 0.4000,
"nl": 0.2000,
"pt": 0.3200,
"tr": 0.2550,
"zh": 0.3550,
}
TurnDetectorModels = Literal["turn-detector-v1", "turn-detector-v1-mini"]
TurnDetectorVersions = Literal["v1", "v1-mini"]
def _normalize_overrides(
overrides: NotGivenOr[float | dict[LanguageCode | str, float]],
) -> NotGivenOr[float | dict[str, float]]:
if not is_given(overrides) or not isinstance(overrides, dict):
return overrides
return {LanguageCode(k).language: float(v) for k, v in overrides.items()}
class ThresholdOptions:
def __init__(
self,
model: TurnDetectorModels,
overrides: NotGivenOr[float | dict[LanguageCode | str, float]] = NOT_GIVEN,
backchannel_overrides: NotGivenOr[float | dict[LanguageCode | str, float]] = NOT_GIVEN,
) -> None:
self._model = model
self._overrides = _normalize_overrides(overrides)
self._bc_overrides = _normalize_overrides(backchannel_overrides)
# server/shipped defaults
self._server_thresholds: dict[str, float] | None = None
self._server_default: float | None = None
if model == "turn-detector-v1-mini":
self._server_thresholds = dict(LOCAL_LANGUAGES)
self._server_default = LOCAL_LANGUAGES["en"]
# backchannel server defaults: cloud-only (the local mini model produces no
# backchannel probability), arrive via ``SessionCreated``.
self._server_bc_thresholds: dict[str, float] | None = None
self._server_bc_default: float | None = None
# materialized values (server defaults layered with user overrides)
self._thresholds: dict[str, float] = {}
self._default: float | None = None
self._bc_thresholds: dict[str, float] = {}
self._bc_default: float | None = None
self._resolve()
@property
def model(self) -> TurnDetectorModels:
return self._model
@property
def overrides(self) -> NotGivenOr[float | dict[str, float]]:
return self._overrides
@property
def backchannel_overrides(self) -> NotGivenOr[float | dict[str, float]]:
return self._bc_overrides
@property
def thresholds(self) -> dict[str, float]:
return self._thresholds
@property
def default_threshold(self) -> float | None:
return self._default
def lookup(self, language: LanguageCode | None) -> float | None:
lang_key = language.language if language else "en"
return self._thresholds.get(lang_key, self.default_threshold)
def lookup_backchannel(self, language: LanguageCode | None) -> float | None:
if not self._bc_thresholds and not self._bc_default:
return None
lang_key = language.language if language else "en"
threshold = self._bc_thresholds.get(lang_key, self._bc_default)
return threshold if threshold and threshold > 0 else None
def supports(self, language: LanguageCode | None) -> bool:
pending = self._model == "turn-detector-v1" and self._server_thresholds is None
return pending or self.lookup(language) is not None
def update_overrides(
self, overrides: NotGivenOr[float | dict[LanguageCode | str, float]]
) -> None:
self._overrides = _normalize_overrides(overrides)
self._resolve()
def update_backchannel_overrides(
self, overrides: NotGivenOr[float | dict[LanguageCode | str, float]]
) -> None:
self._bc_overrides = _normalize_overrides(overrides)
self._resolve()
def _update_defaults(
self,
server_thresholds: dict[str, float],
server_default: float,
backchannel_thresholds: dict[str, float] | None = None,
backchannel_default: float = 0.0,
) -> None:
if not server_thresholds or server_default <= 0:
raise APIError(
"turn detector session created without usable default thresholds",
retryable=False,
)
self._server_thresholds = {
LanguageCode(lang).language: round(value, 4)
for lang, value in server_thresholds.items()
}
self._server_default = round(server_default, 4)
# backchannel defaults are optional; an absent/empty map keeps backchannel disabled
self._server_bc_thresholds = (
{
LanguageCode(lang).language: round(value, 4)
for lang, value in backchannel_thresholds.items()
}
if backchannel_thresholds
else None
)
self._server_bc_default = round(backchannel_default, 4) if backchannel_default > 0 else None
self._resolve()
def _to_local_fallback(self) -> None:
if self._model == "turn-detector-v1-mini":
return
rescaled: dict[str, float] | None = None
if server := self._server_thresholds:
effective = {lang: self.lookup(LanguageCode(lang)) for lang in server}
rescaled = {
lang: LOCAL_LANGUAGES[lang] * (active_t / server[lang])
for lang, active_t in effective.items()
if active_t is not None and lang in LOCAL_LANGUAGES and server[lang] != 0
}
self._model = "turn-detector-v1-mini"
self._server_thresholds = dict(LOCAL_LANGUAGES)
self._server_default = LOCAL_LANGUAGES["en"]
# the mini model produces no backchannel probability
self._server_bc_thresholds = None
self._server_bc_default = None
self._resolve()
if rescaled is not None:
self._thresholds = rescaled
self._default = self.lookup(LanguageCode("en"))
def _resolve(self) -> None:
self._thresholds, self._default = self._resolve_layer(
self._server_thresholds, self._server_default, self._overrides
)
self._bc_thresholds, self._bc_default = self._resolve_layer(
self._server_bc_thresholds, self._server_bc_default, self._bc_overrides
)
@staticmethod
def _resolve_layer(
server_thresholds: dict[str, float] | None,
server_default: float | None,
overrides: NotGivenOr[float | dict[str, float]],
) -> tuple[dict[str, float], float | None]:
"""Layer a user override onto the server defaults.
A scalar override replaces the whole map (every language resolves through
it); a dict override is merged over the server map. Before server defaults
arrive, only a scalar override resolves up front.
"""
scalar_override = is_given(overrides) and not isinstance(overrides, dict)
if server_thresholds is None or server_default is None:
return {}, (float(cast(float, overrides)) if scalar_override else None)
if not is_given(overrides):
return dict(server_thresholds), server_default
if scalar_override:
return {}, float(cast(float, overrides))
override = cast("dict[str, float]", overrides)
return {**server_thresholds, **override}, server_default
livekit-agents/livekit/agents/inference/eot/transports.py
+424
View File
@@ -0,0 +1,424 @@
"""Audio EOT transports: cloud (WebSocket) + local (livekit-local-inference)."""
from __future__ import annotations
import asyncio
import time
import weakref
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
import aiohttp
import numpy as np
from google.protobuf.timestamp_pb2 import Timestamp
from livekit import rtc
from livekit.local_inference import EOT as _EOT
from livekit.protocol.agent_pb.agent_inference import (
AUDIO_ENCODING_PCM_S16LE,
ClientMessage,
EotPrediction,
InferenceStart,
InputAudio,
ServerMessage,
SessionClose,
SessionCreate,
SessionFlush,
SessionSettings,
)
from ... import utils
from ..._exceptions import (
APIConnectionError,
APIError,
APIStatusError,
APITimeoutError,
create_api_error_from_http,
)
from ...log import logger
from ...metrics import EOTInferenceMetrics
from ...metrics.base import Metadata
from ...types import APIConnectOptions
from ...utils import aio, is_given
from .._utils import create_access_token, get_inference_headers
from .base import (
DEFAULT_SAMPLE_RATE,
TurnDetectorOptions,
_BaseStreamingTurnDetectorStream,
_StreamingTurnDetectionTransport,
)
if TYPE_CHECKING:
from .detector import TurnDetector
__all__ = [
"_CloudTransport",
"_CloudTransportOptions",
"_LocalTransport",
"_StreamingTurnDetectionTransport",
]
@dataclass
class _CloudTransportOptions:
"""Cloud-WebSocket-specific options. Held separately from
``TurnDetectorOptions`` so the local transport doesn't see fields that
don't apply to it."""
base_url: str
api_key: str
api_secret: str
conn_options: APIConnectOptions
_CLIENT_BUFFER_SECONDS = 1.2
_CLIENT_BUFFER_SAMPLES = int(_CLIENT_BUFFER_SECONDS * DEFAULT_SAMPLE_RATE)
class _CloudTransport:
"""WebSocket transport for `turn-detector-v1`."""
def __init__(
self,
*,
detector: TurnDetector,
opts: TurnDetectorOptions,
cloud_opts: _CloudTransportOptions,
http_session: aiohttp.ClientSession | None,
) -> None:
self._detector_ref: weakref.ref[TurnDetector] = weakref.ref(detector)
self._opts = opts
self._cloud_opts = cloud_opts
self._conn_options = cloud_opts.conn_options
self._session_holder = http_session
self._ws: aiohttp.ClientWebSocketResponse | None = None
self._num_retries = 0
self._send_ch: aio.Chan[ClientMessage] | None = None
self._stream_ref: weakref.ref[_BaseStreamingTurnDetectorStream] | None = None
def attach(self, stream: _BaseStreamingTurnDetectorStream) -> None:
self._stream_ref = weakref.ref(stream)
def run_inference(self, request_id: str) -> None:
self._send_message(ClientMessage(inference_start=InferenceStart(request_id=request_id)))
def push_frame(self, frame: rtc.AudioFrame) -> None:
pcm_bytes = bytes(frame.data)
if not pcm_bytes:
return
audio_created_at = Timestamp()
audio_created_at.GetCurrentTime()
self._send_message(
ClientMessage(
input_audio=InputAudio(
audio=pcm_bytes,
num_samples=frame.samples_per_channel,
created_at=audio_created_at,
)
)
)
def flush(self) -> None:
self._send_message(ClientMessage(session_flush=SessionFlush()))
def detach(self) -> None:
if self._send_ch is not None:
self._send_ch.close()
self._ws = None
def _ensure_session(self) -> aiohttp.ClientSession:
if self._session_holder is None:
self._session_holder = utils.http_context.http_session()
return self._session_holder
def _build_auth_headers(self) -> dict[str, str]:
return {
**get_inference_headers(),
"Authorization": f"Bearer {create_access_token(self._cloud_opts.api_key, self._cloud_opts.api_secret)}",
}
def _send_message(self, msg: ClientMessage) -> None:
ch = self._send_ch
if ch is None or ch.closed or self._ws is None or self._ws.closed:
return
try:
ch.send_nowait(msg)
except aio.ChanClosed:
pass
async def _connect_ws(self) -> aiohttp.ClientWebSocketResponse:
base_url = self._cloud_opts.base_url
if base_url.startswith(("http://", "https://")):
base_url = base_url.replace("http", "ws", 1)
try:
ws = await asyncio.wait_for(
self._ensure_session().ws_connect(
f"{base_url}/eot",
headers=self._build_auth_headers(),
),
self._conn_options.timeout,
)
session_create_msg = ClientMessage(
session_create=SessionCreate(
settings=SessionSettings(
sample_rate=self._opts.sample_rate,
encoding=AUDIO_ENCODING_PCM_S16LE,
),
)
)
created_at = Timestamp()
created_at.GetCurrentTime()
session_create_msg.created_at.CopyFrom(created_at)
await ws.send_bytes(session_create_msg.SerializeToString())
except aiohttp.ClientResponseError as e:
raise create_api_error_from_http(e.message, status=e.status) from e
except asyncio.TimeoutError as e:
raise APITimeoutError("turn detector connection timed out") from e
except aiohttp.ClientConnectorError as e:
raise APIConnectionError("failed to connect to turn detector") from e
except Exception as e:
raise APIConnectionError("failed to connect to turn detector") from e
return ws
def _warn_transport_latency(self, msg: ServerMessage) -> None:
current_time = Timestamp()
current_time.GetCurrentTime()
if (
transport_latency := current_time.ToMilliseconds()
- msg.client_created_at.ToMilliseconds()
) > 500 and msg.client_created_at.ToMilliseconds() > 0:
logger.warning(
"turn detection transport latency is too high: %sms",
transport_latency,
)
def _process_message(self, msg: ServerMessage) -> None:
stream = self._stream_ref() if self._stream_ref is not None else None
if stream is None:
return
match msg.WhichOneof("message"):
case "eot_prediction":
prediction: EotPrediction = msg.eot_prediction
inference_stats = prediction.inference_stats
request_sent_at_ms = inference_stats.latest_client_created_at.ToMilliseconds()
current_time = Timestamp()
current_time.GetCurrentTime()
detection_delay_ms = current_time.ToMilliseconds() - request_sent_at_ms
inference_duration_ms = inference_stats.server_e2e_latency.ToMilliseconds()
stream._resolve_prediction(
msg.request_id,
prediction.probability,
detection_delay=detection_delay_ms / 1000.0,
inference_duration=inference_duration_ms / 1000.0,
backchannel_probability=prediction.backchannel_probability,
)
client_e2e_ms = inference_stats.client_e2e_latency.ToMilliseconds()
detector = self._detector_ref()
if detector is not None:
detector.emit(
"metrics_collected",
EOTInferenceMetrics(
timestamp=time.time(),
total_duration=client_e2e_ms / 1000.0,
prediction_duration=inference_duration_ms / 1000.0,
detection_delay=detection_delay_ms / 1000.0,
num_requests=1,
metadata=Metadata(
model_name=detector.model,
model_provider=detector.provider,
),
),
)
case "session_created":
self._warn_transport_latency(msg)
created = msg.session_created
thresholds = stream._opts.thresholds
thresholds._update_defaults(
dict(created.default_thresholds),
created.default_threshold,
dict(created.default_backchannel_thresholds),
created.default_backchannel_threshold,
)
logger.debug(
"audio turn detector initialized",
extra={
"model": thresholds.model,
"thresholds": thresholds.thresholds,
"default_threshold": thresholds.default_threshold,
"overrides": thresholds.overrides
if is_given(thresholds.overrides)
else None,
},
)
case "session_closed" | "inference_started" | "inference_stopped":
self._warn_transport_latency(msg)
case "error":
raise APIStatusError(
f"{msg.error.message}",
status_code=msg.error.code,
request_id=msg.request_id,
)
case _:
logger.warning("unexpected turn detector message: %s", msg.WhichOneof("message"))
async def run(self) -> None:
max_retries = self._conn_options.max_retry
while self._num_retries <= max_retries:
try:
return await self._run_once()
except APIError as e:
if max_retries == 0 or not e.retryable:
raise
if self._num_retries == max_retries:
raise APIConnectionError(
f"failed to connect livekit turn detector after {self._num_retries} attempts",
) from e
retry_interval = self._conn_options._interval_for_retry(self._num_retries)
logger.warning(
"livekit turn detector connection failed: %s, retrying in %ss",
e,
retry_interval,
extra={"attempt": self._num_retries},
)
await asyncio.sleep(retry_interval)
self._num_retries += 1
async def _run_once(self) -> None:
stream = self._stream_ref() if self._stream_ref is not None else None
if stream is None:
return
closing_ws = False
send_ch: aio.Chan[ClientMessage] = aio.Chan()
self._send_ch = send_ch
async def drain_audio_task() -> None:
nonlocal closing_ws
await stream._drain_audio_channel()
closing_ws = True
self._send_message(ClientMessage(session_close=SessionClose()))
send_ch.close()
async def sender_task(ws: aiohttp.ClientWebSocketResponse) -> None:
async for msg in send_ch:
if ws.closed:
return
if not msg.HasField("created_at"):
created_at = Timestamp()
created_at.GetCurrentTime()
msg.created_at.CopyFrom(created_at)
try:
await ws.send_bytes(msg.SerializeToString())
except (ConnectionResetError, aiohttp.ClientConnectionError):
return
async def recv_task(ws: aiohttp.ClientWebSocketResponse) -> None:
nonlocal closing_ws
while True:
ws_msg = await ws.receive()
if ws_msg.type in (
aiohttp.WSMsgType.CLOSED,
aiohttp.WSMsgType.CLOSE,
aiohttp.WSMsgType.CLOSING,
):
if closing_ws or self._ensure_session().closed:
return
raise APIStatusError(
message="turn detector connection closed unexpectedly",
status_code=ws.close_code or -1,
body=f"{ws_msg.data=} {ws_msg.extra=}",
retryable=False,
)
if ws_msg.type != aiohttp.WSMsgType.BINARY:
logger.warning("unexpected turn detector message type %s", ws_msg.type)
continue
server_msg = ServerMessage()
server_msg.ParseFromString(ws_msg.data)
self._process_message(server_msg)
ws: aiohttp.ClientWebSocketResponse | None = None
try:
ws = await self._connect_ws()
self._ws = ws
self._num_retries = 0
tasks = [
asyncio.create_task(drain_audio_task()),
asyncio.create_task(sender_task(ws)),
asyncio.create_task(recv_task(ws)),
]
try:
await asyncio.gather(*tasks)
finally:
await aio.gracefully_cancel(*tasks)
finally:
send_ch.close()
if self._send_ch is send_ch:
self._send_ch = None
self._ws = None
if ws is not None:
await ws.close()
class _LocalTransport:
"""In-process ctypes transport for `turn-detector-v1-mini`."""
def __init__(self, *, opts: TurnDetectorOptions) -> None:
self._opts = opts
self._buf = utils.AudioArrayBuffer(
buffer_size=_CLIENT_BUFFER_SAMPLES, sample_rate=DEFAULT_SAMPLE_RATE
)
self._eot = _EOT()
self._stream_ref: weakref.ref[_BaseStreamingTurnDetectorStream] | None = None
self._tasks: set[asyncio.Task[Any]] = set()
def attach(self, stream: _BaseStreamingTurnDetectorStream) -> None:
self._stream_ref = weakref.ref(stream)
def run_inference(self, request_id: str) -> None:
task = asyncio.create_task(self._predict(request_id, self._buf.read()))
self._tasks.add(task)
task.add_done_callback(self._tasks.discard)
async def _predict(self, request_id: str, pcm_snapshot: np.ndarray) -> None:
prob = 0.0
t0 = time.monotonic()
try:
prob = float(await asyncio.to_thread(self._eot.predict, pcm_snapshot))
except Exception:
logger.exception("local audio EOT prediction failed")
inference_duration = time.monotonic() - t0
stream = self._stream_ref() if self._stream_ref is not None else None
if stream is None:
return
stream._resolve_prediction(request_id, prob, inference_duration=inference_duration)
def push_frame(self, frame: rtc.AudioFrame) -> None:
self._buf.push_frame(frame)
def flush(self) -> None:
if len(self._buf) > 0:
self._buf.shift(len(self._buf))
def detach(self) -> None:
for task in list(self._tasks):
task.cancel()
self._tasks.clear()
async def run(self) -> None:
stream = self._stream_ref() if self._stream_ref is not None else None
if stream is None:
return
await stream._drain_audio_channel()
livekit-agents/livekit/agents/inference/stt.py
+3 -8
View File
@@ -235,14 +235,9 @@ def _resolve_vad_for_model(
)
return None
if is_speechmatics and vad_instance is None:
try:
from livekit.plugins.silero import VAD as SileroVAD
except ImportError as e:
raise ImportError(
"livekit-plugins-silero is required: model "
f"{model!r} does not handle endpointing server-side."
) from e
vad_instance = SileroVAD.load()
from .vad import VAD
vad_instance = VAD()
return vad_instance
livekit-agents/livekit/agents/inference/vad.py
+490
View File
@@ -0,0 +1,490 @@
# Copyright 2026 LiveKit, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from __future__ import annotations
import asyncio
import time
import weakref
from dataclasses import dataclass, replace
from typing import Literal
import numpy as np
from livekit import rtc
from livekit.local_inference import VAD as _NativeVAD, VAD_WINDOW_SAMPLES
from .. import utils, vad
from ..log import logger
from ..types import NOT_GIVEN, NotGivenOr
from ..utils import is_given
SLOW_INFERENCE_THRESHOLD = 0.2 # late by 200ms
_MODEL_SAMPLE_RATE = 16000
VADModels = Literal["silero"]
@dataclass
class _VADOptions:
min_speech_duration: float
min_silence_duration: float
prefix_padding_duration: float
max_buffered_speech: float
activation_threshold: float
deactivation_threshold: float
class VAD(vad.VAD):
"""Voice Activity Detection backed by ``livekit-local-inference``.
The native model singleton is loaded once at module import (via the
pybind11 ``.so`` constructor); each stream allocates its own per-instance
LSTM/context state.
"""
def __init__(
self,
*,
model: VADModels = "silero",
min_speech_duration: float = 0.05,
min_silence_duration: float = 0.25,
prefix_padding_duration: float = 0.5,
max_buffered_speech: float = 60.0,
activation_threshold: float = 0.5,
deactivation_threshold: NotGivenOr[float] = NOT_GIVEN,
) -> None:
super().__init__(capabilities=vad.VADCapabilities(update_interval=0.032))
if model != "silero":
raise ValueError(f"Unknown VAD model: {model!r}. Supported: 'silero'.")
if is_given(deactivation_threshold) and deactivation_threshold <= 0:
raise ValueError("deactivation_threshold must be greater than 0")
self._model = model
self._opts = _VADOptions(
min_speech_duration=min_speech_duration,
min_silence_duration=min_silence_duration,
prefix_padding_duration=prefix_padding_duration,
max_buffered_speech=max_buffered_speech,
activation_threshold=activation_threshold,
deactivation_threshold=deactivation_threshold
if is_given(deactivation_threshold)
else max(activation_threshold - 0.15, 0.01),
)
self._streams: weakref.WeakSet[_VADStream] = weakref.WeakSet()
@property
def model(self) -> str:
return self._model
@property
def provider(self) -> str:
return "livekit-local-inference"
def stream(self) -> vad.VADStream:
# Each stream owns its own _VADOptions snapshot so that
# _VADStream.update_options() can read the prior value of
# max_buffered_speech before mutating it. Sharing the dataclass would
# let VAD.update_options() mutate the stream's view first, and the
# stream would never observe an increase.
stream = _VADStream(self, replace(self._opts))
self._streams.add(stream)
return stream
def update_options(
self,
*,
min_speech_duration: NotGivenOr[float] = NOT_GIVEN,
min_silence_duration: NotGivenOr[float] = NOT_GIVEN,
prefix_padding_duration: NotGivenOr[float] = NOT_GIVEN,
max_buffered_speech: NotGivenOr[float] = NOT_GIVEN,
activation_threshold: NotGivenOr[float] = NOT_GIVEN,
deactivation_threshold: NotGivenOr[float] = NOT_GIVEN,
) -> None:
if is_given(min_speech_duration):
self._opts.min_speech_duration = min_speech_duration
if is_given(min_silence_duration):
self._opts.min_silence_duration = min_silence_duration
if is_given(prefix_padding_duration):
self._opts.prefix_padding_duration = prefix_padding_duration
if is_given(max_buffered_speech):
self._opts.max_buffered_speech = max_buffered_speech
if is_given(activation_threshold):
self._opts.activation_threshold = activation_threshold
if is_given(deactivation_threshold):
self._opts.deactivation_threshold = deactivation_threshold
for stream in self._streams:
stream.update_options(
min_speech_duration=min_speech_duration,
min_silence_duration=min_silence_duration,
prefix_padding_duration=prefix_padding_duration,
max_buffered_speech=max_buffered_speech,
activation_threshold=activation_threshold,
deactivation_threshold=deactivation_threshold,
)
@property
def min_silence_duration(self) -> float | None:
return self._opts.min_silence_duration
class _VADStream(vad.VADStream):
def __init__(self, parent: VAD, opts: _VADOptions) -> None:
super().__init__(parent)
self._opts = opts
self._native_vad = _NativeVAD()
self._input_sample_rate = 0
self._speech_buffer: np.ndarray | None = None
self._speech_buffer_max_reached = False
self._prefix_padding_samples = 0 # (input_sample_rate)
def update_options(
self,
*,
min_speech_duration: NotGivenOr[float] = NOT_GIVEN,
min_silence_duration: NotGivenOr[float] = NOT_GIVEN,
prefix_padding_duration: NotGivenOr[float] = NOT_GIVEN,
max_buffered_speech: NotGivenOr[float] = NOT_GIVEN,
activation_threshold: NotGivenOr[float] = NOT_GIVEN,
deactivation_threshold: NotGivenOr[float] = NOT_GIVEN,
) -> None:
old_max_buffered_speech = self._opts.max_buffered_speech
if is_given(min_speech_duration):
self._opts.min_speech_duration = min_speech_duration
if is_given(min_silence_duration):
self._opts.min_silence_duration = min_silence_duration
if is_given(prefix_padding_duration):
self._opts.prefix_padding_duration = prefix_padding_duration
if is_given(max_buffered_speech):
self._opts.max_buffered_speech = max_buffered_speech
if is_given(activation_threshold):
self._opts.activation_threshold = activation_threshold
if is_given(deactivation_threshold):
self._opts.deactivation_threshold = deactivation_threshold
if self._input_sample_rate:
assert self._speech_buffer is not None
self._prefix_padding_samples = int(
self._opts.prefix_padding_duration * self._input_sample_rate
)
self._speech_buffer.resize(
int(self._opts.max_buffered_speech * self._input_sample_rate)
+ self._prefix_padding_samples
)
if self._opts.max_buffered_speech > old_max_buffered_speech:
self._speech_buffer_max_reached = False
@utils.log_exceptions(logger=logger)
async def _main_task(self) -> None:
speech_buffer_index: int = 0
# "pub_" means public, these values are exposed to the users through events
pub_speaking = False
pub_speech_duration = 0.0
pub_silence_duration = 0.0
pub_current_sample = 0
pub_timestamp = 0.0
speech_threshold_duration = 0.0
silence_threshold_duration = 0.0
input_frames: list[rtc.AudioFrame] = []
inference_frames: list[rtc.AudioFrame] = []
resampler: rtc.AudioResampler | None = None
# used to avoid drift when the sample_rate ratio is not an integer
input_copy_remaining_fract = 0.0
extra_inference_time = 0.0
def _reset_state() -> None:
nonlocal speech_buffer_index
nonlocal pub_speaking, pub_speech_duration, pub_silence_duration
nonlocal pub_current_sample, pub_timestamp
nonlocal speech_threshold_duration, silence_threshold_duration
nonlocal input_frames, inference_frames, resampler
nonlocal input_copy_remaining_fract, extra_inference_time
self._native_vad.reset()
speech_buffer_index = 0
self._speech_buffer_max_reached = False
if self._speech_buffer is not None:
self._speech_buffer.fill(0)
pub_speaking = False
pub_speech_duration = 0.0
pub_silence_duration = 0.0
pub_current_sample = 0
pub_timestamp = 0.0
speech_threshold_duration = 0.0
silence_threshold_duration = 0.0
input_frames = []
inference_frames = []
input_copy_remaining_fract = 0.0
extra_inference_time = 0.0
if self._input_sample_rate and self._input_sample_rate != _MODEL_SAMPLE_RATE:
resampler = rtc.AudioResampler(
input_rate=self._input_sample_rate,
output_rate=_MODEL_SAMPLE_RATE,
quality=rtc.AudioResamplerQuality.QUICK,
)
else:
resampler = None
async for input_frame in self._input_ch:
if isinstance(input_frame, self._FlushSentinel):
_reset_state()
continue
if not isinstance(input_frame, rtc.AudioFrame):
continue
if not self._input_sample_rate:
self._input_sample_rate = input_frame.sample_rate
# alloc the buffers now that we know the input sample rate
self._prefix_padding_samples = int(
self._opts.prefix_padding_duration * self._input_sample_rate
)
self._speech_buffer = np.empty(
int(self._opts.max_buffered_speech * self._input_sample_rate)
+ self._prefix_padding_samples,
dtype=np.int16,
)
if self._input_sample_rate != _MODEL_SAMPLE_RATE:
# resampling needed: the input sample rate isn't the same as the model's
# sample rate used for inference
resampler = rtc.AudioResampler(
input_rate=self._input_sample_rate,
output_rate=_MODEL_SAMPLE_RATE,
quality=rtc.AudioResamplerQuality.QUICK, # VAD doesn't need high quality
)
elif self._input_sample_rate != input_frame.sample_rate:
logger.error("a frame with another sample rate was already pushed")
continue
assert self._speech_buffer is not None
input_frames.append(input_frame)
if resampler is not None:
# the resampler may have a bit of latency, but it is OK to ignore since it should be
# negligible
inference_frames.extend(resampler.push(input_frame))
else:
inference_frames.append(input_frame)
while True:
start_time = time.perf_counter()
available_inference_samples = sum(
[frame.samples_per_channel for frame in inference_frames]
)
if available_inference_samples < VAD_WINDOW_SAMPLES:
break # not enough samples to run inference
input_frame = utils.combine_frames(input_frames)
inference_frame = utils.combine_frames(inference_frames)
# native lib takes int16 directly — no float32 conversion
inference_window = np.asarray(
inference_frame.data[:VAD_WINDOW_SAMPLES], dtype=np.int16
)
# run the inference
p = await asyncio.to_thread(self._native_vad.predict, inference_window)
window_duration = VAD_WINDOW_SAMPLES / _MODEL_SAMPLE_RATE
pub_current_sample += VAD_WINDOW_SAMPLES
pub_timestamp += window_duration
resampling_ratio = self._input_sample_rate / _MODEL_SAMPLE_RATE
to_copy = VAD_WINDOW_SAMPLES * resampling_ratio + input_copy_remaining_fract
to_copy_int = int(to_copy)
input_copy_remaining_fract = to_copy - to_copy_int
# copy the inference window to the speech buffer
available_space = len(self._speech_buffer) - speech_buffer_index
to_copy_buffer = min(to_copy_int, available_space)
if to_copy_buffer > 0:
self._speech_buffer[
speech_buffer_index : speech_buffer_index + to_copy_buffer
] = input_frame.data[:to_copy_buffer]
speech_buffer_index += to_copy_buffer
elif not self._speech_buffer_max_reached:
# reached self._opts.max_buffered_speech (padding is included)
self._speech_buffer_max_reached = True
logger.warning(
"max_buffered_speech reached, ignoring further data for the current speech input" # noqa: E501
)
inference_duration = time.perf_counter() - start_time
extra_inference_time = max(
0.0,
extra_inference_time + inference_duration - window_duration,
)
if inference_duration > SLOW_INFERENCE_THRESHOLD:
logger.warning(
"inference is slower than realtime",
extra={"delay": extra_inference_time},
)
def _reset_write_cursor() -> None:
nonlocal speech_buffer_index
assert self._speech_buffer is not None
if speech_buffer_index <= self._prefix_padding_samples:
return
padding_data = self._speech_buffer[
speech_buffer_index - self._prefix_padding_samples : speech_buffer_index
]
self._speech_buffer_max_reached = False
self._speech_buffer[: self._prefix_padding_samples] = padding_data
speech_buffer_index = self._prefix_padding_samples
def _copy_speech_buffer() -> rtc.AudioFrame:
# copy the data from speech_buffer
assert self._speech_buffer is not None
speech_data = self._speech_buffer[:speech_buffer_index].tobytes() # noqa: B023
return rtc.AudioFrame(
sample_rate=self._input_sample_rate,
num_channels=1,
samples_per_channel=speech_buffer_index, # noqa: B023
data=speech_data,
)
if pub_speaking:
pub_speech_duration += window_duration
else:
pub_silence_duration += window_duration
self._event_ch.send_nowait(
vad.VADEvent(
type=vad.VADEventType.INFERENCE_DONE,
samples_index=pub_current_sample,
timestamp=pub_timestamp,
silence_duration=pub_silence_duration,
speech_duration=pub_speech_duration,
probability=p,
inference_duration=inference_duration,
frames=[
rtc.AudioFrame(
data=input_frame.data[:to_copy_int].tobytes(),
sample_rate=self._input_sample_rate,
num_channels=1,
samples_per_channel=to_copy_int,
)
],
speaking=pub_speaking,
raw_accumulated_silence=silence_threshold_duration,
raw_accumulated_speech=speech_threshold_duration,
)
)
if p >= self._opts.activation_threshold or (
pub_speaking and p > self._opts.deactivation_threshold
):
speech_threshold_duration += window_duration
silence_threshold_duration = 0.0
if not pub_speaking:
if speech_threshold_duration >= self._opts.min_speech_duration:
pub_speaking = True
pub_silence_duration = 0.0
pub_speech_duration = speech_threshold_duration
self._event_ch.send_nowait(
vad.VADEvent(
type=vad.VADEventType.START_OF_SPEECH,
samples_index=pub_current_sample,
timestamp=pub_timestamp,
silence_duration=pub_silence_duration,
speech_duration=pub_speech_duration,
frames=[_copy_speech_buffer()],
speaking=True,
)
)
else:
silence_threshold_duration += window_duration
speech_threshold_duration = 0.0
if not pub_speaking:
_reset_write_cursor()
if (
pub_speaking
and silence_threshold_duration >= self._opts.min_silence_duration
):
pub_speaking = False
pub_silence_duration = silence_threshold_duration
self._event_ch.send_nowait(
vad.VADEvent(
type=vad.VADEventType.END_OF_SPEECH,
samples_index=pub_current_sample,
timestamp=pub_timestamp,
silence_duration=pub_silence_duration,
speech_duration=max(
0.0, pub_speech_duration - silence_threshold_duration
),
frames=[_copy_speech_buffer()],
speaking=False,
)
)
pub_speech_duration = 0.0
_reset_write_cursor()
# remove the frames that were used for inference from the input and inference frames
input_frames = []
inference_frames = []
# add the remaining data
if len(input_frame.data) - to_copy_int > 0:
data = input_frame.data[to_copy_int:].tobytes()
input_frames.append(
rtc.AudioFrame(
data=data,
sample_rate=self._input_sample_rate,
num_channels=1,
samples_per_channel=len(data) // 2,
)
)
if len(inference_frame.data) - VAD_WINDOW_SAMPLES > 0:
data = inference_frame.data[VAD_WINDOW_SAMPLES:].tobytes()
inference_frames.append(
rtc.AudioFrame(
data=data,
sample_rate=_MODEL_SAMPLE_RATE,
num_channels=1,
samples_per_channel=len(data) // 2,
)
)
livekit-agents/livekit/agents/metrics/__init__.py
+4
View File
@@ -1,5 +1,6 @@
from .base import (
AgentMetrics,
EOTInferenceMetrics,
EOUMetrics,
InterruptionMetrics,
LLMMetrics,
@@ -10,6 +11,7 @@ from .base import (
)
from .usage import (
AgentSessionUsage,
EOTModelUsage,
InterruptionModelUsage,
LLMModelUsage,
ModelUsage,
@@ -25,6 +27,7 @@ __all__ = [
"AgentMetrics",
"VADMetrics",
"EOUMetrics",
"EOTInferenceMetrics",
"STTMetrics",
"TTSMetrics",
"RealtimeModelMetrics",
@@ -34,6 +37,7 @@ __all__ = [
"TTSModelUsage",
"STTModelUsage",
"InterruptionModelUsage",
"EOTModelUsage",
"ModelUsage",
"AgentSessionUsage",
"ModelUsageCollector",
livekit-agents/livekit/agents/metrics/base.py
+17
View File
@@ -112,6 +112,22 @@ class EOUMetrics(_BaseMetrics):
metadata: Metadata | None = None
class EOTInferenceMetrics(_BaseMetrics):
"""Per-inference metrics emitted by the EOT model on each prediction."""
type: Literal["eot_inference_metrics"] = "eot_inference_metrics"
timestamp: float
total_duration: float
"""Earliest audio creation time in an inference to response receive time."""
detection_delay: float
"""Latest audio creation time in an inference to response receive time."""
prediction_duration: float
"""Server side model inference time."""
num_requests: int = 1
"""Number of inference requests made during one inference."""
metadata: Metadata | None = None
class RealtimeModelMetrics(_BaseMetrics):
class CachedTokenDetails(BaseModel):
audio_tokens: int = 0
@@ -199,6 +215,7 @@ AgentMetrics = (
| TTSMetrics
| VADMetrics
| EOUMetrics
| EOTInferenceMetrics
| RealtimeModelMetrics
| InterruptionMetrics
| AvatarMetrics
livekit-agents/livekit/agents/metrics/usage.py
+33 -2
View File
@@ -7,6 +7,7 @@ from pydantic import BaseModel
from .base import (
AgentMetrics,
EOTInferenceMetrics,
InterruptionMetrics,
LLMMetrics,
RealtimeModelMetrics,
@@ -108,7 +109,19 @@ class InterruptionModelUsage(_BaseModelUsage):
"""Total number of requests sent to the interruption detection model."""
ModelUsage = LLMModelUsage | TTSModelUsage | STTModelUsage | InterruptionModelUsage
class EOTModelUsage(_BaseModelUsage):
"""Usage summary for end-of-turn detection models."""
type: Literal["eot_usage"] = "eot_usage"
provider: str
"""The provider name (e.g., 'livekit')."""
model: str
"""The model name (e.g., 'turn-detector-v1')."""
total_requests: int = 0
"""Total number of inference requests sent to the EOT model."""
ModelUsage = LLMModelUsage | TTSModelUsage | STTModelUsage | InterruptionModelUsage | EOTModelUsage
"""Union type for all model usage types."""
@@ -125,13 +138,19 @@ class ModelUsageCollector:
self._tts_usage: dict[tuple[str, str], TTSModelUsage] = {}
self._stt_usage: dict[tuple[str, str], STTModelUsage] = {}
self._interruption_usage: dict[tuple[str, str], InterruptionModelUsage] = {}
self._eot_usage: dict[tuple[str, str], EOTModelUsage] = {}
def __call__(self, metrics: AgentMetrics) -> None:
self.collect(metrics)
def _extract_provider_model(
self,
metrics: LLMMetrics | STTMetrics | TTSMetrics | RealtimeModelMetrics | InterruptionMetrics,
metrics: LLMMetrics
| STTMetrics
| TTSMetrics
| RealtimeModelMetrics
| InterruptionMetrics
| EOTInferenceMetrics,
) -> tuple[str, str]:
"""Extract provider and model from metrics metadata."""
provider = ""
@@ -169,6 +188,13 @@ class ModelUsageCollector:
self._interruption_usage[key] = InterruptionModelUsage(provider=provider, model=model)
return self._interruption_usage[key]
def _get_eot_usage(self, provider: str, model: str) -> EOTModelUsage:
"""Get or create an EOTModelUsage for the given provider/model combination."""
key = (provider, model)
if key not in self._eot_usage:
self._eot_usage[key] = EOTModelUsage(provider=provider, model=model)
return self._eot_usage[key]
def collect(self, metrics: AgentMetrics) -> None:
if isinstance(metrics, LLMMetrics):
provider, model = self._extract_provider_model(metrics)
@@ -225,6 +251,10 @@ class ModelUsageCollector:
provider, model = self._extract_provider_model(metrics)
interruption_usage = self._get_interruption_usage(provider, model)
interruption_usage.total_requests += metrics.num_requests
elif isinstance(metrics, EOTInferenceMetrics):
provider, model = self._extract_provider_model(metrics)
eot_usage = self._get_eot_usage(provider, model)
eot_usage.total_requests += metrics.num_requests
def flatten(self) -> list[ModelUsage]:
"""Returns a list of usage summaries, one per model/provider combination."""
@@ -233,4 +263,5 @@ class ModelUsageCollector:
result.extend(u.model_copy(deep=True) for u in self._tts_usage.values())
result.extend(u.model_copy(deep=True) for u in self._stt_usage.values())
result.extend(u.model_copy(deep=True) for u in self._interruption_usage.values())
result.extend(u.model_copy(deep=True) for u in self._eot_usage.values())
return result
livekit-agents/livekit/agents/telemetry/trace_types.py
+3
View File
@@ -59,6 +59,9 @@ ATTR_USER_TRANSCRIPT = "lk.user_transcript"
ATTR_TRANSCRIPT_CONFIDENCE = "lk.transcript_confidence"
ATTR_TRANSCRIPTION_DELAY = "lk.transcription_delay"
ATTR_END_OF_TURN_DELAY = "lk.end_of_turn_delay"
ATTR_EOU_SOURCE = "lk.eou.source"
ATTR_EOU_DETECTION_DELAY = "lk.eou.detection_delay"
ATTR_EOU_FROM_CACHE = "lk.eou.from_cache"
# metrics
ATTR_LLM_METRICS = "lk.llm_metrics"
livekit-agents/livekit/agents/utils/__init__.py
+2
View File
@@ -4,6 +4,7 @@ from . import aio, audio, codecs, http_context, http_server, hw, images
from .audio import AudioArrayBuffer, AudioBuffer, combine_frames, merge_frames
from .bounded_dict import BoundedDict
from .connection_pool import ConnectionPool
from .env import resolve_env_var
from .exp_filter import ExpFilter
from .log import log_exceptions
from .misc import is_dev_mode, is_given, is_hosted, nodename, shortuuid, time_ms
@@ -39,6 +40,7 @@ __all__ = [
"wait_for_agent",
"wait_for_participant",
"wait_for_track_publication",
"resolve_env_var",
]
# Cleanup docs of unexported modules
livekit-agents/livekit/agents/utils/env.py
+33
View File
@@ -0,0 +1,33 @@
import os
from ..types import NotGivenOr
from .misc import is_given
def resolve_env_var(val: NotGivenOr[str], *env_vars: str, default: str = "") -> str:
"""
Resolve an environment variable from a list of potential sources.
Args:
val: The value to resolve.
*env_vars: The environment variables to check. Order matters, the first non-None value will be returned.
default: The default value to return if no environment variables are set.
Returns:
The resolved environment variable.
Examples:
>>> resolve_env_var(
... NOT_GIVEN,
... "ABC_URL",
... default="https://agent-gateway.livekit.cloud/v1",
... )
"https://agent-gateway.livekit.cloud/v1"
"""
if is_given(val):
return val
for env_var in env_vars:
curr_val = os.getenv(env_var, None)
if curr_val is not None and curr_val != "":
return curr_val
return default
livekit-agents/livekit/agents/voice/agent_activity.py
+96 -21
View File
@@ -54,6 +54,7 @@ from .events import (
AgentFalseInterruptionEvent,
AgentState,
AgentStateChangedEvent,
EotPredictionEvent,
ErrorEvent,
FunctionToolsExecutedEvent,
MetricsCollectedEvent,
@@ -61,6 +62,7 @@ from .events import (
SpeechCreatedEvent,
UserInputTranscribedEvent,
UserTurnExceededEvent,
_AgentBackchannelOpportunityEvent,
)
from .generation import (
ToolExecutionOutput,
@@ -80,7 +82,14 @@ from .generation import (
)
from .speech_handle import DEFAULT_INPUT_DETAILS, InputDetails, SpeechHandle
from .tool_executor import _resolve_async_tool_options, _ToolExecutor
from .turn import EndpointingOptions, PreemptiveGenerationOptions, TurnDetectionMode
from .turn import (
EndpointingOptions,
PreemptiveGenerationOptions,
TurnDetectionMode,
_resolve_endpointing,
_StreamingTurnDetector,
_StreamingTurnDetectorStream,
)
if TYPE_CHECKING:
from ..llm import mcp
@@ -109,6 +118,7 @@ _OnEnterContextVar = contextvars.ContextVar["_OnEnterData"]("agents_activity_on_
class _ReusableResources:
stt_pipeline: _STTPipeline | None = None
rt_session: llm.RealtimeSession | None = None
turn_detector_stream: _StreamingTurnDetectorStream | None = None
async def cleanup(self) -> None:
tasks = []
@@ -118,6 +128,9 @@ class _ReusableResources:
if self.rt_session is not None:
tasks.append(self.rt_session.aclose())
self.rt_session = None
if self.turn_detector_stream is not None:
tasks.append(self.turn_detector_stream.aclose())
self.turn_detector_stream = None
if tasks:
outputs = await asyncio.gather(*tasks, return_exceptions=True)
@@ -249,7 +262,21 @@ class AgentActivity(RecognitionHooks):
self, turn_detection: TurnDetectionMode | None
) -> TurnDetectionMode | None:
if turn_detection is not None and not isinstance(turn_detection, str):
# return directly if turn_detection is _TurnDetector
if isinstance(turn_detection, _StreamingTurnDetector):
if self.vad is None:
logger.warning(
"TurnDetector requires a VAD model. Pass vad=inference.VAD() to AgentSession/Agent"
" or turn_detection=None to disable the default TurnDetector"
)
return None
if isinstance(self.llm, llm.RealtimeModel) and self.llm.capabilities.turn_detection:
logger.warning(
"turn_detection is a TurnDetector, but the LLM is a RealtimeModel "
"with server-side turn detection enabled, ignoring the turn_detection setting"
)
return None
return turn_detection
mode = turn_detection if isinstance(turn_detection, str) else None
@@ -295,8 +322,13 @@ class AgentActivity(RecognitionHooks):
)
mode = None
# fallback to VAD if server side turn detection is disabled and VAD is available
if not llm_model.capabilities.turn_detection and vad_model and mode is None:
# fallback to VAD if server side turn detection is disabled and user supplied VAD is available
if (
not llm_model.capabilities.turn_detection
and vad_model is not None
and not self.using_default_vad
and mode is None
):
mode = "vad"
elif mode == "realtime_llm":
@@ -354,15 +386,11 @@ class AgentActivity(RecognitionHooks):
@property
def endpointing_opts(self) -> EndpointingOptions:
# session should always have a valid endpointing val based on either defaults or overrides
agent_endpointing = self._agent._turn_handling.get("endpointing", {})
session_endpointing = self.session._opts.turn_handling["endpointing"]
return EndpointingOptions(
mode=agent_endpointing.get("mode", session_endpointing["mode"]),
min_delay=agent_endpointing.get("min_delay", session_endpointing["min_delay"]),
max_delay=agent_endpointing.get("max_delay", session_endpointing["max_delay"]),
alpha=agent_endpointing.get("alpha", session_endpointing["alpha"]),
)
overrides: EndpointingOptions = {
**self.session._opts.endpointing_overrides,
**(self._agent._turn_handling.get("endpointing") or EndpointingOptions()), # type: ignore[typeddict-item]
}
return _resolve_endpointing(overrides, turn_detection=self._turn_detection)
@property
def preemptive_generation_opts(self) -> PreemptiveGenerationOptions:
@@ -655,6 +683,14 @@ class AgentActivity(RecognitionHooks):
):
resources.stt_pipeline = await self._audio_recognition.detach_stt()
# reuse the stream during a handoff whenever we can
if (
self._audio_recognition
and isinstance(self._turn_detection, _StreamingTurnDetector)
and self._turn_detection is new_activity._turn_detection
):
resources.turn_detector_stream = self._audio_recognition.detach_turn_detector()
# rt session
if (
self._rt_session is not None
@@ -768,6 +804,9 @@ class AgentActivity(RecognitionHooks):
self._interruption_detector.on("error", self._on_error)
self._interruption_detector.on("overlapping_speech", self._on_overlap_speech_ended)
if isinstance(self._turn_detection, inference.TurnDetector):
self._turn_detection.on("metrics_collected", self._on_metrics_collected)
if isinstance(self.llm, llm.RealtimeModel):
rt_reused = reuse_resources is not None and reuse_resources.rt_session is not None
if rt_reused:
@@ -840,23 +879,41 @@ class AgentActivity(RecognitionHooks):
self._session._chat_ctx.insert(initial_config)
await self._resume_scheduling_task()
# skip default vad when llm does not need it
wired_vad = self.vad
if (
wired_vad is not None
and self.using_default_vad
and isinstance(self.llm, llm.RealtimeModel)
and self.llm.capabilities.turn_detection
):
wired_vad = None
self._audio_recognition = AudioRecognition(
self._session,
hooks=self,
stt=self._agent.stt_node if self.stt else None,
vad=self.vad,
vad=wired_vad,
using_default_vad=self.using_default_vad,
interruption_detection=self._interruption_detector,
endpointing=create_endpointing(self.endpointing_opts),
turn_detection=self._turn_detection,
stt_model=self.stt.model if self.stt else None,
stt_provider=self.stt.provider if self.stt else None,
)
if reuse_resources and reuse_resources.stt_pipeline is not None:
stt_pipeline = reuse_resources.stt_pipeline if reuse_resources else None
turn_detector_stream = reuse_resources.turn_detector_stream if reuse_resources else None
if stt_pipeline is not None:
logger.debug("reusing STT pipeline from previous activity")
self._audio_recognition.start(stt_pipeline=reuse_resources.stt_pipeline)
reuse_resources.stt_pipeline = None # ownership transferred
else:
self._audio_recognition.start()
if turn_detector_stream is not None:
logger.debug("reusing turn detector stream from previous activity")
self._audio_recognition.start(
stt_pipeline=stt_pipeline,
turn_detector_stream=turn_detector_stream,
)
if reuse_resources:
# ownership transferred to the new AudioRecognition
reuse_resources.stt_pipeline = None
reuse_resources.turn_detector_stream = None
@tracer.start_as_current_span("drain_agent_activity")
async def drain(
@@ -1029,6 +1086,9 @@ class AgentActivity(RecognitionHooks):
self._interruption_detector.off("error", self._on_error)
self._interruption_detector.off("overlapping_speech", self._on_overlap_speech_ended)
if isinstance(self._turn_detection, inference.TurnDetector):
self._turn_detection.off("metrics_collected", self._on_metrics_collected)
if self._rt_session is not None:
await self._rt_session.aclose()
@@ -1634,7 +1694,7 @@ class AgentActivity(RecognitionHooks):
self._session.emit("overlapping_speech", ev)
def _on_input_speech_started(self, _: llm.InputSpeechStartedEvent) -> None:
if self.vad is None:
if self.vad is None or self.using_default_vad:
self._session._update_user_state("speaking")
if self._audio_recognition:
self._audio_recognition.on_start_of_speech(
@@ -1652,7 +1712,7 @@ class AgentActivity(RecognitionHooks):
)
def _on_input_speech_stopped(self, ev: llm.InputSpeechStoppedEvent) -> None:
if self.vad is None:
if self.vad is None or self.using_default_vad:
if self._audio_recognition:
self._audio_recognition.on_end_of_speech(
ended_at=time.time(),
@@ -2008,6 +2068,15 @@ class AgentActivity(RecognitionHooks):
created_at=time.time(),
)
def on_eot_prediction(self, ev: EotPredictionEvent) -> None:
if (host := self._session._session_host) is not None:
host._on_eot_prediction(ev)
def on_agent_backchannel_opportunity(self, ev: _AgentBackchannelOpportunityEvent) -> None:
# TODO: consume the backchannel opportunity internally (e.g. trigger a
# backchannel phrase). Kept internal for now — not surfaced as a public event.
pass
def on_end_of_turn(self, info: _EndOfTurnInfo) -> bool:
# IMPORTANT: This method is sync to avoid it being cancelled by the AudioRecognition
# We explicitly create a new task here
@@ -3928,6 +3997,12 @@ class AgentActivity(RecognitionHooks):
return None
return self._agent.vad if is_given(self._agent.vad) else self._session.vad
@property
def using_default_vad(self) -> bool:
if is_given(self._agent.vad):
return False
return self._session._using_default_vad
def _resolve_interruption_detection(self) -> inference.AdaptiveInterruptionDetector | None:
if not (
self.stt is not None
livekit-agents/livekit/agents/voice/agent_session.py
+27 -24
View File
@@ -142,6 +142,8 @@ class SessionConnectOptions:
@dataclass
class AgentSessionOptions:
turn_handling: TurnHandlingOptions
endpointing_overrides: EndpointingOptions
"""sparse endpointing keys the user provided explicitly"""
max_tool_steps: int
user_away_timeout: float | None
min_consecutive_speech_delay: float
@@ -225,7 +227,7 @@ class AgentSession(rtc.EventEmitter[EventTypes], Generic[Userdata_T]):
self,
*,
stt: NotGivenOr[stt.STT | STTModels | str] = NOT_GIVEN,
vad: NotGivenOr[vad.VAD] = NOT_GIVEN,
vad: NotGivenOr[vad.VAD | None] = NOT_GIVEN,
llm: NotGivenOr[llm.LLM | llm.RealtimeModel | LLMModels | str] = NOT_GIVEN,
tts: NotGivenOr[tts.TTS | TTSModels | str] = NOT_GIVEN,
turn_handling: NotGivenOr[TurnHandlingOptions] = NOT_GIVEN,
@@ -273,7 +275,10 @@ class AgentSession(rtc.EventEmitter[EventTypes], Generic[Userdata_T]):
Args:
stt (stt.STT | str, optional): Speech-to-text backend.
vad (vad.VAD, optional): Voice-activity detector
vad (vad.VAD, optional): Voice-activity detector. Defaults to the
bundled silero VAD (``inference.VAD(model="silero")``) when
omitted. Pass ``vad=None`` to opt out, or pass an explicit
instance to customise options.
llm (llm.LLM | llm.RealtimeModel | str, optional): LLM or RealtimeModel
tts (tts.TTS | str, optional): Text-to-speech engine.
tools (list[llm.FunctionTool | llm.RawFunctionTool], optional): List of
@@ -341,13 +346,8 @@ class AgentSession(rtc.EventEmitter[EventTypes], Generic[Userdata_T]):
turn_handling = (
_migrate_turn_handling(
# backward compatibility for deprecated parameters that had default values
min_endpointing_delay=(
min_endpointing_delay if is_given(min_endpointing_delay) else 0.5
),
max_endpointing_delay=(
max_endpointing_delay if is_given(max_endpointing_delay) else 3.0
),
min_endpointing_delay=min_endpointing_delay,
max_endpointing_delay=max_endpointing_delay,
false_interruption_timeout=false_interruption_timeout,
turn_detection=turn_detection,
discard_audio_if_uninterruptible=discard_audio_if_uninterruptible,
@@ -362,11 +362,14 @@ class AgentSession(rtc.EventEmitter[EventTypes], Generic[Userdata_T]):
else turn_handling
)
endpointing = _resolve_endpointing(turn_handling.get("endpointing"))
raw_turn_detection: TurnDetectionMode | None = turn_handling.get(
"turn_detection", inference.TurnDetector()
)
endpointing_overrides = turn_handling.get("endpointing") or EndpointingOptions()
endpointing = _resolve_endpointing(endpointing_overrides, turn_detection=raw_turn_detection)
interruption = _resolve_interruption(turn_handling.get("interruption"))
preemptive_gen = _resolve_preemptive_generation(turn_handling.get("preemptive_generation"))
user_turn_limit = _resolve_user_turn_limit(turn_handling.get("user_turn_limit"))
raw_turn_detection = turn_handling.get("turn_detection", None)
# This is the "global" chat_context, it holds the entire conversation history
self._chat_ctx = ChatContext.empty()
@@ -378,6 +381,7 @@ class AgentSession(rtc.EventEmitter[EventTypes], Generic[Userdata_T]):
preemptive_generation=preemptive_gen,
user_turn_limit=user_turn_limit,
),
endpointing_overrides=endpointing_overrides,
max_tool_steps=max_tool_steps,
user_away_timeout=user_away_timeout,
min_consecutive_speech_delay=min_consecutive_speech_delay,
@@ -406,6 +410,9 @@ class AgentSession(rtc.EventEmitter[EventTypes], Generic[Userdata_T]):
tts = inference.TTS.from_model_string(tts)
self._stt = stt or None
self._using_default_vad = not is_given(vad)
if not is_given(vad):
vad = inference.VAD(model="silero")
self._vad = vad or None
self._llm = llm or None
self._tts = tts or None
@@ -1100,29 +1107,25 @@ class AgentSession(rtc.EventEmitter[EventTypes], Generic[Userdata_T]):
"min_endpointing_delay and max_endpointing_delay are deprecated, "
"use endpointing_opts instead"
)
endpointing_opts = EndpointingOptions(
mode=self._opts.endpointing["mode"],
min_delay=(
min_endpointing_delay
if is_given(min_endpointing_delay)
else self._opts.endpointing["min_delay"]
),
max_delay=(
max_endpointing_delay
if is_given(max_endpointing_delay)
else self._opts.endpointing["max_delay"]
),
)
endpointing_opts = EndpointingOptions()
if is_given(min_endpointing_delay):
endpointing_opts["min_delay"] = min_endpointing_delay
if is_given(max_endpointing_delay):
endpointing_opts["max_delay"] = max_endpointing_delay
if is_given(endpointing_opts):
if (mode := endpointing_opts.get("mode")) is not None:
self._opts.endpointing["mode"] = mode
self._opts.endpointing_overrides["mode"] = mode
if (min_delay := endpointing_opts.get("min_delay")) is not None:
self._opts.endpointing["min_delay"] = min_delay
self._opts.endpointing_overrides["min_delay"] = min_delay
if (max_delay := endpointing_opts.get("max_delay")) is not None:
self._opts.endpointing["max_delay"] = max_delay
self._opts.endpointing_overrides["max_delay"] = max_delay
if (alpha := endpointing_opts.get("alpha")) is not None:
self._opts.endpointing["alpha"] = alpha
self._opts.endpointing_overrides["alpha"] = alpha
if is_given(turn_detection):
self._turn_detection = turn_detection
livekit-agents/livekit/agents/voice/audio_recognition.py
+407 -69
View File
@@ -7,7 +7,7 @@ import time
from collections import deque
from collections.abc import AsyncIterable, Callable
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any, Protocol
from typing import TYPE_CHECKING, Any, Literal, Protocol
from opentelemetry import trace
from opentelemetry.sdk.trace import ReadableSpan
@@ -16,6 +16,7 @@ from livekit import rtc
from .. import inference, llm, stt, tokenize, utils, vad
from .._exceptions import APIError
from ..inference.eot.base import MIN_SILENCE_DURATION_MS
from ..inference.interruption import (
_AgentSpeechEndedSentinel,
_AgentSpeechStartedSentinel,
@@ -32,8 +33,18 @@ from ..vad import VADStream
from . import io
from ._utils import _set_participant_attributes
from .endpointing import BaseEndpointing
from .events import UserTurnExceededEvent
from .turn import TurnDetectionMode as TurnDetectionMode
from .events import (
EotPredictionEvent,
UserTurnExceededEvent,
_AgentBackchannelOpportunityEvent,
)
from .turn import (
TurnDetectionEvent,
TurnDetectionMode as TurnDetectionMode,
_StreamingTurnDetector,
_StreamingTurnDetectorStream,
_TurnDetector,
)
if TYPE_CHECKING:
from .agent_session import AgentSession
@@ -123,6 +134,8 @@ class RecognitionHooks(Protocol):
def on_interim_transcript(self, ev: stt.SpeechEvent, *, speaking: bool | None) -> None: ...
def on_final_transcript(self, ev: stt.SpeechEvent, *, speaking: bool | None = None) -> None: ...
def on_end_of_turn(self, info: _EndOfTurnInfo) -> bool: ...
def on_eot_prediction(self, ev: EotPredictionEvent) -> None: ...
def on_agent_backchannel_opportunity(self, ev: _AgentBackchannelOpportunityEvent) -> None: ...
def on_preemptive_generation(self, info: _PreemptiveGenerationInfo) -> None: ...
def on_user_turn_exceeded(self, ev: UserTurnExceededEvent) -> None: ...
def retrieve_chat_ctx(self) -> llm.ChatContext: ...
@@ -186,6 +199,7 @@ class AudioRecognition:
endpointing: BaseEndpointing,
stt: io.STTNode | None,
vad: vad.VAD | None,
using_default_vad: bool,
interruption_detection: inference.AdaptiveInterruptionDetector | None,
turn_detection: TurnDetectionMode | None,
stt_model: str | None = None,
@@ -202,6 +216,7 @@ class AudioRecognition:
self._turn_detector = turn_detection if not isinstance(turn_detection, str) else None
self._stt = stt
self._vad = vad
self._using_default_vad = using_default_vad
self._stt_model = stt_model
self._stt_provider = stt_provider
self._turn_detection_mode = turn_detection if isinstance(turn_detection, str) else None
@@ -271,6 +286,14 @@ class AudioRecognition:
self._turn_tracker = _UserTurnTracker()
self._word_tokenizer = tokenize.basic.WordTokenizer()
# streaming audio turn detection
self._turn_detector_stream: _StreamingTurnDetectorStream | None = None
self._turn_detector_prediction_fut: asyncio.Future[TurnDetectionEvent] | None = None
self._turn_detector_flushed: bool = False
self._turn_detector_late_prediction_warned: bool = False
self._last_emitted_prediction: TurnDetectionEvent | None = None
self._user_speaking_event = asyncio.Event()
def update_options(
self,
*,
@@ -284,7 +307,9 @@ class AudioRecognition:
self._endpointing = endpointing
if is_given(turn_detection):
self._turn_detector = turn_detection if not isinstance(turn_detection, str) else None
self.update_turn_detector(
turn_detection if not isinstance(turn_detection, str) else None
)
mode = turn_detection if isinstance(turn_detection, str) else None
if self._turn_detection_mode != mode:
@@ -298,15 +323,26 @@ class AudioRecognition:
self._end_of_turn_task.cancel()
self._end_of_turn_task = None
self._user_turn_committed = False
if self._turn_detector_stream is not None:
self._turn_detector_stream.cancel_inference()
self._turn_detector_prediction_fut = None
def start(self, *, stt_pipeline: _STTPipeline | None = None) -> None:
def start(
self,
*,
stt_pipeline: _STTPipeline | None = None,
turn_detector_stream: _StreamingTurnDetectorStream | None = None,
) -> None:
self.update_stt(self._stt, pipeline=stt_pipeline)
self.update_vad(self._vad)
self.update_interruption_detection(self._interruption_detection)
if isinstance(self._turn_detector, _StreamingTurnDetector) or self._turn_detector is None:
self.update_turn_detector(self._turn_detector, stream=turn_detector_stream)
def stop(self) -> None:
self.update_stt(None)
self.update_vad(None)
self.update_turn_detector(None)
self.update_interruption_detection(None)
@property
@@ -423,12 +459,11 @@ class AudioRecognition:
user_speaking_span: trace.Span | None = None,
interruption: NotGivenOr[bool] = NOT_GIVEN,
) -> None:
should_ignore = is_given(interruption) and not interruption and self._agent_speaking
if self._speaking:
self._endpointing.on_end_of_speech(
ended_at=ended_at,
should_ignore=(
is_given(interruption) and not interruption and self._agent_speaking
),
should_ignore=should_ignore,
)
self.on_end_of_overlap_speech(ended_at=ended_at, user_speaking_span=user_speaking_span)
@@ -628,13 +663,13 @@ class AudioRecognition:
if self._interruption_ch is not None:
self._interruption_ch.send_nowait(frame)
if self._turn_detector_stream is not None:
self._turn_detector_stream.push_audio(frame)
async def aclose(self) -> None:
self._closing.set()
if self._commit_user_turn_atask is not None:
try:
await self._commit_user_turn_atask
except asyncio.CancelledError:
pass
await aio.cancel_and_wait(self._commit_user_turn_atask)
if self._stt_pipeline is not None:
await self._stt_pipeline.aclose()
@@ -652,10 +687,12 @@ class AudioRecognition:
await aio.cancel_and_wait(self._interruption_atask)
if self._end_of_turn_task is not None:
try:
await self._end_of_turn_task
except asyncio.CancelledError:
pass
await aio.cancel_and_wait(self._end_of_turn_task)
if self._turn_detector_stream is not None:
await self._turn_detector_stream.aclose()
self._turn_detector_stream = None
self._turn_detector_prediction_fut = None
if self._backchannel_boundary_timer is not None:
self._backchannel_boundary_timer.cancel()
@@ -693,8 +730,26 @@ class AudioRecognition:
self._tasks.add(task)
self._stt_pipeline = None
def _check_vad_silence_requirement(
self,
detector: NotGivenOr[_TurnDetector | _StreamingTurnDetector | None] = NOT_GIVEN,
) -> None:
if not is_given(detector):
detector = self._turn_detector
if not isinstance(detector, _StreamingTurnDetector) or self._vad is None:
return
if (current := getattr(self._vad, "min_silence_duration", None)) is None:
return
required = (MIN_SILENCE_DURATION_MS + 50) / 1000
if current < required:
raise ValueError(
f"vad min_silence_duration={current}s is too low for the TurnDetector. "
f"Raise the VAD's min_silence_duration to at least {required}s."
)
def update_vad(self, vad: vad.VAD | None) -> None:
self._vad = vad
self._check_vad_silence_requirement()
if vad:
self._vad_stream = None
self._vad_ch = aio.Chan[rtc.AudioFrame]()
@@ -759,6 +814,45 @@ class AudioRecognition:
self._interruption_detection is not None and self._vad is not None
)
def update_turn_detector(
self,
detector: _TurnDetector | _StreamingTurnDetector | None,
*,
stream: _StreamingTurnDetectorStream | None = None,
) -> None:
"""Update the turn detector and turn detector stream if possible.
When *stream* is provided it is adopted as-is (handoff reuse) instead of
opening a fresh stream on *detector*; the live transport stream — and its
per-session cloud->local fallback state — survives the handoff.
"""
self._check_vad_silence_requirement(detector)
self._turn_detector = detector
if (old_stream := self._turn_detector_stream) is not None and old_stream is not stream:
task = asyncio.create_task(old_stream.aclose())
task.add_done_callback(lambda _: self._tasks.discard(task))
self._tasks.add(task)
if stream is None:
stream = detector.stream() if isinstance(detector, _StreamingTurnDetector) else None
if self._turn_detector_stream is not stream:
self._turn_detector_prediction_fut = None
self._turn_detector_flushed = False
self._turn_detector_stream = stream
def detach_turn_detector(self) -> _StreamingTurnDetectorStream | None:
"""Detach the turn detector stream for handoff to another AudioRecognition.
Returns the live stream (transport run loop intact) without closing it.
The caller passes it to the new AudioRecognition via
``start(..., turn_detector_stream=stream)``. The adopting recognition
starts a fresh inference request on its next VAD event, superseding
any request that survived the handoff.
"""
stream, self._turn_detector_stream = self._turn_detector_stream, None
self._turn_detector_prediction_fut = None
return stream
def clear_user_turn(self) -> None:
self._audio_transcript = ""
self._audio_interim_transcript = ""
@@ -769,6 +863,12 @@ class AudioRecognition:
self._last_speaking_time = None
self._vad_speech_started = False
self._user_turn_committed = False
self._last_emitted_prediction = None
if self._turn_detector_stream is not None:
self._turn_detector_stream.flush(reason="clear_user_turn")
self._turn_detector_prediction_fut = None
self._turn_detector_flushed = True
self._turn_tracker = _UserTurnTracker()
# end any in-progress user_turn span so the next speech starts a fresh one
@@ -850,7 +950,11 @@ class AudioRecognition:
transcript = self._audio_transcript
self._audio_interim_transcript = ""
chat_ctx = self._hooks.retrieve_chat_ctx().copy()
self._run_eou_detection(chat_ctx, skip_reply=skip_reply)
self._run_eou_detection(
chat_ctx,
skip_reply=skip_reply,
trigger="manual",
)
self._user_turn_committed = True
if not fut.done():
fut.set_result(transcript)
@@ -951,7 +1055,8 @@ class AudioRecognition:
self._hooks.on_final_transcript(
ev,
speaking=self._speaking
if self._vad or self._turn_detection_mode == "stt"
if (self._vad is not None and not self._using_default_vad)
or self._turn_detection_mode == "stt"
else None,
)
if self._session.amd is not None:
@@ -970,7 +1075,7 @@ class AudioRecognition:
self._audio_interim_transcript = ""
self._audio_preflight_transcript = ""
if not self._vad or self._last_speaking_time is None:
if self._vad is None or self._using_default_vad or self._last_speaking_time is None:
# vad disabled or missed a speech, use stt timestamp
self._last_speaking_time = stt_last_speaking_time
@@ -994,13 +1099,17 @@ class AudioRecognition:
if not self._speaking:
chat_ctx = self._hooks.retrieve_chat_ctx().copy()
self._run_eou_detection(chat_ctx)
self._run_eou_detection(
chat_ctx,
trigger="stt",
)
elif ev.type == stt.SpeechEventType.PREFLIGHT_TRANSCRIPT:
self._hooks.on_interim_transcript(
ev,
speaking=self._speaking
if self._vad or self._turn_detection_mode == "stt"
if (self._vad is not None and not self._using_default_vad)
or self._turn_detection_mode == "stt"
else None,
)
transcript = ev.alternatives[0].text
@@ -1026,7 +1135,7 @@ class AudioRecognition:
self._audio_preflight_transcript = (self._audio_transcript + " " + transcript).lstrip()
self._audio_interim_transcript = transcript
if not self._vad or self._last_speaking_time is None:
if self._vad is None or self._using_default_vad or self._last_speaking_time is None:
# vad disabled or missed a speech, use stt timestamp
self._last_speaking_time = stt_last_speaking_time
@@ -1044,7 +1153,8 @@ class AudioRecognition:
self._hooks.on_interim_transcript(
ev,
speaking=self._speaking
if self._vad or self._turn_detection_mode == "stt"
if (self._vad is not None and not self._using_default_vad)
or self._turn_detection_mode == "stt"
else None,
)
self._audio_interim_transcript = ev.alternatives[0].text
@@ -1076,12 +1186,15 @@ class AudioRecognition:
self._speaking = False
self._user_turn_committed = True
if not self._vad or self._last_speaking_time is None:
if self._vad is None or self._using_default_vad or self._last_speaking_time is None:
# vad disabled or missed a speech, use stt timestamp
self._last_speaking_time = stt_last_speaking_time
chat_ctx = self._hooks.retrieve_chat_ctx().copy()
self._run_eou_detection(chat_ctx)
self._run_eou_detection(
chat_ctx,
trigger="stt",
)
elif ev.type == stt.SpeechEventType.START_OF_SPEECH and self._turn_detection_mode == "stt":
# If the plugin provided a server onset timestamp, use it;
@@ -1110,6 +1223,12 @@ class AudioRecognition:
self._hooks.on_start_of_speech(ev, speech_start_time=speech_start_time)
self._speaking = True
self._user_speaking_event.set()
if self._turn_detector_stream is not None:
self._turn_detector_stream.cancel_inference()
self._turn_detector_prediction_fut = None
self._turn_detector_flushed = False
if self._end_of_turn_task is not None:
self._end_of_turn_task.cancel()
@@ -1126,6 +1245,20 @@ class AudioRecognition:
if self._speech_start_time is None:
self._speech_start_time = time.time() - ev.raw_accumulated_speech
self._user_speaking_event.set()
if self._speaking and self._turn_detector_prediction_fut is not None:
if self._turn_detector_stream is not None:
self._turn_detector_stream.cancel_inference()
self._turn_detector_prediction_fut = None
elif not self._speaking:
self._user_speaking_event.clear()
if ev.raw_accumulated_silence >= MIN_SILENCE_DURATION_MS / 1000 and self._speaking:
if (
self._turn_detector_stream is not None
and self._turn_detector_prediction_fut is None
):
self._turn_detector_prediction_fut = self._turn_detector_stream.predict()
elif ev.type == vad.VADEventType.END_OF_SPEECH:
with trace.use_span(self._ensure_user_turn_span()):
@@ -1133,12 +1266,14 @@ class AudioRecognition:
self._vad_speech_started = False
self._speaking = False
self._user_speaking_event.clear()
self._last_speaking_time = time.time() - ev.silence_duration - ev.inference_duration
if self._vad_base_turn_detection or (
self._turn_detection_mode == "stt" and self._user_turn_committed
):
chat_ctx = self._hooks.retrieve_chat_ctx().copy()
self._run_eou_detection(chat_ctx)
self._run_eou_detection(chat_ctx, trigger="vad")
if self._session.amd is not None:
self._session.amd._on_user_speech_ended(ev.silence_duration)
@@ -1153,16 +1288,44 @@ class AudioRecognition:
if ev.is_interruption:
self._hooks.on_interruption(ev)
def _run_eou_detection(self, chat_ctx: llm.ChatContext, skip_reply: bool = False) -> None:
def _on_missing_eot_prediction(self) -> None:
if self._turn_detector_flushed:
if not self._turn_detector_late_prediction_warned:
self._turn_detector_late_prediction_warned = True
logger.warning(
"eou detection ran after the audio eot turn was already flushed "
"(likely a late stt final). consider raising `min_delay` in the "
"endpointing options to accommodate slow stt. subsequent "
"occurrences will log at debug level.",
)
else:
logger.debug("stt transcript arrived after a turn flush, skipping eot prediction")
else:
logger.debug("no eot inference request in flight, skipping eot prediction")
def _run_eou_detection(
self,
chat_ctx: llm.ChatContext,
*,
trigger: Literal["vad", "stt", "manual"],
skip_reply: bool = False,
) -> None:
if self._stt and not self._audio_transcript and self._turn_detection_mode != "manual":
# stt enabled but no transcript yet
return
chat_ctx = chat_ctx.copy()
chat_ctx.add_message(role="user", content=self._audio_transcript)
if self._audio_transcript:
chat_ctx.add_message(role="user", content=self._audio_transcript)
turn_detector = (
self._turn_detector
if self._audio_transcript and self._turn_detection_mode != "manual"
(
self._turn_detector_stream
if isinstance(self._turn_detector, _StreamingTurnDetector)
else self._turn_detector
)
if self._turn_detection_mode != "manual"
and (self._audio_transcript or isinstance(self._turn_detector, _StreamingTurnDetector))
else None # disable EOU model if manual turn detection enabled
)
@@ -1174,6 +1337,11 @@ class AudioRecognition:
) -> None:
endpointing_delay = self._endpointing.min_delay
user_turn_span = self._ensure_user_turn_span()
end_of_turn_probability: float | None = None
unlikely_threshold: float | None = None
backchannel_threshold: float | None = None
if turn_detector is not None:
if not await turn_detector.supports_language(self._last_language):
logger.info("Turn detector does not support language %s", self._last_language)
@@ -1182,57 +1350,166 @@ class AudioRecognition:
trace.use_span(user_turn_span),
tracer.start_as_current_span("eou_detection") as eou_detection_span,
):
# if there are failures, we should not hold the pipeline up
end_of_turn_probability = 0.0
unlikely_threshold: float | None = None
try:
end_of_turn_probability = await turn_detector.predict_end_of_turn(
chat_ctx
from_cache = False
prediction_event: TurnDetectionEvent | None = None
if isinstance(turn_detector, _StreamingTurnDetectorStream):
fut = self._turn_detector_prediction_fut
if fut is None:
self._on_missing_eot_prediction()
else:
from_cache = fut.done()
done, _ = await asyncio.wait([fut], timeout=endpointing_delay)
if fut in done and not fut.cancelled():
prediction_event = fut.result()
end_of_turn_probability = (
prediction_event.end_of_turn_probability
)
unlikely_threshold = await turn_detector.unlikely_threshold(
self._last_language
)
backchannel_threshold = (
await turn_detector.backchannel_threshold(
self._last_language
)
)
else:
logger.warning(
"eot prediction timed out, committing without a prediction",
extra={"timeout": endpointing_delay},
)
turn_detector.cancel_inference(timed_out=True)
self._turn_detector_prediction_fut = None
else:
try:
end_of_turn_probability = await turn_detector.predict_end_of_turn(
chat_ctx,
timeout=endpointing_delay,
)
unlikely_threshold = await turn_detector.unlikely_threshold(
self._last_language
)
except Exception:
logger.exception("Error predicting end of turn")
if (
end_of_turn_probability is not None
and unlikely_threshold is not None
and end_of_turn_probability < unlikely_threshold
):
endpointing_delay = self._endpointing.max_delay
eou_span_attributes: dict[str, Any] = {
trace_types.ATTR_CHAT_CTX: json.dumps(
llm.ChatContext(chat_ctx.items[-_EOU_MAX_HISTORY_TURNS:])
.copy(
exclude_function_call=True,
exclude_instructions=True,
exclude_empty_message=True,
exclude_handoff=True,
exclude_config_update=True,
)
.to_dict(
exclude_audio=True,
exclude_image=True,
exclude_timestamp=True,
exclude_metrics=True,
)
),
trace_types.ATTR_EOU_DELAY: endpointing_delay,
trace_types.ATTR_EOU_LANGUAGE: self._last_language or "",
trace_types.ATTR_EOU_SOURCE: trigger,
trace_types.ATTR_EOU_FROM_CACHE: from_cache,
}
if end_of_turn_probability is not None:
eou_span_attributes[trace_types.ATTR_EOU_PROBABILITY] = (
end_of_turn_probability
)
unlikely_threshold = await turn_detector.unlikely_threshold(
self._last_language
if unlikely_threshold is not None:
eou_span_attributes[trace_types.ATTR_EOU_UNLIKELY_THRESHOLD] = (
unlikely_threshold
)
eou_detection_span.set_attributes(eou_span_attributes)
logger.debug(
"eot prediction",
extra={
"probability": end_of_turn_probability,
"unlikely_threshold": unlikely_threshold,
"endpointing_delay": endpointing_delay,
"language": self._last_language or "",
"trigger": trigger,
"from_cache": from_cache,
},
)
if (
end_of_turn_probability is not None
and unlikely_threshold is not None
and (
prediction_event is None
or prediction_event is not self._last_emitted_prediction
)
):
self._last_emitted_prediction = prediction_event
inference_duration = (
prediction_event.inference_duration
if prediction_event is not None
and prediction_event.inference_duration is not None
else 0.0
)
# end of speech -> prediction receive time
delay = (
time.time() - last_speaking_time
if last_speaking_time is not None
else 0.0
)
self._hooks.on_eot_prediction(
EotPredictionEvent(
probability=end_of_turn_probability,
threshold=unlikely_threshold,
inference_duration=inference_duration,
delay=delay,
)
)
# surface the backchannel opportunity whenever it clears its
# threshold, regardless of end-of-turn; AgentActivity decides
# whether to acknowledge mid-turn or let it lead the reply
backchannel_probability = (
prediction_event.backchannel_probability
if prediction_event is not None
else None
)
if (
unlikely_threshold is not None
and end_of_turn_probability < unlikely_threshold
backchannel_probability is not None
and backchannel_threshold is not None
and backchannel_probability >= backchannel_threshold
):
endpointing_delay = self._endpointing.max_delay
except Exception:
logger.exception("Error predicting end of turn")
eou_detection_span.set_attributes(
{
trace_types.ATTR_CHAT_CTX: json.dumps(
llm.ChatContext(chat_ctx.items[-_EOU_MAX_HISTORY_TURNS:])
.copy(
exclude_function_call=True,
exclude_instructions=True,
exclude_empty_message=True,
exclude_handoff=True,
exclude_config_update=True,
self._hooks.on_agent_backchannel_opportunity(
_AgentBackchannelOpportunityEvent(
probability=backchannel_probability,
threshold=backchannel_threshold,
end_of_turn_probability=end_of_turn_probability,
end_of_turn_threshold=unlikely_threshold,
language=self._last_language,
)
.to_dict(
exclude_audio=True,
exclude_image=True,
exclude_timestamp=True,
exclude_metrics=True,
)
),
trace_types.ATTR_EOU_PROBABILITY: end_of_turn_probability,
trace_types.ATTR_EOU_UNLIKELY_THRESHOLD: unlikely_threshold or 0,
trace_types.ATTR_EOU_DELAY: endpointing_delay,
trace_types.ATTR_EOU_LANGUAGE: self._last_language or "",
}
)
)
if (
prediction_event is not None
and prediction_event.detection_delay is not None
):
eou_detection_span.set_attribute(
trace_types.ATTR_EOU_DETECTION_DELAY,
prediction_event.detection_delay,
)
extra_sleep = endpointing_delay
if last_speaking_time:
extra_sleep += last_speaking_time - time.time()
delay_completed = False
if extra_sleep > 0:
try:
await asyncio.wait_for(self._closing.wait(), timeout=extra_sleep)
except asyncio.TimeoutError:
delay_completed = True
pass
confidence_avg = (
@@ -1259,6 +1536,18 @@ class AudioRecognition:
)
)
if committed:
logger.debug(
"user turn committed",
extra={
"last_speaking_time": last_speaking_time,
"last_final_transcript_time": last_final_transcript_time,
"speech_start_time": speech_start_time,
"delay_completed": delay_completed,
"source": trigger,
"end_of_turn_probability": end_of_turn_probability,
"unlikely_threshold": unlikely_threshold,
},
)
user_turn_span.set_attributes(
{
trace_types.ATTR_USER_TRANSCRIPT: self._audio_transcript,
@@ -1287,15 +1576,64 @@ class AudioRecognition:
self._vad_speech_started = False
self._last_speaking_time = None
if self._turn_detector_stream is not None:
self._turn_detector_stream.flush(reason="turn committed")
self._turn_detector_prediction_fut = None
self._turn_detector_flushed = True
self._user_turn_committed = False
async def _bounce_eou_task_with_speaking_guard(
last_speaking_time: float | None = None,
last_final_transcript_time: float | None = None,
speech_start_time: float | None = None,
) -> None:
if self._speaking:
logger.debug(
"user is still speaking, skipping end of turn task",
extra={
"last_speaking_time": last_speaking_time,
"last_final_transcript_time": last_final_transcript_time,
"speech_start_time": speech_start_time,
},
)
return
tasks = [
(speaking_task := asyncio.create_task(self._user_speaking_event.wait())),
asyncio.create_task(
_bounce_eou_task(
last_speaking_time, last_final_transcript_time, speech_start_time
)
),
]
try:
done, _ = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
if speaking_task in done:
logger.debug(
"user spoke during endpointing, cancelling end of turn task",
extra={
"last_speaking_time": last_speaking_time,
"last_final_transcript_time": last_final_transcript_time,
"speech_start_time": speech_start_time,
},
)
return
finally:
await aio.cancel_and_wait(*tasks)
if self._end_of_turn_task is not None:
# TODO(theomonnom): disallow cancel if the extra sleep is done
self._end_of_turn_task.cancel()
task_func = (
_bounce_eou_task_with_speaking_guard
if isinstance(self._turn_detector, _StreamingTurnDetector)
else _bounce_eou_task
)
# copy the last_speaking_time before awaiting (the value can change)
self._end_of_turn_task = asyncio.create_task(
_bounce_eou_task(
task_func(
self._last_speaking_time,
self._last_final_transcript_time,
self._user_turn_start,
livekit-agents/livekit/agents/voice/events.py
+34 -1
View File
@@ -320,6 +320,39 @@ class UserInputTranscribedEvent(BaseModel):
created_at: float = Field(default_factory=time.time)
class EotPredictionEvent(BaseModel):
type: Literal["eot_prediction"] = "eot_prediction"
probability: float
threshold: float
inference_duration: float
"""Server-side model inference time."""
delay: float
"""End of user speech → prediction received latency (s), anchored on the
VAD-backdated last_speaking_time."""
created_at: float = Field(default_factory=time.time)
class _AgentBackchannelOpportunityEvent(BaseModel):
"""Internal: a window in which the agent could backchannel (a short
acknowledgment such as "mm-hmm"), as predicted by the turn detector. Passed to
``AgentActivity`` only — not surfaced as a public ``AgentSession`` event yet.
``AgentActivity`` owns the decision of what to do with it. The end-of-turn margin
(``end_of_turn_threshold - end_of_turn_probability``) gives a progressive risk axis:
a large positive margin means the user is clearly still going, so riskier
backchannels (yeah/okay/right) are safe; a small margin (or a negative one, where
``end_of_turn_probability >= end_of_turn_threshold`` and a reply is imminent) calls
for safe, less ambiguous ones (hmm/uh-huh) that won't collide with the reply."""
type: Literal["agent_backchannel_opportunity"] = "agent_backchannel_opportunity"
probability: float
threshold: float
end_of_turn_probability: float
end_of_turn_threshold: float
language: str | None = None
created_at: float = Field(default_factory=time.time)
class AgentFalseInterruptionEvent(BaseModel):
type: Literal["agent_false_interruption"] = "agent_false_interruption"
resumed: bool
@@ -443,7 +476,7 @@ class ErrorEvent(BaseModel):
@field_serializer("source")
def _serialize_source(self, source: Any) -> Any:
if isinstance(source, (LLM, STT, TTS, RealtimeModel, AdaptiveInterruptionDetector)):
if isinstance(source, LLM | STT | TTS | RealtimeModel | AdaptiveInterruptionDetector):
return {"model": source.model, "provider": source.provider}
if isinstance(source, BaseModel):
return source.model_dump()
livekit-agents/livekit/agents/voice/remote_session.py
+29 -1
View File
@@ -27,6 +27,7 @@ from ..llm import (
from ..log import logger
from ..metrics import (
AgentSessionUsage,
EOTModelUsage,
InterruptionModelUsage,
LLMModelUsage,
STTModelUsage,
@@ -38,6 +39,7 @@ from .events import (
AgentState,
AgentStateChangedEvent,
ConversationItemAddedEvent,
EotPredictionEvent,
ErrorEvent,
FunctionToolsExecutedEvent,
SessionUsageUpdatedEvent,
@@ -551,7 +553,23 @@ class SessionHost:
)
)
# TODO: @chenghao-mou add EOT prediction event
def _on_eot_prediction(self, event: EotPredictionEvent) -> None:
inference_duration = Duration()
inference_duration.FromNanoseconds(int(event.inference_duration * 1e9))
delay = Duration()
delay.FromNanoseconds(int(event.delay * 1e9))
self._send_event(
agent_pb.AgentSessionEvent(
eot_prediction=agent_pb.AgentSessionEvent.EotPrediction(
probability=event.probability,
threshold=event.threshold,
inference_duration=inference_duration,
delay=delay,
)
)
)
def _on_session_usage_updated(self, event: SessionUsageUpdatedEvent) -> None:
self._send_event(
@@ -894,6 +912,16 @@ def _session_usage_to_proto(usage: AgentSessionUsage) -> agent_pb.AgentSessionUs
)
)
)
elif isinstance(mu, EOTModelUsage):
model_usages.append(
agent_pb.ModelUsage(
eot=agent_pb.EotModelUsage(
provider=mu.provider,
model=mu.model,
total_requests=mu.total_requests,
)
)
)
return agent_pb.AgentSessionUsage(model_usage=model_usages)
livekit-agents/livekit/agents/voice/turn.py
+91 -7
View File
@@ -1,15 +1,38 @@
from __future__ import annotations
from typing import Literal, Protocol
import asyncio
from dataclasses import dataclass
from typing import Literal, Protocol, runtime_checkable
from typing_extensions import TypedDict
from livekit import rtc
from ..language import LanguageCode
from ..llm import ChatContext
from ..types import NOT_GIVEN, NotGivenOr
from ..types import (
DEFAULT_API_CONNECT_OPTIONS,
NOT_GIVEN,
APIConnectOptions,
NotGivenOr,
)
from ..utils import is_given
@dataclass
class TurnDetectionEvent:
type: Literal["eot_prediction"]
end_of_turn_probability: float
last_speaking_time: float
detection_delay: float | None = None
"""Latest input audio creation time -> prediction receive time."""
inference_duration: float | None = None
"""Server-side model inference time."""
backchannel_probability: float | None = None
"""How appropriate it is for the agent to backchannel at this pause.
``None`` when the detector does not produce one (e.g. the local mini model)."""
class _TurnDetector(Protocol):
@property
def model(self) -> str:
@@ -28,7 +51,48 @@ class _TurnDetector(Protocol):
) -> float: ...
TurnDetectionMode = Literal["stt", "vad", "realtime_llm", "manual"] | _TurnDetector
@runtime_checkable
class _StreamingTurnDetectorStream(Protocol):
"""I/O stream for the streaming turn detector."""
@property
def model(self) -> str: ...
@property
def provider(self) -> str: ...
@property
def is_fallback(self) -> bool: ...
async def unlikely_threshold(self, language: LanguageCode | None) -> float | None: ...
async def backchannel_threshold(self, language: LanguageCode | None) -> float | None: ...
async def supports_language(self, language: LanguageCode | None) -> bool: ...
def predict(self) -> asyncio.Future[TurnDetectionEvent]: ...
def cancel_inference(self, *, timed_out: bool = False) -> None: ...
def flush(self, reason: str | None = None) -> None: ...
def push_audio(self, frame: rtc.AudioFrame) -> None: ...
def end_input(self) -> None: ...
async def aclose(self) -> None: ...
@runtime_checkable
class _StreamingTurnDetector(Protocol):
"""Turn detector that processes streaming data."""
@property
def model(self) -> str: ...
@property
def provider(self) -> str: ...
def stream(
self,
*,
conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS,
) -> _StreamingTurnDetectorStream: ...
TurnDetectionMode = (
Literal["stt", "vad", "realtime_llm", "manual"] | _TurnDetector | _StreamingTurnDetector
)
"""
The mode of turn detection to use.
@@ -73,6 +137,13 @@ _ENDPOINTING_DEFAULTS: EndpointingOptions = {
"alpha": 0.9,
}
_STREAMING_ENDPOINTING_DEFAULTS: EndpointingOptions = {
"mode": "fixed",
"min_delay": 0.3,
"max_delay": 2.5,
"alpha": 0.9,
}
class InterruptionOptions(TypedDict, total=False):
"""Configuration for interruption handling.
@@ -222,11 +293,24 @@ def _resolve_preemptive_generation(
return PreemptiveGenerationOptions(**{**_PREEMPTIVE_GENERATION_DEFAULTS, **config})
def _resolve_endpointing(config: EndpointingOptions | None = None) -> EndpointingOptions:
"""Fill in defaults for missing keys."""
def _resolve_endpointing(
config: EndpointingOptions | None = None,
*,
turn_detection: TurnDetectionMode | None = None,
) -> EndpointingOptions:
"""Fill in defaults for missing keys.
When ``turn_detection`` is a streaming turn detector, keys the caller did
not provide fall back to the tighter streaming defaults instead of the
legacy ones."""
base = (
_STREAMING_ENDPOINTING_DEFAULTS
if isinstance(turn_detection, _StreamingTurnDetector)
else _ENDPOINTING_DEFAULTS
)
if config is None:
return EndpointingOptions(**_ENDPOINTING_DEFAULTS)
return EndpointingOptions(**{**_ENDPOINTING_DEFAULTS, **config})
return EndpointingOptions(**base)
return EndpointingOptions(**{**base, **config})
def _resolve_interruption(
livekit-agents/livekit/agents/worker.py
+8 -1
View File
@@ -741,7 +741,14 @@ class AgentServer(utils.EventEmitter[EventTypes]):
)
if self._mp_ctx_str == "forkserver":
plugin_packages = [p.package for p in Plugin.registered_plugins] + ["av"]
# `livekit.agents.inference._warmup` is a side-effect module:
# importing it from the forkserver process calls `init_vad()` and
# `init_eot()`, paging the native model weights into the
# forkserver. Forked job processes inherit those pages via COW.
plugin_packages = [p.package for p in Plugin.registered_plugins] + [
"av",
"livekit.agents.inference._warmup",
]
logger.info("preloading plugins", extra={"packages": plugin_packages})
self._mp_ctx.set_forkserver_preload(plugin_packages)
livekit-agents/pyproject.toml
+3 -2
View File
@@ -29,7 +29,8 @@ dependencies = [
"certifi>=2025.6.15",
"livekit==1.1.9",
"livekit-api>=1.0.7,<2",
"livekit-protocol>=1.1.15,<2",
"livekit-local-inference>=0.2.5",
"livekit-protocol>=1.1.17,<2",
"livekit-blingfire~=1.1,<2",
"protobuf>=3",
"pyjwt>=2.0",
@@ -56,7 +57,7 @@ dependencies = [
"typer>=0.15.1",
"click~=8.1",
"sounddevice>=0.5",
"watchfiles>=1.0"
"watchfiles>=1.0",
]
[project.optional-dependencies]
livekit-plugins/livekit-plugins-aws/README.md
+3 -3
View File
@@ -90,7 +90,6 @@ The `realtime_joke_teller.py` example demonstrates both realtime and pipeline mo
1. **Install dependencies:**
```bash
pip install livekit-plugins-aws[realtime] \
livekit-plugins-silero \
jokeapi \
duckduckgo-search \
python-weather \
@@ -331,13 +330,14 @@ if __name__ == "__main__":
For more control over individual components, use pipeline mode:
```python
from livekit.plugins import aws, silero
from livekit.agents import inference
from livekit.plugins import aws
session = AgentSession(
stt=aws.STT(), # Amazon Transcribe
llm=aws.LLM(), # Nova 2 Lite (default)
tts=aws.TTS(), # Amazon Polly
vad=silero.VAD.load(),
vad=inference.VAD(),
)
```
livekit-plugins/livekit-plugins-baseten/README.md
+5 -5
View File
@@ -82,9 +82,9 @@ stt = baseten.STT(
```python
import os
from livekit import agents
from livekit.agents import AgentSession, Agent, RoomInputOptions
from livekit.plugins import baseten, openai, noise_cancellation, silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
from livekit.agents import AgentSession, Agent, RoomInputOptions, inference
from livekit.plugins import baseten, openai, noise_cancellation
from livekit.agents.inference import TurnDetector
BASETEN_API_KEY = os.getenv("BASETEN_API_KEY")
whisper_model_id = "your-whisper-model-id" # or use chain_id for chain deployments
@@ -116,8 +116,8 @@ async def entrypoint(ctx: agents.JobContext):
".api.baseten.co/environments/production/predict"
),
),
vad=silero.VAD.load(),
turn_detection=MultilingualModel(),
vad=inference.VAD(),
turn_detection=TurnDetector(),
)
await session.start(
livekit-plugins/livekit-plugins-inworld/README.md
+5 -12
View File
@@ -127,13 +127,13 @@ from livekit.agents import (
AgentServer,
AgentSession,
JobContext,
JobProcess,
cli,
inference,
metrics,
room_io,
)
from livekit.plugins import inworld, silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
from livekit.agents.inference import TurnDetector
from livekit.plugins import inworld
logger = logging.getLogger("inworld-agent")
@@ -158,13 +158,6 @@ class InworldAgent(Agent):
server = AgentServer()
def prewarm(proc: JobProcess):
proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm
@server.rtc_session()
async def entrypoint(ctx: JobContext):
ctx.log_context_fields = {"room": ctx.room.name}
@@ -173,8 +166,8 @@ async def entrypoint(ctx: JobContext):
stt=inworld.STT(model="inworld/inworld-stt-1"),
llm="openai/gpt-4.1-mini",
tts=inworld.TTS(voice="Clive"),
turn_detection=MultilingualModel(),
vad=ctx.proc.userdata["vad"],
turn_detection=TurnDetector(),
vad=inference.VAD(),
)
usage_collector = metrics.UsageCollector()
livekit-plugins/livekit-plugins-krisp/README.md
+3 -3
View File
@@ -43,8 +43,8 @@ It must be obtained and installed separately from Krisp (https://krisp.ai/develo
For cleaning up user audio before STT/VAD processing using the FrameProcessor approach:
```python
from livekit.agents import AgentSession, Agent, JobContext, room_io
from livekit.plugins import krisp, silero, openai
from livekit.agents import AgentSession, Agent, JobContext, inference, room_io
from livekit.plugins import krisp, openai
@server.rtc_session()
async def entrypoint(ctx: JobContext):
@@ -56,7 +56,7 @@ async def entrypoint(ctx: JobContext):
)
session = AgentSession(
vad=silero.VAD.load(),
vad=inference.VAD(),
stt=openai.STT(),
llm=openai.LLM(model="gpt-4o-mini"),
tts=openai.TTS(),
livekit-plugins/livekit-plugins-krisp/examples/krisp_agent_example.py
+3 -3
View File
@@ -13,7 +13,6 @@ Prerequisites:
2. Install required packages:
- livekit-agents (with PR #4145 support for FrameProcessor)
- livekit-plugins-krisp
- livekit-plugins-silero (for VAD)
- livekit-plugins-openai (or your preferred STT/LLM/TTS)
Usage:
@@ -30,9 +29,10 @@ from livekit.agents import (
AgentSession,
JobContext,
cli,
inference,
room_io,
)
from livekit.plugins import krisp, openai, silero
from livekit.plugins import krisp, openai
logger = logging.getLogger("krisp-agent-example")
load_dotenv()
@@ -66,7 +66,7 @@ async def entrypoint(ctx: JobContext):
# Configure the agent session
session = AgentSession(
vad=silero.VAD.load(),
vad=inference.VAD(),
stt=openai.STT(model="whisper-1"),
llm=openai.LLM(model="gpt-4o-mini"),
tts=openai.TTS(voice="alloy"),
livekit-plugins/livekit-plugins-silero/livekit/plugins/silero/__init__.py
+12
View File
@@ -17,6 +17,8 @@
See https://docs.livekit.io/agents/build/turns/vad/ for more information.
"""
import warnings
from .vad import VAD, VADStream
from .version import __version__
@@ -26,6 +28,16 @@ from livekit.agents import Plugin
from .log import logger
warnings.warn(
"livekit-plugins-silero is deprecated and will be removed in v2.0. "
"AgentSession now defaults to the bundled silero VAD, so you can drop the "
"explicit `vad=` argument entirely; pass `vad=None` to opt out, or use "
'`from livekit.agents import inference; inference.VAD(model="silero", ...)`'
" to customise options.",
DeprecationWarning,
stacklevel=2,
)
class SileroPlugin(Plugin):
def __init__(self) -> None:
livekit-plugins/livekit-plugins-silero/livekit/plugins/silero/vad.py
+31 -1
View File
@@ -71,7 +71,7 @@ class VAD(agents.vad.VAD):
deactivation_threshold: NotGivenOr[float] = NOT_GIVEN,
# deprecated
padding_duration: NotGivenOr[float] = NOT_GIVEN,
) -> VAD:
) -> agents.vad.VAD:
"""
Load and initialize the Silero VAD model.
@@ -128,6 +128,32 @@ class VAD(agents.vad.VAD):
if is_given(deactivation_threshold) and deactivation_threshold <= 0:
raise ValueError("deactivation_threshold must be greater than 0")
# When the requested settings are compatible with the native
# implementation in `livekit-local-inference`, delegate to the new
# `livekit.agents.inference.VAD(model="silero")` so users get the
# COW-shared / faster path without changing their call sites. The
# native lib only supports 16 kHz with the bundled model file, so
# custom sample rate or `onnx_file_path` falls back to the legacy
# onnxruntime path.
if sample_rate == 16000 and not is_given(onnx_file_path):
if not force_cpu:
logger.warning(
"force_cpu=False is ignored when using the bundled native "
"VAD; the model runs CPU-only. Pass `onnx_file_path=...` "
"to keep the legacy onnxruntime path that honors force_cpu."
)
from livekit.agents import inference
return inference.VAD(
model="silero",
min_speech_duration=min_speech_duration,
min_silence_duration=min_silence_duration,
prefix_padding_duration=prefix_padding_duration,
max_buffered_speech=max_buffered_speech,
activation_threshold=activation_threshold,
deactivation_threshold=deactivation_threshold,
)
session = onnx_model.new_inference_session(force_cpu, onnx_file_path=onnx_file_path or None)
opts = _VADOptions(
min_speech_duration=min_speech_duration,
@@ -221,6 +247,10 @@ class VAD(agents.vad.VAD):
deactivation_threshold=deactivation_threshold,
)
@property
def min_silence_duration(self) -> float | None:
return self._opts.min_silence_duration
class VADStream(agents.vad.VADStream):
def __init__(self, vad: VAD, opts: _VADOptions, model: onnx_model.OnnxModel) -> None:
livekit-plugins/livekit-plugins-speechmatics/README.md
+5 -5
View File
@@ -37,9 +37,9 @@ The `end_of_utterance_silence_trigger` parameter controls the amount of silence
Usage:
```python
from livekit.agents import AgentSession
from livekit.plugins.turn_detector.multilingual import MultilingualModel
from livekit.plugins import speechmatics, silero
from livekit.agents import AgentSession, inference
from livekit.agents.inference import TurnDetector
from livekit.plugins import speechmatics
agent = AgentSession(
stt=speechmatics.STT(
@@ -47,8 +47,8 @@ agent = AgentSession(
speaker_active_format="[Speaker {speaker_id}] {text}",
speaker_passive_format="[Speaker {speaker_id} *PASSIVE*] {text}",
),
vad=silero.VAD.load(),
turn_detection=MultilingualModel(),
vad=inference.VAD(),
turn_detection=TurnDetector(),
min_endpointing_delay=0.3,
max_endpointing_delay=5.0,
...
livekit-plugins/livekit-plugins-turn-detector/.gitignore
+5
View File
@@ -0,0 +1,5 @@
# Setuptools build outputs
/build/
*.egg-info/
*.egg
dist/
livekit-plugins/livekit-plugins-turn-detector/README.md
+7 -16
View File
@@ -1,5 +1,7 @@
# Turn detector plugin for LiveKit Agents
> ⚠️ **Deprecated.** This plugin is deprecated and will be removed in a future release. Use [`livekit.agents.inference.TurnDetector`](https://docs.livekit.io/agents/build/turns/turn-detector/) instead — it ships with `livekit-agents`, requires no additional install, and replaces both the English and Multilingual text-based models with a unified audio end-of-turn detector.
This plugin introduces end-of-turn detection for LiveKit Agents using a custom open-weight model to determine when a user has finished speaking.
Traditional voice agents use VAD (voice activity detection) for end-of-turn detection. However, VAD models lack language understanding, often causing false positives where the agent interrupts the user before they finish speaking.
@@ -8,39 +10,28 @@ By leveraging a language model specifically trained for this task, this plugin o
See [https://docs.livekit.io/agents/build/turns/turn-detector/](https://docs.livekit.io/agents/build/turns/turn-detector/) for more information.
## Installation
```bash
pip install livekit-plugins-turn-detector
```
## Usage
### Multilingual model
We've trained a multilingual model that supports the following languages: `English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Indonesian, Russian, Turkish, Hindi`
The multilingual model requires ~400MB of RAM and completes inferences in ~25ms.
The recommended replacement is `TurnDetector`, available from `livekit-agents` directly:
```python
from livekit.plugins.turn_detector.multilingual import MultilingualModel
from livekit.agents.inference import TurnDetector
session = AgentSession(
...
turn_detection=MultilingualModel(),
turn_detection=TurnDetector(),
)
```
### Usage with RealtimeModel
The turn detector can be used even with speech-to-speech models such as OpenAI's Realtime API. You'll need to provide a separate STT to ensure our model has access to the text content.
`TurnDetector` works with speech-to-speech models such as OpenAI's Realtime API:
```python
session = AgentSession(
...
stt=deepgram.STT(model="nova-3", language="multi"),
llm=openai.realtime.RealtimeModel(),
turn_detection=MultilingualModel(),
turn_detection=TurnDetector(),
)
```
livekit-plugins/livekit-plugins-turn-detector/livekit/plugins/turn_detector/__init__.py
+13
View File
@@ -15,8 +15,14 @@
"""Contextually-aware turn detection for LiveKit Agents
See https://docs.livekit.io/agents/build/turns/turn-detector/ for more information.
.. deprecated::
This plugin is deprecated and will be removed in a future release. Use
``livekit.agents.inference.TurnDetector`` instead.
"""
import warnings
from livekit.agents import Plugin
from .base import EOUPlugin
@@ -26,6 +32,13 @@ from .version import __version__
__all__ = ["english", "multilingual", "__version__"]
warnings.warn(
"`livekit.plugins.turn_detector` is deprecated and will be removed in a "
"future release. Use `livekit.agents.inference.TurnDetector` instead.",
DeprecationWarning,
stacklevel=2,
)
Plugin.register_plugin(EOUPlugin(_EUORunnerEn))
Plugin.register_plugin(EOUPlugin(_EUORunnerMultilingual))
livekit-plugins/livekit-plugins-turn-detector/pyproject.toml
+2 -2
View File
@@ -13,7 +13,6 @@ authors = [{ name = "LiveKit", email = "hello@livekit.io" }]
keywords = ["voice", "ai", "realtime", "audio", "video", "livekit", "webrtc"]
classifiers = [
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Topic :: Multimedia :: Sound/Audio",
"Topic :: Multimedia :: Video",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
@@ -25,7 +24,8 @@ dependencies = [
"livekit-agents>=1.6.0",
"transformers>=4.47.1,!=4.57.2,!=4.57.3", # 4.57.2-4.57.3 have a bug with local_files_only=True (huggingface/transformers#42369)
"numpy>=1.26",
"onnxruntime>=1.18",
"onnxruntime>=1.18; python_version >= '3.11'",
"onnxruntime>=1.18,<1.24; python_version < '3.11'",
"jinja2",
]
livekit-plugins/livekit-plugins-ultravox/README.md
+6 -14
View File
@@ -30,8 +30,7 @@ export LK_ULTRAVOX_DEBUG=true
```python
import asyncio
from livekit.agents import Agent, AgentSession, JobContext, JobProcess, WorkerOptions, cli
from livekit.plugins import silero
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, inference
from livekit.plugins.ultravox.realtime import RealtimeModel
async def entrypoint(ctx: JobContext):
@@ -39,7 +38,7 @@ async def entrypoint(ctx: JobContext):
session: AgentSession[None] = AgentSession(
allow_interruptions=True,
vad=ctx.proc.userdata["vad"],
vad=inference.VAD(),
llm=RealtimeModel(
model_id="fixie-ai/ultravox",
voice="Mark",
@@ -53,18 +52,14 @@ async def entrypoint(ctx: JobContext):
room=ctx.room,
)
def prewarm(proc: JobProcess) -> None:
proc.userdata["vad"] = silero.VAD.load()
if __name__ == "__main__":
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm))
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
```
### Voice Assistant with Tools
```python
from livekit.agents import function_tool, Agent, AgentSession, JobContext, JobProcess, WorkerOptions, cli
from livekit.plugins import silero
from livekit.agents import function_tool, Agent, AgentSession, JobContext, WorkerOptions, cli, inference
from livekit.plugins.ultravox.realtime import RealtimeModel
@function_tool
@@ -82,7 +77,7 @@ async def entrypoint(ctx: JobContext):
session: AgentSession[None] = AgentSession(
allow_interruptions=True,
vad=ctx.proc.userdata["vad"],
vad=inference.VAD(),
llm=RealtimeModel(model_id="fixie-ai/ultravox"),
)
@@ -94,11 +89,8 @@ async def entrypoint(ctx: JobContext):
room=ctx.room,
)
def prewarm(proc: JobProcess) -> None:
proc.userdata["vad"] = silero.VAD.load()
if __name__ == "__main__":
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm))
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
```
makefile
+1 -1
View File
@@ -324,4 +324,4 @@ doctor: ## Check development environment health
else \
echo "$(BOLD)$(RED)⚠️ Found $$ISSUES issue(s). Please fix the errors above.$(RESET)"; \
exit 1; \
fi
fi
pyproject.toml
+6
View File
@@ -163,6 +163,7 @@ pythonpath = ["."]
testpaths = ["tests"]
markers = [
"unit: fast tests with no external providers (select with --unit)",
"audio_eot: hermetic audio end-of-turn / turn-detection tests (select with --audio_eot)",
"plugin(name): provider integration test for `name` (select with --plugin [name])",
"realtime(name): realtime-model test, optionally for provider `name` (--realtime [name])",
"stt(name): speech-to-text test, optionally for provider `name` (--stt [name])",
@@ -223,6 +224,11 @@ ignore_missing_imports = true
module = "speechmatics.*"
follow_untyped_imports = true
[[tool.mypy.overrides]]
module = "bithuman.*"
follow_untyped_imports = true
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = "livekit.plugins"
follow_untyped_imports = true
tests/fake_session.py
+5
View File
@@ -51,6 +51,11 @@ def create_session(
)
# allowing overriding default endpointing and interruption options
turn_handling = turn_handling or {}
# Use VAD-based endpointing by default. The AgentSession default is the
# turn-detector-v1-mini model; it runs locally but predicts end-of-turn from
# acoustic features, so it can't fire deterministically on synthetic test
# audio. Model accuracy is covered by the audio_eot suite instead.
turn_handling.setdefault("turn_detection", None)
turn_handling["endpointing"] = EndpointingOptions(
**{**default_endpointing, **turn_handling.get("endpointing", {})}
)
tests/fake_turn_detector_ws.py
+161
View File
@@ -0,0 +1,161 @@
"""In-process fakes for testing the cloud transport without aiohttp.
Used by cloud-transport tests to drive ``_CloudTransport.run`` deterministically:
- ``FakeTurnDetectorWS`` impersonates an ``aiohttp.ClientWebSocketResponse``.
Captures outbound ``send_bytes`` payloads as parsed ``ClientMessage``
protobufs, and yields scripted server frames from ``receive()``.
- ``ControlledCloudTransport`` overrides ``_connect_ws`` so each connect
attempt is scripted: an exception is raised, ``None`` returns the fake ws.
- ``make_stream(...)`` constructs an ``_BaseStreamingTurnDetectorStream`` in cloud
mode bound to a controlled transport.
- ``wait_until_connected(...)`` blocks until the transport's ``_ws`` is set,
so tests can assert against post-connect state without arbitrary sleeps.
"""
from __future__ import annotations
import asyncio
from typing import Any
from unittest.mock import MagicMock
import aiohttp
from livekit.agents.inference.eot.base import (
TurnDetectorOptions,
_BaseStreamingTurnDetectorStream,
)
from livekit.agents.inference.eot.languages import ThresholdOptions
from livekit.agents.inference.eot.transports import _CloudTransport, _CloudTransportOptions
from livekit.agents.types import APIConnectOptions
from livekit.protocol.agent_pb.agent_inference import ClientMessage
class FakeTurnDetectorWS:
"""Fake ``aiohttp.ClientWebSocketResponse`` for cloud-transport tests."""
def __init__(self) -> None:
self.sent: list[ClientMessage] = []
self._incoming: asyncio.Queue[aiohttp.WSMessage] = asyncio.Queue()
self.closed = False
self.close_code: int | None = None
async def send_bytes(self, data: bytes) -> None:
if self.closed:
raise ConnectionResetError("ws closed")
msg = ClientMessage()
msg.ParseFromString(data)
self.sent.append(msg)
async def receive(self) -> aiohttp.WSMessage:
return await self._incoming.get()
async def close(self) -> None:
self.closed = True
def feed_unexpected_close(self, code: int = 1006) -> None:
"""Simulate a server-initiated drop. ``recv_task`` will raise."""
self.close_code = code
self._incoming.put_nowait(
aiohttp.WSMessage(type=aiohttp.WSMsgType.CLOSE, data=code, extra=None)
)
self.closed = True
class ControlledCloudTransport(_CloudTransport):
"""``_CloudTransport`` subclass that scripts the ``_connect_ws`` outcome.
``connect_script`` is consumed left-to-right. An exception is raised; a
``None`` returns the fake ws. Once exhausted, subsequent calls return
the fake ws.
"""
def __init__(
self,
*,
fake_ws: FakeTurnDetectorWS,
connect_script: list[BaseException | None] | None = None,
**kwargs: Any,
) -> None:
self._fake_ws = fake_ws
self._connect_script: list[BaseException | None] = list(connect_script or [])
self._connect_calls = 0
super().__init__(**kwargs)
async def _connect_ws(self) -> aiohttp.ClientWebSocketResponse:
self._connect_calls += 1
if self._connect_script:
r = self._connect_script.pop(0)
if isinstance(r, BaseException):
raise r
return self._fake_ws # type: ignore[return-value]
def make_stream(
*,
fake_ws: FakeTurnDetectorWS | None = None,
connect_script: list[BaseException | None] | None = None,
max_retry: int = 3,
retry_interval: float = 0.0,
) -> tuple[_BaseStreamingTurnDetectorStream, FakeTurnDetectorWS, ControlledCloudTransport]:
"""Construct a cloud-mode stream with a controlled transport.
Returns the stream, the fake ws, and the transport so callers can read
post-connect state from either side.
"""
fake_ws = fake_ws or FakeTurnDetectorWS()
detector = MagicMock()
detector.model = "turn-detector-v1"
detector.provider = "livekit"
session_mock = MagicMock()
session_mock.closed = False
opts = TurnDetectorOptions(sample_rate=16000, thresholds=ThresholdOptions("turn-detector-v1"))
conn_options = APIConnectOptions(max_retry=max_retry, retry_interval=retry_interval)
cloud_opts = _CloudTransportOptions(
base_url="",
api_key="x",
api_secret="x",
conn_options=conn_options,
)
transport = ControlledCloudTransport(
fake_ws=fake_ws,
connect_script=connect_script,
detector=detector,
opts=opts,
cloud_opts=cloud_opts,
http_session=session_mock,
)
stream = _BaseStreamingTurnDetectorStream(
detector=detector,
opts=opts,
transport=transport,
model="turn-detector-v1",
)
return stream, fake_ws, transport
async def wait_until_connected(
transport: ControlledCloudTransport, *, timeout: float = 1.0
) -> None:
"""Block until ``transport._ws`` is set."""
loop = asyncio.get_event_loop()
deadline = loop.time() + timeout
while transport._ws is None:
if loop.time() > deadline:
raise TimeoutError("transport did not connect within timeout")
await asyncio.sleep(0)
async def drain_send_queue(transport: ControlledCloudTransport, *, timeout: float = 1.0) -> None:
"""Yield until the outbound channel is empty (sender task has drained it)."""
loop = asyncio.get_event_loop()
deadline = loop.time() + timeout
while True:
ch = transport._send_ch
if ch is None or ch.qsize() == 0:
# Give the sender task one more tick to flush whatever it was awaiting.
await asyncio.sleep(0)
return
if loop.time() > deadline:
raise TimeoutError("send queue did not drain within timeout")
await asyncio.sleep(0)
tests/test_agent_session.py
+183
View File
@@ -737,6 +737,7 @@ async def test_backchannel_boundary_suppresses_start_boundary_backchannel() -> N
endpointing=BaseEndpointing(min_delay=0.1, max_delay=1.0),
stt=None,
vad=None,
using_default_vad=False,
interruption_detection=None,
turn_detection="vad",
)
@@ -768,6 +769,7 @@ async def _make_stt_eos_recognition() -> AudioRecognition:
endpointing=BaseEndpointing(min_delay=0.0, max_delay=0.0),
stt=None,
vad=None,
using_default_vad=False,
interruption_detection=None,
turn_detection="stt",
)
@@ -824,6 +826,7 @@ async def test_backchannel_boundary_releases_end_boundary_transcript() -> None:
endpointing=BaseEndpointing(min_delay=0.1, max_delay=1.0),
stt=None,
vad=None,
using_default_vad=False,
interruption_detection=None,
turn_detection="vad",
)
@@ -962,6 +965,7 @@ async def test_force_flush_held_transcripts_emits_buffered_events() -> None:
endpointing=BaseEndpointing(min_delay=0.1, max_delay=1.0),
stt=None,
vad=None,
using_default_vad=False,
interruption_detection=None,
turn_detection="manual",
)
@@ -1447,6 +1451,185 @@ async def test_silent_tool_call_pause_state_does_not_leak_into_tool_reply() -> N
assert false_interruption_events[-1].resumed is True
async def test_default_vad_is_auto_provisioned() -> None:
from livekit.agents.voice.agent_session import AgentSession
session = AgentSession()
try:
assert session.vad is not None
assert session._using_default_vad is True
finally:
await session.aclose()
async def test_explicit_vad_none_opts_out() -> None:
from livekit.agents.voice.agent_session import AgentSession
session = AgentSession(vad=None)
try:
assert session.vad is None
assert session._using_default_vad is False
finally:
await session.aclose()
async def test_user_supplied_vad_clears_default_flag() -> None:
from livekit.agents.voice.agent_session import AgentSession
from .fake_vad import FakeVAD
user_vad = FakeVAD(fake_user_speeches=[])
session = AgentSession(vad=user_vad)
try:
assert session.vad is user_vad
assert session._using_default_vad is False
finally:
await session.aclose()
async def test_default_turn_detection_builds_default_eot() -> None:
"""No turn_detection given → session auto-provisions a default TurnDetector."""
from livekit.agents.voice.agent_session import AgentSession
from livekit.agents.voice.turn import _StreamingTurnDetector
session = AgentSession()
try:
assert isinstance(session.turn_detection, _StreamingTurnDetector)
finally:
await session.aclose()
async def test_turn_detection_none_opts_out() -> None:
"""Explicit None opts out of turn detection (no default detector built)."""
from livekit.agents.voice.agent_session import AgentSession
session = AgentSession(turn_handling={"turn_detection": None})
try:
assert session.turn_detection is None
finally:
await session.aclose()
async def test_user_supplied_turn_detector_passes_through() -> None:
from livekit.agents import inference
from livekit.agents.voice.agent_session import AgentSession
user_detector = inference.TurnDetector(version="v1-mini")
session = AgentSession(turn_handling={"turn_detection": user_detector})
try:
assert session.turn_detection is user_detector
finally:
await session.aclose()
async def test_streaming_detector_uses_streaming_endpointing_defaults() -> None:
"""Default session → streaming detector → tighter 0.3/2.5 endpointing defaults."""
from livekit.agents.voice.agent_session import AgentSession
session = AgentSession()
try:
assert session._opts.endpointing["min_delay"] == 0.3
assert session._opts.endpointing["max_delay"] == 2.5
assert session._opts.endpointing_overrides == {}
finally:
await session.aclose()
async def test_non_streaming_detector_uses_legacy_endpointing_defaults() -> None:
"""A non-streaming mode keeps the legacy 0.5/3.0 defaults."""
from livekit.agents.voice.agent_session import AgentSession
session = AgentSession(turn_handling={"turn_detection": "vad"})
try:
assert session._opts.endpointing["min_delay"] == 0.5
assert session._opts.endpointing["max_delay"] == 3.0
finally:
await session.aclose()
async def test_explicit_endpointing_overrides_streaming_default_per_key() -> None:
"""An explicit delay is honored; the unset one still gets the streaming default."""
from livekit.agents.voice.agent_session import AgentSession
session = AgentSession(turn_handling={"endpointing": {"min_delay": 0.4}})
try:
assert session._opts.endpointing["min_delay"] == 0.4
assert session._opts.endpointing["max_delay"] == 2.5
assert session._opts.endpointing_overrides == {"min_delay": 0.4}
finally:
await session.aclose()
async def test_user_streaming_detector_uses_streaming_defaults() -> None:
"""A user-constructed streaming detector also triggers the streaming defaults."""
from livekit.agents import inference
from livekit.agents.voice.agent_session import AgentSession
session = AgentSession(
turn_handling={"turn_detection": inference.TurnDetector(version="v1-mini")}
)
try:
assert session._opts.endpointing["min_delay"] == 0.3
assert session._opts.endpointing["max_delay"] == 2.5
finally:
await session.aclose()
async def test_deprecated_turn_detection_vad_uses_legacy_defaults() -> None:
"""Deprecated turn_detection arg + no delays → legacy defaults (non-streaming)."""
from livekit.agents.voice.agent_session import AgentSession
session = AgentSession(turn_detection="vad")
try:
assert session._opts.endpointing["min_delay"] == 0.5
assert session._opts.endpointing["max_delay"] == 3.0
finally:
await session.aclose()
async def test_agent_turn_detection_override_resolves_endpointing_per_activity() -> None:
"""endpointing_opts uses the activity's resolved detector, not just the session's."""
from livekit.agents.voice.agent_session import AgentSession
from .fake_vad import FakeVAD
# session default → streaming detector; provide VAD so it validates
session = AgentSession(vad=FakeVAD(fake_user_speeches=[]))
try:
streaming_activity = AgentActivity(Agent(instructions="test"), session)
assert streaming_activity.endpointing_opts["min_delay"] == 0.3
assert streaming_activity.endpointing_opts["max_delay"] == 2.5
# an agent overriding to VAD falls back to legacy defaults for this activity
vad_activity = AgentActivity(Agent(instructions="test", turn_detection="vad"), session)
assert vad_activity.endpointing_opts["min_delay"] == 0.5
assert vad_activity.endpointing_opts["max_delay"] == 3.0
finally:
await session.aclose()
async def test_runtime_endpointing_opts_survive_handoff() -> None:
"""update_options changes are recorded as overrides, so a new activity keeps them."""
from livekit.agents.voice.agent_session import AgentSession
from .fake_vad import FakeVAD
session = AgentSession(vad=FakeVAD(fake_user_speeches=[]))
try:
session.update_options(endpointing_opts={"mode": "dynamic", "alpha": 0.5, "min_delay": 0.4})
# a fresh activity (as built on agent handoff) re-resolves from overrides
activity = AgentActivity(Agent(instructions="test"), session)
assert activity.endpointing_opts["mode"] == "dynamic"
assert activity.endpointing_opts["alpha"] == 0.5
assert activity.endpointing_opts["min_delay"] == 0.4
# untouched key still gets the streaming default
assert activity.endpointing_opts["max_delay"] == 2.5
finally:
await session.aclose()
class FlushMultiSegmentAgent(Agent):
"""Agent whose llm_node flushes the reply into two segments via FlushSentinel."""

Some files were not shown because too many files have changed in this diff Show More