How livekit/agents works
Purpose and Runtime
This is the Python LiveKit Agents framework: server-side, realtime programmable participants for conversational voice and multimodal sessions. It is a Python/asyncio monorepo containing the core SDK, separately packaged provider plugins, native extensions, and runnable examples. Agents connect to LiveKit rooms over the LiveKit Python SDK; deployments use LiveKit Cloud or self-hosted LiveKit infrastructure, with optional telephony and data/RPC integration.
Core package: livekit-agents/livekit/agents/__init__.py, livekit-agents/pyproject.toml.
Application Entry and Job Lifecycle
An application constructs an AgentServer, decorates a room-session entrypoint, and invokes the CLI. The CLI supports local console execution, hot-reloading development servers, and production worker operation. The worker/server layer receives or schedules jobs, connects an agent process to a LiveKit room, and invokes user entrypoints with job context.
Paths: livekit-agents/livekit/agents/cli/cli.py, livekit-agents/livekit/agents/worker.py, livekit-agents/livekit/agents/job.py.
Session Orchestration and Agent Control
AgentSession is the central runtime coordinator: it owns the active agent, conversation state, model configuration, user data, and turn/run lifecycles. Agents supply instructions, tools, lifecycle hooks, and can hand off to another agent. Tool execution and speech/run handles form the boundary between LLM decisions and asynchronous side effects.
Paths: livekit-agents/livekit/agents/voice/agent_session.py, livekit-agents/livekit/agents/voice/agent.py, livekit-agents/livekit/agents/voice/tool_executor.py.
Realtime Media and Turn Pipeline
Room I/O bridges LiveKit participant audio/data with the session. Incoming audio flows through recognition, VAD/endpointing, STT, and turn detection; final user turns drive LLM or realtime-model generation. Output text streams through TTS and playout back to the room. Interruption, filler speech, transcription synchronization, DTMF/IVR, recording, and avatar paths are session-adjacent specializations.
Paths: livekit-agents/livekit/agents/voice/room_io/room_io.py, livekit-agents/livekit/agents/voice/audio_recognition.py, livekit-agents/livekit/agents/voice/generation.py.
Model Abstractions and Inference Boundary
The core exposes provider-neutral streaming contracts for STT, TTS, VAD, LLM/chat, and duplex realtime models, including fallback and stream adapters. inference provides LiveKit-hosted model access; external providers are implemented outside the core package. LLM context/tool schemas normalize provider formats, while realtime adapters combine speech and generation into a different control path from the staged STT→LLM→TTS pipeline.
Paths: livekit-agents/livekit/agents/stt/stt.py, livekit-agents/livekit/agents/tts/tts.py, livekit-agents/livekit/agents/llm/llm.py.
Provider Plugin Ecosystem
Each livekit-plugins/livekit-plugins-* subtree is an independently packaged adapter for external AI, avatar, browser, audio-processing, or workflow services. Plugins map vendor APIs and streaming protocols into core interfaces rather than coupling vendors to session orchestration. Representative integrations include OpenAI realtime/chat/STT/TTS, Deepgram STT/TTS, Google models, AWS, and avatar APIs.
Paths: livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py, livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/stt.py, livekit-plugins/livekit-plugins-silero/livekit/plugins/silero/vad.py.
Isolation, Observability, and Evaluation
Workers can isolate jobs and inference in supervised processes/threads using IPC pools, separating failures and expensive model execution from the coordinator. Metrics, OpenTelemetry traces, PII handling, and HTTP/RPC telemetry observe sessions and provider calls. Built-in evaluation primitives support deterministic event assertions plus LLM judging; examples demonstrate complete vertical agent implementations.
Paths: livekit-agents/livekit/agents/ipc/proc_pool.py, livekit-agents/livekit/agents/telemetry/traces.py, livekit-agents/livekit/agents/evals/evaluation.py.
Main entry points: agent_session.py AgentSession · llm.py Model contracts · worker.py AgentServer worker · room_io.py Room I/O
Components 20
Core SDK
Async runtime, session orchestration, and model contracts.
- CLI · Runs console, development, and production worker modes.
livekit-agents/livekit/agents/cli/cli.py - AgentServer worker · Schedules jobs and connects agent processes to rooms.
livekit-agents/livekit/agents/worker.py - Job context · Carries room job state into the application entrypoint.
livekit-agents/livekit/agents/job.py - AgentSession · Owns agent state, turns, models, and run lifecycle.
livekit-agents/livekit/agents/voice/agent_session.py - Agent · Supplies instructions, tools, hooks, and handoffs.
livekit-agents/livekit/agents/voice/agent.py - Room I/O · Bridges LiveKit media and data with the session.
livekit-agents/livekit/agents/voice/room_io/room_io.py - Audio recognition · Applies recognition, VAD, endpointing, and turn detection.
livekit-agents/livekit/agents/voice/audio_recognition.py - Generation & playout · Streams model output through TTS and room playout.
livekit-agents/livekit/agents/voice/generation.py - Tool executor · Executes LLM-selected tools asynchronously.
livekit-agents/livekit/agents/voice/tool_executor.py - Model contracts · Streaming STT, TTS, LLM, VAD, and realtime interfaces.
livekit-agents/livekit/agents/llm/llm.py - STT contract
livekit-agents/livekit/agents/stt/stt.py - TTS contract
livekit-agents/livekit/agents/tts/tts.py
Provider Plugins
Separately packaged vendor adapters.
- OpenAI plugin · Implements realtime model integration.
livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py - Deepgram plugin · Implements speech recognition integration.
livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/stt.py - Silero plugin · Implements voice activity detection.
livekit-plugins/livekit-plugins-silero/livekit/plugins/silero/vad.py
Isolation & Quality
Supervision, observability, and evaluation.
- Process pool · Runs jobs and inference in isolated processes or threads.
livekit-agents/livekit/agents/ipc/proc_pool.py - Tracing · Emits telemetry, traces, and PII-aware session observations.
livekit-agents/livekit/agents/telemetry/traces.py - Evaluation · Supports event assertions and LLM judging.
livekit-agents/livekit/agents/evals/evaluation.py
Other components
- Agent application · Defines room entrypoints and agent behavior.
- LiveKit room · Provides participant audio, data, and RPC transport.
Connections 23
- Agent application → CLI: invokes
- CLI → AgentServer worker: starts
- AgentServer worker → Job context: creates
- AgentServer worker → LiveKit room: joins room
- Job context → Agent application: calls entrypoint
- Agent application → AgentSession: creates
- AgentSession → Agent: runs active
- AgentSession → Room I/O: coordinates
- LiveKit room → Room I/O: audio and data
- Room I/O → Audio recognition: incoming audio
- Audio recognition → AgentSession: final user turn
- AgentSession → Generation & playout: response run
- Generation & playout → Room I/O: speech playout
- AgentSession → Tool executor: tool calls
- AgentSession → Model contracts: uses models
- Model contracts → STT contract: includes
- Model contracts → TTS contract: includes
- OpenAI plugin → Model contracts: implements
- Deepgram plugin → STT contract: implements
- Silero plugin → Model contracts: implements VAD
- AgentServer worker → Process pool: supervises
- AgentSession → Tracing: emits telemetry
- Evaluation → AgentSession: assesses behavior