# livekit/agents architecture

> Architecture diagram of the GitHub repository livekit/agents, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/livekit/agents
- Repository: https://github.com/livekit/agents

## Overview

#### Purpose and Runtime
This is the Python LiveKit Agents framework: server-side, realtime programmable participants for conversational voice and multimodal sessions. It is a Python/asyncio monorepo containing the core SDK, separately packaged provider plugins, native extensions, and runnable examples. Agents connect to LiveKit rooms over the LiveKit Python SDK; deployments use LiveKit Cloud or self-hosted LiveKit infrastructure, with optional telephony and data/RPC integration.

Core package: `livekit-agents/livekit/agents/__init__.py`, `livekit-agents/pyproject.toml`.

#### Application Entry and Job Lifecycle
An application constructs an `AgentServer`, decorates a room-session entrypoint, and invokes the CLI. The CLI supports local console execution, hot-reloading development servers, and production worker operation. The worker/server layer receives or schedules jobs, connects an agent process to a LiveKit room, and invokes user entrypoints with job context.

Paths: `livekit-agents/livekit/agents/cli/cli.py`, `livekit-agents/livekit/agents/worker.py`, `livekit-agents/livekit/agents/job.py`.

#### Session Orchestration and Agent Control
`AgentSession` is the central runtime coordinator: it owns the active agent, conversation state, model configuration, user data, and turn/run lifecycles. Agents supply instructions, tools, lifecycle hooks, and can hand off to another agent. Tool execution and speech/run handles form the boundary between LLM decisions and asynchronous side effects.

Paths: `livekit-agents/livekit/agents/voice/agent_session.py`, `livekit-agents/livekit/agents/voice/agent.py`, `livekit-agents/livekit/agents/voice/tool_executor.py`.

#### Realtime Media and Turn Pipeline
Room I/O bridges LiveKit participant audio/data with the session. Incoming audio flows through recognition, VAD/endpointing, STT, and turn detection; final user turns drive LLM or realtime-model generation. Output text streams through TTS and playout back to the room. Interruption, filler speech, transcription synchronization, DTMF/IVR, recording, and avatar paths are session-adjacent specializations.

Paths: `livekit-agents/livekit/agents/voice/room_io/room_io.py`, `livekit-agents/livekit/agents/voice/audio_recognition.py`, `livekit-agents/livekit/agents/voice/generation.py`.

#### Model Abstractions and Inference Boundary
The core exposes provider-neutral streaming contracts for STT, TTS, VAD, LLM/chat, and duplex realtime models, including fallback and stream adapters. `inference` provides LiveKit-hosted model access; external providers are implemented outside the core package. LLM context/tool schemas normalize provider formats, while realtime adapters combine speech and generation into a different control path from the staged STT→LLM→TTS pipeline.

Paths: `livekit-agents/livekit/agents/stt/stt.py`, `livekit-agents/livekit/agents/tts/tts.py`, `livekit-agents/livekit/agents/llm/llm.py`.

#### Provider Plugin Ecosystem
Each `livekit-plugins/livekit-plugins-*` subtree is an independently packaged adapter for external AI, avatar, browser, audio-processing, or workflow services. Plugins map vendor APIs and streaming protocols into core interfaces rather than coupling vendors to session orchestration. Representative integrations include OpenAI realtime/chat/STT/TTS, Deepgram STT/TTS, Google models, AWS, and avatar APIs.

Paths: `livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py`, `livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/stt.py`, `livekit-plugins/livekit-plugins-silero/livekit/plugins/silero/vad.py`.

#### Isolation, Observability, and Evaluation
Workers can isolate jobs and inference in supervised processes/threads using IPC pools, separating failures and expensive model execution from the coordinator. Metrics, OpenTelemetry traces, PII handling, and HTTP/RPC telemetry observe sessions and provider calls. Built-in evaluation primitives support deterministic event assertions plus LLM judging; examples demonstrate complete vertical agent implementations.

Paths: `livekit-agents/livekit/agents/ipc/proc_pool.py`, `livekit-agents/livekit/agents/telemetry/traces.py`, `livekit-agents/livekit/agents/evals/evaluation.py`.

Main entry points: [`livekit-agents/livekit/agents/voice/agent_session.py`](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/agent_session.py) (AgentSession), [`livekit-agents/livekit/agents/llm/llm.py`](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/llm/llm.py) (Model contracts), [`livekit-agents/livekit/agents/worker.py`](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/worker.py) (AgentServer worker), [`livekit-agents/livekit/agents/voice/room_io/room_io.py`](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/room_io/room_io.py) (Room I/O)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core SDK"]
  node_cli["CLI<br/>runtime entry<br/>[cli.py]"]
  node_worker["AgentServer worker<br/>job runtime<br/>[worker.py]"]
  node_job["Job context<br/>lifecycle context<br/>[job.py]"]
  node_session["AgentSession<br/>session coordinator<br/>[agent_session.py]"]
  node_agent["Agent<br/>behavior definition<br/>[agent.py]"]
  node_room_io["Room I/O<br/>media bridge<br/>[room_io.py]"]
  node_recognition["Audio recognition<br/>input pipeline"]
  node_generation["Generation &amp; playout<br/>output pipeline<br/>[generation.py]"]
  node_tools["Tool executor<br/>side-effect boundary<br/>[tool_executor.py]"]
  node_model_contracts["Model contracts<br/>provider-neutral API<br/>[llm.py]"]
  node_stt["STT contract<br/>model API<br/>[stt.py]"]
  node_tts["TTS contract<br/>model API<br/>[tts.py]"]
end

subgraph group_plugins["Provider Plugins"]
  node_openai["OpenAI plugin<br/>provider adapter<br/>[realtime_model.py]"]
  node_deepgram["Deepgram plugin<br/>provider adapter<br/>[stt.py]"]
  node_silero["Silero plugin<br/>provider adapter<br/>[vad.py]"]
end

subgraph group_operations["Isolation &amp; Quality"]
  node_isolation["Process pool<br/>supervised isolation<br/>[proc_pool.py]"]
  node_traces["Tracing<br/>observability<br/>[traces.py]"]
  node_evals["Evaluation<br/>quality framework<br/>[evaluation.py]"]
end

node_app["Agent application<br/>user code"]
node_livekit{{"LiveKit room<br/>external realtime service"}}

node_app -->|"invokes"| node_cli
node_cli -->|"starts"| node_worker
node_worker -->|"creates"| node_job
node_worker -->|"joins room"| node_livekit
node_job -->|"calls entrypoint"| node_app
node_app -->|"creates"| node_session
node_session -->|"runs active"| node_agent
node_session -->|"coordinates"| node_room_io
node_livekit -->|"audio and data"| node_room_io
node_room_io -->|"incoming audio"| node_recognition
node_recognition -->|"final user turn"| node_session
node_session -->|"response run"| node_generation
node_generation -->|"speech playout"| node_room_io
node_session -->|"tool calls"| node_tools
node_session -->|"uses models"| node_model_contracts
node_model_contracts -->|"includes"| node_stt
node_model_contracts -->|"includes"| node_tts
node_openai -.->|"implements"| node_model_contracts
node_deepgram -.->|"implements"| node_stt
node_silero -.->|"implements VAD"| node_model_contracts
node_worker -->|"supervises"| node_isolation
node_session -.->|"emits telemetry"| node_traces
node_evals -.->|"assesses behavior"| node_session

click node_cli "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/cli/cli.py"
click node_worker "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/worker.py"
click node_job "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/job.py"
click node_session "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/agent_session.py"
click node_agent "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/agent.py"
click node_room_io "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/room_io/room_io.py"
click node_recognition "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/audio_recognition.py"
click node_generation "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/generation.py"
click node_tools "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/tool_executor.py"
click node_model_contracts "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/llm/llm.py"
click node_stt "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/stt/stt.py"
click node_tts "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/tts/tts.py"
click node_openai "https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py"
click node_deepgram "https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/stt.py"
click node_silero "https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-silero/livekit/plugins/silero/vad.py"
click node_isolation "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/ipc/proc_pool.py"
click node_traces "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/telemetry/traces.py"
click node_evals "https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/evals/evaluation.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_worker,node_job,node_session,node_agent,node_room_io,node_recognition,node_generation,node_tools,node_model_contracts,node_stt,node_tts toneBlue
class node_openai,node_deepgram,node_silero toneAmber
class node_isolation,node_traces,node_evals toneMint
class node_app,node_livekit toneNeutral
```

## Components

### Core SDK

Async runtime, session orchestration, and model contracts.

- [CLI](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/cli/cli.py): runtime entry · Runs console, development, and production worker modes. · `livekit-agents/livekit/agents/cli/cli.py`
- [AgentServer worker](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/worker.py): job runtime · Schedules jobs and connects agent processes to rooms. · `livekit-agents/livekit/agents/worker.py`
- [Job context](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/job.py): lifecycle context · Carries room job state into the application entrypoint. · `livekit-agents/livekit/agents/job.py`
- [AgentSession](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/agent_session.py): session coordinator · Owns agent state, turns, models, and run lifecycle. · `livekit-agents/livekit/agents/voice/agent_session.py`
- [Agent](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/agent.py): behavior definition · Supplies instructions, tools, hooks, and handoffs. · `livekit-agents/livekit/agents/voice/agent.py`
- [Room I/O](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/room_io/room_io.py): media bridge · Bridges LiveKit media and data with the session. · `livekit-agents/livekit/agents/voice/room_io/room_io.py`
- [Audio recognition](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/audio_recognition.py): input pipeline · Applies recognition, VAD, endpointing, and turn detection. · `livekit-agents/livekit/agents/voice/audio_recognition.py`
- [Generation & playout](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/generation.py): output pipeline · Streams model output through TTS and room playout. · `livekit-agents/livekit/agents/voice/generation.py`
- [Tool executor](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/tool_executor.py): side-effect boundary · Executes LLM-selected tools asynchronously. · `livekit-agents/livekit/agents/voice/tool_executor.py`
- [Model contracts](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/llm/llm.py): provider-neutral API · Streaming STT, TTS, LLM, VAD, and realtime interfaces. · `livekit-agents/livekit/agents/llm/llm.py`
- [STT contract](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/stt/stt.py): model API · `livekit-agents/livekit/agents/stt/stt.py`
- [TTS contract](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/tts/tts.py): model API · `livekit-agents/livekit/agents/tts/tts.py`

### Provider Plugins

Separately packaged vendor adapters.

- [OpenAI plugin](https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py): provider adapter · Implements realtime model integration. · `livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py`
- [Deepgram plugin](https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/stt.py): provider adapter · Implements speech recognition integration. · `livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/stt.py`
- [Silero plugin](https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-silero/livekit/plugins/silero/vad.py): provider adapter · Implements voice activity detection. · `livekit-plugins/livekit-plugins-silero/livekit/plugins/silero/vad.py`

### Isolation & Quality

Supervision, observability, and evaluation.

- [Process pool](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/ipc/proc_pool.py): supervised isolation · Runs jobs and inference in isolated processes or threads. · `livekit-agents/livekit/agents/ipc/proc_pool.py`
- [Tracing](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/telemetry/traces.py): observability · Emits telemetry, traces, and PII-aware session observations. · `livekit-agents/livekit/agents/telemetry/traces.py`
- [Evaluation](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/evals/evaluation.py): quality framework · Supports event assertions and LLM judging. · `livekit-agents/livekit/agents/evals/evaluation.py`

### Other components

- **Agent application**: user code · Defines room entrypoints and agent behavior.
- **LiveKit room**: external realtime service · Provides participant audio, data, and RPC transport.

## Connections

- Agent application → CLI: invokes
- CLI → AgentServer worker: starts
- AgentServer worker → Job context: creates
- AgentServer worker → LiveKit room: joins room
- Job context → Agent application: calls entrypoint
- Agent application → AgentSession: creates
- AgentSession → Agent: runs active
- AgentSession → Room I/O: coordinates
- LiveKit room → Room I/O: audio and data
- Room I/O → Audio recognition: incoming audio
- Audio recognition → AgentSession: final user turn
- AgentSession → Generation & playout: response run
- Generation & playout → Room I/O: speech playout
- AgentSession → Tool executor: tool calls
- AgentSession → Model contracts: uses models
- Model contracts → STT contract: includes
- Model contracts → TTS contract: includes
- OpenAI plugin → Model contracts: implements
- Deepgram plugin → STT contract: implements
- Silero plugin → Model contracts: implements VAD
- AgentServer worker → Process pool: supervises
- AgentSession → Tracing: emits telemetry
- Evaluation → AgentSession: assesses behavior
