How livekit/agents-js works
1. What this repository is
- This is a TypeScript monorepo for the LiveKit Agents SDK for Node.js.
- It is centered on building stateful, realtime voice agents that join LiveKit rooms as participants and interact with users over audio/data channels.
2. Top-level architecture
- The repo is split into:
- agents/: the core framework runtime and public SDK.
- plugins/: provider integrations for LLM, STT, TTS, VAD, realtime models, and avatars.
- examples/: runnable agent applications and scenarios.
- tests/: end-to-end validation against a real LiveKit deployment.
- This separation is important: the core SDK defines abstractions, while plugins implement vendor-specific backends.
3. Core runtime: workers, jobs, and agent sessions
- The core execution model is worker-driven, not request/response web serving.
- A Worker connects to a LiveKit server, waits for jobs, and launches agent instances for individual sessions.
- An entrypoint in the agent app defines how a job is handled, and JobContext / JobProcess provide room access plus per-process state.
- The worker lifecycle is designed for long-lived, stateful sessions and graceful shutdown.
4. Realtime communication boundary
- The main external system is LiveKit server.
- The agent joins a room and exchanges:
- audio tracks for speech input/output,
- data/RPC messages for structured control,
- and possibly avatar or other media streams depending on plugins.
- The repo’s README and examples show that LiveKit is the session transport and orchestration layer.
5. Voice pipeline inside agents/
- The core voice stack is composed of:
- VAD / turn detection to decide when the user is speaking and when to respond,
- STT to convert user audio to text,
- LLM to reason and generate replies,
- TTS to synthesize response audio back into the room.
- These are wired together by voice/ modules, especially voice/agent.ts, voice/agent_session.ts, voice/generation.ts, and voice/turn_config/.
- The architecture supports both classic modular pipelines and realtime model variants.
6. Inference abstraction layer
- agents/src/inference/ is the provider-agnostic model layer.
- It contains core types and wrappers for:
- LLM
- STT
- TTS
- interruption detection
- realtime inference transport
- The inference/interruption/ subtree handles streaming interruption logic, showing that the framework treats turn-taking as a first-class realtime concern.
7. Provider plugins as isolated boundaries
- Each provider lives in its own package under plugins/, for example:
- OpenAI: LLM, STT, TTS, realtime, responses, websocket APIs
- Google: LLM, TTS, Gemini realtime
- Deepgram: STT/TTS
- Cartesia, ElevenLabs, Neuphonic, Rime, Resemble, Inworld, Sarvam: mostly TTS/STT
- Silero: VAD
- LiveKit: semantic end-of-utterance / turn detection
- Anam, Bey, Hedra, Lemonslice, Trugen: avatar integrations
- Phonic, XAI: realtime / multimodal models
- These packages are independently built and versioned, but consumed by the core agents package as adapters.
8. Agent behavior and LLM orchestration
- The llm/ and voice/ layers define higher-level agent primitives:
- Agent: instructions, tools, and lifecycle hooks
- AgentSession: conversation/session container
- tool contexts and handoff behavior for multi-agent flows
- remote chat context and fallback adapters
- This is where the framework turns raw model calls into a conversational state machine.
9. Media and stream utilities
- Several shared subsystems support realtime media processing:
- audio.ts, vad.ts, transcription.ts
- stream/ helpers for merging, channels, deferred streams, and identity transforms
- tokenize/ for text segmentation at word/sentence/paragraph level
- These modules are the plumbing that lets audio, text, and event streams stay synchronized.
10. Process isolation and IPC
- The ipc/ and connection_pool.ts code indicates a deliberate boundary between the main worker and heavier execution paths.
- There are executors and process-pool abstractions for:
- job execution
- inference execution
- supervised subprocess management
- This suggests the runtime can isolate unstable or CPU-heavy work from the main worker process.
11. Telemetry, metrics, and logging
- The framework includes built-in observability:
- metrics/ for model usage and usage collection
- telemetry/ for logs, traces, and OpenTelemetry HTTP export
- log.ts and pino/OpenTelemetry transport support
- These are framework-level concerns, not app-level concerns, and are important for production deployment.
12. Examples as architecture references
- examples/src/ contains runnable reference apps:
- basic agents, tool calling, multi-agent handoff, realtime agents, background audio, push-to-talk, avatars, domain-specific demos like drive-thru and frontdesk
- They demonstrate the intended composition pattern: define agent → choose plugins → start session in a room → generate replies / handle turn-taking.
13. Testing and validation
- The repo uses Vitest for unit tests across core modules and plugins.
- tests/e2e/ contains Docker-based end-to-end tests that exercise the system against LiveKit in a realistic environment.
- This indicates the architecture is validated both at the library boundary and at the full worker-to-room integration boundary.
14. Build and release structure
- The project uses pnpm workspaces, turbo, tsup, and API Extractor for package builds and typed package outputs.
- Changesets drive versioning/release management.
- This matters architecturally because the repo is designed as a multi-package SDK distribution, not a single app.
15. How to draw the main diagram
- The central flow is:
- LiveKit server / room
- → Worker
- → AgentSession
- → VAD / STT / LLM / TTS / turn detection
- → audio/data back to room
- Surround that core with provider plugins as interchangeable adapters and telemetry/process pools as supporting subsystems.
Main entry points: index.ts Inference · generation.ts Voice stack · worker.ts Worker · agent_session.ts Agent session
Components 26
Core SDK
Worker-driven runtime, session model, voice pipeline, and inference abstractions.
- Worker · Connects to LiveKit, waits for jobs, and launches agent sessions.
agents/src/worker.ts - Job model · Defines job handling, process state, and room access for an agent run.
agents/src/job.ts - Proc pool · Shared process-pool and IPC boundary for heavy or isolated execution.
agents/src/ipc/proc_pool.ts - Job exec · Executes jobs in supervised subprocesses when isolation is needed.
agents/src/ipc/job_executor.ts - Infer exec · Runs model work in separate processes or lazy-loaded workers.
agents/src/ipc/inference_executor.ts - Agent session · Conversation container that binds agent logic to a room session.
agents/src/voice/agent_session.ts - Agent · Defines instructions, tools, and lifecycle hooks for a conversational agent.
agents/src/voice/agent.ts - Turn config · Endpointing, interruption, and turn-handling policies for realtime conversation.
agents/src/voice/turn_config/turn_handling.ts - Voice stack · Coordinates audio input, VAD, transcription, generation, and output back to the room.
agents/src/voice/generation.ts - VAD · Voice activity detection and speech boundary detection.
agents/src/vad.ts - STT · Streaming speech recognition abstraction.
agents/src/stt/stt.ts - LLM · Provider-agnostic language model abstraction and chat context handling.
agents/src/llm/llm.ts - TTS · Streaming speech synthesis abstraction.
agents/src/tts/tts.ts - Inference · Provider-neutral wrappers for LLM, STT, TTS, realtime transport, and interruption logic.
agents/src/inference/index.ts - Telemetry · Logging, traces, and OpenTelemetry export support.
agents/src/telemetry/index.ts - Metrics · Model usage collection and reporting helpers.
agents/src/metrics/index.ts
Plugins
Provider adapters for models, turn detection, and avatars.
- OpenAI · LLM, STT, TTS, responses, websocket, and realtime model adapters.
plugins/openai/src/index.ts - Google · Gemini LLM, TTS, and realtime adapters.
plugins/google/src/index.ts - Deepgram · STT and TTS adapters.
plugins/deepgram/src/index.ts - Silero · Neural VAD adapter.
plugins/silero/src/index.ts - LiveKit turn detector · Semantic end-of-utterance and turn-detection integration.
plugins/livekit/src/index.ts - Avatar plugins · Avatar integrations such as Anam, Bey, Hedra, Lemonslice, and Trugen.
plugins/anam/src/index.ts - Realtime plugins · Realtime multimodal model adapters such as Phonic and XAI.
plugins/phonic/src/index.ts
Examples
Runnable agent applications and reference compositions.
- Examples · Reference agents and scenario demos showing how to compose the SDK.
examples/src/basic_agent.ts
Validation
Unit and end-to-end testing against LiveKit.
- E2E tests · Docker-based end-to-end tests against a real LiveKit deployment.
tests/e2e/test_agent.mjs
Other components
- LiveKit server · External room transport for audio, data, and session orchestration.
Connections 31
- LiveKit server → Worker: jobs · Worker connects to LiveKit and receives job assignments.
- Worker → Job model: starts · Worker creates per-job handling context and lifecycle.
- Worker → Proc pool: isolates · Worker can offload heavy execution into pooled subprocesses.
- Proc pool → Job exec: runs · Pool coordinates job execution processes.
- Proc pool → Infer exec: runs · Pool coordinates inference worker processes.
- Job model → Agent session: binds · Job context becomes the session container for an agent run.
- Agent session → Agent: hosts · Session drives agent behavior and lifecycle hooks.
- Agent session → Turn config: policies · Session applies endpointing and interruption settings.
- Agent session → Voice stack: orchestrates · Session coordinates the realtime voice pipeline.
- Voice stack → VAD: detects · Voice pipeline uses VAD to identify user speech turns.
- Voice stack → STT: transcribes · Audio input is converted to text.
- Voice stack → LLM: reasons · Conversation state is sent to the language model.
- Voice stack → TTS: speaks · Generated text is synthesized back into audio.
- Voice stack → LiveKit server: audio/data · Pipeline exchanges media and control data through the room.
- Inference → LLM: abstracts · Provider-neutral model interface for language generation.
- Inference → STT: abstracts · Provider-neutral speech recognition interface.
- Inference → TTS: abstracts · Provider-neutral speech synthesis interface.
- Inference → Telemetry: reports · Inference operations emit tracing and logging signals.
- Inference → Metrics: counts · Model usage is collected for reporting.
- OpenAI → Inference: adapts · OpenAI plugin implements core inference contracts.
- Google → Inference: adapts · Google plugin implements core inference contracts.
- Deepgram → Inference: adapts · Deepgram plugin implements speech contracts.
- Silero → VAD: adapts · Silero provides the VAD backend.
- LiveKit turn detector → Turn config: adapts · LiveKit turn detector plugs into turn-handling policy.
- Avatar plugins → LiveKit server: media · Avatar integrations extend room media handling.
- Realtime plugins → Inference: adapts · Realtime providers integrate through the model layer.
- Examples → Worker: uses · Examples show how an app starts workers and sessions.
- Examples → Agent: defines · Examples compose agent behavior and tools.
- Examples → Voice stack: demonstrates · Examples exercise turn-taking and generation flows.
- E2E tests → LiveKit server: validates · End-to-end tests run against a real LiveKit deployment.
- E2E tests → Worker: covers · E2E tests exercise the worker-to-room integration path.