# livekit/agents-js architecture

> Architecture diagram of the GitHub repository livekit/agents-js, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/livekit/agents-js
- Repository: https://github.com/livekit/agents-js

## Overview

1. **What this repository is**
   - This is a **TypeScript monorepo** for the **LiveKit Agents SDK for Node.js**.
   - It is centered on building **stateful, realtime voice agents** that join **LiveKit rooms** as participants and interact with users over audio/data channels.

2. **Top-level architecture**
   - The repo is split into:
     - **`agents/`**: the core framework runtime and public SDK.
     - **`plugins/`**: provider integrations for LLM, STT, TTS, VAD, realtime models, and avatars.
     - **`examples/`**: runnable agent applications and scenarios.
     - **`tests/`**: end-to-end validation against a real LiveKit deployment.
   - This separation is important: the **core SDK defines abstractions**, while **plugins implement vendor-specific backends**.

3. **Core runtime: workers, jobs, and agent sessions**
   - The core execution model is **worker-driven**, not request/response web serving.
   - A **Worker** connects to a LiveKit server, waits for jobs, and launches agent instances for individual sessions.
   - An **entrypoint** in the agent app defines how a job is handled, and `JobContext` / `JobProcess` provide room access plus per-process state.
   - The worker lifecycle is designed for long-lived, stateful sessions and graceful shutdown.

4. **Realtime communication boundary**
   - The main external system is **LiveKit server**.
   - The agent joins a **room** and exchanges:
     - **audio tracks** for speech input/output,
     - **data/RPC messages** for structured control,
     - and possibly **avatar** or other media streams depending on plugins.
   - The repo’s README and examples show that LiveKit is the session transport and orchestration layer.

5. **Voice pipeline inside `agents/`**
   - The core voice stack is composed of:
     - **VAD / turn detection** to decide when the user is speaking and when to respond,
     - **STT** to convert user audio to text,
     - **LLM** to reason and generate replies,
     - **TTS** to synthesize response audio back into the room.
   - These are wired together by `voice/` modules, especially `voice/agent.ts`, `voice/agent_session.ts`, `voice/generation.ts`, and `voice/turn_config/`.
   - The architecture supports both classic modular pipelines and **realtime model** variants.

6. **Inference abstraction layer**
   - `agents/src/inference/` is the provider-agnostic model layer.
   - It contains core types and wrappers for:
     - **LLM**
     - **STT**
     - **TTS**
     - **interruption detection**
     - **realtime inference transport**
   - The `inference/interruption/` subtree handles streaming interruption logic, showing that the framework treats turn-taking as a first-class realtime concern.

7. **Provider plugins as isolated boundaries**
   - Each provider lives in its own package under `plugins/`, for example:
     - **OpenAI**: LLM, STT, TTS, realtime, responses, websocket APIs
     - **Google**: LLM, TTS, Gemini realtime
     - **Deepgram**: STT/TTS
     - **Cartesia, ElevenLabs, Neuphonic, Rime, Resemble, Inworld, Sarvam**: mostly TTS/STT
     - **Silero**: VAD
     - **LiveKit**: semantic end-of-utterance / turn detection
     - **Anam, Bey, Hedra, Lemonslice, Trugen**: avatar integrations
     - **Phonic, XAI**: realtime / multimodal models
   - These packages are independently built and versioned, but consumed by the core `agents` package as adapters.

8. **Agent behavior and LLM orchestration**
   - The `llm/` and `voice/` layers define higher-level agent primitives:
     - **Agent**: instructions, tools, and lifecycle hooks
     - **AgentSession**: conversation/session container
     - **tool contexts** and **handoff** behavior for multi-agent flows
     - **remote chat context** and **fallback adapters**
   - This is where the framework turns raw model calls into a conversational state machine.

9. **Media and stream utilities**
   - Several shared subsystems support realtime media processing:
     - `audio.ts`, `vad.ts`, `transcription.ts`
     - `stream/` helpers for merging, channels, deferred streams, and identity transforms
     - `tokenize/` for text segmentation at word/sentence/paragraph level
   - These modules are the plumbing that lets audio, text, and event streams stay synchronized.

10. **Process isolation and IPC**
   - The `ipc/` and `connection_pool.ts` code indicates a deliberate boundary between the main worker and heavier execution paths.
   - There are executors and process-pool abstractions for:
     - **job execution**
     - **inference execution**
     - **supervised subprocess management**
   - This suggests the runtime can isolate unstable or CPU-heavy work from the main worker process.

11. **Telemetry, metrics, and logging**
   - The framework includes built-in observability:
     - `metrics/` for model usage and usage collection
     - `telemetry/` for logs, traces, and OpenTelemetry HTTP export
     - `log.ts` and pino/OpenTelemetry transport support
   - These are framework-level concerns, not app-level concerns, and are important for production deployment.

12. **Examples as architecture references**
   - `examples/src/` contains runnable reference apps:
     - basic agents, tool calling, multi-agent handoff, realtime agents, background audio, push-to-talk, avatars, domain-specific demos like drive-thru and frontdesk
   - They demonstrate the intended composition pattern: **define agent → choose plugins → start session in a room → generate replies / handle turn-taking**.

13. **Testing and validation**
   - The repo uses **Vitest** for unit tests across core modules and plugins.
   - `tests/e2e/` contains Docker-based end-to-end tests that exercise the system against LiveKit in a realistic environment.
   - This indicates the architecture is validated both at the library boundary and at the full worker-to-room integration boundary.

14. **Build and release structure**
   - The project uses **pnpm workspaces**, **turbo**, **tsup**, and **API Extractor** for package builds and typed package outputs.
   - **Changesets** drive versioning/release management.
   - This matters architecturally because the repo is designed as a **multi-package SDK distribution**, not a single app.

15. **How to draw the main diagram**
   - The central flow is:
     - **LiveKit server / room**
     - → **Worker**
     - → **AgentSession**
     - → **VAD / STT / LLM / TTS / turn detection**
     - → **audio/data back to room**
   - Surround that core with **provider plugins** as interchangeable adapters and **telemetry/process pools** as supporting subsystems.

Main entry points: [`agents/src/inference/index.ts`](https://github.com/livekit/agents-js/blob/main/agents/src/inference/index.ts) (Inference), [`agents/src/voice/generation.ts`](https://github.com/livekit/agents-js/blob/main/agents/src/voice/generation.ts) (Voice stack), [`agents/src/worker.ts`](https://github.com/livekit/agents-js/blob/main/agents/src/worker.ts) (Worker), [`agents/src/voice/agent_session.ts`](https://github.com/livekit/agents-js/blob/main/agents/src/voice/agent_session.ts) (Agent session)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core SDK"]
  node_worker["Worker<br/>job runtime<br/>[worker.ts]"]
  node_job["Job model<br/>job context<br/>[job.ts]"]
  node_ipc_pool["Proc pool<br/>process isolation<br/>[proc_pool.ts]"]
  node_job_exec["Job exec<br/>ipc executor<br/>[job_executor.ts]"]
  node_infer_exec["Infer exec<br/>ipc executor"]
  node_agent_session["Agent session<br/>session state<br/>[agent_session.ts]"]
  node_agent["Agent<br/>behavior layer<br/>[agent.ts]"]
  node_turn_config["Turn config<br/>turn policy<br/>[turn_handling.ts]"]
  node_voice_stack["Voice stack<br/>pipeline<br/>[generation.ts]"]
  node_vad(("VAD<br/>audio turn detection<br/>[vad.ts]"))
  node_stt["STT<br/>speech-to-text<br/>[stt.ts]"]
  node_llm["LLM<br/>language model<br/>[llm.ts]"]
  node_tts["TTS<br/>text-to-speech<br/>[tts.ts]"]
  node_inference["Inference<br/>model layer<br/>[index.ts]"]
  node_telemetry["Telemetry<br/>observability<br/>[index.ts]"]
  node_metrics["Metrics<br/>usage tracking<br/>[index.ts]"]
end

subgraph group_plugins["Plugins"]
  node_openai["OpenAI<br/>provider plugin<br/>[index.ts]"]
  node_google["Google<br/>provider plugin<br/>[index.ts]"]
  node_deepgram["Deepgram<br/>provider plugin<br/>[index.ts]"]
  node_silero["Silero<br/>provider plugin<br/>[index.ts]"]
  node_livekit_turn["LiveKit turn detector<br/>provider plugin<br/>[index.ts]"]
  node_avatar_plugins["Avatar plugins<br/>provider adapters<br/>[index.ts]"]
  node_realtime_plugins["Realtime plugins<br/>provider adapters<br/>[index.ts]"]
end

subgraph group_examples["Examples"]
  node_examples_app["Examples<br/>runnable apps<br/>[basic_agent.ts]"]
end

subgraph group_validation["Validation"]
  node_e2e_tests["E2E tests<br/>integration validation<br/>[test_agent.mjs]"]
end

node_livekit_server[("LiveKit server<br/>realtime backend")]

node_livekit_server -->|"jobs"| node_worker
node_worker -->|"starts"| node_job
node_worker -->|"isolates"| node_ipc_pool
node_ipc_pool -->|"runs"| node_job_exec
node_ipc_pool -->|"runs"| node_infer_exec
node_job -->|"binds"| node_agent_session
node_agent_session -->|"hosts"| node_agent
node_agent_session -->|"policies"| node_turn_config
node_agent_session -->|"orchestrates"| node_voice_stack
node_voice_stack -->|"detects"| node_vad
node_voice_stack -->|"transcribes"| node_stt
node_voice_stack -->|"reasons"| node_llm
node_voice_stack -->|"speaks"| node_tts
node_voice_stack -->|"audio/data"| node_livekit_server
node_inference -->|"abstracts"| node_llm
node_inference -->|"abstracts"| node_stt
node_inference -->|"abstracts"| node_tts
node_inference -->|"reports"| node_telemetry
node_inference -->|"counts"| node_metrics
node_openai -->|"adapts"| node_inference
node_google -->|"adapts"| node_inference
node_deepgram -->|"adapts"| node_inference
node_silero -->|"adapts"| node_vad
node_livekit_turn -->|"adapts"| node_turn_config
node_avatar_plugins -->|"media"| node_livekit_server
node_realtime_plugins -->|"adapts"| node_inference
node_examples_app -->|"uses"| node_worker
node_examples_app -->|"defines"| node_agent
node_examples_app -->|"demonstrates"| node_voice_stack
node_e2e_tests -->|"validates"| node_livekit_server
node_e2e_tests -->|"covers"| node_worker

click node_worker "https://github.com/livekit/agents-js/blob/main/agents/src/worker.ts"
click node_job "https://github.com/livekit/agents-js/blob/main/agents/src/job.ts"
click node_ipc_pool "https://github.com/livekit/agents-js/blob/main/agents/src/ipc/proc_pool.ts"
click node_job_exec "https://github.com/livekit/agents-js/blob/main/agents/src/ipc/job_executor.ts"
click node_infer_exec "https://github.com/livekit/agents-js/blob/main/agents/src/ipc/inference_executor.ts"
click node_agent_session "https://github.com/livekit/agents-js/blob/main/agents/src/voice/agent_session.ts"
click node_agent "https://github.com/livekit/agents-js/blob/main/agents/src/voice/agent.ts"
click node_turn_config "https://github.com/livekit/agents-js/blob/main/agents/src/voice/turn_config/turn_handling.ts"
click node_voice_stack "https://github.com/livekit/agents-js/blob/main/agents/src/voice/generation.ts"
click node_vad "https://github.com/livekit/agents-js/blob/main/agents/src/vad.ts"
click node_stt "https://github.com/livekit/agents-js/blob/main/agents/src/stt/stt.ts"
click node_llm "https://github.com/livekit/agents-js/blob/main/agents/src/llm/llm.ts"
click node_tts "https://github.com/livekit/agents-js/blob/main/agents/src/tts/tts.ts"
click node_inference "https://github.com/livekit/agents-js/blob/main/agents/src/inference/index.ts"
click node_telemetry "https://github.com/livekit/agents-js/blob/main/agents/src/telemetry/index.ts"
click node_metrics "https://github.com/livekit/agents-js/blob/main/agents/src/metrics/index.ts"
click node_openai "https://github.com/livekit/agents-js/blob/main/plugins/openai/src/index.ts"
click node_google "https://github.com/livekit/agents-js/blob/main/plugins/google/src/index.ts"
click node_deepgram "https://github.com/livekit/agents-js/blob/main/plugins/deepgram/src/index.ts"
click node_silero "https://github.com/livekit/agents-js/blob/main/plugins/silero/src/index.ts"
click node_livekit_turn "https://github.com/livekit/agents-js/blob/main/plugins/livekit/src/index.ts"
click node_avatar_plugins "https://github.com/livekit/agents-js/blob/main/plugins/anam/src/index.ts"
click node_realtime_plugins "https://github.com/livekit/agents-js/blob/main/plugins/phonic/src/index.ts"
click node_examples_app "https://github.com/livekit/agents-js/blob/main/examples/src/basic_agent.ts"
click node_e2e_tests "https://github.com/livekit/agents-js/blob/main/tests/e2e/test_agent.mjs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_worker,node_job,node_ipc_pool,node_job_exec,node_infer_exec,node_agent_session,node_agent,node_turn_config,node_voice_stack,node_vad,node_stt,node_llm,node_tts,node_inference,node_telemetry,node_metrics toneBlue
class node_openai,node_google,node_deepgram,node_silero,node_livekit_turn,node_avatar_plugins,node_realtime_plugins toneAmber
class node_examples_app toneMint
class node_e2e_tests toneRose
class node_livekit_server toneNeutral
```

## Components

### Core SDK

Worker-driven runtime, session model, voice pipeline, and inference abstractions.

- [Worker](https://github.com/livekit/agents-js/blob/main/agents/src/worker.ts): job runtime · Connects to LiveKit, waits for jobs, and launches agent sessions. · `agents/src/worker.ts`
- [Job model](https://github.com/livekit/agents-js/blob/main/agents/src/job.ts): job context · Defines job handling, process state, and room access for an agent run. · `agents/src/job.ts`
- [Proc pool](https://github.com/livekit/agents-js/blob/main/agents/src/ipc/proc_pool.ts): process isolation · Shared process-pool and IPC boundary for heavy or isolated execution. · `agents/src/ipc/proc_pool.ts`
- [Job exec](https://github.com/livekit/agents-js/blob/main/agents/src/ipc/job_executor.ts): ipc executor · Executes jobs in supervised subprocesses when isolation is needed. · `agents/src/ipc/job_executor.ts`
- [Infer exec](https://github.com/livekit/agents-js/blob/main/agents/src/ipc/inference_executor.ts): ipc executor · Runs model work in separate processes or lazy-loaded workers. · `agents/src/ipc/inference_executor.ts`
- [Agent session](https://github.com/livekit/agents-js/blob/main/agents/src/voice/agent_session.ts): session state · Conversation container that binds agent logic to a room session. · `agents/src/voice/agent_session.ts`
- [Agent](https://github.com/livekit/agents-js/blob/main/agents/src/voice/agent.ts): behavior layer · Defines instructions, tools, and lifecycle hooks for a conversational agent. · `agents/src/voice/agent.ts`
- [Turn config](https://github.com/livekit/agents-js/blob/main/agents/src/voice/turn_config/turn_handling.ts): turn policy · Endpointing, interruption, and turn-handling policies for realtime conversation. · `agents/src/voice/turn_config/turn_handling.ts`
- [Voice stack](https://github.com/livekit/agents-js/blob/main/agents/src/voice/generation.ts): pipeline · Coordinates audio input, VAD, transcription, generation, and output back to the room. · `agents/src/voice/generation.ts`
- [VAD](https://github.com/livekit/agents-js/blob/main/agents/src/vad.ts): audio turn detection · Voice activity detection and speech boundary detection. · `agents/src/vad.ts`
- [STT](https://github.com/livekit/agents-js/blob/main/agents/src/stt/stt.ts): speech-to-text · Streaming speech recognition abstraction. · `agents/src/stt/stt.ts`
- [LLM](https://github.com/livekit/agents-js/blob/main/agents/src/llm/llm.ts): language model · Provider-agnostic language model abstraction and chat context handling. · `agents/src/llm/llm.ts`
- [TTS](https://github.com/livekit/agents-js/blob/main/agents/src/tts/tts.ts): text-to-speech · Streaming speech synthesis abstraction. · `agents/src/tts/tts.ts`
- [Inference](https://github.com/livekit/agents-js/blob/main/agents/src/inference/index.ts): model layer · Provider-neutral wrappers for LLM, STT, TTS, realtime transport, and interruption logic. · `agents/src/inference/index.ts`
- [Telemetry](https://github.com/livekit/agents-js/blob/main/agents/src/telemetry/index.ts): observability · Logging, traces, and OpenTelemetry export support. · `agents/src/telemetry/index.ts`
- [Metrics](https://github.com/livekit/agents-js/blob/main/agents/src/metrics/index.ts): usage tracking · Model usage collection and reporting helpers. · `agents/src/metrics/index.ts`

### Plugins

Provider adapters for models, turn detection, and avatars.

- [OpenAI](https://github.com/livekit/agents-js/blob/main/plugins/openai/src/index.ts): provider plugin · LLM, STT, TTS, responses, websocket, and realtime model adapters. · `plugins/openai/src/index.ts`
- [Google](https://github.com/livekit/agents-js/blob/main/plugins/google/src/index.ts): provider plugin · Gemini LLM, TTS, and realtime adapters. · `plugins/google/src/index.ts`
- [Deepgram](https://github.com/livekit/agents-js/blob/main/plugins/deepgram/src/index.ts): provider plugin · STT and TTS adapters. · `plugins/deepgram/src/index.ts`
- [Silero](https://github.com/livekit/agents-js/blob/main/plugins/silero/src/index.ts): provider plugin · Neural VAD adapter. · `plugins/silero/src/index.ts`
- [LiveKit turn detector](https://github.com/livekit/agents-js/blob/main/plugins/livekit/src/index.ts): provider plugin · Semantic end-of-utterance and turn-detection integration. · `plugins/livekit/src/index.ts`
- [Avatar plugins](https://github.com/livekit/agents-js/blob/main/plugins/anam/src/index.ts): provider adapters · Avatar integrations such as Anam, Bey, Hedra, Lemonslice, and Trugen. · `plugins/anam/src/index.ts`
- [Realtime plugins](https://github.com/livekit/agents-js/blob/main/plugins/phonic/src/index.ts): provider adapters · Realtime multimodal model adapters such as Phonic and XAI. · `plugins/phonic/src/index.ts`

### Examples

Runnable agent applications and reference compositions.

- [Examples](https://github.com/livekit/agents-js/blob/main/examples/src/basic_agent.ts): runnable apps · Reference agents and scenario demos showing how to compose the SDK. · `examples/src/basic_agent.ts`

### Validation

Unit and end-to-end testing against LiveKit.

- [E2E tests](https://github.com/livekit/agents-js/blob/main/tests/e2e/test_agent.mjs): integration validation · Docker-based end-to-end tests against a real LiveKit deployment. · `tests/e2e/test_agent.mjs`

### Other components

- **LiveKit server**: realtime backend · External room transport for audio, data, and session orchestration.

## Connections

- LiveKit server → Worker: jobs · Worker connects to LiveKit and receives job assignments.
- Worker → Job model: starts · Worker creates per-job handling context and lifecycle.
- Worker → Proc pool: isolates · Worker can offload heavy execution into pooled subprocesses.
- Proc pool → Job exec: runs · Pool coordinates job execution processes.
- Proc pool → Infer exec: runs · Pool coordinates inference worker processes.
- Job model → Agent session: binds · Job context becomes the session container for an agent run.
- Agent session → Agent: hosts · Session drives agent behavior and lifecycle hooks.
- Agent session → Turn config: policies · Session applies endpointing and interruption settings.
- Agent session → Voice stack: orchestrates · Session coordinates the realtime voice pipeline.
- Voice stack → VAD: detects · Voice pipeline uses VAD to identify user speech turns.
- Voice stack → STT: transcribes · Audio input is converted to text.
- Voice stack → LLM: reasons · Conversation state is sent to the language model.
- Voice stack → TTS: speaks · Generated text is synthesized back into audio.
- Voice stack → LiveKit server: audio/data · Pipeline exchanges media and control data through the room.
- Inference → LLM: abstracts · Provider-neutral model interface for language generation.
- Inference → STT: abstracts · Provider-neutral speech recognition interface.
- Inference → TTS: abstracts · Provider-neutral speech synthesis interface.
- Inference → Telemetry: reports · Inference operations emit tracing and logging signals.
- Inference → Metrics: counts · Model usage is collected for reporting.
- OpenAI → Inference: adapts · OpenAI plugin implements core inference contracts.
- Google → Inference: adapts · Google plugin implements core inference contracts.
- Deepgram → Inference: adapts · Deepgram plugin implements speech contracts.
- Silero → VAD: adapts · Silero provides the VAD backend.
- LiveKit turn detector → Turn config: adapts · LiveKit turn detector plugs into turn-handling policy.
- Avatar plugins → LiveKit server: media · Avatar integrations extend room media handling.
- Realtime plugins → Inference: adapts · Realtime providers integrate through the model layer.
- Examples → Worker: uses · Examples show how an app starts workers and sessions.
- Examples → Agent: defines · Examples compose agent behavior and tools.
- Examples → Voice stack: demonstrates · Examples exercise turn-taking and generation flows.
- E2E tests → LiveKit server: validates · End-to-end tests run against a real LiveKit deployment.
- E2E tests → Worker: covers · E2E tests exercise the worker-to-room integration path.
