# videosdk-live/agents architecture

> Architecture diagram of the GitHub repository videosdk-live/agents, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/videosdk-live/agents
- Repository: https://github.com/videosdk-live/agents

## Overview

##### Project Type & Purpose
- A Python SDK / framework (monorepo) for building real-time, multimodal AI voice/video agents.
- Core library (`videosdk-agents`) + many optional plugins (`videosdk-plugins-*`) + sample apps under `examples/`.

##### Key Files & Directories
- Root
– `.github/`, CI workflows, issue/pr templates
– `README.md`, CONTRIBUTING, LICENSE, scripts (`build_docs.py`, `serve_docs.py`)
– `examples/` – dozens of agent usage samples (cascading/realtime pipelines, SIP, avatar, MCP, A2A, human-in-loop).
- `videosdk-agents/`
– `pyproject.toml` + package code under `videosdk/agents/…`
– Core classes: Agent, AgentSession, pipelines (cascading/realtime), event_bus, tools, STT/LLM/TTS/VAD abstractions, mcp, a2a, metrics, CLI.
- `videosdk-plugins/`
– One subfolder per provider (anthropic, openai, google, aws, silero, turn-detector, etc.)
– Each plugin ships code under `videosdk/plugins/<category>/<provider>`, registered via entry points in its `pyproject.toml`.

##### Architecture at a Glance
A. Client Layer
– Web/mobile/desktop applications (React, Flutter, JS/Unity/IoT)
– Connect to VideoSDK Signaling & Media servers
B. VideoSDK Infrastructure (external)
– Signaling server (WebSocket), SFU/MCU media servers
– Room ID, auth token – binds users & agents into the same conference
C. Agent Worker Layer (this repo)
1. JobContext & WorkerJob (bootstraps)
2. AgentSession
- Receives media streams from VideoSDK server
- Routes audio frames → pipeline
3. Pipeline(s)
- Cascading (STT → LLM → TTS)
- Realtime (unified low-latency models)
4. Plugin System (entry-point lookup)
- STT plugins
- LLM plugins
- TTS plugins
- VAD / turn-detector / avatar / denoise / SIP / MCP / A2A
5. Event Bus & Function Tools
- Internal & external `@function_tool` actions
- Observability: OpenTelemetry traces & metrics
6. Output
- Synthesized audio/video sent back via AgentSession → VideoSDK server → client apps

##### External Integrations
- AI Providers: OpenAI, Google Gemini, AWS Nova, Azure Live, Anthropic, Deepgram, ElevenLabs, Cartesia, etc.
- Telephony (SIP gateway)
- Virtual avatar (Simli)
- External data/tools via MCP (Model Context Protocol)
- Developer CLI (console mode)

##### Architectural Patterns & Principles
- Plugin architecture via Python entry points – decouples core from specific AI providers
- Asynchronous, event-driven design (asyncio) for low-latency media processing
- Separation of concerns: media handling, AI pipeline, tools, observability, session management
- Extensibility points: function tools, plugin categories, example templates

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Layer
    subgraph "Client Layer"
        direction TB
        Browser["Web/Mobile/Desktop Apps"]:::client
    end

    %% VideoSDK Infrastructure
    subgraph "VideoSDK Infrastructure" 
        direction TB
        Signaling["Signaling Server (WebSocket)"]:::infra
        Media["SFU/MCU Media Servers"]:::infra
    end

    %% Agent Worker Layer
    subgraph "Agent Worker Layer" 
        direction TB
        JobCtx["JobContext"]:::core
        WorkerJob["WorkerJob"]:::core
        AgentSession["AgentSession"]:::core
        PipelineSub["Pipeline Module"]:::core
        Plugins["Plugin System"]:::plugin
        EventBus["Event Bus & Tools"]:::core
        Output["Output to VideoSDK"]:::core
    end

    %% Pipelines Subgraph
    subgraph "Pipelines" 
        direction TB
        BasePipeline["Base Pipeline"]:::core
        Cascading["Cascading Pipeline"]:::core
        Realtime["Realtime Pipeline"]:::core
        RealtimeModel["Realtime Base Model"]:::core
    end

    %% Plugins Subgraph
    subgraph "Plugins" 
        direction TB
        STTPlugin["STT Plugins"]:::plugin
        LLMPlugin["LLM Plugins"]:::plugin
        TTSPlugin["TTS Plugins"]:::plugin
        VADPlugin["VAD/Turn-Detector Plugins"]:::plugin
        AvatarPlugin["Avatar Plugin"]:::plugin
        Denoise["Denoise Plugin"]:::plugin
    end

    %% External AI & Services
    subgraph "External AI & Services"
        direction TB
        AIServices["AI Service Providers"]:::external
        Telephony["Telephony (SIP)"]:::external
        MCPService["MCP / External Tools"]:::external
        A2AService["Agent-to-Agent Service"]:::external
    end

    %% Flows
    Browser -->|"RTP/Opus"| Signaling
    Signaling -->|"Media Streams"| AgentSession
    AgentSession -->|"Audio Frames"| BasePipeline
    BasePipeline -->|"STT"| STTPlugin
    STTPlugin -->|"Text (JSON)"| Cascading
    Cascading -->|"Text"| LLMPlugin
    LLMPlugin -->|"Response Text"| TTSPlugin
    TTSPlugin -->|"Audio Frames"| Realtime
    Realtime -->|"Low-Latency Audio"| AgentSession
    AgentSession -->|"Media Out"| Media
    Media --> Browser

    AgentSession -->|"Avatar Frames"| AvatarPlugin
    AgentSession -->|"VAD Events"| VADPlugin
    AgentSession -->|"Turn Events"| VADPlugin

    AgentSession -->|"MCP Calls"| MCPService
    AgentSession -->|"A2A Streams"| A2AService

    %% Subgraph connections
    JobCtx --> WorkerJob --> AgentSession
    AgentSession --> PipelineSub
    PipelineSub --> BasePipeline
    PipelineSub --> Cascading
    PipelineSub --> Realtime
    PipelineSub --> RealtimeModel
    AgentSession --> EventBus --> Plugins
    Plugins --> STTPlugin
    Plugins --> LLMPlugin
    Plugins --> TTSPlugin
    Plugins --> VADPlugin
    Plugins --> AvatarPlugin
    Plugins --> Denoise
    AgentSession --> Output

    %% Click Events
    click JobCtx "https://github.com/videosdk-live/agents/blob/main/videosdk-agents/videosdk/agents/job.py"
    click WorkerJob "https://github.com/videosdk-live/agents/blob/main/videosdk-agents/videosdk/agents/worker.py"
    click AgentSession "https://github.com/videosdk-live/agents/blob/main/videosdk-agents/videosdk/agents/agent_session.py"
    click BasePipeline "https://github.com/videosdk-live/agents/blob/main/videosdk-agents/videosdk/agents/pipeline.py"
    click Cascading "https://github.com/videosdk-live/agents/blob/main/videosdk-agents/videosdk/agents/cascading_pipeline.py"
    click Realtime "https://github.com/videosdk-live/agents/blob/main/videosdk-agents/videosdk/agents/realtime_pipeline.py"
    click RealtimeModel "https://github.com/videosdk-live/agents/blob/main/videosdk-agents/videosdk/agents/realtime_base_model.py"
    click EventBus "https://github.com/videosdk-live/agents/blob/main/videosdk-agents/videosdk/agents/event_bus.py"
    click STTPlugin "https://github.com/videosdk-live/agents/blob/main/videosdk-agents/videosdk/agents/stt/stt.py"
    click LLMPlugin "https://github.com/videosdk-live/agents/blob/main/videosdk-agents/videosdk/agents/llm/llm.py"
    click TTSPlugin "https://github.com/videosdk-live/agents/blob/main/videosdk-agents/videosdk/agents/tts/tts.py"
    click VADPlugin "https://github.com/videosdk-live/agents/blob/main/videosdk-agents/videosdk/agents/vad.py"
    click Plugins "https://github.com/videosdk-live/agents/tree/main/videosdk-plugins/"
    click Denoise "https://github.com/videosdk-live/agents/blob/main/videosdk-plugins-rnnoise/videosdk/plugins/rnnoise/denoise.py"
    click AvatarPlugin "https://github.com/videosdk-live/agents/blob/main/videosdk-plugins-simli/videosdk/plugins/simli/simli.py"

    %% Styles
    classDef client fill:#e0f7fa,stroke:#006064,color:#006064
    classDef infra fill:#eceff1,stroke:#607d8b,color:#37474f
    classDef core fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef plugin fill:#c8e6c9,stroke:#43a047,color:#1b5e20
    classDef external fill:#ffe0b2,stroke:#fb8c00,color:#e65100
```
