How videosdk-live/agents works
Project Type & Purpose
• A Python SDK / framework (monorepo) for building real-time, multimodal AI voice/video agents.
• Core library (videosdk-agents) + many optional plugins (videosdk-plugins-*) + sample apps under examples/.
Key Files & Directories
• Root
– .github/, CI workflows, issue/pr templates
– README.md, CONTRIBUTING, LICENSE, scripts (build_docs.py, serve_docs.py)
– examples/ – dozens of agent usage samples (cascading/realtime pipelines, SIP, avatar, MCP, A2A, human-in-loop).
• videosdk-agents/
– pyproject.toml + package code under videosdk/agents/…
– Core classes: Agent, AgentSession, pipelines (cascading/realtime), event_bus, tools, STT/LLM/TTS/VAD abstractions, mcp, a2a, metrics, CLI.
• videosdk-plugins/
– One subfolder per provider (anthropic, openai, google, aws, silero, turn-detector, etc.)
– Each plugin ships code under videosdk/plugins/<category>/<provider>, registered via entry points in its pyproject.toml.
Architecture at a Glance
A. Client Layer
– Web/mobile/desktop applications (React, Flutter, JS/Unity/IoT)
– Connect to VideoSDK Signaling & Media servers
B. VideoSDK Infrastructure (external)
– Signaling server (WebSocket), SFU/MCU media servers
– Room ID, auth token – binds users & agents into the same conference
C. Agent Worker Layer (this repo)
1. JobContext & WorkerJob (bootstraps)
2. AgentSession
• Receives media streams from VideoSDK server
• Routes audio frames → pipeline
3. Pipeline(s)
• Cascading (STT → LLM → TTS)
• Realtime (unified low-latency models)
4. Plugin System (entry-point lookup)
• STT plugins
• LLM plugins
• TTS plugins
• VAD / turn-detector / avatar / denoise / SIP / MCP / A2A
5. Event Bus & Function Tools
• Internal & external @function_tool actions
• Observability: OpenTelemetry traces & metrics
6. Output
• Synthesized audio/video sent back via AgentSession → VideoSDK server → client apps
External Integrations
• AI Providers: OpenAI, Google Gemini, AWS Nova, Azure Live, Anthropic, Deepgram, ElevenLabs, Cartesia, etc.
• Telephony (SIP gateway)
• Virtual avatar (Simli)
• External data/tools via MCP (Model Context Protocol)
• Developer CLI (console mode)
Architectural Patterns & Principles
• Plugin architecture via Python entry points – decouples core from specific AI providers
• Asynchronous, event-driven design (asyncio) for low-latency media processing
• Separation of concerns: media handling, AI pipeline, tools, observability, session management
• Extensibility points: function tools, plugin categories, example templates