# videosdk-community/ai-agent architecture

> Architecture diagram of the GitHub repository videosdk-community/ai-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/videosdk-community/ai-agent
- Repository: https://github.com/videosdk-community/ai-agent

## Overview

##### Project Type and Purpose
- This is a Python-based, real-time voice AI agent that joins video meetings (via Video SDK/WebRTC) to conduct an automated “interview.”
- It captures live audio, streams it to Speech-to-Text (Deepgram), uses a Large Language Model (OpenAI/GPT-4) to generate conversational responses, and converts those back into speech (Eleven Labs) before re-injecting into the meeting.

##### File Structure
- Top-level
- main.py         ← Orchestrates initialization and module wiring
- .env.example    ← Configuration template for room ID, tokens, API keys
- requirements.txt
- agent/
- agent.py        ← Core session manager: joins room, receives audio, sends responses
- audio_stream_track.py ← Wraps Video SDK’s custom audio track for ingestion & output
- stt/
- stt.py          ← Defines STT interface, streaming pipeline
- deepgram_stt.py ← Deepgram-specific implementation
- intelligence/
- intelligence.py        ← Encapsulates prompt management, context window
- intelligence_client.py ← Handles LLM API calls (OpenAI or custom)
- tts/
- tts.py               ← Defines TTS interface
- elevenlabs_tts.py    ← Eleven Labs-specific implementation

##### Key Insights from README
- The pipeline: Video SDK ↔ STT ↔ LLM ↔ TTS ↔ Video SDK
- External services: Deepgram (Streaming STT), OpenAI (GPT-4), Eleven Labs (multilingual TTS)
- Supports an optional Vector DB for RAG between STT and LLM.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Video SDK
    VideoSDK["Video SDK / WebRTC Client"]:::external

    %% Configuration Inputs
    Env[".env Configuration"]:::config
    Req["requirements.txt"]:::config

    %% Agent Orchestrator
    subgraph "Agent Orchestrator"
        direction TB
        Orchestrator["Orchestrator\n(main.py)"]:::internal
    end

    %% Agent Layer
    subgraph "Agent Layer"
        direction TB
        AgentMgr["Agent Session Manager\n(agent/agent.py)"]:::internal
        AudioTrack["AudioStreamTrack\n(agent/audio_stream_track.py)"]:::internal
    end

    %% STT Layer
    subgraph "Speech-to-Text Layer"
        direction TB
        STTInterface["STT Interface\n(stt/stt.py)"]:::internal
        DeepgramImpl["Deepgram STT\n(stt/deepgram_stt.py)"]:::external
    end

    %% Optional Vector DB
    VectorDB["Vector DB (RAG)"]:::external

    %% Intelligence Layer
    subgraph "Intelligence Layer"
        direction TB
        PromptMgr["Prompt & Context Manager\n(intelligence/intelligence.py)"]:::internal
        LLMClient["LLM API Client\n(intelligence/intelligence_client.py)"]:::external
    end

    %% TTS Layer
    subgraph "Text-to-Speech Layer"
        direction TB
        TTSInterface["TTS Interface\n(tts/tts.py)"]:::internal
        ElevenImpl["Eleven Labs TTS\n(tts/elevenlabs_tts.py)"]:::external
    end

    %% Data Flows
    VideoSDK -->|Audio frames| AudioTrack
    AudioTrack -->|Wrapped stream| AgentMgr
    AgentMgr -->|Stream| STTInterface
    STTInterface -->|Transcript chunks| DeepgramImpl
    DeepgramImpl -->|Transcript text| STTInterface
    STTInterface -->|Transcript text| PromptMgr
    PromptMgr -->|Prompt+Context| LLMClient
    LLMClient -->|Response text| PromptMgr
    PromptMgr -->|Response text| TTSInterface
    TTSInterface -->|Audio stream| ElevenImpl
    ElevenImpl -->|Synthesized audio| TTSInterface
    TTSInterface -->|Audio frames| AgentMgr
    AgentMgr -->|Audio out| VideoSDK

    %% Optional RAG path
    STTInterface -->|Transcript text| VectorDB
    VectorDB -->|Retrieved info| PromptMgr

    %% Config to Orchestrator
    Env -->|API keys, Room ID| Orchestrator
    Req -->|Dependencies| Orchestrator

    %% Click Events
    click Orchestrator "https://github.com/videosdk-community/ai-agent/blob/main/main.py"
    click AgentMgr "https://github.com/videosdk-community/ai-agent/blob/main/agent/agent.py"
    click AudioTrack "https://github.com/videosdk-community/ai-agent/blob/main/agent/audio_stream_track.py"
    click STTInterface "https://github.com/videosdk-community/ai-agent/blob/main/stt/stt.py"
    click DeepgramImpl "https://github.com/videosdk-community/ai-agent/blob/main/stt/deepgram_stt.py"
    click PromptMgr "https://github.com/videosdk-community/ai-agent/blob/main/intelligence/intelligence.py"
    click LLMClient "https://github.com/videosdk-community/ai-agent/blob/main/intelligence/intelligence_client.py"
    click TTSInterface "https://github.com/videosdk-community/ai-agent/blob/main/tts/tts.py"
    click ElevenImpl "https://github.com/videosdk-community/ai-agent/blob/main/tts/elevenlabs_tts.py"
    click Env "https://github.com/videosdk-community/ai-agent/blob/main/.env.example"
    click Req "https://github.com/videosdk-community/ai-agent/blob/main/requirements.txt"

    %% Styles
    classDef internal fill:#D0E8FF,stroke:#0366d6,color:#000;
    classDef external fill:#D7FFD9,stroke:#28a745,color:#000;
    classDef config fill:#FFF3CD,stroke:#856404,color:#000;
```
