How videosdk-community/ai-agent works
Project Type and Purpose
• This is a Python-based, real-time voice AI agent that joins video meetings (via Video SDK/WebRTC) to conduct an automated “interview.”
• It captures live audio, streams it to Speech-to-Text (Deepgram), uses a Large Language Model (OpenAI/GPT-4) to generate conversational responses, and converts those back into speech (Eleven Labs) before re-injecting into the meeting.
File Structure
• Top-level
• main.py ← Orchestrates initialization and module wiring
• .env.example ← Configuration template for room ID, tokens, API keys
• requirements.txt
• agent/
• agent.py ← Core session manager: joins room, receives audio, sends responses
• audio_stream_track.py ← Wraps Video SDK’s custom audio track for ingestion & output
• stt/
• stt.py ← Defines STT interface, streaming pipeline
• deepgram_stt.py ← Deepgram-specific implementation
• intelligence/
• intelligence.py ← Encapsulates prompt management, context window
• intelligence_client.py ← Handles LLM API calls (OpenAI or custom)
• tts/
• tts.py ← Defines TTS interface
• elevenlabs_tts.py ← Eleven Labs-specific implementation
Key Insights from README
• The pipeline: Video SDK ↔ STT ↔ LLM ↔ TTS ↔ Video SDK
• External services: Deepgram (Streaming STT), OpenAI (GPT-4), Eleven Labs (multilingual TTS)
• Supports an optional Vector DB for RAG between STT and LLM.