How humanaigc-engineering/openavatarchat works
Project type and purpose
• OpenAvatarChat is a full-stack, modular interactive digital-human conversation application. It runs a single Python process that exposes a browser UI (via Gradio) and orchestrates real-time audio/video pipelines—including VAD, ASR, LLM, TTS—and avatar rendering. It supports two rendering modes (server-rendered WebRTC video or client-rendered H5 avatar) and two conversation modes (multimodal MiniCPM-o speech2speech or classical ASR→LLM→TTS).
Main components
A. Frontend Layer
• Browser UI (Gradio application)
• Two client handlers:
– H5 avatar renderer (LAM client)
– WebRTC videochat client
B. Service Layer
• Gradio server (src/demo.py + src/service modules)
• RTC Service (src/service/rtc_service) managing WebRTC peer connections and TURN
C. Engine Layer
• ChatEngine (src/chat_engine/chat_engine.py) as orchestrator
• HandlerManager (loads per-config handler modules)
• ChatSession (maintains per-user state)
D. Handler Layer (dynamically loaded plugins under src/handlers)
• VAD (SileroVAD)
• ASR (SenseVoice)
• LLM (MiniCPM-o local or OpenAI-compatible API)
• TTS (CosyVoice local or API, EdgeTTS, Bailian API)
• Avatar (LiteAvatar or LAM Audio2Expression)
E. Data Models
• chat_data, chat_signal, runtime_data bundles passed between engine and handlers
F. External Systems
• External LLM APIs (OpenAI-compatible, Bailian)
• CosyVoice API
• Edge TTS service
• Model storage (local MiniCPM-o under /models)
• TURN server for NAT traversal
G. Infrastructure & Configuration
• Config files (config/*.yaml)
• SSL certificates (ssl_certs/)
• Docker (Dockerfile + build_and_run.sh)
• Scripts (model download, cert generation, coturn setup)
Relationships and data flow
1. User speaks → audio stream via WebRTC → RTC Service
2. Audio frames → VAD → ASR → text transcript
3. Transcript (+ optional video frame) → ChatEngine → LLM Handler → response (text or audio)
4. If response is text → TTS → audio waveform
5. Audio (+ text metadata) → Avatar Handler → video frames (2D or 3D avatar)
6. Packaged audio/video → RTC Service → Browser UI
7. In H5 client mode, the client pulls pre-computed avatar assets instead of streaming video
Architectural patterns and design principles
• Plugin-based modular architecture: HandlerManager discovers and loads interchangeable handler modules according to YAML config
• Layered separation:
• Presentation (Browser/Gradio UI)
• Transport (RTC Service)
• Orchestration (ChatEngine)
• Business logic (Handlers)
• Config-driven behavior: Different pipelines (MiniCPM-o mode vs ASR+LLM+TTS mode) are selected at startup via config files
Essential elements to capture
• Separation of concerns (UI vs transport vs orchestration vs handlers)
• Two conversation modes and two rendering modes
• Dynamic, config-driven handler loading
• Data flow sequence from user input to avatar output
• External dependencies (APIs, models, TURN server)