# humanaigc-engineering/openavatarchat architecture

> Architecture diagram of the GitHub repository humanaigc-engineering/openavatarchat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/humanaigc-engineering/openavatarchat
- Repository: https://github.com/humanaigc-engineering/openavatarchat

## Overview

##### Project type and purpose
- OpenAvatarChat is a full-stack, modular interactive digital-human conversation application. It runs a single Python process that exposes a browser UI (via Gradio) and orchestrates real-time audio/video pipelines—including VAD, ASR, LLM, TTS—and avatar rendering. It supports two rendering modes (server-rendered WebRTC video or client-rendered H5 avatar) and two conversation modes (multimodal MiniCPM-o speech2speech or classical ASR→LLM→TTS).

##### Main components
A. Frontend Layer
- Browser UI (Gradio application)
- Two client handlers:
– H5 avatar renderer (LAM client)
– WebRTC videochat client
B. Service Layer
- Gradio server (src/demo.py + src/service modules)
- RTC Service (src/service/rtc_service) managing WebRTC peer connections and TURN
C. Engine Layer
- ChatEngine (src/chat_engine/chat_engine.py) as orchestrator
- HandlerManager (loads per-config handler modules)
- ChatSession (maintains per-user state)
D. Handler Layer (dynamically loaded plugins under src/handlers)
- VAD (SileroVAD)
- ASR (SenseVoice)
- LLM (MiniCPM-o local or OpenAI-compatible API)
- TTS (CosyVoice local or API, EdgeTTS, Bailian API)
- Avatar (LiteAvatar or LAM Audio2Expression)
E. Data Models
- chat_data, chat_signal, runtime_data bundles passed between engine and handlers
F. External Systems
- External LLM APIs (OpenAI-compatible, Bailian)
- CosyVoice API
- Edge TTS service
- Model storage (local MiniCPM-o under /models)
- TURN server for NAT traversal
G. Infrastructure & Configuration
- Config files (config/*.yaml)
- SSL certificates (ssl_certs/)
- Docker (Dockerfile + build_and_run.sh)
- Scripts (model download, cert generation, coturn setup)

##### Relationships and data flow
1. User speaks → audio stream via WebRTC → RTC Service
2. Audio frames → VAD → ASR → text transcript
3. Transcript (+ optional video frame) → ChatEngine → LLM Handler → response (text or audio)
4. If response is text → TTS → audio waveform
5. Audio (+ text metadata) → Avatar Handler → video frames (2D or 3D avatar)
6. Packaged audio/video → RTC Service → Browser UI
7. In H5 client mode, the client pulls pre-computed avatar assets instead of streaming video

##### Architectural patterns and design principles
- Plugin-based modular architecture: HandlerManager discovers and loads interchangeable handler modules according to YAML config
- Layered separation:
- Presentation (Browser/Gradio UI)
- Transport (RTC Service)
- Orchestration (ChatEngine)
- Business logic (Handlers)
- Config-driven behavior: Different pipelines (MiniCPM-o mode vs ASR+LLM+TTS mode) are selected at startup via config files

##### Essential elements to capture
- Separation of concerns (UI vs transport vs orchestration vs handlers)
- Two conversation modes and two rendering modes
- Dynamic, config-driven handler loading
- Data flow sequence from user input to avatar output
- External dependencies (APIs, models, TURN server)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        UI["Browser UI (Gradio)"]:::frontend
        H5["H5 Avatar Renderer (LAM client)"]:::frontend
        RTCClient["WebRTC Videochat Client"]:::frontend
    end

    %% Service Layer
    subgraph "Service Layer"
        Gradio["Gradio Server"]:::service
        RTCService["RTC Service"]:::service
    end

    %% Engine Layer
    subgraph "Engine Layer"
        ChatEngine["ChatEngine"]:::engine
        HandlerManager["HandlerManager"]:::engine
        ChatSession["ChatSession"]:::engine
    end

    %% Handler Layer
    subgraph "Handler Layer"
        VAD["VAD (SileroVAD)"]:::handler
        ASR["ASR (SenseVoice)"]:::handler
        LLM1["LLM (MiniCPM-o)"]:::handler
        LLM2["LLM (OpenAI-compatible)"]:::handler
        TTS1["TTS (CosyVoice)"]:::handler
        TTS2["TTS (EdgeTTS)"]:::handler
        TTS3["TTS (Bailian)"]:::handler
        Avatar1["Avatar (LiteAvatar)"]:::handler
        Avatar2["Avatar (LAM Audio2Expression)"]:::handler
    end

    %% Data Models
    subgraph "Data Models"
        ChatData["chat_data bundle"]:::data
        ChatSignal["chat_signal"]:::data
        RuntimeData["runtime_data bundle"]:::data
    end

    %% External Systems
    subgraph "External Systems"
        OpenAIAPI["OpenAI-compatible API"]:::external
        BailianAPI["Bailian API"]:::external
        CosyVoiceAPI["CosyVoice API"]:::external
        EdgeTTS["Edge TTS Service"]:::external
        ModelStorage["Model Storage (MiniCPM-o)"]:::external
        TURN["TURN Server"]:::external
    end

    %% Infrastructure & Config
    subgraph "Infra & Config"
        Config["Config Files (YAML)"]:::infra
        SSLCerts["SSL Certificates"]:::infra
        Docker["Docker Setup"]:::infra
        Scripts["Auxiliary Scripts"]:::infra
    end

    %% Connections: Frontend -> Service
    UI -->|"ui requests"| Gradio
    RTCClient -->|"media stream"| RTCService
    H5 -->|pull assets| ModelStorage

    %% Service -> Engine
    Gradio -->|"API calls"| ChatEngine
    RTCService -->|"audio/video packets"| ChatEngine
    RTCService -.->|uses TURN| TURN

    %% Engine internals
    ChatEngine -->|"loads handlers"| HandlerManager
    HandlerManager -->|manages| ChatSession
    ChatEngine -->|"session state"| ChatSession

    %% Data flow: Multimodal vs Classical
    ChatEngine -->|"audio frames"| VAD
    VAD -->|"speech segments"| ASR
    ASR -->|"transcript"| ChatEngine

    ChatEngine -->|"for MiniCPM-o"| LLM1
    LLM1 -->|"response data"| ChatEngine

    ChatEngine -->|"for OpenAI"| LLM2
    LLM2 -->|"API call"| OpenAIAPI
    LLM2 -->|"API call"| BailianAPI
    LLM2 -->|"response text"| ChatEngine

    ChatEngine -->|"text output"| TTS1
    ChatEngine -->|"text output"| TTS2
    ChatEngine -->|"text output"| TTS3
    TTS1 -->|"audio waveform"| ChatEngine
    TTS2 -->|"audio waveform"| ChatEngine
    TTS3 -->|"audio waveform"| ChatEngine
    TTS1 -->|"may call"| CosyVoiceAPI
    TTS3 -->|"may call"| CosyVoiceAPI
    TTS2 -->|"calls"| EdgeTTS

    ChatEngine -->|"audio+text"| Avatar1
    ChatEngine -->|"audio+text"| Avatar2
    Avatar1 -->|"avatar frames"| ChatEngine
    Avatar2 -->|"avatar assets"| ModelStorage

    ChatEngine -->|"pack media"| RTCService
    RTCService -->|"stream to ui"| RTCClient

    %% Dynamic Data Models
    ChatEngine -->|uses| ChatData
    ChatEngine -->|signals| ChatSignal
    ChatEngine -->|context| RuntimeData

    %% Infra & Config usage
    HandlerManager -->|reads| Config
    Gradio -->|uses| SSLCerts
    Docker -->|builds| Gradio
    Scripts -->|setup| TURN
    Scripts -->|download models| ModelStorage

    %% Click Events
    click UI "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/demo.py"
    click H5 "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/client/h5_rendering_client/client_handler_lam.py"
    click RTCClient "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/client/rtc_client/client_handler_rtc.py"
    click Gradio "https://github.com/humanaigc-engineering/openavatarchat/tree/main/src/service/"
    click RTCService "https://github.com/humanaigc-engineering/openavatarchat/tree/main/src/service/rtc_service/"
    click ChatEngine "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/chat_engine/chat_engine.py"
    click HandlerManager "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/chat_engine/core/handler_manager.py"
    click ChatSession "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/chat_engine/core/chat_session.py"
    click ChatData "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/chat_engine/data_models/chat_data/chat_data_model.py"
    click ChatSignal "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/chat_engine/data_models/chat_signal.py"
    click RuntimeData "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/chat_engine/data_models/runtime_data/data_bundle.py"
    click VAD "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/vad/silerovad/vad_handler_silero.py"
    click ASR "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/asr/sensevoice/asr_handler_sensevoice.py"
    click LLM1 "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/llm/minicpm/llm_handler_minicpm.py"
    click LLM2 "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/llm/openai_compatible/llm_handler_openai_compatible.py"
    click TTS1 "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/tts/cosyvoice/tts_handler_cosyvoice.py"
    click TTS2 "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/tts/edgetts/tts_handler_edgetts.py"
    click TTS3 "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/tts/bailian_tts/tts_handler_cosyvoice_bailian.py"
    click Avatar1 "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/avatar/liteavatar/avatar_handler_liteavatar.py"
    click Avatar2 "https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/avatar/lam/avatar_handler_lam_audio2expression.py"

    %% Styles
    classDef frontend fill:#D0E8FF,stroke:#0288D1,color:#000
    classDef service fill:#D7F3D7,stroke:#2E7D32,color:#000
    classDef engine fill:#FFE0B2,stroke:#EF6C00,color:#000
    classDef handler fill:#E1BEE7,stroke:#8E24AA,color:#000
    classDef data fill:#FFF9C4,stroke:#F9A825,color:#000
    classDef external fill:#ECEFF1,stroke:#546E7A,color:#000
    classDef infra fill:#CFD8DC,stroke:#455A64,color:#000
```
