# mu-l/livetalking architecture

> Architecture diagram of the GitHub repository mu-l/livetalking, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mu-l/livetalking
- Repository: https://github.com/mu-l/livetalking

## Overview

##### Project type and purpose
- This is a full-stack, real-time interactive “digital human” streaming system.
- It exposes a Python-based backend (app.py + various model modules) that orchestrates ASR → language (LLM) → TTS → avatar rendering pipelines, and a web-based frontend for WebRTC/RTMP streaming of audio/video.

##### File structure
- Top-level entry points:
- app.py / webrtc.py – main server processes (HTTP/WebSocket/WebRTC signaling, transport selection)
- Dockerfile – containerization
- requirements.txt – Python dependencies (PyTorch, etc.)
- Service modules (all under project root):
- ASR: baseasr.py, hubertasr.py, lipasr.py, lipreal.py, lightreal.py, museasr.py, musereal.py
- Whisper ASR: musetalk/whisper/...
- TTS: ttsreal.py (plus fish-speech, Tencent, 豆包, etc.)
- LLM: llm.py (conversation management, optional chat model)
- Avatar rendering: wav2lip/, ultralight/, musetalk/ (models/unet.py, vae.py, simple_musetalk.py), ernerf branch (in separate git branch)
- Utilities: logger.py, muse/utils, wav2lip/face_detection, museTalk/utils, ultralight/audio2feature & genavatar
- Frontend: web/ (HTML/JS pages for ASR, chat, webrtc, RTMP push, virtual camera)
- Data & static assets: data/avatars, models/ (place for downloaded .pth), assets/ (demo, FAQ)

##### README for additional insights
- Features: multi-model support (wav2lip256, musetalk, Ultr​alight, ernerf), voice cloning, interruptible speech, full-body splicing, action choreography, multi-concurrency
- Transports: WebRTC, RTMP, virtual camera output
- External services: optional SRS server, HuggingFace mirror, cloud-hosted Docker images

##### Main components
a. Clients:
- Browser UI (web/*.html + JS)
- Custom client (native executable)
b. Signaling & Transport Layer:
- HTTP / WebSocket API (app.py)
- WebRTC signaling & media connections (webrtc.py)
- RTMP/WebSocket push (SRS or built-in)
c. Backend Orchestrator:
- Request dispatcher (audio/video ↔ pipelines)
- Module registry (chooses ASR, TTS, avatar model per request)
- Concurrency manager (asyncio / multi-thread)
d. ASR Subsystem:
- Whisper or custom ASR modules
- API: audio frames in → text out
e. Language / Dialogue Subsystem:
- llm.py (optional LLM/chat model)
- Handles interruptions, context management
f. TTS Subsystem:
- ttsreal.py + plugins (fish-speech, Tencent, 豆包)
- audio synthesis: text in → waveform out
g. Avatar Rendering Subsystem:
- wav2lip module (lip sync)
- musetalk (UNet + VAE)
- ultralight (lightweight digital human)
- ernerf (NeRF-based multi-view)
- Preprocessing (face detection, segmentation)
- Model loading & inference
h. Media Mixer & Encoder:
- Composes audio + video frames
- Splices full-body, overlays custom “idle” animations when silent
- Virtual camera output plugin
i. Transport Outbound:
- WebRTC media streams back to client
- RTMP push to SRS or CDN
j. Data & Configuration:
- data/avatars (avatar assets)
- data/custom_config.json
- models/*.pth
k. External Dependencies:
- SRS streaming server (optional)
- HuggingFace model hub / mirrors
- Docker registry

##### Relationships & data flows
1) Client → HTTP/WebSocket (start session, choose transport & model)
2) Client → audio media → Signaling/Transport
3) Transport layer → Orchestrator → ASR → text transcript
4) Transcript → (optional) LLM → dialog text
5) Text → TTS → audio waveform
6) Audio + silent intervals → Avatar renderer → video frames
7) Media Mixer → combined A/V stream
8) Stream → Transport outbound → Client

##### Architectural patterns & principles
- Modular “plugin” architecture for ASR/TTS/avatar engines
- Pipeline pattern for sequential media processing
- Async / concurrency to handle multiple concurrent sessions without GPU memory blow-up
- Separation of concerns: signaling, inference, media encoding, transport

##### Essential elements in detail
- Concurrency manager interacting with each inference component
- Realtime constraints (25+ fps target, CPU vs GPU bottlenecks)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        BrowserUI["Browser UI"]:::frontend
        CustomClient["Custom Client"]:::frontend
    end

    %% Signaling & Transport Layer
    subgraph "Signaling & Transport Layer"
        direction TB
        HTTPAPI["HTTP/WebSocket API"]:::frontend
        WebRTC["WebRTC Signaling & Media"]:::frontend
    end

    %% Backend Orchestration
    subgraph "Backend Orchestrator"  
        direction TB
        Orchestrator["Request Dispatcher / Module Registry / Concurrency Manager"]:::orchestration
    end

    %% Inference Subsystems
    subgraph "Inference Subsystems"
        direction TB

        subgraph "ASR Subsystem"  
            direction TB
            ASRPlugins["ASR Plugins"]:::inference
            subgraph "Custom ASR Engines"
                direction TB
                BaseASR["baseasr.py"]:::inference
                HubERTASR["hubertasr.py"]:::inference
                LipASR["lipasr.py"]:::inference
                LipReal["lipreal.py"]:::inference
                LightReal["lightreal.py"]:::inference
                MuseASR["museasr.py"]:::inference
                MuseReal["musereal.py"]:::inference
            end
            subgraph "Whisper Engine"
                direction TB
                Whisper["Whisper Models"]:::inference
            end
        end

        LLM["Language / Dialogue (llm.py)"]:::inference

        TTS["TTS Engine (ttsreal.py)"]:::inference

        subgraph "Avatar Rendering Subsystem"
            direction TB
            Wav2Lip["Wav2Lip & Utilities"]:::inference
            MuseTalk["MuseTalk Model & Utils"]:::inference
            UltraLight["UltraLight Model & Utils"]:::inference
        end
    end

    %% Media Mixer & Encoder
    MediaMixer["Media Mixer & Encoder (basereal.py)"]:::orchestration

    %% Data & Configuration Storage
    subgraph "Data & Config Storage"
        direction TB
        Avatars["Avatar Assets"]:::storage
        Config["Custom Config"]:::storage
        ModelsStore["Model Store (.pth files)"]:::storage
    end

    %% Utilities
    Utils["Utilities & Logging (logger.py)"]:::orchestration

    %% External Services
    SRS["SRS Streaming Server"]:::external
    HFGateway["HuggingFace Model Hub"]:::external

    %% Docker Container Boundary
    subgraph "Docker Container"
        direction TB
        HTTPAPI
        WebRTC
        Orchestrator
        ASRPlugins
        LLM
        TTS
        Wav2Lip
        MuseTalk
        UltraLight
        MediaMixer
        Utils
        Avatars
        Config
        ModelsStore
    end

    %% Data Flows
    BrowserUI -->|"start session"| HTTPAPI
    CustomClient -->|"start session"| HTTPAPI
    BrowserUI -->|"media signaling"| WebRTC
    CustomClient -->|"media signaling"| WebRTC

    WebRTC -->|"audio frames"| Orchestrator
    HTTPAPI -->|"control requests"| Orchestrator

    Orchestrator -->|"audio frames"| ASRPlugins
    ASRPlugins -->|"text transcript"| LLM
    ASRPlugins -->|"text transcript"| TTS
    LLM -->|"dialog text"| TTS
    TTS -->|"audio waveform"| MediaMixer

    Orchestrator -->|"audio + silent intervals"| Wav2Lip
    Orchestrator -->|"audio + silent intervals"| MuseTalk
    Orchestrator -->|"audio + silent intervals"| UltraLight
    Wav2Lip -->|"video frames"| MediaMixer
    MuseTalk -->|"video frames"| MediaMixer
    UltraLight -->|"video frames"| MediaMixer

    MediaMixer -->|"A/V stream"| WebRTC
    MediaMixer -->|"A/V stream"| SRS

    Orchestrator -->|"module logs"| Utils
    Orchestrator -->|"load models"| ModelsStore
    Orchestrator -->|"read config"| Config
    Orchestrator -->|"read avatars"| Avatars
    Orchestrator -->|"fetch models"| HFGateway

    %% Click Events
    click BrowserUI "https://github.com/mu-l/livetalking/tree/main/web/"
    click HTTPAPI "https://github.com/mu-l/livetalking/blob/main/app.py"
    click WebRTC "https://github.com/mu-l/livetalking/blob/main/webrtc.py"
    click Orchestrator "https://github.com/mu-l/livetalking/blob/main/app.py"
    click BaseASR "https://github.com/mu-l/livetalking/blob/main/baseasr.py"
    click HubERTASR "https://github.com/mu-l/livetalking/blob/main/hubertasr.py"
    click LipASR "https://github.com/mu-l/livetalking/blob/main/lipasr.py"
    click LipReal "https://github.com/mu-l/livetalking/blob/main/lipreal.py"
    click LightReal "https://github.com/mu-l/livetalking/blob/main/lightreal.py"
    click MuseASR "https://github.com/mu-l/livetalking/blob/main/museasr.py"
    click MuseReal "https://github.com/mu-l/livetalking/blob/main/musereal.py"
    click Whisper "https://github.com/mu-l/livetalking/tree/main/musetalk/whisper/"
    click LLM "https://github.com/mu-l/livetalking/blob/main/llm.py"
    click TTS "https://github.com/mu-l/livetalking/blob/main/ttsreal.py"
    click Wav2Lip "https://github.com/mu-l/livetalking/tree/main/wav2lip/"
    click MuseTalk "https://github.com/mu-l/livetalking/tree/main/musetalk/"
    click UltraLight "https://github.com/mu-l/livetalking/tree/main/ultralight/"
    click Utils "https://github.com/mu-l/livetalking/blob/main/logger.py"
    click Avatars "https://github.com/mu-l/livetalking/tree/main/data/avatars/"
    click Config "https://github.com/mu-l/livetalking/blob/main/data/custom_config.json"
    click ModelsStore "https://github.com/mu-l/livetalking/tree/main/models/"

    %% Styles
    classDef frontend fill:#AED6F1,stroke:#1F618D,color:#1F618D
    classDef orchestration fill:#ABEBC6,stroke:#196F3D,color:#196F3D
    classDef inference fill:#FAD7A0,stroke:#B9770E,color:#B9770E
    classDef storage fill:#D5DBDB,stroke:#717D7E,color:#717D7E
    classDef external fill:#F2D7D5,stroke:#943126,color:#943126
```
