# yolo-zyl/livetalking architecture

> Architecture diagram of the GitHub repository yolo-zyl/livetalking, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yolo-zyl/livetalking
- Repository: https://github.com/yolo-zyl/livetalking

## Overview

##### Project Type and Purpose
- This is a full-stack, real-time interactive streaming “digital human” platform.
- It ingests live audio, runs ASR → LLM chat → TTS/voice-clone → lip-sync & face/head animation (via ErNeRF, MuseTalk, or Wav2Lip) → streams synchronized audio/video back to the browser.
- Supports WebRTC & RTMP ingestion/output, multiple model backends, and custom video playback when silent.

##### High-Level Component Analysis
a. Front-End (web/)
- Static HTML/JS clients for: ASR demo, WebRTC push/pull, echo tests, chat UI.
- Uses recorder-core.js, WebRTC APIs, SRS SDK to publish/subcribe media channels.
b. Streaming Server (SRS)
- External containerized SRS instance (Docker) handling RTMP/WebRTC ingest and distribution.
c. Back-End Orchestrator (app.py + support scripts)
- app.py: central asyncio server receiving incoming media streams, routing through pipelines, servicing HTTP/WebSocket control endpoints (ports 8010 etc.).
- Dockerfile and deployment scripts.
d. Audio Processing
- baseasr.py / lipasr.py / museasr.py / nerfasr.py: wrappers around ASR engines (e.g., Whisper, wav2vec).
- llm/: ChatGPT/GPT/Qwen clients for generating bot replies.
- ttsreal.py / muse tts / edgetts integration for text-to-speech and voice cloning.
e. Video Generation
- ernerf/: Neural Radiance Field based talking head (freqencoder, gridencoder, raymarching, nerf_triplane).
- musetalk/: VAE+UNet-based face models + full-body compositing.
- wav2lip/: Conv-based lip-sync model + face detection.
f. Data & Models
- data/pretrained: PTH checkpoints for ErNeRF & custom ASR.
- muse tts & museasr pretrained checkpoints.
- wav2lip model weights.
g. External Services
- Hugging Face mirrors (HF_ENDPOINT override).
- Docker registry for SRS and base images.

##### Key Architectural Patterns
- Modular “plugin” pipelines for ASR, TTS, video engines: choose at runtime via configuration (data/custom_config.json).
- Asynchronous, streaming-first design: non-blocking I/O for media.
- Microservice-like separation: front-end client, media bus (SRS), compute back-end.

- Client Browser (HTML/JS)
- SRS Streaming Server
- Backend App Server
- ASR & LLM Pipeline
- TTS + Voice Clone
- Video Renderer Module (ErNeRF | MuseTalk | Wav2Lip)
- Data/Model Storage
- External HF Mirror
1. Browser → (WebRTC / RTMP) → SRS Ingest
2. SRS → (WebSocket / HTTP pull) → app.py
3. app.py → ASR → LLM → TTS → Voice Clone
4. app.py + Audio → Video Renderer → Synthesized Frames
5. app.py → (encode) → SRS Publish
6. Browser ← (WebRTC / RTMP) ← SRS
c. Asynchronous Steps:
d. Technologies:
- ASR: Whisper, wav2vec; LLM: ChatGPT/GPT/Qwen; TTS: edgetts; Rendering: PyTorch (CUDA), Pytorch3D, custom C++/CUDA kernels; Front-end: recorder-core.js, srs.sdk.js; Deployment: Docker.
e. Patterns:
- Plugin modules in separate subdirectories.
- Shared data models under data/.
- Dockerized components (SRS, app).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User-Facing Components
    subgraph "User-Facing" 
        Browser["Browser\n(HTML/JS,\nrecorder-core.js,\nsrs.sdk.js)"]:::frontend
    end
    click Browser "https://github.com/yolo-zyl/livetalking/tree/main/web/"

    %% Streaming Infrastructure
    subgraph "Streaming Infrastructure"
        SRS["SRS Streaming Server\n(Dockerized)"]:::streaming
    end

    %% Backend Orchestrator
    subgraph "Backend Orchestrator"
        AppServer["app.py\n(async HTTP/WebSocket)"]:::backend
        WebRTCControl["webrtc.py"]:::backend
        DockerFile["Dockerfile"]:::backend
    end
    click AppServer "https://github.com/yolo-zyl/livetalking/blob/main/app.py"
    click WebRTCControl "https://github.com/yolo-zyl/livetalking/blob/main/webrtc.py"
    click DockerFile "https://github.com/yolo-zyl/livetalking/tree/main/Dockerfile"

    %% Compute Pipelines
    subgraph "Compute Pipelines"
        %% ASR Modules
        subgraph "ASR Pipeline" 
            BaseASR["baseasr.py"]:::compute
            LipASR["lipasr.py"]:::compute
            MuseASR["museasr.py"]:::compute
            NerfASR["nerfasr.py"]:::compute
        end
        click BaseASR "https://github.com/yolo-zyl/livetalking/blob/main/baseasr.py"
        click LipASR "https://github.com/yolo-zyl/livetalking/blob/main/lipasr.py"
        click MuseASR "https://github.com/yolo-zyl/livetalking/blob/main/museasr.py"
        click NerfASR "https://github.com/yolo-zyl/livetalking/blob/main/nerfasr.py"

        %% LLM Clients
        LLM["LLM Clients\n(ChatGPT/GPT/Qwen)"]:::compute
    end
    click LLM "https://github.com/yolo-zyl/livetalking/tree/main/llm/"

    %% TTS & Voice Cloning
    subgraph "TTS & Voice Cloning"
        TTSReal["ttsreal.py"]:::compute
        BaseReal["basereal.py"]:::compute
        MuseReal["musereal.py"]:::compute
    end
    click TTSReal "https://github.com/yolo-zyl/livetalking/blob/main/ttsreal.py"
    click BaseReal "https://github.com/yolo-zyl/livetalking/blob/main/basereal.py"
    click MuseReal "https://github.com/yolo-zyl/livetalking/blob/main/musereal.py"

    %% Video Renderer Module
    subgraph "Video Renderer"
        ErNeRF["ErNeRF\n(Nerf-based\nTalking Head)"]:::compute
        MuseTalk["MuseTalk\n(VAE+UNet\nFace/Body)"]:::compute
        Wav2Lip["Wav2Lip\n(Conv Lip-Sync)"]:::compute
    end
    click ErNeRF "https://github.com/yolo-zyl/livetalking/tree/main/ernerf/"
    click MuseTalk "https://github.com/yolo-zyl/livetalking/tree/main/musetalk/"
    click Wav2Lip "https://github.com/yolo-zyl/livetalking/tree/main/wav2lip/"

    %% Storage & Config
    subgraph "Data & Config"
        Config["custom_config.json"]:::storage
        Pretrained["ngp_kf.pth"]:::storage
    end
    click Config "https://github.com/yolo-zyl/livetalking/blob/main/data/custom_config.json"
    click Pretrained "https://github.com/yolo-zyl/livetalking/blob/main/data/pretrained/ngp_kf.pth"

    %% External Services
    subgraph "External Services"
        HF["Hugging Face Mirror"]:::external
        DockerReg["Docker Registry"]:::external
    end

    %% Data Flows
    Browser -->|WebRTC/RTMP| SRS
    SRS -->|WebSocket/HTTP| AppServer
    AppServer -->|"async queue"| BaseASR
    AppServer -->|"async queue"| LipASR
    AppServer -->|"async queue"| MuseASR
    AppServer -->|"async queue"| NerfASR
    BaseASR -->|text| LLM
    LipASR -->|text| LLM
    MuseASR -->|text| LLM
    NerfASR -->|text| LLM
    LLM -->|tts request| TTSReal
    LLM -->|tts request| BaseReal
    LLM -->|tts request| MuseReal
    TTSReal -->|audio| ErNeRF
    BaseReal -->|audio| MuseTalk
    MuseReal -->|audio| Wav2Lip
    ErNeRF -->|frames| AppServer
    MuseTalk -->|frames| AppServer
    Wav2Lip -->|frames| AppServer
    AppServer -->|encode| SRS
    SRS -->|WebRTC/RTMP| Browser

    %% Dependencies
    Config --> BaseASR
    Config --> LipASR
    Config --> MuseASR
    Config --> NerfASR
    Config --> TTSReal
    Config --> BaseReal
    Config --> MuseReal
    Config --> ErNeRF
    Config --> MuseTalk
    Config --> Wav2Lip

    Pretrained --> ErNeRF
    Pretrained --> MuseTalk
    Pretrained --> Wav2Lip

    HF --> BaseASR
    HF --> LipASR
    HF --> MuseASR
    HF --> NerfASR
    HF --> TTSReal
    HF --> BaseReal
    HF --> MuseReal
    HF --> ErNeRF
    HF --> MuseTalk
    HF --> Wav2Lip

    DockerReg --> SRS
    DockerReg --> AppServer

    %% Styles
    classDef frontend fill:#ccffcc,stroke:#333,stroke-width:1px
    classDef streaming fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef backend fill:#ffebcc,stroke:#333,stroke-width:1px
    classDef compute fill:#ffe6cc,stroke:#333,stroke-width:1px
    classDef storage fill:#e0e0e0,stroke:#333,stroke-width:1px
    classDef external fill:#f0f0f0,stroke:#333,stroke-width:1px
```
