# lipku/metahuman-stream architecture

> Architecture diagram of the GitHub repository lipku/metahuman-stream, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lipku/metahuman-stream
- Repository: https://github.com/lipku/metahuman-stream

## Overview

##### Project Type and Purpose
This is a complex AI-powered streaming application that creates real-time interactive digital humans with audio-video synchronization. The system supports multiple digital human models (ernerf, musetalk, wav2lip, Ultralight-Digital-Human) and features like voice cloning and video streaming.

##### Main Components

A. Frontend Layer:
- Web Interface (web/ directory)
- WebRTC/RTMP Streaming Client
- User Interaction Components

B. Backend Services:
- Main Application Server (app.py)
- WebRTC/RTMP Streaming Server (SRS)
- Model Serving Layer

C. AI Model Components:
- Digital Human Models
- ERNeRF
- MuseTalk
- Wav2Lip
- Ultralight-Digital-Human
- Voice Processing
- ASR (Automatic Speech Recognition)
- TTS (Text-to-Speech)
- LLM Integration (llm/ directory)

D. Data Processing Pipeline:
- Face Detection & Tracking
- Audio Processing
- Video Generation
- Stream Synchronization

**[User Interface Layer]**
- ↕
**[Streaming Layer (WebRTC/RTMP)]**
- ↕
**[Application Server]**
- ↕
**[Model Orchestration Layer]**
- ↓
**┌─────────────────────────────────┐**
- AI Processing Pipeline       │
- ┬──────────┬───────────┤
- Video  │  Audio   │   LLM     │
- Models  │ Process. │ Services  │
- ┴──────────┴───────────┘

##### Key Relationships to Highlight

a. Data Flow:
- User input → ASR → LLM → TTS → Digital Human Generation
- Audio/Video synchronization path
- Streaming data flow

b. Component Integration:
- Model interaction patterns
- Service communication
- Resource sharing

##### Technical Details

- CUDA/GPU utilization
- WebRTC/RTMP protocols
- Model preprocessing pipelines
- Real-time processing components
- Concurrent processing handling

##### Implementation Notes

- Use standardized UML notation where appropriate
- System requirements (CUDA, Python version)
- GPU-dependent components
- Memory management for concurrent operations
- Model preheating mechanisms

This architecture should be represented as a living document that can be updated as the project evolves with new features and optimizations.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph UI["User Interface Layer"]
        WEB["Web Interface"]:::frontend
        STREAM["WebRTC/RTMP Client"]:::frontend
        ASR_WEB["ASR Web Client"]:::frontend
    end

    subgraph SL["Streaming Layer"]
        SRS["SRS Server"]:::streaming
        WRTC["WebRTC Implementation"]:::streaming
    end

    subgraph AL["Application Layer"]
        APP["Main Application Server"]:::server
        BASE["Base Real-time Processing"]:::server
    end

    subgraph ML["Model Orchestration Layer"]
        NREAL["NeRF Real-time"]:::process
        LREAL["Lip Real-time"]:::process
        MREAL["Muse Real-time"]:::process
        TREAL["TTS Real-time"]:::process
    end

    subgraph AIP["AI Processing Pipeline"]
        subgraph VM["Video Models"]
            ERN["ERNeRF"]:::model
            MUS["MuseTalk"]:::model
            WAV["Wav2Lip"]:::model
            ULT["Ultralight"]:::model
        end

        subgraph FD["Face Detection & Tracking"]
            ERN_FT["ERNeRF Face Tracking"]:::process
            MUS_FD["MuseTalk Face Detection"]:::process
            WAV_FD["Wav2Lip Face Detection"]:::process
        end

        subgraph AP["Audio Processing"]
            WHSP["Whisper ASR"]:::process
            A2F["Audio Feature Extraction"]:::process
        end

        subgraph VG["Video Generation"]
            ERN_R["ERNeRF Renderer"]:::process
            MUS_G["MuseTalk Generator"]:::process
            WAV_G["Wav2Lip Generator"]:::process
            ULT_G["Ultralight Generator"]:::process
        end

        subgraph LL["LLM Services"]
            GPT["ChatGPT"]:::llm
            GEM["Gemini"]:::llm
            QWN["Qwen"]:::llm
        end
    end

    %% Connections
    WEB --> SRS
    STREAM --> WRTC
    ASR_WEB --> APP
    SRS --> APP
    WRTC --> APP
    
    APP --> BASE
    BASE --> NREAL & LREAL & MREAL & TREAL
    
    NREAL --> ERN
    LREAL --> WAV
    MREAL --> MUS
    TREAL --> ULT
    
    ERN --> ERN_FT
    MUS --> MUS_FD
    WAV --> WAV_FD
    
    ERN_FT --> ERN_R
    MUS_FD --> MUS_G
    WAV_FD --> WAV_G
    ULT --> ULT_G
    
    WHSP --> APP
    A2F --> ULT_G
    
    APP --> GPT & GEM & QWN

    %% Click Events
    click WEB "https://github.com/lipku/metahuman-stream/tree/main/web/"
    click ASR_WEB "https://github.com/lipku/metahuman-stream/tree/main/web/asr/"
    click WRTC "https://github.com/lipku/metahuman-stream/blob/main/web/srs.sdk.js"
    click APP "https://github.com/lipku/metahuman-stream/blob/main/app.py"
    click BASE "https://github.com/lipku/metahuman-stream/blob/main/basereal.py"
    click NREAL "https://github.com/lipku/metahuman-stream/blob/main/nerfreal.py"
    click LREAL "https://github.com/lipku/metahuman-stream/blob/main/lipreal.py"
    click MREAL "https://github.com/lipku/metahuman-stream/blob/main/musereal.py"
    click TREAL "https://github.com/lipku/metahuman-stream/blob/main/ttsreal.py"
    click ERN "https://github.com/lipku/metahuman-stream/tree/main/ernerf/"
    click MUS "https://github.com/lipku/metahuman-stream/tree/main/musetalk/"
    click WAV "https://github.com/lipku/metahuman-stream/tree/main/wav2lip/"
    click ULT "https://github.com/lipku/metahuman-stream/tree/main/ultralight/"
    click ERN_FT "https://github.com/lipku/metahuman-stream/tree/main/ernerf/data_utils/face_tracking/"
    click MUS_FD "https://github.com/lipku/metahuman-stream/tree/main/musetalk/utils/face_detection/"
    click WAV_FD "https://github.com/lipku/metahuman-stream/tree/main/wav2lip/face_detection/"
    click WHSP "https://github.com/lipku/metahuman-stream/tree/main/musetalk/whisper/"
    click A2F "https://github.com/lipku/metahuman-stream/blob/main/ultralight/audio2feature.py"
    click ERN_R "https://github.com/lipku/metahuman-stream/blob/main/ernerf/nerf_triplane/renderer.py"
    click MUS_G "https://github.com/lipku/metahuman-stream/blob/main/musetalk/simple_musetalk.py"
    click WAV_G "https://github.com/lipku/metahuman-stream/blob/main/wav2lip/genavatar.py"
    click ULT_G "https://github.com/lipku/metahuman-stream/blob/main/ultralight/genavatar.py"
    click GPT "https://github.com/lipku/metahuman-stream/blob/main/llm/ChatGPT.py"
    click GEM "https://github.com/lipku/metahuman-stream/blob/main/llm/Gemini.py"
    click QWN "https://github.com/lipku/metahuman-stream/blob/main/llm/Qwen.py"

    %% Styles
    classDef frontend fill:#3498db
    classDef streaming fill:#2ecc71
    classDef server fill:#e74c3c
    classDef process fill:#9b59b6
    classDef model fill:#f1c40f
    classDef llm fill:#95a5a6
```
