# anliyuan/ultralight-digital-human architecture

> Architecture diagram of the GitHub repository anliyuan/ultralight-digital-human, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anliyuan/ultralight-digital-human
- Repository: https://github.com/anliyuan/ultralight-digital-human

## Overview

Key Components to Include in the System Design:

##### Data Processing Pipeline
- Video Input Processing
- Audio Feature Extraction (with two paths: Wenet and HuBERT)
- Face Detection & Landmark Extraction
- Data Preprocessing Module

##### Model Architecture
- Core Components:
- SyncNet Module (for synchronization)
- UNet Architecture (main generation model)
- Feature Pipeline
- Training Pipeline
- Inference Pipeline

##### Integration Layer
- Mobile Optimization Components
- Real-time Processing Units
- Audio-Video Synchronization

Specific Design Guidelines:

**Input Layer**
- ↓
**Processing Layer**
- ↓
**Model Layer**
- ↓
**Output Layer**

##### Technical Details
- Input specifications (video fps requirements: 20fps for Wenet, 25fps for HuBERT)
- Model checkpoint locations
- Configuration file dependencies
- ONNX runtime integration points

##### Specific Modules to Highlight
- Face Detection Module (detect_face.py)
- Landmark Generation (get_landmark.py)
- Feature Pipeline (FeaturePipeline.py)
- Wenet/HuBERT Audio Processing
- UNet Architecture (unet.py)
- SyncNet Architecture (syncnet.py)

##### Mobile Optimization Layer
- A separate section showing how the system is optimized for mobile deployment
- Streaming inference capabilities
- Channel reduction optimizations

##### Configuration Management
- How the various YAML configurations (in data_utils/conf) integrate with the system
- Model checkpoint management

##### The training and inference paths as separate but related flows

Remember to maintain a balance between showing the technical complexity of the system while keeping it clear and understandable at a glance.

This system design should reflect both the training and inference workflows, as they are distinct processes in the codebase but share common components.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and classes
    classDef processing fill:#a8d1f0,stroke:#333
    classDef model fill:#90EE90,stroke:#333
    classDef training fill:#FFB347,stroke:#333
    classDef inference fill:#DDA0DD,stroke:#333
    classDef config fill:#E6E6FA,stroke:#333

    %% Legend
    subgraph Legend
        L1[Processing]:::processing
        L2[Model Components]:::model
        L3[Training]:::training
        L4[Inference]:::inference
        L5[Configuration]:::config
    end

    %% Data Processing Pipeline
    subgraph DataProcessing["Data Processing Pipeline"]
        VideoInput[Video Input]:::processing
        AudioInput[Audio Input]:::processing
        FaceDetect[Face Detection]:::processing
        LandmarkExt[Landmark Extraction]:::processing
        
        subgraph AudioFeatures["Audio Feature Extraction"]
            WenetPath[Wenet Path]:::processing
            HubertPath[Hubert Path]:::processing
        end
        
        FeaturePipe[Feature Pipeline]:::processing
        
        VideoInput --> FaceDetect
        AudioInput --> AudioFeatures
        FaceDetect --> LandmarkExt
        LandmarkExt --> FeaturePipe
        AudioFeatures --> FeaturePipe
    end

    %% Model Architecture
    subgraph ModelArch["Model Architecture"]
        SyncNet[SyncNet Module]:::model
        UNet[Digital Human Generator]:::model
        FeaturePipeline[Feature Pipeline Base]:::model
        
        FeaturePipeline --> SyncNet
        FeaturePipeline --> UNet
    end

    %% Training Pipeline
    subgraph Training["Training Pipeline"]
        DataLoader[Dataset Management]:::training
        TrainScript[Training Script]:::training
        CheckpointMgr[Checkpoint Management]:::training
        
        DataLoader --> TrainScript
        TrainScript --> CheckpointMgr
    end

    %% Inference Pipeline
    subgraph Inference["Inference Pipeline"]
        InferScript[Inference Script]:::inference
        ModelExport[Model Export]:::inference
        RealTimeProc[Real-time Processing]:::inference
        
        InferScript --> ModelExport
        ModelExport --> RealTimeProc
    end

    %% Configuration
    subgraph Config["Configuration"]
        WenetConfig[Wenet Config]:::config
        FaceRef[Face Reference Data]:::config
    end

    %% Component Connections
    DataProcessing --> Training
    DataProcessing --> ModelArch
    ModelArch --> Training
    ModelArch --> Inference
    Config --> Training
    Config --> Inference

    %% Click Events
    click FaceDetect "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/data_utils/detect_face.py"
    click LandmarkExt "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/data_utils/get_landmark.py"
    click WenetPath "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/data_utils/wenet_infer.py"
    click HubertPath "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/data_utils/hubert.py"
    click FeaturePipe "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/data_utils/FeaturePipeline.py"
    click SyncNet "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/syncnet.py"
    click UNet "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/unet.py"
    click FeaturePipeline "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/data_utils/base_module.py"
    click TrainScript "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/train.py"
    click DataLoader "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/datasetsss.py"
    click CheckpointMgr "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/data_utils/wenet/utils/checkpoint.py"
    click InferScript "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/inference.py"
    click ModelExport "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/pth2onnx.py"
    click WenetConfig "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/data_utils/conf/wenetspeech_unified_conformer/train.yaml"
    click FaceRef "https://github.com/anliyuan/Ultralight-Digital-Human/blob/master/data_utils/mean_face.txt"
```
