How anliyuan/ultralight-digital-human works
Key Components to Include in the System Design:
Data Processing Pipeline
• Video Input Processing
• Audio Feature Extraction (with two paths: Wenet and HuBERT)
• Face Detection & Landmark Extraction
• Data Preprocessing Module
Model Architecture
• Core Components:
• SyncNet Module (for synchronization)
• UNet Architecture (main generation model)
• Feature Pipeline
• Training Pipeline
• Inference Pipeline
Integration Layer
• Mobile Optimization Components
• Real-time Processing Units
• Audio-Video Synchronization
Specific Design Guidelines:
Input Layer
• ↓
Processing Layer
• ↓
Model Layer
• ↓
Output Layer
Technical Details
• Input specifications (video fps requirements: 20fps for Wenet, 25fps for HuBERT)
• Model checkpoint locations
• Configuration file dependencies
• ONNX runtime integration points
Specific Modules to Highlight
• Face Detection Module (detect_face.py)
• Landmark Generation (get_landmark.py)
• Feature Pipeline (FeaturePipeline.py)
• Wenet/HuBERT Audio Processing
• UNet Architecture (unet.py)
• SyncNet Architecture (syncnet.py)
Mobile Optimization Layer
• A separate section showing how the system is optimized for mobile deployment
• Streaming inference capabilities
• Channel reduction optimizations
Configuration Management
• How the various YAML configurations (in data_utils/conf) integrate with the system
• Model checkpoint management
The training and inference paths as separate but related flows
Remember to maintain a balance between showing the technical complexity of the system while keeping it clear and understandable at a glance.
This system design should reflect both the training and inference workflows, as they are distinct processes in the codebase but share common components.