How powerhouseman/comfyui-advancedliveportrait works
Project type and purpose
• This is a ComfyUI plugin (“AdvancedLivePortrait”) for real-time facial expression editing, motion transfer, and animation generation using deep-learning models.
• It extends ComfyUI with new node types (defined in nodes.py) that orchestrate a sequence of neural-network modules to process source images/videos and driving videos.
File structure
• Top level
• nodes.py – defines the custom ComfyUI nodes that wrap the core pipeline
• pyproject.toml, requirements.txt – Python packaging and dependencies (PyTorch, ComfyUI, etc.)
• install.bat – install script for Windows
• LivePortrait/
• config/ – inference parameters (inference_config.py) and models.yaml (paths, hyperparameters)
• modules/ – deep-learning building blocks (appearance_feature_extractor, motion_extractor, dense_motion, spade_generator, warping_network, stitching_retargeting_network)
• utils/ – helper routines for I/O, video handling, camera processing, face analysis, timing, resource management
• sample/
• workflows/ – example JSON graphs for ComfyUI (various use cases)
• original_sample_asset/ – driving video frames and source images
• sample_exp_data.exp – serialized facial‐expression data
• .github/workflows/publish.yml – CI for publishing the plugin to ComfyUI-Manager
Key insights from README
• Purpose: “Faster with real-time preview,” supports editing expressions in photos and videos, animation generation from multiple expressions, extract & reuse expression data.
• Installation via ComfyUI-Manager; sample workflows placed under custom_nodes.
• Workflows use JSON commands to link nodes, specifying motion indices, frame linking, save/load expression nodes.
• Relies on models converted by Kijai and the original LivePortrait models.
a. Main components
1. ComfyUI Core – node graph engine, GUI, scheduler
2. AdvancedLivePortrait Plugin
• Node Definitions (nodes.py)
• Module Library (LivePortrait/modules)
• Config Loader (LivePortrait/config)
• Utils Layer (LivePortrait/utils)
3. Pre-trained Models – files referenced by models.yaml
4. Sample Data & Workflows – demonstration JSON graphs, videos, expression data
5. External Services/Libraries – PyTorch runtime, GPU, face-tracking, ComfyUI-Manager registry
b. Relationships and interactions
1. User places nodes in ComfyUI canvas → invokes node callbacks in nodes.py
2. Each node loads config parameters and calls into specific module classes
3. Modules chain data:
Source Image/Video → Appearance Feature Extractor → Motion Extractor
→ Dense Motion → SPADE Generator → Warping Network
→ Stitching & Retargeting Network → Final Output Frames
4. Utils handle file I/O, video decoding/encoding, timing, and resource logging
5. Sample workflows glue these nodes together in predefined sequences
6. Models are loaded from locations in models.yaml; inference_config.py supplies runtime flags
c. Architectural patterns & principles
• Modular decomposition: each neural-network stage lives in its own module file
• Pipeline pattern: data flows sequentially through feature extraction, motion analysis, synthesis
• Plugin-based extension: clean separation between ComfyUI core and AdvancedLivePortrait code
• Configuration-driven: models.yaml and inference_config.py control hyperparameters and file paths
d. Relevant technologies/frameworks
• Python 3.x, PyTorch (model inference)
• ComfyUI (node graph framework)
• OpenCV/FFmpeg (video I/O via utils/video.py)
• Face‐tracking library (in utils/face_analysis_diy.py)