How antgroup/echomimic works
Project type and purpose
• EchoMimic is a research-grade Python library and toolkit for generating lifelike, audio- and/or landmark-driven portrait animations using diffusion models plus a transformer-based motion module.
• It provides command-line inference scripts (e.g., infer_audio2vid.py), a demo script for motion synchronization (demo_motion_sync.py), and a Gradio-based Web GUI (webgui.py).
File structure
• Top-level directories:
• assets: test inputs (images, audio, videos)
• configs: YAML files for inference and prompt settings
• src: core code, subdivided into models, pipelines, utils
• Key modules:
• src/models: UNet variants (2D, 3D, echo), transformer modules, motion_module, face_locator, Whisper audio processor
• src/pipelines: orchestrate the overall inference workflow for different modes (audio only, pose only, combined)
• src/utils: helper functions for image I/O, drawing, face landmarks, motion computation
• Inference scripts at top level invoke pipelines, load configs, read pretrained weights, write videos.
• The Web GUI (webgui.py) wraps inference scripts in a Gradio server.
README for additional insights
• Describes Audio-Driven, Pose-Driven, and Audio+Pose-Driven modes
• Points out 10× acceleration variants (infer_*_acc.py)
• Dependencies: diffusion checkpoints, Whisper Tiny, ffmpeg for video assembly
• Inference configurable via YAML under configs/prompts and configs/inference
Components
1. Input Layer
• Reference image(s)
• Driver audio file (and/or landmark sequence)
2. Preprocessing
• Whisper audio2feature module (audio → feature vectors)
• Face landmark detection (MediaPipe or face_landmarker)
• Optional: pose loading or drawing function
3. Pipeline Orchestrator (src/pipelines)
• Context builder: packages audio features + landmarks + prompts
• Mode-specific pipelines (pipeline_echo_mimic, pipeline_echo_mimic_pose, etc.)
4. Core Model Stack
• Motion Module (transformer_2d or transformer_3d)
• Diffusion UNet(s): denoising_unet, reference_unet, unet_3d_echo
• Attention modules, self-attention, mutual attention
• Face locator network
5. Postprocessing
• Frame decoding → images → assemble into video (ffmpeg)
• Draw_utils for landmarks overlay (debug/demo)
6. Output Layer
• Generated video file
7. Configuration & Checkpoints
• YAML config files (prompts + inference settings)
• Pretrained model weights directory
8. Web GUI Layer
• Gradio server (webgui.py) exposing above pipeline as REST/HTML UI
b. Relationships and interactions
• Inputs feed Preprocessing → Pipeline Orchestrator
• Orchestrator loads Config → instantiates Core Model Stack → yields raw frames
• Postprocessing consumes frames → outputs final video
• Web GUI invokes same Orchestrator via CLI calls or direct imports
c. Architectural patterns & design principles
• Modular pipeline pattern: separate preprocessing, core model, postprocessing
• Strategy pattern: different pipeline classes for audio-only, pose-only, combined
• Configuration-driven: decode YAML at runtime to change prompts, test cases
• Reusable utility modules for image/audio/landmark I/O
d. Technologies, frameworks, libraries
• PyTorch for model definitions & inference
• OpenAI Whisper for audio feature extraction
• Diffusers style UNet implementations
• MediaPipe or TFLite for face landmark detection
• ffmpeg for assembling video
• Gradio for Web UI