How neeek2303/emoportraits works
Project Type and Purpose
• EMOPortraits is a research‐oriented deep learning codebase for generating one-shot, emotion-enhanced head avatars from a source image and a driving video.
• It provides training and inference pipelines (train.py, run_video_driven_pipeline.py), modular dataset loaders, network definitions, loss functions, and utility scripts.
File Structure Analysis
• Top-level scripts:
• train.py: orchestrates the training loop.
• run_video_driven_pipeline.py: entry point for inference.
• environment.yml / conda-pack archive: reproducible environment.
• Data and preprocessing:
• data/: raw videos and precomputed keypoints.
• datasets/: Python modules that wrap various face/emotion datasets.
• utils/point_transforms.py, utils/prepare_for_landmark.py: data augmentation and landmark alignment.
• Model definition:
• networks/: PyTorch modules for encoders, decoders, discriminators (basic_avatar, volumetric_avatar).
• networks/head_pose_regressor.py, networks/common: shared layers.
• models/stage_1 & stage_2: higher-level wrappers combining networks, losses, and visualizations per training stage.
• Loss functions:
• losses/: adversarial, perceptual, gaze, head-pose matching, PSNR, LPIPS, plus submodules (rt_gene_loss).
• Experiments and logging:
• experiments/: CLI args snapshots for reproducibility.
• utils/logger.py, utils/logger_wandb.py: logging & Weights & Biases integration.
• Notebooks:
• notebooks/: interactive demos and inference scripts (E_emo_infer_video.ipynb).
Key README Insights
• Multimodal inputs: extreme/asymmetric emotions from FEED dataset.
• Two-stage training: “stage_1” for volumetric avatar pretraining, “stage_2” for fine-tuning.
• External dependencies: face_detection, roi_tanh_warping, face_parsing (installed via pip –e).
• Pretrained weights (logs.zip, logs_s2.zip) are unzipped into the root.
a. Main Components
• Data Layer: Raw Video/Image assets (data/), precomputed keypoints (.npy).
• Dataset Loader: datasets/*.py modules, output: PyTorch DataLoader of (image, landmarks, emotion labels).
• Preprocessing & Augmentation: utils/*.py (grid_sample, point_transforms, face parsing & warping).
• Model Stage 1: networks/basic_avatar + losses → volumetric avatar embedding.
• Model Stage 2: networks/volumetric_avatar + stage_2 wrappers → refined animated avatar.
• Loss Modules: adversarial, perceptual, gaze, head_pose, etc., plugged into training loop.
• Training Controller: train.py—initializes datasets, networks, optimizers, loss functions, checkpointing, logger.
• Inference Controller: run_video_driven_pipeline.py—notebook scripts—loads pretrained models, processes source + driver, renders output video.
• Logging & Monitoring: WandB or local logs, checkpoint directories under logs/ and logs_s2/.
• External Tools: face_detection, roi_tanh_warping, face_parsing repos.
b. Relationships & Data Flow
1. Raw data → Dataset Loader → Preprocessing → Batches of Tensors
2. Batches → Stage 1 Model → Intermediate Avatar Representation → Loss Modules → Backprop
3. Stage 1 Checkpoints → Stage 2 Model → Final Avatar Frames → Loss Modules → Backprop
4. Training Controller ↔ Logger → Checkpoints & Metrics
5. Inference Controller loads Stage 2 Checkpoint, routes inputs through Preprocessing → Model → Video Renderer → Output
c. Architectural Patterns & Principles
• Modular separation: data, preprocessing, models, losses, scripts.
• Two-stage pipeline: progressive training.
• Plugin architecture for losses and network blocks.
• Clear entry points (train.py, run_video_driven_pipeline.py, notebooks).
d. Key Technologies & Frameworks
• Python 3, PyTorch, NumPy, OpenCV (implied), FFmpeg (for video I/O).
• External pip-installable repos: face_detection, roi_tanh_warping, face_parsing.
• Weights & Biases (wandb) or local logger.