How stelabou/hyperreenact works
Project Type & Purpose
• This is a research-oriented deep-learning pipeline (PyTorch) for neural face reenactment.
• It ingests one source image + a target video (or sequence of frames), and produces a talking-head video of the source identity in the target’s expressions/pose.
Top-Level Components
A. Input/Data
• Source Image
• Target Video or Frame Sequence
B. Preprocessing & Feature Extraction
1. Face Detection & Cropping
• SFD Detector (libs/face_models/sfd)
• FAN Landmark Estimator (libs/face_models/fan_model)
→ produce aligned/cropped facial images and 2D landmarks
2. Pose Encoder
• DECA Module (libs/DECA/decalib/deca.py + utils)
→ outputs 3D pose parameters (rotation, expression codes)
3. Appearance Encoder
• ArcFace (libs/models/appearance_encoder.py)
→ outputs identity embedding
4. Latent Inversion
• e4e Inversion (libs/models/encoders/psp_encoders.py)
→ maps source image → initial StyleGAN2 latent (w code)
C. Hypernetwork Reenactment
• HyperNetwork Module (libs/models/hypernetworks + hypernetwork_reenact.py)
• Refinement Sub-network (enhances identity details)
• Retargeting Sub-network (applies pose parameters)
→ generates per-layer weight deltas or latent offsets
D. Generator
• StyleGAN2 Generator (libs/models/gan/StyleGAN2/model.py + op modules)
• Takes refined + retargeted latent codes → synthesizes output frames
E. Output
• Post-processing & Video Assembly (run_inference.py)
→ composite frames into video, apply any cropping overlays, save to disk
Relationships & Data Flow
Source Image → Face Detector → Aligned Face →
→ (branch 1) Appearance Encoder
→ (branch 2) e4e Latent Inversion
Target Video → Frame Extractor → for each frame → Face Detector → Landmarks → Pose Encoder
Appearance Embedding + Pose Parameters + Initial Latent → Hypernetwork → Refined Latent → StyleGAN2 → Output Frame
Architectural Patterns & Principles
• Pipeline Pattern: sequential stages, each transforms data.
• Modular Encapsulation: detectors, encoders, hypernetwork, generator are separate modules.
• Data Parallelism (optional training detail): note that StyleGAN2 and hypernetwork can be run on multiple GPUs (libs/models/gan/distributed.py).
• Configuration Driven: model paths and hyperparameters loaded via libs/configs/config_models.py.
Key Technologies & Libraries
• Python 3.8+, PyTorch ≥1.5, CUDA/CuDNN
• Face-Alignment / SFD for detection, DECA for 3D pose, ArcFace for identity, e4e for inversion, StyleGAN2 for generation, custom Hypernetworks for refinement/retargeting.