# stelabou/hyperreenact architecture

> Architecture diagram of the GitHub repository stelabou/hyperreenact, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stelabou/hyperreenact
- Repository: https://github.com/stelabou/hyperreenact

## Overview

##### Project Type & Purpose
- This is a research-oriented deep-learning pipeline (PyTorch) for neural face reenactment.
- It ingests one source image + a target video (or sequence of frames), and produces a talking-head video of the source identity in the target’s expressions/pose.

##### Top-Level Components
A. Input/Data
- Source Image
- Target Video or Frame Sequence
B. Preprocessing & Feature Extraction
1. Face Detection & Cropping
- SFD Detector (libs/face_models/sfd)
- FAN Landmark Estimator (libs/face_models/fan_model)
→ produce aligned/cropped facial images and 2D landmarks
2. Pose Encoder
- DECA Module (libs/DECA/decalib/deca.py + utils)
→ outputs 3D pose parameters (rotation, expression codes)
3. Appearance Encoder
- ArcFace (libs/models/appearance_encoder.py)
→ outputs identity embedding
4. Latent Inversion
- e4e Inversion (libs/models/encoders/psp_encoders.py)
→ maps source image → initial StyleGAN2 latent (`w` code)

C. Hypernetwork Reenactment
- HyperNetwork Module (libs/models/hypernetworks + hypernetwork_reenact.py)
- Refinement Sub-network (enhances identity details)
- Retargeting Sub-network (applies pose parameters)
→ generates per-layer weight deltas or latent offsets

D. Generator
- StyleGAN2 Generator (libs/models/gan/StyleGAN2/model.py + op modules)
- Takes refined + retargeted latent codes → synthesizes output frames

E. Output
- Post-processing & Video Assembly (run_inference.py)
→ composite frames into video, apply any cropping overlays, save to disk

##### Relationships & Data Flow
Source Image → Face Detector → Aligned Face →
​→ (branch 1) Appearance Encoder
​→ (branch 2) e4e Latent Inversion
Target Video → Frame Extractor → for each frame → Face Detector → Landmarks → Pose Encoder
Appearance Embedding + Pose Parameters + Initial Latent → Hypernetwork → Refined Latent → StyleGAN2 → Output Frame

##### Architectural Patterns & Principles
- Pipeline Pattern: sequential stages, each transforms data.
- Modular Encapsulation: detectors, encoders, hypernetwork, generator are separate modules.
- Data Parallelism (optional training detail): note that StyleGAN2 and hypernetwork can be run on multiple GPUs (libs/models/gan/distributed.py).
- Configuration Driven: model paths and hyperparameters loaded via libs/configs/config_models.py.

##### Key Technologies & Libraries
- Python 3.8+, PyTorch ≥1.5, CUDA/CuDNN
- Face-Alignment / SFD for detection, DECA for 3D pose, ArcFace for identity, e4e for inversion, StyleGAN2 for generation, custom Hypernetworks for refinement/retargeting.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Input/Data
    subgraph "1. Input/Data"
        SI[/"Source Image"/]:::input
        TV[/"Target Video or Frame Sequence"/]:::input
    end

    %% Preprocessing & Feature Extraction
    subgraph "2. Preprocessing & Feature Extraction"
        FD["SFD Detector"]:::preprocess
        FDdet["SFD Detect Script"]:::preprocess
        FAN["FAN Landmark Estimator"]:::preprocess
        FANutil["FAN Utils"]:::preprocess
        Crop["FFHQ Cropping"]:::preprocess
        Crop2["Image Utils"]:::preprocess
        FrameExt["Frame Extractor"]:::preprocess
        AF["Aligned Face"]:::preprocess
        LM["2D Landmarks"]:::preprocess
        PE["DECA Pose Encoder"]:::preprocess
        PE_model["DECA Models"]:::preprocess
        PE_utils["DECA Utils"]:::preprocess
        PE_entry["DECA Entry"]:::preprocess
        AE["Appearance Encoder (ArcFace)"]:::neural
        LI["Latent Inversion (e4e)"]:::neural
        LIh["Encoder Helpers"]:::preprocess
    end

    %% Hypernetwork Reenactment
    subgraph "3. Hypernetwork Reenactment"
        HMain["Hypernetwork Controller"]:::neural
        HShared["Shared Weights Block"]:::neural
        HRefine["Refinement Sub-network"]:::neural
        HRetarget["Retargeting Blocks"]:::neural
    end

    %% Generator
    subgraph "4. Generator (StyleGAN2)"
        GS2["StyleGAN2 Model"]:::generator
        GS2ops["StyleGAN2 Ops"]:::neural
        GDist["Distributed Utils"]:::neural
    end

    %% Config & Pretrained
    CFG[(Config Loader)]:::util
    PM[(Pretrained Models)]:::util

    %% Output
    subgraph "5. Output"
        RI["Inference Runner"]:::util
        FU["Utilities"]:::util
        Criteria["Criteria/Metrics"]:::util
        Final["Final Reenacted Video"]:::util
    end

    %% Data Flow
    SI --> FD
    TV --> FrameExt --> FD
    FD -->|"cropped faces"| AF
    FDdet --> FD
    Crop --> AF
    Crop2 --> AF
    FAN --> AF
    FANutil --> FAN
    AF --> AE
    AF --> LI
    AF --> LM
    LM --> PE
    PE -->|"3D pose params"| HMain
    PE_model --> PE
    PE_utils --> PE
    PE_entry --> PE
    AE -->|"identity embedding"| HMain
    LI -->|"initial latent"| HMain
    LIh --> LI
    HShared --> HMain
    HRefine --> HMain
    HRetarget --> HMain
    HMain -->|"latent offsets"| GS2
    GS2ops --> GS2
    GDist --> GS2
    GS2 -->|"synthesized frame"| RI
    RI -->|"frames to video"| Final
    FU --> RI
    Criteria --> RI

    %% Config & Pretrained connections
    CFG --> FD
    CFG --> FAN
    CFG --> PE
    CFG --> AE
    CFG --> LI
    CFG --> HMain
    CFG --> GS2
    PM --> FD
    PM --> FAN
    PM --> PE
    PM --> AE
    PM --> LI
    PM --> HMain
    PM --> GS2

    %% Click Events
    click FD "https://github.com/stelabou/hyperreenact/blob/master/libs/face_models/sfd/sfd_detector.py"
    click FDdet "https://github.com/stelabou/hyperreenact/blob/master/libs/face_models/sfd/detect.py"
    click FAN "https://github.com/stelabou/hyperreenact/blob/master/libs/face_models/fan_model/models.py"
    click FANutil "https://github.com/stelabou/hyperreenact/blob/master/libs/face_models/fan_model/utils.py"
    click Crop "https://github.com/stelabou/hyperreenact/blob/master/libs/utilities/ffhq_cropping.py"
    click Crop2 "https://github.com/stelabou/hyperreenact/blob/master/libs/utilities/image_utils.py"
    click PE "https://github.com/stelabou/hyperreenact/blob/master/libs/DECA/decalib/deca.py"
    click PE_model "https://github.com/stelabou/hyperreenact/blob/master/libs/DECA/decalib/models/*.py"
    click PE_utils "https://github.com/stelabou/hyperreenact/blob/master/libs/DECA/decalib/utils/*.py"
    click PE_entry "https://github.com/stelabou/hyperreenact/blob/master/libs/DECA/estimate_DECA.py"
    click AE "https://github.com/stelabou/hyperreenact/blob/master/libs/models/appearance_encoder.py"
    click LI "https://github.com/stelabou/hyperreenact/blob/master/libs/models/encoders/psp_encoders.py"
    click LIh "https://github.com/stelabou/hyperreenact/blob/master/libs/models/encoders/helpers.py"
    click HMain "https://github.com/stelabou/hyperreenact/blob/master/libs/models/hypernetwork_reenact.py"
    click HShared "https://github.com/stelabou/hyperreenact/blob/master/libs/models/hypernetworks/shared_weights_hypernet.py"
    click HRefine "https://github.com/stelabou/hyperreenact/blob/master/libs/models/hypernetworks/refinement_blocks.py"
    click HRetarget "https://github.com/stelabou/hyperreenact/blob/master/libs/models/hypernetworks/hypernetwork_blocks.py"
    click GS2 "https://github.com/stelabou/hyperreenact/blob/master/libs/models/gan/StyleGAN2/model.py"
    click GS2ops "https://github.com/stelabou/hyperreenact/tree/master/libs/models/gan/StyleGAN2/op/*"
    click GDist "https://github.com/stelabou/hyperreenact/blob/master/libs/models/gan/distributed.py"
    click CFG "https://github.com/stelabou/hyperreenact/blob/master/libs/configs/config_models.py"
    click RI "https://github.com/stelabou/hyperreenact/blob/master/run_inference.py"
    click FU "https://github.com/stelabou/hyperreenact/blob/master/libs/utilities/utils.py"
    click Criteria "https://github.com/stelabou/hyperreenact/blob/master/libs/criteria/helpers.py"
    click TV "https://github.com/stelabou/hyperreenact/blob/master/inference_examples/target_video_1.mp4"

    %% Styles
    classDef input fill:#bbe1fa,stroke:#3282b8;
    classDef preprocess fill:#d4f7dc,stroke:#2d6a4f;
    classDef neural fill:#ffd8a8,stroke:#cc8e35;
    classDef generator fill:#e2d0f2,stroke:#6a4c93;
    classDef util fill:#e0e0e0,stroke:#828282;
    classDef input,preprocess,neural,generator,util stroke-width:2px;
```
