# neeek2303/emoportraits architecture

> Architecture diagram of the GitHub repository neeek2303/emoportraits, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/neeek2303/emoportraits
- Repository: https://github.com/neeek2303/emoportraits

## Overview

##### Project Type and Purpose
- EMOPortraits is a research‐oriented deep learning codebase for generating one-shot, emotion-enhanced head avatars from a source image and a driving video.
- It provides training and inference pipelines (train.py, run_video_driven_pipeline.py), modular dataset loaders, network definitions, loss functions, and utility scripts.

##### File Structure Analysis
- Top-level scripts:
- train.py: orchestrates the training loop.
- run_video_driven_pipeline.py: entry point for inference.
- environment.yml / conda-pack archive: reproducible environment.
- Data and preprocessing:
- data/: raw videos and precomputed keypoints.
- datasets/: Python modules that wrap various face/emotion datasets.
- utils/point_transforms.py, utils/prepare_for_landmark.py: data augmentation and landmark alignment.
- Model definition:
- networks/: PyTorch modules for encoders, decoders, discriminators (basic_avatar, volumetric_avatar).
- networks/head_pose_regressor.py, networks/common: shared layers.
- models/stage_1 & stage_2: higher-level wrappers combining networks, losses, and visualizations per training stage.
- Loss functions:
- losses/: adversarial, perceptual, gaze, head-pose matching, PSNR, LPIPS, plus submodules (rt_gene_loss).
- Experiments and logging:
- experiments/: CLI args snapshots for reproducibility.
- utils/logger.py, utils/logger_wandb.py: logging & Weights & Biases integration.
- Notebooks:
- notebooks/: interactive demos and inference scripts (E_emo_infer_video.ipynb).

##### Key README Insights
- Multimodal inputs: extreme/asymmetric emotions from FEED dataset.
- Two-stage training: “stage_1” for volumetric avatar pretraining, “stage_2” for fine-tuning.
- External dependencies: face_detection, roi_tanh_warping, face_parsing (installed via pip –e).
- Pretrained weights (logs.zip, logs_s2.zip) are unzipped into the root.

a. Main Components
- Data Layer: Raw Video/Image assets (data/), precomputed keypoints (.npy).
- Dataset Loader: datasets/*.py modules, output: PyTorch DataLoader of (image, landmarks, emotion labels).
- Preprocessing & Augmentation: utils/*.py (grid_sample, point_transforms, face parsing & warping).
- Model Stage 1: networks/basic_avatar + losses → volumetric avatar embedding.
- Model Stage 2: networks/volumetric_avatar + stage_2 wrappers → refined animated avatar.
- Loss Modules: adversarial, perceptual, gaze, head_pose, etc., plugged into training loop.
- Training Controller: train.py—initializes datasets, networks, optimizers, loss functions, checkpointing, logger.
- Inference Controller: run_video_driven_pipeline.py—notebook scripts—loads pretrained models, processes source + driver, renders output video.
- Logging & Monitoring: WandB or local logs, checkpoint directories under logs/ and logs_s2/.
- External Tools: face_detection, roi_tanh_warping, face_parsing repos.

b. Relationships & Data Flow
1. Raw data → Dataset Loader → Preprocessing → Batches of Tensors
2. Batches → Stage 1 Model → Intermediate Avatar Representation → Loss Modules → Backprop
3. Stage 1 Checkpoints → Stage 2 Model → Final Avatar Frames → Loss Modules → Backprop
4. Training Controller ↔ Logger → Checkpoints & Metrics
5. Inference Controller loads Stage 2 Checkpoint, routes inputs through Preprocessing → Model → Video Renderer → Output

c. Architectural Patterns & Principles
- Modular separation: data, preprocessing, models, losses, scripts.
- Two-stage pipeline: progressive training.
- Plugin architecture for losses and network blocks.
- Clear entry points (train.py, run_video_driven_pipeline.py, notebooks).

d. Key Technologies & Frameworks
- Python 3, PyTorch, NumPy, OpenCV (implied), FFmpeg (for video I/O).
- External pip-installable repos: face_detection, roi_tanh_warping, face_parsing.
- Weights & Biases (wandb) or local logger.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data & Preprocessing Layer
    subgraph "Data & Preprocessing" 
        direction TB
        DL["Raw Data (data/)"]:::data
        subgraph "Dataset Loader" 
            direction TB
            D1["extreme_faces_pairs.py"]:::data
            D2["ffhq_crops.py"]:::data
            D3["infer_dataset.py"]:::data
            D4["mead_faces_pairs.py"]:::data
            D5["voxceleb2hq_pairs.py"]:::data
        end
        subgraph "Preprocessing & Augmentation" 
            direction TB
            P1["point_transforms.py"]:::pre
            P2["prepare_for_landmark.py"]:::pre
            P3["grid_sample.py"]:::pre
            P4["utils_keypoints.py"]:::pre
            P5["misc.py"]:::pre
        end
    end

    %% Model Stage 1
    subgraph "Model Stage 1" 
        direction TB
        subgraph "Basic Avatar Networks"
            direction TB
            M1["encoder.py"]:::model
            M2["decoder.py"]:::model
            M3["keypoints_encoder.py"]:::model
            M4["motion_field_estimator.py"]:::model
            M5["multiscale_discriminator.py"]:::model
        end
        subgraph "Stage1 Wrapper & Visuals"
            direction TB
            W1["va.py"]:::model
            W2["va_arguments.py"]:::model
            W3["va_losses_and_visuals.py"]:::model
        end
    end

    %% Model Stage 2
    subgraph "Model Stage 2" 
        direction TB
        subgraph "Volumetric Avatar Networks"
            direction TB
            S21["decoder_s2.py"]:::model
            S22["expression_embedder.py"]:::model
            S23["global_encoder.py"]:::model
            S24["local_encoder.py"]:::model
            S25["volume_renderer.py"]:::model
        end
        subgraph "Stage2 Wrapper & Visuals"
            direction TB
            W21["volumetric_avatar_two.py"]:::model
            W22["va_losses_and_visuals_two.py"]:::model
        end
    end

    %% Loss Modules
    subgraph "Loss Modules" 
        direction TB
        L1["adversarial.py"]:::loss
        L2["perceptual.py"]:::loss
        L3["gaze.py"]:::loss
        L4["head_pose_matching.py"]:::loss
        L5["psnr.py"]:::loss
        L6["lpips.py"]:::loss
        L7["warping_loss.py"]:::loss
        L8["warping_regularizer.py"]:::loss
        subgraph "rt_gene_loss" 
            direction TB
            L9["rt_gene_loss/"]:::loss
            L10["rt_gene_loss_fb/"]:::loss
        end
    end

    %% Controllers & Logging
    subgraph "Controllers & Logging"
        direction TB
        TC["train.py"]:::ctrl
        IC["run_video_driven_pipeline.py"]:::ctrl
        subgraph "Logging & Experiments"
            direction TB
            LG1["logger.py"]:::ctrl
            LG2["logger_wandb.py"]:::ctrl
            LG3["args.txt"]:::ctrl
        end
    end

    %% External Repos
    subgraph "External Tools" 
        direction TB
        EV1["face_detection"]:::ext
        EV2["roi_tanh_warping"]:::ext
        EV3["face_parsing"]:::ext
    end

    %% Flow Connections
    DL --> D1
    DL --> D2
    DL --> D3
    DL --> D4
    DL --> D5
    D1 --> P1
    D2 --> P1
    D3 --> P1
    P1 --> P2 --> P3 --> P4 --> P5
    P5 --> M1
    M1 --> M3 --> M4 --> M2
    M5 --> W1
    M2 --> W1
    W1 --> W3
    W3 --> L1 & L2 & L3 & L4 & L5 & L6 & L7 & L8 & L9
    L1 --> TC
    L2 --> TC
    W3 --> TC
    TC --> W2
    W2 --> S21
    S21 --> S22 --> S23 --> S24 --> S25
    S25 --> W21 --> W22
    W22 --> L1
    W22 --> L2
    W22 --> L3
    W22 --> L4
    W22 --> TC
    TC --> LG1
    TC --> LG2
    TC --> LG3
    IC --> P1
    IC --> W21
    IC --> IC
    IC --> LG1

    %% Click Events
    click DL "https://github.com/neeek2303/emoportraits/tree/main/data/"
    click D1 "https://github.com/neeek2303/emoportraits/blob/main/datasets/extrime_faces_pairs.py"
    click D2 "https://github.com/neeek2303/emoportraits/blob/main/datasets/ffhq_crops.py"
    click D3 "https://github.com/neeek2303/emoportraits/blob/main/datasets/infer_dataset.py"
    click D4 "https://github.com/neeek2303/emoportraits/blob/main/datasets/mead_faces_pairs.py"
    click D5 "https://github.com/neeek2303/emoportraits/blob/main/datasets/voxceleb2hq_pairs.py"
    click P1 "https://github.com/neeek2303/emoportraits/blob/main/utils/point_transforms.py"
    click P2 "https://github.com/neeek2303/emoportraits/blob/main/utils/prepare_for_landmark.py"
    click P3 "https://github.com/neeek2303/emoportraits/blob/main/utils/grid_sample.py"
    click P4 "https://github.com/neeek2303/emoportraits/blob/main/utils/utils_keypoints.py"
    click P5 "https://github.com/neeek2303/emoportraits/blob/main/utils/misc.py"
    click M1 "https://github.com/neeek2303/emoportraits/blob/main/networks/basic_avatar/encoder.py"
    click M2 "https://github.com/neeek2303/emoportraits/blob/main/networks/basic_avatar/decoder.py"
    click M3 "https://github.com/neeek2303/emoportraits/blob/main/networks/basic_avatar/keypoints_encoder.py"
    click M4 "https://github.com/neeek2303/emoportraits/blob/main/networks/basic_avatar/motion_field_estimator.py"
    click M5 "https://github.com/neeek2303/emoportraits/blob/main/networks/basic_avatar/multiscale_discriminator.py"
    click W1 "https://github.com/neeek2303/emoportraits/blob/main/models/stage_1/volumetric_avatar/va.py"
    click W2 "https://github.com/neeek2303/emoportraits/blob/main/models/stage_1/volumetric_avatar/va_arguments.py"
    click W3 "https://github.com/neeek2303/emoportraits/blob/main/models/stage_1/volumetric_avatar/va_losses_and_visuals.py"
    click S21 "https://github.com/neeek2303/emoportraits/blob/main/networks/volumetric_avatar/decoder_s2.py"
    click S22 "https://github.com/neeek2303/emoportraits/blob/main/networks/volumetric_avatar/expression_embedder.py"
    click S23 "https://github.com/neeek2303/emoportraits/blob/main/networks/volumetric_avatar/global_encoder.py"
    click S24 "https://github.com/neeek2303/emoportraits/blob/main/networks/volumetric_avatar/local_encoder.py"
    click S25 "https://github.com/neeek2303/emoportraits/blob/main/networks/volumetric_avatar/volume_renderer.py"
    click W21 "https://github.com/neeek2303/emoportraits/blob/main/models/stage_2/base/volumetric_avatar_two.py"
    click W22 "https://github.com/neeek2303/emoportraits/blob/main/models/stage_2/base/va_losses_and_visuals_two.py"
    click L1 "https://github.com/neeek2303/emoportraits/blob/main/losses/adversarial.py"
    click L2 "https://github.com/neeek2303/emoportraits/blob/main/losses/perceptual.py"
    click L3 "https://github.com/neeek2303/emoportraits/blob/main/losses/gaze.py"
    click L4 "https://github.com/neeek2303/emoportraits/blob/main/losses/head_pose_matching.py"
    click L5 "https://github.com/neeek2303/emoportraits/blob/main/losses/psnr.py"
    click L6 "https://github.com/neeek2303/emoportraits/blob/main/losses/lpips.py"
    click L7 "https://github.com/neeek2303/emoportraits/blob/main/losses/warping_loss.py"
    click L8 "https://github.com/neeek2303/emoportraits/blob/main/losses/warping_regularizer.py"
    click L9 "https://github.com/neeek2303/emoportraits/tree/main/losses/rt_gene_loss/"
    click L10 "https://github.com/neeek2303/emoportraits/tree/main/losses/rt_gene_loss_fb/"
    click TC "https://github.com/neeek2303/emoportraits/blob/main/train.py"
    click IC "https://github.com/neeek2303/emoportraits/blob/main/run_video_driven_pipeline.py"
    click LG1 "https://github.com/neeek2303/emoportraits/blob/main/utils/logger.py"
    click LG2 "https://github.com/neeek2303/emoportraits/blob/main/utils/logger_wandb.py"
    click LG3 "https://github.com/neeek2303/emoportraits/blob/main/experiments/args.txt"

    %% Styles
    classDef data fill:#D0E6FA,stroke:#3B82F6,color:#1E40AF;
    classDef pre fill:#E0F2F1,stroke:#009688,color:#004D40;
    classDef model fill:#D1FAE5,stroke:#10B981,color:#065F46;
    classDef loss fill:#FEE2E2,stroke:#EF4444,color:#991B1B;
    classDef ctrl fill:#FEF3C7,stroke:#F59E0B,color:#92400E;
    classDef ext fill:none,stroke-dasharray: 5 5,stroke:#6B7280,color:#374151;
```
