# abdallahdib/nextface architecture

> Architecture diagram of the GitHub repository abdallahdib/nextface, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abdallahdib/nextface
- Repository: https://github.com/abdallahdib/nextface

## Overview

##### Project Type and Purpose
- NextFace is a specialized PyTorch library designed for high-fidelity 3D face reconstruction from single or multiple monocular images.
- Its purpose is to estimate comprehensive scene attributes: 3D geometry, various reflectance maps (diffuse, specular, and roughness), camera parameters, head pose, and scene illumination via differentiable ray tracing.

##### File Structure
- Top-level files include key configuration files (config.py, optimConfig.ini, optimConfigShadows.ini), data processing scripts (optimizer.py, pipeline.py, projection.py, renderer.py, replay.py) and utility modules (utils.py, image.py, gaussiansmoothing.py, sphericalharmonics.py, textureloss.py).
- There is a dedicated directory “baselMorphableModel” which holds essential data files (e.g., landmark files and UV parametrizations) that underpin the morphable model for faces.
- Other files like demo.ipynb and environment.yml indicate that there is support for interactive demos and environment setup.

##### README Insights
- The README clearly explains the three-stage optimization process and describes how landmarks (via mediapipe or FAN), ray tracing, and statistical albedo estimation work together.
- It also lists prerequisite models (Basel face model and albedo model) and explains output artifacts (rendered images, 3D OBJ meshes, diffuse/specular/roughness maps).
- The text indicates integration with external services/libraries such as PyTorch, redner for ray tracing, and optional mediapipe for landmark detection.

a. Main Components:
- Input Module: Handles image input and pre-processing. This includes loading images from disk and possibly handling batch inputs.
- Landmark Detection Module: Utilizes either mediapipe (by default) or FAN (switchable via the config file) to extract 2D facial landmarks.
- Morphable Model & Preprocessing Module: Contains the Basel morphable model files (located in baselMorphableModel) which define the face structure and UV parametrizations.
- Optimization Core:
– Stage 1: Coarse alignment using landmarks and initial estimation of head pose and expression.
– Stage 3: Per-pixel refinement of albedos using consistency, smoothness, and symmetry regularizers.
- Rendering Module: Leveraging differentiable ray tracing (with redner) to generate estimated renders and projections of the reconstructed face.
- Output Module: Manages output files (rendered images, OBJ 3D meshes, and texture maps) and produces the final results.

b. Relationships and Data Flow:
- Initially, input images are fed into the Landmark Detection Module.
- Detected landmarks guide the Morphable Model & Preprocessing Module to initialize the facial structure.
- Data flows into the Optimization Core where, stage by stage, the model parameters are refined.
- The refined model and associated parameters are passed on to the Rendering Module, which simulates scene lighting (via spherical harmonics) and ray tracing.
- Finally, the output module saves generated renderings and 3D mesh files.

c. Architectural Patterns and Technologies:
- The optimization process is an iterative multi-stage pipeline – clearly delineate the three stages.
- Use directional flows to indicate how initial estimates inform later refinements.
- Use of PyTorch’s autograd engine for gradient-based optimization and the integration of differentiable ray tracing.
- Recognize the configuration’s role (via optimConfig.ini) in parameterizing each stage and allowing for the choice between landmark detectors.

d. Technologies and External Integrations:
- PyTorch for deep learning and optimization.
- redner (or a similar ray tracing library) for generating differentiable renders.
- mediapipe / FAN for landmark extraction.
- External morphable model data from Basel and albedo models.
- Tools like environment.yml for dependency management.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Module
    inputMod["Input Module"]:::module

    %% Landmark Detection Module
    subgraph "Landmark Detection Module"
        mediapipe["Mediapipe Landmark (landmarksmediapipe.py)"]:::module
        fan["FAN Landmark (landmarksfan.py)"]:::module
    end

    %% Morphable Model & Preprocessing Module
    subgraph "Morphable Model & Preprocessing Module"
        basel["Basel Morphable Model"]:::module
        morph["Morphable Model Init (morphablemodel.py)"]:::module
    end

    %% Optimization Core
    subgraph "Optimization Core"
        stage1["Stage 1: Coarse Alignment"]:::module
        stage2["Stage 2: Photo Consistency"]:::module
        stage3["Stage 3: Per-pixel Refinement"]:::module
    end

    %% Rendering Module
    subgraph "Rendering Module"
        renderer["Renderer (renderer.py)"]:::module
        projection["Projection (projection.py)"]:::module
    end

    %% Output Module
    outputMod["Output Module"]:::module

    %% Camera and Scene Setup
    camera["Camera Setup (camera.py)"]:::module

    %% External Integrations
    pytorch["PyTorch"]:::external
    redner["redner"]:::external

    %% Utility and Supporting Modules
    subgraph "Utilities"
        utils["Utils (utils.py)"]:::utility
        smoothing["GaussianSmoothing (gaussiansmoothing.py)"]:::utility
        spherical["SphericalHarmonics (sphericalharmonics.py)"]:::utility
        texture["TextureLoss (textureloss.py)"]:::utility
    end

    %% Optimization Tools (side components)
    opt_py["optimizer.py"]:::fileModule
    pipe_py["pipeline.py"]:::fileModule
    opt_ini["optimConfig.ini"]:::config
    opt_shad["optimConfigShadows.ini"]:::config

    %% Connections - Main Pipeline
    inputMod -->|"inputImage"| mediapipe
    inputMod -->|"inputImage"| fan

    mediapipe -->|"landmarksGuide"| basel
    fan -->|"landmarksGuide"| basel
    mediapipe -->|"landmarksGuide"| morph
    fan -->|"landmarksGuide"| morph

    basel -->|"preparesModel"| stage1
    morph -->|"preprocessesModel"| stage1

    stage1 -->|"refines"| stage2
    stage2 -->|"improves"| stage3

    stage3 -->|"passesParams"| renderer
    stage3 -->|"passesParams"| projection

    renderer -->|"generatesRenderings"| outputMod
    projection -->|"projectsImages"| outputMod

    camera ---|"providesParams"| renderer
    camera ---|"providesParams"| projection

    %% External integrations connections
    pytorch ---|"autogradOpt"| stage1
    pytorch ---|"autogradOpt"| stage2
    pytorch ---|"autogradOpt"| stage3

    redner ---|"rayTracing"| renderer
    redner ---|"rayTracing"| projection

    %% Optimization Tools integration (dashed connections)
    opt_py ---|"controlsOpt"| stage1
    pipe_py ---|"orchestrates"| stage1
    opt_ini ---|"configures"| stage1
    opt_shad ---|"configures"| stage1

    %% Utilities connections (dashed)
    utils ---|"helpers"| stage1
    spherical ---|"illumination"| stage2
    smoothing ---|"regularizes"| stage3
    texture ---|"lossCalc"| stage3

    %% Click Events
    click inputMod "https://github.com/abdallahdib/nextface/blob/main/image.py"
    click mediapipe "https://github.com/abdallahdib/nextface/blob/main/landmarksmediapipe.py"
    click fan "https://github.com/abdallahdib/nextface/blob/main/landmarksfan.py"
    click basel "https://github.com/abdallahdib/nextface/tree/main/baselMorphableModel"
    click morph "https://github.com/abdallahdib/nextface/blob/main/morphablemodel.py"
    click opt_py "https://github.com/abdallahdib/nextface/blob/main/optimizer.py"
    click pipe_py "https://github.com/abdallahdib/nextface/blob/main/pipeline.py"
    click opt_ini "https://github.com/abdallahdib/nextface/blob/main/optimConfig.ini"
    click opt_shad "https://github.com/abdallahdib/nextface/blob/main/optimConfigShadows.ini"
    click renderer "https://github.com/abdallahdib/nextface/blob/main/renderer.py"
    click projection "https://github.com/abdallahdib/nextface/blob/main/projection.py"
    click outputMod "https://github.com/abdallahdib/nextface/tree/main/output"
    click camera "https://github.com/abdallahdib/nextface/blob/main/camera.py"
    click utils "https://github.com/abdallahdib/nextface/blob/main/utils.py"
    click smoothing "https://github.com/abdallahdib/nextface/blob/main/gaussiansmoothing.py"
    click spherical "https://github.com/abdallahdib/nextface/blob/main/sphericalharmonics.py"
    click texture "https://github.com/abdallahdib/nextface/blob/main/textureloss.py"

    %% Styles
    classDef module fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef external fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef utility fill:#FADBD8,stroke:#922B21,stroke-width:2px;
    classDef fileModule fill:#D6EAF8,stroke:#2874A6,stroke-width:2px;
    classDef config fill:#ABEBC6,stroke:#196F3D,stroke-width:2px;
```
