# cleardusk/3ddfa_v2 architecture

> Architecture diagram of the GitHub repository cleardusk/3ddfa_v2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cleardusk/3ddfa_v2
- Repository: https://github.com/cleardusk/3ddfa_v2

## Overview

##### Project type and purpose
- 3DDFA_V2 is a Python‑based full-stack computer‑vision toolkit for fast, accurate, stable 3D dense face alignment and rendering.
- It combines:
- Face detection (FaceBoxes module)
- 3D‑MM parameter regression (TDDFA module)
- 3D mesh construction and rendering (Sim3DR + Cython/C++ utils)
- CLI/interactive demos (demo.py, demo_video.py, demo_webcam_smooth.py, Gradio web app)

##### File structure
- Top‑level modules:
- FaceBoxes/ – face detector with build scripts and ONNX support
- Sim3DR/ – low‑level C++/Cython rasterizer for 3D mesh
- bfm/ – 3D Morphable Model loader & onnx variant
- TDDFA*.py – core 3DMM regressor and ONNX stub
- configs/weights/ – model configs (YAML, MAT, PKL) & pretrained weights (.pth)
- utils/ – utilities (I/O, rendering, pose, PNCC, UV mapping, serialization)
- demo*.py & examples/ – entry‑point scripts and sample data
- Build scripts: build_cpu_nms.sh, build_sim3dr.sh, build.sh compile Cython/C++ extensions.
- Dependencies defined in requirements.txt; major ones: PyTorch, onnxruntime, numpy, OpenCV.

##### README for additional insights
- Describes two execution modes: native PyTorch and accelerated ONNXRuntime (`--onnx` flag).
- Provides latency numbers, usage patterns, configuration mechanics.
- Mentions integration of FaceBoxes.PyTorch, custom C++ rasterizer, Gradio demo.

##### Main components and interactions
a. Components
1. Input Interfaces
- CLI demos (demo.py, demo_video*, demo_webcam_smooth.py)
- Notebook/Gradio web app
2. Face Detector
- FaceBoxes (Python + optional ONNXRuntime)
- NMS modules (Cython/C++)
3. 3DMM Regressor
- TDDFA (PyTorch model) or TDDFA_ONNX (onnxruntime)
- BFM loader (bfm.py / bfm_onnx.py)
4. Mesh & Rendering
- Sim3DR (rasterize.pyx + rasterize_kernel.cpp)
- utils/render.py / render_ctypes.so
5. Data Stores
- configs/ (YAML, MAT, PKL files)
- weights/ (MobileNet and FaceBoxes .pth models)
6. Utilities
- utils/io (image/video I/O), utils/pose, utils/pncc, utils/uv, utils/serialization
b. Relationships & Data Flow
1. User invokes a demo script → parses CLI args (input file, flags, config)
2. Load config (YAML) → load weights (.pth for PyTorch or .onnx for ONNX)
3. Read image/video frame via utils/io
6. BFM module + parameters → generate 3D mesh vertices & triangles
7. Sim3DR rasterizer or utils/render_ctypes → project & render mesh into target view (depth, uv, pncc, pose, 2D/3D landmarks)
8. Overlay results / serialize to ply/obj / display or write file
c. Architectural patterns & design principles
- Layered pipeline: detection → regression → mesh construction → rendering
- Pluggable backends: PyTorch vs ONNXRuntime, Cython vs pure Python
- Separation of concerns: modular subpackages for detector, regressor, model loader, renderer, utilities
- Config‑driven design: all model hyperparams and file paths live in configs/
d. Key technologies & libraries
- PyTorch for model training/inference
- ONNXRuntime for accelerated inference
- Cython & C++ for performance‑critical NMS and rasterizer
- OpenCV, numpy for data handling
- YAML/PKL/MAT for configuration & 3DMM data

##### Detail is critical
- Don’t collapse the NMS Cython component or the Sim3DR rasterizer into generic “performance code.”
- Both onnx and PyTorch branches explicitly.
- Demo scripts and build scripts as first‑class components in orchestration.
- Configuration sources (configs/, command‑line flags, environment variables like OMP_NUM_THREADS).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Build Phase
    subgraph "Build Phase"
        B1["build.sh"]:::green
        B2["FaceBoxes/build_cpu_nms.sh"]:::green
        B3["Sim3DR/build_sim3dr.sh"]:::green
        B4["utils/asset/build_render_ctypes.sh"]:::green
    end

    %% Input Interfaces
    subgraph "Input Interfaces"
        CLI["demo.py, demo_video.py, demo_video_smooth.py, demo_webcam_smooth.py"]:::blue
        NB["demo.ipynb"]:::blue
        GR["gradiodemo.py"]:::blue
    end

    %% Configuration Store
    Configs[(configs/)]:::orange
    Weights[(weights/)]:::orange

    %% Decision on Backend
    ONNX_Flag{"--onnx flag?"}:::gray

    %% Detector Layer
    subgraph "Face Detector"
        FB_PY["FaceBoxes/FaceBoxes.py"]:::blue
        FB_ONNX["FaceBoxes/FaceBoxes_ONNX.py"]:::blue
        FB_MODEL["FaceBoxes/models/faceboxes.py"]:::blue
        FB_ONNX_SUPPORT["FaceBoxes/onnx.py"]:::blue
        NMS_CY["FaceBoxes/utils/nms/cpu_nms.pyx"]:::green
        NMS_PY["FaceBoxes/utils/nms/py_cpu_nms.py"]:::blue
    end

    %% Regressor Layer
    subgraph "3DMM Regressor"
        TDDFA_PY["TDDFA.py"]:::blue
        TDDFA_ONNX["TDDFA_ONNX.py"]:::blue
    end

    %% BFM Loader
    subgraph "3D Morphable Model Loader"
        BFM_PY["bfm/bfm.py"]:::blue
        BFM_ONNX["bfm/bfm_onnx.py"]:::blue
        BFM_DATA[(configs/BFM_UV.mat, configs/bfm_noneck_v3.pkl)]:::orange
    end

    %% Rendering Layer
    subgraph "Mesh Construction & Rendering"
        subgraph "Sim3DR Rasterizer"
            SIM_IF["Sim3DR/Sim3DR.py"]:::blue
            RAST_PYX["Sim3DR/lib/rasterize.pyx"]:::green
            RAST_CPP["Sim3DR/lib/rasterize_kernel.cpp"]:::green
            RAST_H["Sim3DR/lib/rasterize.h"]:::green
        end
        subgraph "CTypes Renderer"
            REND_C["utils/asset/render.c"]:::green
            REND_PYCT["utils/render_ctypes.py"]:::blue
        end
        subgraph "Pure-Python Render Utils"
            REND_UTIL["utils/render.py"]:::blue
            DEPTH["utils/depth.py"]:::blue
            UV["utils/uv.py"]:::blue
        end
    end

    %% Utilities
    subgraph "Utilities"
        IO["utils/io.py"]:::blue
        POSE["utils/pose.py"]:::blue
        PNCC["utils/pncc.py"]:::blue
        UV_U["utils/uv.py"]:::blue
        SERIAL["utils/serialization.py"]:::blue
        ONNX_UTIL["utils/onnx.py"]:::blue
        FUNCS["utils/functions.py"]:::blue
        TUTIL["utils/tddfa_util.py"]:::blue
    end

    %% Dependencies
    subgraph "External Dependencies"
        PYTORCH[/"PyTorch"/]:::gray
        ONNXRUNTIME[/"ONNXRuntime"/]:::gray
        CYTHON[/"Cython"/]:::gray
        OPENCV[/"OpenCV, NumPy"/]:::gray
    end

    %% Output
    subgraph "Output"
        OVERLAY["Overlay & Display"]:::blue
        SAVE["Serialize to PLY/OBJ"]:::blue
    end

    %% Data Flow
    CLI -->|invoke| Configs
    CLI -->|invoke| Weights
    CLI -->|invoke| ONNX_Flag
    NB -->|invoke| Configs
    NB -->|invoke| Weights
    NB -->|invoke| ONNX_Flag
    GR -->|invoke| Configs
    GR -->|invoke| Weights
    GR -->|invoke| ONNX_Flag

    Configs -->|load YAML| ONNX_Flag
    Weights -->|load .pth/.onnx| ONNX_Flag

    ONNX_Flag -->|false| FB_PY
    ONNX_Flag -->|true| FB_ONNX
    FB_PY --> FB_MODEL
    FB_ONNX --> FB_ONNX_SUPPORT
    FB_PY --> NMS_PY
    FB_PY --> NMS_CY
    FB_ONNX --> NMS_PY
    FB_ONNX --> NMS_CY
    FB_MODEL -->|bboxes| IO

    IO -->|"image frame"| FB_PY
    IO -->|"image frame"| FB_ONNX

    FB_PY -->|bboxes| TDDFA_PY
    FB_ONNX -->|bboxes| TDDFA_ONNX

    ONNX_Flag -->|false| TDDFA_PY
    ONNX_Flag -->|true| TDDFA_ONNX

    TDDFA_PY -->|params| BFM_PY
    TDDFA_ONNX -->|params| BFM_ONNX

    BFM_PY -->|vertices| SIM_IF
    BFM_ONNX -->|vertices| SIM_IF
    BFM_DATA --> BFM_PY
    BFM_DATA --> BFM_ONNX

    SIM_IF -->|mesh| RAST_PYX
    SIM_IF -->|mesh| RAST_CPP
    RAST_PYX -->|rasterized| OVERLAY
    RAST_CPP -->|rasterized| OVERLAY

    REND_C -->|rasterized| REND_PYCT
    REND_PYCT -->|image| OVERLAY

    REND_UTIL -->|image| OVERLAY
    DEPTH -->|depth map| OVERLAY
    UV -->|uv map| OVERLAY

    OVERLAY --> SAVE

    %% Dependencies usage
    TDDFA_PY --> PYTORCH
    TDDFA_ONNX --> ONNXRUNTIME
    FB_ONNX_SUPPORT --> ONNXRUNTIME
    NMS_CY --> CYTHON
    RAST_PYX --> CYTHON
    IO --> OPENCV
    IO --> OPENCV

    %% Click Events
    click FB_PY "https://github.com/cleardusk/3ddfa_v2/blob/master/FaceBoxes/FaceBoxes.py"
    click FB_ONNX "https://github.com/cleardusk/3ddfa_v2/blob/master/FaceBoxes/FaceBoxes_ONNX.py"
    click FB_MODEL "https://github.com/cleardusk/3ddfa_v2/blob/master/FaceBoxes/models/faceboxes.py"
    click FB_ONNX_SUPPORT "https://github.com/cleardusk/3ddfa_v2/blob/master/FaceBoxes/onnx.py"
    click NMS_CY "https://github.com/cleardusk/3ddfa_v2/blob/master/FaceBoxes/utils/nms/cpu_nms.pyx"
    click NMS_PY "https://github.com/cleardusk/3ddfa_v2/blob/master/FaceBoxes/utils/nms/py_cpu_nms.py"
    click BFM_PY "https://github.com/cleardusk/3ddfa_v2/blob/master/bfm/bfm.py"
    click BFM_ONNX "https://github.com/cleardusk/3ddfa_v2/blob/master/bfm/bfm_onnx.py"
    click SIM_IF "https://github.com/cleardusk/3ddfa_v2/blob/master/Sim3DR/Sim3DR.py"
    click RAST_PYX "https://github.com/cleardusk/3ddfa_v2/blob/master/Sim3DR/lib/rasterize.pyx"
    click RAST_CPP "https://github.com/cleardusk/3ddfa_v2/blob/master/Sim3DR/lib/rasterize_kernel.cpp"
    click RAST_H "https://github.com/cleardusk/3ddfa_v2/blob/master/Sim3DR/lib/rasterize.h"
    click REND_C "https://github.com/cleardusk/3ddfa_v2/blob/master/utils/asset/render.c"
    click REND_PYCT "https://github.com/cleardusk/3ddfa_v2/blob/master/utils/render_ctypes.py"
    click Configs "https://github.com/cleardusk/3ddfa_v2/tree/master/configs/"
    click Weights "https://github.com/cleardusk/3ddfa_v2/tree/master/weights/"
    click B2 "https://github.com/cleardusk/3ddfa_v2/blob/master/FaceBoxes/build_cpu_nms.sh"
    click B3 "https://github.com/cleardusk/3ddfa_v2/blob/master/Sim3DR/build_sim3dr.sh"
    click B4 "https://github.com/cleardusk/3ddfa_v2/blob/master/utils/asset/build_render_ctypes.sh"

    %% Styles
    classDef blue fill:#D0E8FF,stroke:#0366D6,stroke-width:1px
    classDef green fill:#DFF2BF,stroke:#4F8A10,stroke-width:1px
    classDef orange fill:#FFE1B3,stroke:#D26900,stroke-width:1px
    classDef gray fill:#F0F0F0,stroke:#888888,stroke-width:1px,stroke-dasharray: 2 2
    class B1,B2,B3,B4 green
    class PYTORCH,ONNXRUNTIME,CYTHON,OPENCV gray
```
