How cleardusk/3ddfa_v2 works
Project type and purpose
• 3DDFA_V2 is a Python‑based full-stack computer‑vision toolkit for fast, accurate, stable 3D dense face alignment and rendering.
• It combines:
• Face detection (FaceBoxes module)
• 3D‑MM parameter regression (TDDFA module)
• 3D mesh construction and rendering (Sim3DR + Cython/C++ utils)
• CLI/interactive demos (demo.py, demo_video.py, demo_webcam_smooth.py, Gradio web app)
File structure
• Top‑level modules:
• FaceBoxes/ – face detector with build scripts and ONNX support
• Sim3DR/ – low‑level C++/Cython rasterizer for 3D mesh
• bfm/ – 3D Morphable Model loader & onnx variant
• TDDFA*.py – core 3DMM regressor and ONNX stub
• configs/weights/ – model configs (YAML, MAT, PKL) & pretrained weights (.pth)
• utils/ – utilities (I/O, rendering, pose, PNCC, UV mapping, serialization)
• demo*.py & examples/ – entry‑point scripts and sample data
• Build scripts: build_cpu_nms.sh, build_sim3dr.sh, build.sh compile Cython/C++ extensions.
• Dependencies defined in requirements.txt; major ones: PyTorch, onnxruntime, numpy, OpenCV.
README for additional insights
• Describes two execution modes: native PyTorch and accelerated ONNXRuntime (--onnx flag).
• Provides latency numbers, usage patterns, configuration mechanics.
• Mentions integration of FaceBoxes.PyTorch, custom C++ rasterizer, Gradio demo.
Main components and interactions
a. Components
1. Input Interfaces
• CLI demos (demo.py, demo_video*, demo_webcam_smooth.py)
• Notebook/Gradio web app
2. Face Detector
• FaceBoxes (Python + optional ONNXRuntime)
• NMS modules (Cython/C++)
3. 3DMM Regressor
• TDDFA (PyTorch model) or TDDFA_ONNX (onnxruntime)
• BFM loader (bfm.py / bfm_onnx.py)
4. Mesh & Rendering
• Sim3DR (rasterize.pyx + rasterize_kernel.cpp)
• utils/render.py / render_ctypes.so
5. Data Stores
• configs/ (YAML, MAT, PKL files)
• weights/ (MobileNet and FaceBoxes .pth models)
6. Utilities
• utils/io (image/video I/O), utils/pose, utils/pncc, utils/uv, utils/serialization
b. Relationships & Data Flow
1. User invokes a demo script → parses CLI args (input file, flags, config)
2. Load config (YAML) → load weights (.pth for PyTorch or .onnx for ONNX)
3. Read image/video frame via utils/io
6. BFM module + parameters → generate 3D mesh vertices & triangles
7. Sim3DR rasterizer or utils/render_ctypes → project & render mesh into target view (depth, uv, pncc, pose, 2D/3D landmarks)
8. Overlay results / serialize to ply/obj / display or write file
c. Architectural patterns & design principles
• Layered pipeline: detection → regression → mesh construction → rendering
• Pluggable backends: PyTorch vs ONNXRuntime, Cython vs pure Python
• Separation of concerns: modular subpackages for detector, regressor, model loader, renderer, utilities
• Config‑driven design: all model hyperparams and file paths live in configs/
d. Key technologies & libraries
• PyTorch for model training/inference
• ONNXRuntime for accelerated inference
• Cython & C++ for performance‑critical NMS and rasterizer
• OpenCV, numpy for data handling
• YAML/PKL/MAT for configuration & 3DMM data
Detail is critical
• Don’t collapse the NMS Cython component or the Sim3DR rasterizer into generic “performance code.”
• Both onnx and PyTorch branches explicitly.
• Demo scripts and build scripts as first‑class components in orchestration.
• Configuration sources (configs/, command‑line flags, environment variables like OMP_NUM_THREADS).