How choyingw/synergynet works
Project Type and Purpose
• SynergyNet is a research-grade full‐stack Python toolkit for 3D facial geometry: it performs face detection, 3DMM fitting (synergy between 3D Morphable Models and 3D landmarks), UV‐texture GAN training, and 3D rendering. It bundles C/C++/Cython modules for speed (FaceBoxes and Sim3DR), PyTorch models (backbone_nets, synergy3DMM), and a suite of scripts for inference, benchmarking, training, and artistic/real‐face texture rendering.
A. Frontend / Inference API
• Scripts: singleImage.py, singleImage_simple.py, benchmark.py, artistic.py, uv_texture_realFaces.py
• CLI / Python API entry point: synergy3DMM.SynergyNet
B. Face Detection
• Module: FaceBoxes/
• FaceBoxes.py (Python wrapper)
• build_cpu_nms.sh, cpu_nms Cython extensions
C. 3DMM + Landmark Synergy Network
• Core: synergy3DMM.py (defines SynergyNet class)
• Model definitions: model_building.py, backbone_nets/ (ResNeSt, MobileNet, ResNet, GhostNet, PointNet)
• Loss & training utilities: loss_definition.py, train_script.sh, backbone_nets, utils/params.py
D. 3D Renderer
• Module: Sim3DR/
• Sim3DR.py (Python wrapper)
• build_sim3dr.sh, C++/Cython rasterize.* for fast mesh rendering
E. UV‐Texture GAN (optional training phase)
• Scripts & instructions in README under “Training UV-texture GAN”
• External dependency: pix2pix repository invoked via custom scripts (no code here, but note as external service)
F. Utilities & I/O
• utils/ folder: IO, rendering helpers, configuration, timing, box_utils, nms_wrapper, prior_box, etc.
• pretrained/ & weights/: store model checkpoints
Data Flow & Interaction
1. Input Image(s) →
[Face Detection (FaceBoxes)] → face bbox + crop →
[Synergy Network] → 3D landmarks + 3DMM coefficients →
[Optional UV-Texture GAN] → UV map refinement →
[Mesh Construction & Rendering (Sim3DR)] → 3D mesh & rendered output →
[Output: landmarks, mesh points, Euler angles, rendered frames]
2. Training Pipeline (separate flow):
Dataset (3DDFA aug data) → loss_definition.py → model_building.py + backbone_nets → optimizer/training loop → checkpoint to pretrained/
Architectural Patterns & Principles
• Modular separation:
• C/C++ speedups (FaceBoxes, Sim3DR) are isolated in their own directories and built via shell scripts.
• Deep‐learning logic and network definitions are in Python modules under backbone_nets and synergy3DMM.
• Utilities are shared across detection, modeling, and rendering.
• Dependency inversion: high‐level scripts invoke unified API (SynergyNet), which hides detection, modeling, and rendering details.
• Extensibility: new backbones can drop into backbone_nets; the build scripts allow adding further C/C++ plugins.