# choyingw/synergynet architecture

> Architecture diagram of the GitHub repository choyingw/synergynet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/choyingw/synergynet
- Repository: https://github.com/choyingw/synergynet

## Overview

##### Project Type and Purpose
- SynergyNet is a research-grade full‐stack Python toolkit for 3D facial geometry: it performs face detection, 3DMM fitting (synergy between 3D Morphable Models and 3D landmarks), UV‐texture GAN training, and 3D rendering. It bundles C/C++/Cython modules for speed (FaceBoxes and Sim3DR), PyTorch models (backbone_nets, synergy3DMM), and a suite of scripts for inference, benchmarking, training, and artistic/real‐face texture rendering.

A. Frontend / Inference API
- Scripts: singleImage.py, singleImage_simple.py, benchmark.py, artistic.py, uv_texture_realFaces.py
- CLI / Python API entry point: `synergy3DMM.SynergyNet`
B. Face Detection
- Module: FaceBoxes/
- FaceBoxes.py (Python wrapper)
- build_cpu_nms.sh, cpu_nms Cython extensions
C. 3DMM + Landmark Synergy Network
- Core: synergy3DMM.py (defines SynergyNet class)
- Model definitions: model_building.py, backbone_nets/ (ResNeSt, MobileNet, ResNet, GhostNet, PointNet)
- Loss & training utilities: loss_definition.py, train_script.sh, backbone_nets, utils/params.py
D. 3D Renderer
- Module: Sim3DR/
- Sim3DR.py (Python wrapper)
- build_sim3dr.sh, C++/Cython rasterize.* for fast mesh rendering
E. UV‐Texture GAN (optional training phase)
- Scripts & instructions in README under “Training UV-texture GAN”
- External dependency: pix2pix repository invoked via custom scripts (no code here, but note as external service)
F. Utilities & I/O
- utils/ folder: IO, rendering helpers, configuration, timing, box_utils, nms_wrapper, prior_box, etc.
- pretrained/ & weights/: store model checkpoints

##### Data Flow & Interaction
1. Input Image(s) →
[Face Detection (FaceBoxes)] → face bbox + crop →
[Synergy Network] → 3D landmarks + 3DMM coefficients →
[Optional UV-Texture GAN] → UV map refinement →
[Mesh Construction & Rendering (Sim3DR)] → 3D mesh & rendered output →
[Output: landmarks, mesh points, Euler angles, rendered frames]
2. Training Pipeline (separate flow):
Dataset (3DDFA aug data) → loss_definition.py → model_building.py + backbone_nets → optimizer/training loop → checkpoint to pretrained/

##### Architectural Patterns & Principles
- Modular separation:
- C/C++ speedups (FaceBoxes, Sim3DR) are isolated in their own directories and built via shell scripts.
- Deep‐learning logic and network definitions are in Python modules under backbone_nets and synergy3DMM.
- Utilities are shared across detection, modeling, and rendering.
- Dependency inversion: high‐level scripts invoke unified API (SynergyNet), which hides detection, modeling, and rendering details.
- Extensibility: new backbones can drop into backbone_nets; the build scripts allow adding further C/C++ plugins.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Scripts / CLI
    subgraph "Scripts / CLI"
        SI["singleImage.py"]:::script
        SIS["singleImage_simple.py"]:::script
        BP["benchmark.py"]:::script
        ART["artistic.py"]:::script
        UVS["uv_texture_realFaces.py"]:::script
        TS["train_script.sh"]:::script
        MT["main_train.py"]:::script
    end

    %% C/C++ Extensions
    subgraph "C/C++ Extensions"
        subgraph FaceBoxes
            FBpy["FaceBoxes.py"]:::cExt
            FBbuild["build_cpu_nms.sh"]:::script
            NMS["cpu_nms.pyx"]:::cExt
            BU1["box_utils.py"]:::cExt
            NU["nms_wrapper.py"]:::cExt
            PB["prior_box.py"]:::cExt
            TM["timer.py"]:::cExt
            FBmodel["faceboxes.py"]:::cExt
            FBw["FaceBoxesProd.pth"]:::external
        end
        subgraph Sim3DR
            SRpy["Sim3DR.py"]:::cExt
            SRbuild["build_sim3dr.sh"]:::script
            Rpyx["rasterize.pyx"]:::cExt
            Rcpp["rasterize_kernel.cpp"]:::cExt
            Rh["rasterize.h"]:::cExt
            Light["lighting.py"]:::cExt
        end
    end

    %% Core Python Library
    subgraph "Core Python Library"
        SY["synergy3DMM.py"]:::py
        MB["model_building.py"]:::py
        LD["loss_definition.py"]:::py
        subgraph Backbones
            GB["ghostnet_backbone.py"]:::py
            MV1["mobilenetv1_backbone.py"]:::py
            MV2["mobilenetv2_backbone.py"]:::py
            RN["resnet_backbone.py"]:::py
            PN["pointnet_backbone.py"]:::py
            RNS["ResNeSt modules"]:::py
        end
        subgraph Utils
            IO["io.py"]:::py
            INF["inference.py"]:::py
            REN["render.py"]:::py
            DDFA["ddfa.py"]:::py
            PM["params.py"]:::py
        end
    end

    %% External Dependencies
    subgraph "External"
        PT["PyTorch"]:::external
        GAN["pix2pix GAN"]:::external
    end

    %% Data Flow
    SI -->|calls| SY
    SIS -->|calls| SY
    BP -->|calls| SY
    ART -->|calls| SY
    SY -->|detect face| FBpy
    FBpy -->|bbox + crop| SY
    SY -->|infer 3DMM & landmarks| SY
    SY -->|build model| MB
    MB -->|uses| RNS
    MB -->|uses| GB
    MB -->|uses| MV1
    MB -->|uses| MV2
    MB -->|uses| RN
    MB -->|uses| PN
    MB -->|logs| LD
    SY -->|get_all_outputs| INF
    SY -->|calls renderer| SRpy
    SRpy -->|rasterize mesh| Rpyx
    Rpyx -->|calls| Rcpp
    Rcpp -->|includes| Rh
    SRpy -->|lighting| Light
    SRpy -->|outputs| REN
    SY -->|outputs| IO

    %% Optional UV-Texture GAN path
    SY -->|produce UV map| GAN
    GAN -->|refine UV| UVS
    UVS -->|feeds into| SRpy

    %% Training flow
    TS -->|train pipeline| MB
    TS -->|uses| LD
    TS -->|uses| PM
    TS -->|stores checkpoint| pretrained["pretrained/"]:::external
    MT -->|train pipeline| MB

    %% Build script flows
    FBbuild -.-> NMS
    SRbuild -.-> Rpyx

    %% Click events
    click SI "https://github.com/choyingw/synergynet/blob/main/./singleImage.py"
    click SIS "https://github.com/choyingw/synergynet/blob/main/./singleImage_simple.py"
    click BP "https://github.com/choyingw/synergynet/blob/main/./benchmark.py"
    click ART "https://github.com/choyingw/synergynet/blob/main/./artistic.py"
    click UVS "https://github.com/choyingw/synergynet/blob/main/./uv_texture_realFaces.py"
    click SY "https://github.com/choyingw/synergynet/blob/main/./synergy3DMM.py"
    click FBpy "https://github.com/choyingw/synergynet/blob/main/FaceBoxes/FaceBoxes.py"
    click FBbuild "https://github.com/choyingw/synergynet/blob/main/FaceBoxes/build_cpu_nms.sh"
    click NMS "https://github.com/choyingw/synergynet/blob/main/FaceBoxes/utils/nms/cpu_nms.pyx"
    click BU1 "https://github.com/choyingw/synergynet/blob/main/FaceBoxes/utils/box_utils.py"
    click NU "https://github.com/choyingw/synergynet/blob/main/FaceBoxes/utils/nms_wrapper.py"
    click PB "https://github.com/choyingw/synergynet/blob/main/FaceBoxes/utils/prior_box.py"
    click TM "https://github.com/choyingw/synergynet/blob/main/FaceBoxes/utils/timer.py"
    click FBmodel "https://github.com/choyingw/synergynet/blob/main/FaceBoxes/models/faceboxes.py"
    click FBw "https://github.com/choyingw/synergynet/blob/main/FaceBoxes/weights/FaceBoxesProd.pth"
    click SRpy "https://github.com/choyingw/synergynet/blob/main/Sim3DR/Sim3DR.py"
    click SRbuild "https://github.com/choyingw/synergynet/blob/main/Sim3DR/build_sim3dr.sh"
    click Rpyx "https://github.com/choyingw/synergynet/blob/main/Sim3DR/lib/rasterize.pyx"
    click Rcpp "https://github.com/choyingw/synergynet/blob/main/Sim3DR/lib/rasterize_kernel.cpp"
    click Rh "https://github.com/choyingw/synergynet/blob/main/Sim3DR/lib/rasterize.h"
    click IO "https://github.com/choyingw/synergynet/blob/main/utils/io.py"
    click INF "https://github.com/choyingw/synergynet/blob/main/utils/inference.py"
    click REN "https://github.com/choyingw/synergynet/blob/main/utils/render.py"
    click DDFA "https://github.com/choyingw/synergynet/blob/main/utils/ddfa.py"
    click PM "https://github.com/choyingw/synergynet/blob/main/utils/params.py"
    click MB "https://github.com/choyingw/synergynet/blob/main/./model_building.py"
    click LD "https://github.com/choyingw/synergynet/blob/main/./loss_definition.py"
    click TS "https://github.com/choyingw/synergynet/blob/main/./train_script.sh"
    click MT "https://github.com/choyingw/synergynet/blob/main/./main_train.py"
    click GB "https://github.com/choyingw/synergynet/blob/main/backbone_nets/ghostnet_backbone.py"
    click MV1 "https://github.com/choyingw/synergynet/blob/main/backbone_nets/mobilenetv1_backbone.py"
    click MV2 "https://github.com/choyingw/synergynet/blob/main/backbone_nets/mobilenetv2_backbone.py"
    click RN "https://github.com/choyingw/synergynet/blob/main/backbone_nets/resnet_backbone.py"
    click PN "https://github.com/choyingw/synergynet/blob/main/backbone_nets/pointnet_backbone.py"
    click RNS "https://github.com/choyingw/synergynet/tree/main/backbone_nets/ResNeSt/"

    %% Styles
    classDef cExt fill:#b2f2bb,stroke:#333,stroke-width:1px
    classDef py fill:#bae1ff,stroke:#333,stroke-width:1px
    classDef script fill:#ffd8b1,stroke:#333,stroke-width:1px
    classDef external fill:#e2e2e2,stroke:#333,stroke-width:1px
```
