# yyuezhi/genvdm architecture

> Architecture diagram of the GitHub repository yyuezhi/genvdm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yyuezhi/genvdm
- Repository: https://github.com/yyuezhi/genvdm

## Overview

##### Project Type and Purpose
- GenVDM is a research‐oriented, full-stack machine-learning pipeline written in Python (PyTorch).
- It supports:
- Data preprocessing and dataset building (voxelization, rendering)
- Model training (using multi-GPU PyTorch)
- Model inference (generating Vector Displacement Maps from single images)
- Interactive visualization in Blender

##### File Structure
Top-level components:
- asset/ (images for docs)
- config/ (global YAML settings)
- dataset/ (custom C/Cython tools, data preprocessing, rendering scripts)
- instant-nsr-pl/ (third-party repo integrated as a submodule for neural surface reconstruction)
- src/ (core VDM code: data loaders, utilities)
- zero123plus/ (additional model/pipeline code)
- Scripts: train.py, inference.py, extract_VDM.py, generate.sh, launch.py
- Environment setup: install_env.sh, requirements.txt

##### Extract Key Insights from README
- Training: `python train.py --base config/example_run.yaml ...`
- Inference: `bash generate.sh <image> <exp> <checkpoint>`
- Interactive Modeling: `demo.blend` + output `.exr` in Blender
- External dependencies: Hugging Face (for checkpoints), Blender

##### Main Components & Their Relationships

A. Data Ingestion & Preprocessing
- Input: Raw 2D RGBA images (./input) and rendered datasets (./dataset/rendering_result)
- Tools:
– dataset/voxelize.py, patch_gen.py, multi_rendering.py
– C/Cython modules (dataset/cutils.c, flood_fill.c, .pyx)
– Scripts: dataset/setup.py, env_set.sh
- Output: Preprocessed data on disk (voxel grids, rendered views)

B. Configuration Management
- config/VDM_setting.yaml, example_run.yaml (global + dataset/training/inference settings)
- instant-nsr-pl/configs/example_run.yaml
- Used by train.py, inference.py, launch.py

C. Model Training
- Entry point: train.py (PyTorch Lightning or custom training loop)
- Core code:
– src/data/VDM.py (datasets, data loaders)
– src/utils/train_util.py (training helpers)
– instant-nsr-pl/models/… (geometry, NeRF, NeuS, texture modules)
– zero123plus/pipeline.py, model.py (additional pipelines or backbones)
– instant-nsr-pl/systems/criterions.py, neus_ortho.py (losses & training systems)
- Compute: multi-GPU, distributed training
- Output: Model checkpoints (./checkpoints/<exp_name>/)

D. Model Inference
- Entry point: inference.py, generate.sh
- Uses: Pretrained checkpoints + config/example_run.yaml
- Core utilities: src/utils/infer_util.py
- Output: Vector Displacement Maps (.exr, .png) in an output directory

E. Visualization & Interactive Modeling
- demo.blend (Blender 3.6 file configured for VDM)
- Extract script: extract_VDM.py (maybe converts raw output into Blender‐readable format)
- User opens .blend, loads .exr, sculpts/displaces in Blender UI

F. External Services / Storage
- Hugging Face dataset URLs (for pre-trained weights & raw data)
- GitHub (code repository)
- Blender (local desktop application)

##### Architectural Patterns & Design Principles
- Modular Python packages (src/, zero123plus/, instant-nsr-pl/)
- Separation of concerns: preprocessing vs. training vs. inference vs. visualization
- YAML-driven configuration for reproducibility
- C/Cython extensions for performance-critical steps (voxelization, flood fill)
- Script-based orchestration (bash, Python entry scripts)

##### Essential Elements
- Clear component labels (script names, directories, modules)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        direction TB
        dp1["voxelize.py"]:::data
        dp2["patch_gen.py"]:::data
        dp3["multi_rendering.py"]:::data
        dp4["cutils.c/.pyx"]:::data
        dp5["flood_fill.c/.pyx"]:::data
        dp6["setup.py"]:::data
        dp7["env_set.sh"]:::data
        dp8["preprocess.yaml"]:::data
        dp9["render.yaml"]:::data
    end

    %% Configuration Management
    subgraph "Configuration Management"
        direction TB
        cfg1["VDM_setting.yaml"]:::config
        cfg2["example_run.yaml"]:::config
        cfg3["instant-nsr-pl/example_run.yaml"]:::config
    end

    %% Model Training
    subgraph "Model Training"
        direction TB
        tr1["train.py"]:::compute
        tr2["VDM.py"]:::compute
        tr3["train_util.py"]:::compute
        tr4["geometry.py"]:::compute
        tr5["nerf.py"]:::compute
        tr6["neus.py"]:::compute
        tr7["texture.py"]:::compute
        tr8["criterions.py"]:::compute
        tr9["neus_ortho.py"]:::compute
        tr10["pipeline.py"]:::compute
        tr11["model.py"]:::compute
    end

    %% Model Inference
    subgraph "Model Inference"
        direction TB
        inf1["inference.py"]:::compute
        inf2["generate.sh"]:::compute
        inf3["infer_util.py"]:::compute
        inf4["example_run.yaml"]:::config
    end

    %% Visualization & Interactive Modeling
    subgraph "Visualization & Interactive Modeling"
        direction TB
        vis1["demo.blend"]:::external
        vis2["extract_VDM.py"]:::compute
    end

    %% External Services
    subgraph "External Services"
        direction TB
        ext1["Hugging Face"]:::external
        ext2["GitHub"]:::external
        ext3["Blender App"]:::external
    end

    %% Data Flow
    dp1 -->|"produces voxel grids"| tr2
    dp2 --> tr2
    dp3 --> tr2
    dp4 --> dp1
    dp5 --> dp1
    dp6 --> dp1
    dp7 --> dp1
    dp8 --> dp1
    dp9 --> dp3

    %% Config Usage
    cfg1 -->|loads settings| tr1
    cfg2 -->|loads settings| tr1
    cfg3 -->|loads settings| tr1
    cfg2 -->|loads settings| inf1
    ext1 -->|provides checkpoints| tr1
    ext1 -->|provides checkpoints| inf1

    %% Training to Inference
    tr1 -->|"outputs checkpoints"| inf1

    %% Inference to Visualization
    inf1 -->|"writes .exr/.png"| vis2
    vis2 --> vis1

    %% External Integration
    ext3 --> vis1
    ext2 --> dp1
    ext2 --> tr1
    ext2 --> inf1

    %% Click Events
    click dp1 "https://github.com/yyuezhi/genvdm/blob/main/dataset/voxelize.py"
    click dp2 "https://github.com/yyuezhi/genvdm/blob/main/dataset/patch_gen.py"
    click dp3 "https://github.com/yyuezhi/genvdm/blob/main/dataset/multi_rendering.py"
    click dp4 "https://github.com/yyuezhi/genvdm/blob/main/dataset/cutils.c"
    click dp5 "https://github.com/yyuezhi/genvdm/blob/main/dataset/flood_fill.c"
    click dp6 "https://github.com/yyuezhi/genvdm/blob/main/dataset/setup.py"
    click dp7 "https://github.com/yyuezhi/genvdm/blob/main/dataset/env_set.sh"
    click dp8 "https://github.com/yyuezhi/genvdm/blob/main/dataset/config/preprocess.yaml"
    click dp9 "https://github.com/yyuezhi/genvdm/blob/main/dataset/config/render.yaml"
    click cfg1 "https://github.com/yyuezhi/genvdm/blob/main/config/VDM_setting.yaml"
    click cfg2 "https://github.com/yyuezhi/genvdm/blob/main/config/example_run.yaml"
    click cfg3 "https://github.com/yyuezhi/genvdm/blob/main/instant-nsr-pl/configs/example_run.yaml"
    click tr1 "https://github.com/yyuezhi/genvdm/blob/main/train.py"
    click tr2 "https://github.com/yyuezhi/genvdm/blob/main/src/data/VDM.py"
    click tr3 "https://github.com/yyuezhi/genvdm/blob/main/src/utils/train_util.py"
    click tr4 "https://github.com/yyuezhi/genvdm/blob/main/instant-nsr-pl/models/geometry.py"
    click tr5 "https://github.com/yyuezhi/genvdm/blob/main/instant-nsr-pl/models/nerf.py"
    click tr6 "https://github.com/yyuezhi/genvdm/blob/main/instant-nsr-pl/models/neus.py"
    click tr7 "https://github.com/yyuezhi/genvdm/blob/main/instant-nsr-pl/models/texture.py"
    click tr8 "https://github.com/yyuezhi/genvdm/blob/main/instant-nsr-pl/systems/criterions.py"
    click tr9 "https://github.com/yyuezhi/genvdm/blob/main/instant-nsr-pl/systems/neus_ortho.py"
    click tr10 "https://github.com/yyuezhi/genvdm/blob/main/zero123plus/pipeline.py"
    click tr11 "https://github.com/yyuezhi/genvdm/blob/main/zero123plus/model.py"
    click inf1 "https://github.com/yyuezhi/genvdm/blob/main/inference.py"
    click inf2 "https://github.com/yyuezhi/genvdm/blob/main/generate.sh"
    click inf3 "https://github.com/yyuezhi/genvdm/blob/main/src/utils/infer_util.py"
    click inf4 "https://github.com/yyuezhi/genvdm/blob/main/config/example_run.yaml"
    click vis1 "https://github.com/yyuezhi/genvdm/blob/main/demo.blend"
    click vis2 "https://github.com/yyuezhi/genvdm/blob/main/extract_VDM.py"

    %% Styles
    classDef data fill:#D0E8FF,stroke:#0366d6,color:#0366d6;
    classDef compute fill:#E6FFED,stroke:#22863A,color:#22863A;
    classDef config fill:#F6F8FA,stroke:#6A737D,color:#6A737D;
    classDef external fill:#FFF5B1,stroke:#D4A72C,color:#8E6C08;
```
