How yyuezhi/genvdm works
Project Type and Purpose
• GenVDM is a research‐oriented, full-stack machine-learning pipeline written in Python (PyTorch).
• It supports:
• Data preprocessing and dataset building (voxelization, rendering)
• Model training (using multi-GPU PyTorch)
• Model inference (generating Vector Displacement Maps from single images)
• Interactive visualization in Blender
File Structure
Top-level components:
• asset/ (images for docs)
• config/ (global YAML settings)
• dataset/ (custom C/Cython tools, data preprocessing, rendering scripts)
• instant-nsr-pl/ (third-party repo integrated as a submodule for neural surface reconstruction)
• src/ (core VDM code: data loaders, utilities)
• zero123plus/ (additional model/pipeline code)
• Scripts: train.py, inference.py, extract_VDM.py, generate.sh, launch.py
• Environment setup: install_env.sh, requirements.txt
Extract Key Insights from README
• Training: python train.py --base config/example_run.yaml ...
• Inference: bash generate.sh <image> <exp> <checkpoint>
• Interactive Modeling: demo.blend + output .exr in Blender
• External dependencies: Hugging Face (for checkpoints), Blender
Main Components & Their Relationships
A. Data Ingestion & Preprocessing
• Input: Raw 2D RGBA images (./input) and rendered datasets (./dataset/rendering_result)
• Tools:
– dataset/voxelize.py, patch_gen.py, multi_rendering.py
– C/Cython modules (dataset/cutils.c, flood_fill.c, .pyx)
– Scripts: dataset/setup.py, env_set.sh
• Output: Preprocessed data on disk (voxel grids, rendered views)
B. Configuration Management
• config/VDM_setting.yaml, example_run.yaml (global + dataset/training/inference settings)
• instant-nsr-pl/configs/example_run.yaml
• Used by train.py, inference.py, launch.py
C. Model Training
• Entry point: train.py (PyTorch Lightning or custom training loop)
• Core code:
– src/data/VDM.py (datasets, data loaders)
– src/utils/train_util.py (training helpers)
– instant-nsr-pl/models/… (geometry, NeRF, NeuS, texture modules)
– zero123plus/pipeline.py, model.py (additional pipelines or backbones)
– instant-nsr-pl/systems/criterions.py, neus_ortho.py (losses & training systems)
• Compute: multi-GPU, distributed training
• Output: Model checkpoints (./checkpoints/<exp_name>/)
D. Model Inference
• Entry point: inference.py, generate.sh
• Uses: Pretrained checkpoints + config/example_run.yaml
• Core utilities: src/utils/infer_util.py
• Output: Vector Displacement Maps (.exr, .png) in an output directory
E. Visualization & Interactive Modeling
• demo.blend (Blender 3.6 file configured for VDM)
• Extract script: extract_VDM.py (maybe converts raw output into Blender‐readable format)
• User opens .blend, loads .exr, sculpts/displaces in Blender UI
F. External Services / Storage
• Hugging Face dataset URLs (for pre-trained weights & raw data)
• GitHub (code repository)
• Blender (local desktop application)
Architectural Patterns & Design Principles
• Modular Python packages (src/, zero123plus/, instant-nsr-pl/)
• Separation of concerns: preprocessing vs. training vs. inference vs. visualization
• YAML-driven configuration for reproducibility
• C/Cython extensions for performance-critical steps (voxelization, flood fill)
• Script-based orchestration (bash, Python entry scripts)
Essential Elements
• Clear component labels (script names, directories, modules)