How lzzcd001/meshdiffusion works
Project Type and Purpose
• MeshDiffusion is a research-oriented full-stack ML pipeline for 3D mesh generation.
• It implements a score-based diffusion model that directly parameterizes meshes via Deep Marching Tetrahedra (DMTet).
• The codebase splits into:
• Data preparation (raw mesh → tetrahedral grid → cubic volumes)
• Diffusion model (training, sampling, evaluation)
• DMTet fitting & rendering (via the nvdiffrec submodule)
• Visualization (Blender scripts)
File Structure & Architectural Patterns
• Top-level modules:
• configs/ — model and diffusion hyperparameters
• data/ — scripts to generate tet masks & convert to 3D grids
• lib/diffusion/ — core diffusion code: models, trainer, sampler, losses
• main_diffusion.py — orchestrates train, uncond_gen, cond_gen modes
• metadata/ — builds JSON meta lists of dataset samples
• nvdiffrec/ — DMTet fitting, rendering, evaluation pipeline
• blender_viz/ — Blender Python scripts for final mesh visualization
• Architectural pattern: a modular pipeline with clear separation of concerns—data → model → reconstruction → visualization.
Key Components
a. Data Preparation
• Raw ShapeNet meshes → nvdiffrec/fit_dmtets.py → DMTet dictionaries → data/tets_to_3dgrid.py → 3D grid volumes.
• Metadata builder (metadata/save_meta.py) → JSON index files.
b. Diffusion Model
• Configuration loader (configs/*.py) → injects hyperparameters via ml_collections.
• Trainer (lib/diffusion/trainer.py) → iterates over grid dataset, computes losses (lib/diffusion/losses.py).
• Sampler/Inference (lib/diffusion/sampling.py, lib/diffusion/evaler.py) → unconditional (--mode=uncond_gen) or conditional (--mode=cond_gen).
• Entry point: main_diffusion.py.
c. DMTet Reconstruction & Rendering
• Conditional workflow: use partial DMTet from fitting to feed diffusion.
• After sampling, back to nvdiffrec/eval.py → convert grids to DMTet mesh → .obj files.
• Rendering pipeline in nvdiffrec/lib/render/*, including CUDA-accelerated BSDFs.
d. Visualization
• Blender script (blender_viz/blender_script.py) loads .obj and produces scene renders.
Architectural Patterns & Principles
• Modular pipeline: each stage is a stand-alone script or Python module.
• Clear separation of configuration (configs/) from code.
• Reuse of nvdiffrec library for geometry and rendering.
• Use of metadata JSON to decouple data indexing from actual I/O.
Technologies & Frameworks
• Python 3.8+
• PyTorch (model & training) + CUDA 11.6
• PyTorch3D (3D ops)
• ml_collections (config management)
• Blender Python API (visualization)
• NVlabs nvdiffrec (DMTet fitting & CUDA rendering kernels)
Essential Elements to Capture
• All major Python modules (configs/, data/, lib/diffusion/, nvdiffrec/)
• Data stores (grid volumes, metadata JSON, checkpoints, .obj meshes)
• Execution scripts (main_diffusion.py, fit_dmtets.py, eval.py, blender_script.py)
• External tool dependencies (PyTorch, Blender, CUDA)