How kylesargent/zeronvs works
Project Type and Purpose
• Type: Research-focused ML codebase for “Zero-Shot 360-Degree View Synthesis from a Single Real Image.”
• Purpose: Given one input image, distill a Neural Radiance Field (NeRF) via Score Distillation Sampling (SDS) from pre-trained diffusion models, then render novel views and evaluate quality on standard datasets.
High-Level Architecture Overview
• Configuration Layer
– YAML files under configs/ drive every run.
– resources.py encapsulates dataset/model paths.
• Orchestration Layer
– launch.py: main entry point for training/inference/eval.
– Bash scripts (launch_inference.sh, launch_eval_*.sh) wrap launch.py with dataset-specific arguments.
• Core Pipeline Components (in threestudio/)
– Data Loaders (threestudio/data): DTU, MipNeRF360, single-image loader.
– Model Factories (threestudio/models):
· Guidance modules (stable diffusion, zero123, ControlNet, InstructPix2Pix…)
· Geometry modules (SDF, implicit volumes, mesh representations)
· Material modules (neural radiance, PBR, latent adapters)
· Renderers (nerf/volume/gan/rasterizer/patch) produce images or gradients.
– Prompt Processors (threestudio/models/prompt_processors) prepare text or image prompts.
– System Definitions (threestudio/systems): pipeline recipes (dreamfusion, latentnerf, magic3d, zero123, etc.), each wiring together guidance + geometry + renderer + optimizer.
• Utilities (threestudio/utils): config loader, callbacks, saving, GPU-accelerated ops (tiny-cuda-nn, nerfacc), GAN utilities.
• External Submodule: zeronvs_diffusion (Zero123-style diffusion training) integrated via git submodule.
Main Components
1. Configuration Manager
– Reads YAML, initializes resources (paths, hyperparameters).
2. Data & Resource Loader
– Inputs: image(s) or datasets, camera parameters.
– Outputs: rays, images, poses.
3. Prompt & Guidance Module
– Interfaces to pre-trained diffusion: zero123, stable diffusion variants, controlnet.
– Provides SDS gradients.
4. NeRF Model Components
– Geometry network (implicit SDF/volume).
– Material network.
– Background network.
5. Renderer
– Takes model outputs + camera rays → synthetic views.
– May use nerfacc or tiny-cuda-nn kernels.
6. Optimizer & Training Loop
– Iteratively applies SDS gradients to model parameters.
– Logs losses, checkpoints.
7. Exporter & Evaluator
– Mesh exporter (mesh_exporter.py), metrics calculator, metrics.json.
8. Scripts / Orchestration
– Shell scripts for dataset-specific evaluation, single-image inference.
Relationships & Data Flow
• Start: Config Manager loads a selected YAML.
• Config → Resource Loader yields data batches.
• For Each Batch:
– Renderer generates images from current NeRF model.
– Prompt Processor creates conditioning.
– Guidance Module computes guidance gradients w.r.t. SDS loss.
– Optimizer applies gradients to geometry/material/background networks.
• Periodically: Exporter dumps mesh, saves checkpoint.
• Post-training: Evaluator script aggregates metrics.json per scene.
Architectural Patterns & Design Principles
• Modular “plugin” architecture: you swap in new guidance, renderer, or geometry by adding a module under threestudio/models or threestudio/systems.
• Configuration-driven: no hard-coded parameters; uses plain YAML.
• Separation of Concerns: data handling, model definition, training logic, and evaluation are distinct layers.
• GPU-accelerated custom kernels: tiny-cuda-nn for NeRF MLPs, nerfacc for fast ray marching.
• Reuse of existing open-source bases: zero123 and threestudio core.
Relevant Technologies & Frameworks
• Python 3.8, PyTorch 2.0.1 + CUDA 11.8
• tiny-cuda-nn (custom CUDA bindings)
• nerfacc for ray-marching acceleration
• Bash scripts for orchestration
• gdown for data download
• YAML for configuration