# kylesargent/zeronvs architecture

> Architecture diagram of the GitHub repository kylesargent/zeronvs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kylesargent/zeronvs
- Repository: https://github.com/kylesargent/zeronvs

## Overview

##### Project Type and Purpose
- Type: Research-focused ML codebase for “Zero-Shot 360-Degree View Synthesis from a Single Real Image.”
- Purpose: Given one input image, distill a Neural Radiance Field (NeRF) via Score Distillation Sampling (SDS) from pre-trained diffusion models, then render novel views and evaluate quality on standard datasets.

##### High-Level Architecture Overview
- Configuration Layer
– YAML files under configs/ drive every run.
– resources.py encapsulates dataset/model paths.
- Orchestration Layer
– launch.py: main entry point for training/inference/eval.
– Bash scripts (launch_inference.sh, launch_eval_*.sh) wrap launch.py with dataset-specific arguments.
- Core Pipeline Components (in threestudio/)
– Data Loaders (threestudio/data): DTU, MipNeRF360, single-image loader.
– Model Factories (threestudio/models):
· Guidance modules (stable diffusion, zero123, ControlNet, InstructPix2Pix…)
· Geometry modules (SDF, implicit volumes, mesh representations)
· Material modules (neural radiance, PBR, latent adapters)
· Renderers (nerf/volume/gan/rasterizer/patch) produce images or gradients.
– Prompt Processors (threestudio/models/prompt_processors) prepare text or image prompts.
– System Definitions (threestudio/systems): pipeline recipes (dreamfusion, latentnerf, magic3d, zero123, etc.), each wiring together guidance + geometry + renderer + optimizer.
- Utilities (threestudio/utils): config loader, callbacks, saving, GPU-accelerated ops (tiny-cuda-nn, nerfacc), GAN utilities.
- External Submodule: zeronvs_diffusion (Zero123-style diffusion training) integrated via git submodule.

##### Main Components
1. Configuration Manager
– Reads YAML, initializes resources (paths, hyperparameters).
2. Data & Resource Loader
– Inputs: image(s) or datasets, camera parameters.
– Outputs: rays, images, poses.
3. Prompt & Guidance Module
– Interfaces to pre-trained diffusion: zero123, stable diffusion variants, controlnet.
– Provides SDS gradients.
4. NeRF Model Components
– Geometry network (implicit SDF/volume).
– Material network.
– Background network.
5. Renderer
– Takes model outputs + camera rays → synthetic views.
– May use nerfacc or tiny-cuda-nn kernels.
6. Optimizer & Training Loop
– Iteratively applies SDS gradients to model parameters.
– Logs losses, checkpoints.
7. Exporter & Evaluator
– Mesh exporter (mesh_exporter.py), metrics calculator, metrics.json.
8. Scripts / Orchestration
– Shell scripts for dataset-specific evaluation, single-image inference.

##### Relationships & Data Flow
- Start: Config Manager loads a selected YAML.
- Config → Resource Loader yields data batches.
- For Each Batch:
– Renderer generates images from current NeRF model.
– Prompt Processor creates conditioning.
– Guidance Module computes guidance gradients w.r.t. SDS loss.
– Optimizer applies gradients to geometry/material/background networks.
- Periodically: Exporter dumps mesh, saves checkpoint.
- Post-training: Evaluator script aggregates metrics.json per scene.

##### Architectural Patterns & Design Principles
- Modular “plugin” architecture: you swap in new guidance, renderer, or geometry by adding a module under threestudio/models or threestudio/systems.
- Configuration-driven: no hard-coded parameters; uses plain YAML.
- Separation of Concerns: data handling, model definition, training logic, and evaluation are distinct layers.
- GPU-accelerated custom kernels: tiny-cuda-nn for NeRF MLPs, nerfacc for fast ray marching.
- Reuse of existing open-source bases: zero123 and threestudio core.

##### Relevant Technologies & Frameworks
- Python 3.8, PyTorch 2.0.1 + CUDA 11.8
- tiny-cuda-nn (custom CUDA bindings)
- nerfacc for ray-marching acceleration
- Bash scripts for orchestration
- gdown for data download
- YAML for configuration

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Layer
    subgraph "Configuration Layer"
        Config["Configuration Manager"]:::config
    end

    %% Data & Resource Loader
    subgraph "Data & Resource Loader"
        Data["Data Loader\n(images, poses)"]:::data
    end

    %% Model Components
    subgraph "Model Components"
        subgraph "NeRF Components"
            Geom["Geometry Network"]:::compute
            Mat["Material Network"]:::compute
            Back["Background Network"]:::compute
        end
        subgraph "Renderer"
            Rend["Renderer"]:::compute
        end
    end

    %% Prompt & Guidance
    subgraph "Prompt & Guidance Module"
        Prompt["Prompt Processor"]:::compute
        External["Pre-trained Diffusion"]:::external
        Guidance["Guidance Module"]:::compute
    end

    %% Optimizer & Training Loop
    Optim["Optimizer & Training Loop"]:::compute

    %% Exporter & Evaluator
    subgraph "Exporter & Evaluator"
        Export["Mesh Exporter & Saving"]:::compute
        Eval["Evaluator & Metrics"]:::compute
    end

    %% Scripts / Orchestration
    subgraph "Scripts / Orchestration"
        Scripts1["launch.py"]:::scripts
        Scripts2["Shell Scripts"]:::scripts
    end

    %% Data Flow
    Config -->|"loads configs"| Data
    Data -->|"provides rays/images"| Rend
    Rend -->|"renders views"| Guidance
    Prompt -->|"processes prompts"| Guidance
    External -->|"pre-trained models"| Guidance
    Guidance -->|"SDS gradients"| Optim
    Optim -->|"updates params"| Geom
    Optim -->|"updates params"| Mat
    Optim -->|"updates params"| Back
    Geom -->|"geometry output"| Rend
    Mat -->|"material output"| Rend
    Back -->|"background output"| Rend
    Optim -->|"periodic export"| Export
    Export -->|"metrics.json"| Eval

    %% Orchestration
    Scripts1 -->|"invokes"| Config
    Scripts1 -->|"invokes"| Optim
    Scripts2 -->|"wrappers"| Scripts1

    %% Click Events
    click Config "https://github.com/kylesargent/zeronvs/tree/main/configs/"
    click Config "https://github.com/kylesargent/zeronvs/blob/main/resources.py"
    click Config "https://github.com/kylesargent/zeronvs/blob/main/threestudio/utils/config.py"
    click Data "https://github.com/kylesargent/zeronvs/tree/main/threestudio/data/"
    click Prompt "https://github.com/kylesargent/zeronvs/tree/main/threestudio/models/prompt_processors/"
    click Guidance "https://github.com/kylesargent/zeronvs/tree/main/threestudio/models/guidance/"
    click Geom "https://github.com/kylesargent/zeronvs/tree/main/threestudio/models/geometry/"
    click Mat "https://github.com/kylesargent/zeronvs/tree/main/threestudio/models/materials/"
    click Back "https://github.com/kylesargent/zeronvs/tree/main/threestudio/models/background/"
    click Rend "https://github.com/kylesargent/zeronvs/tree/main/threestudio/models/renderers/"
    click Optim "https://github.com/kylesargent/zeronvs/blob/main/launch.py"
    click Optim "https://github.com/kylesargent/zeronvs/blob/main/threestudio/systems/optimizers.py"
    click Export "https://github.com/kylesargent/zeronvs/blob/main/threestudio/models/exporters/mesh_exporter.py"
    click Export "https://github.com/kylesargent/zeronvs/blob/main/threestudio/utils/saving.py"
    click Scripts1 "https://github.com/kylesargent/zeronvs/blob/main/launch.py"
    click Scripts2 "https://github.com/kylesargent/zeronvs/blob/main/launch_inference.sh"

    %% Styles
    classDef config fill:#D6E9FF,stroke:#000,stroke-width:1px;
    classDef data fill:#DFFFE0,stroke:#000,stroke-width:1px;
    classDef compute fill:#FFD8A8,stroke:#000,stroke-width:1px;
    classDef external fill:#E0E0E0,stroke:#000,stroke-width:1px;
    classDef scripts fill:#F0D0FF,stroke:#000,stroke-width:1px;
```
