# nv-tlabs/scube architecture

> Architecture diagram of the GitHub repository nv-tlabs/scube, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nv-tlabs/scube
- Repository: https://github.com/nv-tlabs/scube

## Overview

##### Main Components Identification

a) VAE (Variational Autoencoder) Stage
b) Geometry Reconstruction Stage (Latent Voxel Diffusion Model)
c) Appearance Reconstruction Stage (Gaussian Splatting Model - GSM)

##### Data Flow Architecture

**Input Layer:**
- Waymo Dataset
- Image Data
- LiDAR Data
- Pose Information
- Camera Intrinsics

**Processing Pipeline:**
**Data Preparation → VAE → Diffusion Model → GSM → Output Visualization**

##### Key Components

a) Data Processing Components:
- WebDataset Converter
- SegFormer (Sky Mask Generation)
- Metric3Dv2 (GT Depth Processing)

b) Model Components:
- Coarse Stage VAE (64x64x64)
- Fine Stage VAE (256x256x128)
- Image-conditioned Voxel Diffusion
- Semantic-conditioned Upsampling
- GSM Renderer

c) Output Components:
- Grid Visualization
- 3D Gaussian Splatting
- Interactive Visualizer

##### Architectural Patterns to Highlight

- Multi-stage Pipeline Architecture
- Hierarchical Processing (Coarse to Fine)
- Parallel Processing Capabilities
- CUDA-accelerated Components

##### Implementation Details

**Extensions:**
- NKSR-CUDA Implementation
- SDF Generation
- Common CUDA Utilities

**Core Modules:**
- Autoencoding
- Diffusion Modules
- GSM Modules
- Rendering Systems

##### Additional Elements

- Configuration Management
- Training Pipeline
- Inference Pipeline
- Visualization Tools

##### Technical Details to Note

Specific technical parameters:
- Resolution scales (64³, 256x256x128)
- Model architectures (UNet, VAE)
- CUDA acceleration points
- Memory management systems

This system design should effectively communicate both the high-level architecture and the detailed technical implementation of the SCube system while maintaining clarity and readability.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Input["Input Layer"]
        WD[("Waymo Dataset")]
        ID[("Image Data")]
        LD[("LiDAR Data")]
        PI[("Pose Info")]
        CI[("Camera Intrinsics")]
    end

    subgraph DataProc["Data Processing"]
        WDC["WebDataset Converter"]:::data
        SFM["SegFormer"]:::data
        M3D["Metric3Dv2"]:::data
    end
    click DataProc "https://github.com/nv-tlabs/SCube/tree/main/scube/data/"
    click WDC "https://github.com/nv-tlabs/SCube/blob/main/datagen/waymo2nerfstudio.py"

    subgraph VAE["VAE Stage"]
        CV["Coarse VAE (64³)"]:::model
        FV["Fine VAE (256x256x128)"]:::model
    end
    click VAE "https://github.com/nv-tlabs/SCube/tree/main/scube/modules/autoencoding/"
    click CV "https://github.com/nv-tlabs/SCube/blob/main/configs/train/vae/vae_16x16x16_dense.yaml"
    click FV "https://github.com/nv-tlabs/SCube/blob/main/configs/train/vae/vae_128x128x128_sparse.yaml"

    subgraph Diffusion["Geometry Reconstruction"]
        IVD["Image-conditioned Voxel Diffusion"]:::model
        SU["Semantic Upsampling"]:::model
    end
    click Diffusion "https://github.com/nv-tlabs/SCube/tree/main/scube/modules/diffusionmodules/"
    click IVD "https://github.com/nv-tlabs/SCube/blob/main/scube/models/diffusion.py"

    subgraph GSM["Appearance Reconstruction"]
        GR["GSM Renderer"]:::render
        GS["3D Gaussian Splatting"]:::render
    end
    click GSM "https://github.com/nv-tlabs/SCube/tree/main/scube/modules/gsm_modules/"
    click GR "https://github.com/nv-tlabs/SCube/blob/main/scube/modules/render/gsplat_renderer.py"

    subgraph Vis["Visualization"]
        GV["Grid Visualization"]:::vis
        IV["Interactive Visualizer"]:::vis
    end
    click Vis "https://github.com/nv-tlabs/SCube/blob/main/scube/utils/vis_util.py"
    click GV "https://github.com/nv-tlabs/SCube/blob/main/inference/visualize_grid_compare.py"
    click IV "https://github.com/nv-tlabs/SCube/blob/main/inference/visualize_scene.py"

    subgraph CUDA["CUDA Extensions"]
        NK["NKSR-CUDA"]:::cuda
        SG["SDF Generation"]:::cuda
        CU["Common Utilities"]:::cuda
    end
    click CUDA "https://github.com/nv-tlabs/SCube/tree/main/ext/common/"
    click NK "https://github.com/nv-tlabs/SCube/tree/main/ext/nksr-cuda/"
    click SG "https://github.com/nv-tlabs/SCube/tree/main/ext/sdfgen/"

    subgraph Config["Configuration"]
        TC["Training Config"]:::config
        MC["Model Config"]:::config
    end
    click Config "https://github.com/nv-tlabs/SCube/tree/main/configs/waymo_scube/"
    click TC "https://github.com/nv-tlabs/SCube/tree/main/configs/train/"

    %% Data Flow Connections
    WD --> WDC
    ID --> WDC
    LD --> WDC
    PI --> WDC
    CI --> WDC
    
    WDC --> SFM
    WDC --> M3D
    
    SFM --> CV
    M3D --> CV
    CV --> FV
    
    FV --> IVD
    IVD --> SU
    
    SU --> GR
    GR --> GS
    
    GS --> GV
    GS --> IV

    %% Support System Connections
    CUDA -.-> VAE
    CUDA -.-> Diffusion
    CUDA -.-> GSM
    Config -.-> VAE
    Config -.-> Diffusion
    Config -.-> GSM

    %% Styling
    classDef data fill:#3498db,stroke:#2980b9,color:white
    classDef model fill:#e67e22,stroke:#d35400,color:white
    classDef render fill:#9b59b6,stroke:#8e44ad,color:white
    classDef vis fill:#2ecc71,stroke:#27ae60,color:white
    classDef cuda fill:#95a5a6,stroke:#7f8c8d,color:white
    classDef config fill:#f1c40f,stroke:#f39c12,color:white

    %% Legend
    subgraph Legend
        L1["Data Processing"]:::data
        L2["Model Components"]:::model
        L3["Rendering"]:::render
        L4["Visualization"]:::vis
        L5["CUDA Extensions"]:::cuda
        L6["Configuration"]:::config
    end
```
