# yukun-huang/dreamcube architecture

> Architecture diagram of the GitHub repository yukun-huang/dreamcube, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yukun-huang/dreamcube
- Repository: https://github.com/yukun-huang/dreamcube

## Overview

##### Project Type and Purpose
- DreamCube is a research-oriented, diffusion-based 3D panorama generation tool.
- It provides both a Gradio UI and command-line interface (CLI) for inference:
- Takes a single-view input image, loads a pretrained Stable Diffusion pipeline, applies “Multi-plane Synchronization,” and outputs RGB-D cubemaps, equirectangular panoramas, and 3D scenes (mesh & 3dgs).
- Core research contribution: adapting 2D diffusion models to multi-plane panoramic representations (cubemaps) via custom attention, convolution, and normalization processors.

##### High-Level File/Module Structure
- app.py
- Entry point – parses CLI/Gradio flags, orchestrates the inference flow.
- external/
- equilib & py360convert: geometry utilities for cubemap↔equirectangular↔perspective conversions and grid sampling.
- models/
- dreamcube.py: wraps the diffusion pipeline and synchronizes across planes.
- multiplane_sync.ipynb: demo notebook for custom processors.
- utils/
- Pre-/post-processing: camera intrinsics, 3D mesh routines, depth handling, panorama transformations, custom PyTorch/C++ ops.
- requirements.txt
- Auto-generated dependency list; installation instructions in README.

##### Core Architectural Components
a. Interfaces
- CLI & Gradio UI (app.py)
b. Orchestration Layer
- InferenceController: parses inputs, loads weights, invokes model and utils.
c. Diffusion Pipeline & Synchronization
- Hugging Face StableDiffusionPipeline (UNet & VAE)
- Multi-plane Synchronization module (custom processors applied to attention, conv, GN)
d. Geometry & Conversion Utilities
- external/equilib: cubemap↔equirectangular↔perspective transforms + grid samplers
- external/py360convert: spherical & planar conversions
e. Pre/Post-Processing (utils/)
- Camera & projection transforms (utils/camera.py, utils/pano_to_3d.py)
- Depth & mesh generation (utils/depth.py, utils/mesh.py, utils/video.py)
- Custom ops for performance (utils/ops/*.cpp, .cu, .py)
f. Outputs
- Saved files under ./outputs: RGB-D cubemap images, panorama images, 3D mesh & 3dgs files
g. Infrastructure
- GPU Compute (NVIDIA L40S, CUDA)
- External Model Store (HuggingFace) for downloading pretrained weights

##### Relationships & Data Flow
- User → (CLI / Gradio UI) → InferenceController
- InferenceController → load StableDiffusionPipeline & weights
- InferenceController → apply Multi-plane Sync Processors → Diffusion UNet & VAE
- InferenceController → for each cubemap face:
- Pre-process with geometry utils → feed to diffusion → post-process outputs
- Outputs → depth maps & 3D scene assembly via utils → write to disk / display in UI
- All heavy computations (diffusion, grid sampling) are executed on GPU

##### Architectural Patterns & Principles
- Modular Separation:
- Interface vs. orchestration vs. computation vs. utilities
- Extensibility:
- external/ modules can be swapped or extended for new geometry backends (NumPy, PyTorch, C++)
- Data Flow Pipeline:
- Clear, sequential processing steps (pre-process → inference → post-process)
- Resource Abstraction:
- GPU vs. CPU ops isolated (utils/ops)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top Layer: User Interfaces
    subgraph "User Interfaces"
        CLI["CLI Interface"]:::interface
        Gradio["Gradio UI"]:::interface
    end

    %% Orchestration Layer
    subgraph "Orchestration & Entry" 
        InferenceController["InferenceController\n(app.py)"]:::orchestration
    end

    %% Core ML Components
    subgraph "Core ML Pipeline"
        DreamCubeModel["DreamCubeModel\n(models/dreamcube.py)"]:::ml
        MultiPlaneSync["Multi-Plane Sync\n(models/multiplane_sync.py)"]:::ml
        DreamCubeModel -->|uses| MultiPlaneSync
    end

    %% Geometry Conversion Utilities
    subgraph "Geometry & Conversions" 
        subgraph "external/equilib"
            E_C2E["cube2equi/"]:::utils
            E_E2C["equi2cube/"]:::utils
            E_E2E["equi2equi/"]:::utils
            E_E2P["equi2pers/"]:::utils
            E_GS["grid_sample/"]:::utils
            E_NP["numpy_utils/"]:::utils
            E_TU["torch_utils/"]:::utils
        end
        subgraph "external/py360convert"
            P_C2E["c2e.py"]:::utils
            P_E2C["e2c.py"]:::utils
            P_E2P["e2p.py"]:::utils
            P_UTIL["utils.py"]:::utils
        end
    end

    %% Pre/Post-Processing Utilities
    subgraph "Pre/Post-Processing & Ops"
        Cam["camera.py"]:::utils
        Pano3D["pano_to_3d.py"]:::utils
        Depth["depth.py"]:::utils
        Mesh["mesh.py"]:::utils
        Video["video.py"]:::utils
        subgraph "utils/ops"
            OpsCustom["custom_ops.py"]:::utils
            OpsGrad["grid_sample_gradfix.py"]:::utils
            OpsMisc["misc.py"]:::utils
            OpsResample["resample_padding.cpp/.cu/.h/.py"]:::utils
        end
        subgraph "Supporting Projections"
            U_CUBE["cube.py"]:::utils
            U_EQUI["equi.py"]:::utils
            U_PERS["pers.py"]:::utils
        end
    end

    %% Infrastructure & External Services
    subgraph "Infrastructure & Outputs"
        GPU["GPU Compute\n(NVIDIA L40S, CUDA)"]:::infra
        ModelStore(("HuggingFace Model Store")):::infra
        Outputs(("Outputs Folder\n(cubemap, panorama, mesh)")):::infra
    end

    %% Data Flow Connections
    CLI -->|input image| InferenceController
    Gradio -->|input image| InferenceController
    InferenceController -->|load weights| ModelStore
    InferenceController -->|init pipeline| DreamCubeModel
    DreamCubeModel -->|apply sync| MultiPlaneSync
    InferenceController -->|pre-process| E_C2E & E_E2C & E_E2E & E_E2P & E_GS & E_NP & E_TU & P_C2E & P_E2C & P_E2P & P_UTIL & Cam & Pano3D
    MultiPlaneSync -->|infer| GPU
    MultiPlaneSync -->|calls| OpsCustom & OpsGrad & OpsMisc & OpsResample
    MultiPlaneSync -->|support proj| U_CUBE & U_EQUI & U_PERS
    MultiPlaneSync -->|post-process| Depth & Mesh & Video
    Depth & Mesh & Video -->|write| Outputs
    Outputs -->|available to| CLI
    Outputs -->|available to| Gradio

    %% Click Events
    click CLI "https://github.com/yukun-huang/dreamcube/blob/main/app.py"
    click Gradio "https://github.com/yukun-huang/dreamcube/blob/main/app.py"
    click InferenceController "https://github.com/yukun-huang/dreamcube/blob/main/app.py"
    click DreamCubeModel "https://github.com/yukun-huang/dreamcube/blob/main/models/dreamcube.py"
    click MultiPlaneSync "https://github.com/yukun-huang/dreamcube/blob/main/models/multiplane_sync.py"
    click E_C2E "https://github.com/yukun-huang/dreamcube/tree/main/external/equilib/cube2equi/"
    click E_E2C "https://github.com/yukun-huang/dreamcube/tree/main/external/equilib/equi2cube/"
    click E_E2E "https://github.com/yukun-huang/dreamcube/tree/main/external/equilib/equi2equi/"
    click E_E2P "https://github.com/yukun-huang/dreamcube/tree/main/external/equilib/equi2pers/"
    click E_GS "https://github.com/yukun-huang/dreamcube/tree/main/external/equilib/grid_sample/"
    click E_NP "https://github.com/yukun-huang/dreamcube/tree/main/external/equilib/numpy_utils/"
    click E_TU "https://github.com/yukun-huang/dreamcube/tree/main/external/equilib/torch_utils/"
    click P_C2E "https://github.com/yukun-huang/dreamcube/blob/main/external/py360convert/c2e.py"
    click P_E2C "https://github.com/yukun-huang/dreamcube/blob/main/external/py360convert/e2c.py"
    click P_E2P "https://github.com/yukun-huang/dreamcube/blob/main/external/py360convert/e2p.py"
    click P_UTIL "https://github.com/yukun-huang/dreamcube/blob/main/external/py360convert/utils.py"
    click Cam "https://github.com/yukun-huang/dreamcube/blob/main/utils/camera.py"
    click Pano3D "https://github.com/yukun-huang/dreamcube/blob/main/utils/pano_to_3d.py"
    click Depth "https://github.com/yukun-huang/dreamcube/blob/main/utils/depth.py"
    click Mesh "https://github.com/yukun-huang/dreamcube/blob/main/utils/mesh.py"
    click Video "https://github.com/yukun-huang/dreamcube/blob/main/utils/video.py"
    click OpsCustom "https://github.com/yukun-huang/dreamcube/blob/main/utils/ops/custom_ops.py"
    click OpsGrad "https://github.com/yukun-huang/dreamcube/blob/main/utils/ops/grid_sample_gradfix.py"
    click OpsMisc "https://github.com/yukun-huang/dreamcube/blob/main/utils/ops/misc.py"
    click OpsResample "https://github.com/yukun-huang/dreamcube/blob/main/utils/ops/resample_padding.cpp"
    click U_CUBE "https://github.com/yukun-huang/dreamcube/blob/main/utils/cube.py"
    click U_EQUI "https://github.com/yukun-huang/dreamcube/blob/main/utils/equi.py"
    click U_PERS "https://github.com/yukun-huang/dreamcube/blob/main/utils/pers.py"

    %% Styles
    classDef interface fill:#D0E8FF,stroke:#4A90E2
    classDef orchestration fill:#D0E8FF,stroke:#4A90E2
    classDef ml fill:#C6ECC6,stroke:#2E7D32
    classDef utils fill:#FFE5B4,stroke:#E07A00
    classDef infra fill:#E0E0E0,stroke:#777777
```
