How yukun-huang/dreamcube works
Project Type and Purpose
• DreamCube is a research-oriented, diffusion-based 3D panorama generation tool.
• It provides both a Gradio UI and command-line interface (CLI) for inference:
• Takes a single-view input image, loads a pretrained Stable Diffusion pipeline, applies “Multi-plane Synchronization,” and outputs RGB-D cubemaps, equirectangular panoramas, and 3D scenes (mesh & 3dgs).
• Core research contribution: adapting 2D diffusion models to multi-plane panoramic representations (cubemaps) via custom attention, convolution, and normalization processors.
High-Level File/Module Structure
• app.py
• Entry point – parses CLI/Gradio flags, orchestrates the inference flow.
• external/
• equilib & py360convert: geometry utilities for cubemap↔equirectangular↔perspective conversions and grid sampling.
• models/
• dreamcube.py: wraps the diffusion pipeline and synchronizes across planes.
• multiplane_sync.ipynb: demo notebook for custom processors.
• utils/
• Pre-/post-processing: camera intrinsics, 3D mesh routines, depth handling, panorama transformations, custom PyTorch/C++ ops.
• requirements.txt
• Auto-generated dependency list; installation instructions in README.
Core Architectural Components
a. Interfaces
• CLI & Gradio UI (app.py)
b. Orchestration Layer
• InferenceController: parses inputs, loads weights, invokes model and utils.
c. Diffusion Pipeline & Synchronization
• Hugging Face StableDiffusionPipeline (UNet & VAE)
• Multi-plane Synchronization module (custom processors applied to attention, conv, GN)
d. Geometry & Conversion Utilities
• external/equilib: cubemap↔equirectangular↔perspective transforms + grid samplers
• external/py360convert: spherical & planar conversions
e. Pre/Post-Processing (utils/)
• Camera & projection transforms (utils/camera.py, utils/pano_to_3d.py)
• Depth & mesh generation (utils/depth.py, utils/mesh.py, utils/video.py)
• Custom ops for performance (utils/ops/*.cpp, .cu, .py)
f. Outputs
• Saved files under ./outputs: RGB-D cubemap images, panorama images, 3D mesh & 3dgs files
g. Infrastructure
• GPU Compute (NVIDIA L40S, CUDA)
• External Model Store (HuggingFace) for downloading pretrained weights
Relationships & Data Flow
• User → (CLI / Gradio UI) → InferenceController
• InferenceController → load StableDiffusionPipeline & weights
• InferenceController → apply Multi-plane Sync Processors → Diffusion UNet & VAE
• InferenceController → for each cubemap face:
• Pre-process with geometry utils → feed to diffusion → post-process outputs
• Outputs → depth maps & 3D scene assembly via utils → write to disk / display in UI
• All heavy computations (diffusion, grid sampling) are executed on GPU
Architectural Patterns & Principles
• Modular Separation:
• Interface vs. orchestration vs. computation vs. utilities
• Extensibility:
• external/ modules can be swapped or extended for new geometry backends (NumPy, PyTorch, C++)
• Data Flow Pipeline:
• Clear, sequential processing steps (pre-process → inference → post-process)
• Resource Abstraction:
• GPU vs. CPU ops isolated (utils/ops)