How trickygo/pano2room works
Project Type & Purpose
• The project is an advanced computer vision and graphics system for novel view synthesis from a single indoor panorama. Its main function is to convert an input panorama into a 3D geometric structure (3DGS) and render novel views along with depth maps.
• It involves several techniques such as image inpainting (with both SDFT and LAMA methods), depth estimation, and rendering using differentiable gaussian rasterization. The use of libraries like Pytorch3D further indicates high-level 3D rendering and model manipulation.
High-Level Architectural Components
• Input Module:
• Handles the panorama images and the camera trajectory (files located in the input folder and input/Camera_Trajectory).
• Prepares the input data (intrinsics, extrinsics) for the pipeline.
• Preprocessing & Data Conversion:
• Contains modules for converting between different image representations (cube to equirectangular, equirectangular to perspective), found under modules/equilib (with submodules: cube2equi, equi2cube, equi2equi, equi2pers).
• These modules take in the raw panoramic data and transform it into intermediate representations for further processing.
• Inpainting & Image Refinement:
• The system offers inpainting functions using SDFT (see create_SDFT_pairs.py, train_SDFT.py, and modules/inpainters/SDFT_inpainter.py).
• Additionally, there is a LAMA-based inpainter module (modules/inpainters/lama) which likely assists in restoring missing regions in the panorama.
• Training scripts and shell files under scripts/ (create_SDFT_pairs.sh, train_SDFT.sh) support fine-tuning these models.
• Depth & Geometry Prediction:
• Modules in modules/geo_predictors provide predictors for depth and other geometric properties (e.g., PanoFusionDistancePredictor, pano_geo_refiner, and omni-data based predictors).
• These modules extract or refine depth information essential for 3D rendering.
• 3D Rendering Pipeline:
• The rendering components use gaussian_renderer (with its network_gui) to possibly provide an interactive or demo visualization.
• The system also relies on external libraries such as Pytorch3D and diff-gaussian-rasterization-w-depth for mesh or 3DGS rendering with depth.
• Scene & Utility Management:
• The scene folder includes modules for handling camera parameters, dataset reading, and the core 3D model (scene/gaussian_model.py). This ties together the processing chain from input up to rendering.
• The utils folder groups common utility functions (camera_utils, geo_utils, warp_utils, etc.) that support various stages (data ingestion, transformation, and loss functions for training modules).
• Training & Configuration:
• The checkpoints directory contains pretrained weights and configuration files (big-lama-config.yaml) hinting at a modular training/inference design.
• Shell scripts in the scripts folder orchestrate the overall training and demo runs (run_Pano2Room.sh).
Relationships & Data Flow
• Data Flow: The architecture begins with an input (panorama + camera trajectory) which is preprocessed into intermediate representations.
• The processed data flows into two parallel (or sequentially coupled) branches:
• Inpainting and refinement (SDFT/LAMA models) that improve the inputs.
• Depth and geometry prediction mixing guidance from geo_predictors.
• The outputs from these branches feed into the 3D rendering pipeline that constructs the 3D geometry scene (3DGS) and produces the final novel views and depth maps.
• Training pipelines exist to adjust the inpainting modules; these pipelines additionally feed back into the inference pipeline when fine-tuning is required.
Architectural Patterns & Key Technologies
• Modularity: Each component—conversion, inpainting, depth prediction, and rendering—is encapsulated in its own module/subdirectory. This modular design promotes reusability and easier debugging.
• Pipeline/Coupling: Data processing pipelines are clearly distinguished between preprocessing, model inference, and rendering. Also, training scripts (both for overall demo and for fine-tuning SDFT) are clearly separated.
• External Dependencies: Integration of Pytorch3D, diff-gaussian-rasterization-w-depth, and possibly PyTorch (since many modules have torch and numpy variants) is central to the system’s operation.