# trickygo/pano2room architecture

> Architecture diagram of the GitHub repository trickygo/pano2room, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/trickygo/pano2room
- Repository: https://github.com/trickygo/pano2room

## Overview

##### Project Type & Purpose
- The project is an advanced computer vision and graphics system for novel view synthesis from a single indoor panorama. Its main function is to convert an input panorama into a 3D geometric structure (3DGS) and render novel views along with depth maps.
- It involves several techniques such as image inpainting (with both SDFT and LAMA methods), depth estimation, and rendering using differentiable gaussian rasterization. The use of libraries like Pytorch3D further indicates high-level 3D rendering and model manipulation.

##### High-Level Architectural Components
- Input Module:
- Handles the panorama images and the camera trajectory (files located in the input folder and input/Camera_Trajectory).
- Prepares the input data (intrinsics, extrinsics) for the pipeline.

- Preprocessing & Data Conversion:
- Contains modules for converting between different image representations (cube to equirectangular, equirectangular to perspective), found under modules/equilib (with submodules: cube2equi, equi2cube, equi2equi, equi2pers).
- These modules take in the raw panoramic data and transform it into intermediate representations for further processing.

- Inpainting & Image Refinement:
- The system offers inpainting functions using SDFT (see create_SDFT_pairs.py, train_SDFT.py, and modules/inpainters/SDFT_inpainter.py).
- Additionally, there is a LAMA-based inpainter module (modules/inpainters/lama) which likely assists in restoring missing regions in the panorama.
- Training scripts and shell files under scripts/ (create_SDFT_pairs.sh, train_SDFT.sh) support fine-tuning these models.

- Depth & Geometry Prediction:
- Modules in modules/geo_predictors provide predictors for depth and other geometric properties (e.g., PanoFusionDistancePredictor, pano_geo_refiner, and omni-data based predictors).
- These modules extract or refine depth information essential for 3D rendering.

- 3D Rendering Pipeline:
- The rendering components use gaussian_renderer (with its network_gui) to possibly provide an interactive or demo visualization.
- The system also relies on external libraries such as Pytorch3D and diff-gaussian-rasterization-w-depth for mesh or 3DGS rendering with depth.

- Scene & Utility Management:
- The scene folder includes modules for handling camera parameters, dataset reading, and the core 3D model (scene/gaussian_model.py). This ties together the processing chain from input up to rendering.
- The utils folder groups common utility functions (camera_utils, geo_utils, warp_utils, etc.) that support various stages (data ingestion, transformation, and loss functions for training modules).

- Training & Configuration:
- The checkpoints directory contains pretrained weights and configuration files (big-lama-config.yaml) hinting at a modular training/inference design.
- Shell scripts in the scripts folder orchestrate the overall training and demo runs (run_Pano2Room.sh).

##### Relationships & Data Flow
- Data Flow: The architecture begins with an input (panorama + camera trajectory) which is preprocessed into intermediate representations.
- The processed data flows into two parallel (or sequentially coupled) branches:
- Inpainting and refinement (SDFT/LAMA models) that improve the inputs.
- Depth and geometry prediction mixing guidance from geo_predictors.
- The outputs from these branches feed into the 3D rendering pipeline that constructs the 3D geometry scene (3DGS) and produces the final novel views and depth maps.
- Training pipelines exist to adjust the inpainting modules; these pipelines additionally feed back into the inference pipeline when fine-tuning is required.

##### Architectural Patterns & Key Technologies
- Modularity: Each component—conversion, inpainting, depth prediction, and rendering—is encapsulated in its own module/subdirectory. This modular design promotes reusability and easier debugging.
- Pipeline/Coupling: Data processing pipelines are clearly distinguished between preprocessing, model inference, and rendering. Also, training scripts (both for overall demo and for fine-tuning SDFT) are clearly separated.
- External Dependencies: Integration of Pytorch3D, diff-gaussian-rasterization-w-depth, and possibly PyTorch (since many modules have torch and numpy variants) is central to the system’s operation.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Input Layer
    subgraph "Input Layer"
        IM["Input Module"]:::input
        CT["Camera Trajectory"]:::input
    end
    IM --> CT

    %% Data Preprocessing Layer
    subgraph "Data Preprocessing"
        PC["Preprocessing & Data Conversion"]:::preprocess
    end
    CT --> PC

    %% Processing Branches
    subgraph "Inpainting & Detail"
        %% SDFT Inpainting Subgraph
        subgraph "SDFT Inpainting"
            SDFT1["Pair Creation (SDFT)"]:::inpainting
            SDFT2["Training (SDFT)"]:::inpainting
            SDFT3["SDFT Inpainter"]:::inpainting
            SDFT1 --> SDFT2 --> SDFT3
        end
        %% LAMA Inpainting Subgraph
        subgraph "LAMA Inpainting"
            LAMA1["LAMA Inpainter Directory"]:::inpainting
            LAMA2["LAMA Inpainter File"]:::inpainting
            LAMA3["Pano Pers Fusion Inpainter"]:::inpainting
            LAMA1 --> LAMA2 --> LAMA3
        end
    end

    subgraph "Depth Estimation"
        DG["Depth & Geometry Prediction"]:::depth
    end

    %% Connections from Preprocessing to Branches
    PC --> SDFT1
    PC --> LAMA1
    PC --> DG

    %% 3D Rendering Layer
    subgraph "3D Rendering"
        GR["Gaussian Renderer"]:::rendering
        NG["Network GUI"]:::rendering
        GR --> NG
    end
    SDFT3 --> GR
    LAMA3 --> GR
    DG --> GR

    %% Scene Management Layer
    subgraph "Scene Management"
        SC["Scene"]:::scene
        UT["Utils"]:::scene
        SC --> UT
    end
    GR --> SC

    %% Training & Configuration Management Layer
    subgraph "Training & Config"
        ST["Scripts"]:::training
        CP["Checkpoints"]:::training
        ST --> CP
    end
    ST -.-> SDFT1
    ST -.-> DG
    CP -.-> GR

    %% Click Events
    click IM "https://github.com/trickygo/pano2room/tree/main/input"
    click CT "https://github.com/trickygo/pano2room/tree/main/input/Camera_Trajectory"
    click PC "https://github.com/trickygo/pano2room/tree/main/modules/equilib"
    click SDFT1 "https://github.com/trickygo/pano2room/blob/main/create_SDFT_pairs.py"
    click SDFT2 "https://github.com/trickygo/pano2room/blob/main/train_SDFT.py"
    click SDFT3 "https://github.com/trickygo/pano2room/blob/main/modules/inpainters/SDFT_inpainter.py"
    click LAMA1 "https://github.com/trickygo/pano2room/tree/main/modules/inpainters/lama"
    click LAMA2 "https://github.com/trickygo/pano2room/blob/main/modules/inpainters/lama_inpainter.py"
    click LAMA3 "https://github.com/trickygo/pano2room/blob/main/modules/inpainters/pano_pers_fusion_inpainter.py"
    click DG "https://github.com/trickygo/pano2room/tree/main/modules/geo_predictors"
    click GR "https://github.com/trickygo/pano2room/tree/main/gaussian_renderer"
    click NG "https://github.com/trickygo/pano2room/blob/main/gaussian_renderer/network_gui.py"
    click SC "https://github.com/trickygo/pano2room/tree/main/scene"
    click UT "https://github.com/trickygo/pano2room/tree/main/utils"
    click ST "https://github.com/trickygo/pano2room/tree/main/scripts"
    click CP "https://github.com/trickygo/pano2room/blob/main/checkpoints/big-lama-config.yaml"

    %% Styles
    classDef input fill:#f9c74f,stroke:#000,stroke-width:2px;
    classDef preprocess fill:#f9844a,stroke:#000,stroke-width:2px;
    classDef inpainting fill:#90be6d,stroke:#000,stroke-width:2px;
    classDef depth fill:#43aa8b,stroke:#000,stroke-width:2px;
    classDef rendering fill:#577590,stroke:#000,stroke-width:2px;
    classDef scene fill:#277da1,stroke:#000,stroke-width:2px;
    classDef training fill:#f94144,stroke:#000,stroke-width:2px;
```
