# chensjtu/gaussianobject architecture

> Architecture diagram of the GitHub repository chensjtu/gaussianobject, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chensjtu/gaussianobject
- Repository: https://github.com/chensjtu/gaussianobject

## Overview

##### Project Type and Purpose
- This is a full‐stack research framework for high‐quality 3D object reconstruction from four views using Gaussian splatting and diffusion‐based repair.
- Key features (from README): visual hull initialization, coarse 3D Gaussian Splatting (3DGS), leave‐one‐out analysis, LoRA fine‐tuning, Gaussian repair with diffusion, and final rendering.

##### Main Components
a. Data Preparation & Preprocessing
- Mask Generation (Segment‐Anything)
- Pose Estimation (DUSt3R or MASt3R)
- Downsampling / Depth Prediction (preprocess/ folder)
b. Visual Hull Initialization
- visual_hull.py & scene/colmap_loader.py → initial point cloud (.ply)
c. Coarse 3DGS Training
- train_gs.py uses cldm, ldm, gaussian_renderer, threestudio modules
d. Leave-One-Out Analysis
- leave_one_out_stage1.py & leave_one_out_stage2.py
e. LoRA Fine‐Tuning of Diffusion & Control Models
- train_lora.py integrates ControlNet, CLIP, stable‐diffusion via submodules
f. Gaussian Repair Model Training
- train_repair.py uses configs/gaussian-object.yaml, dust3r/mast3r, threestudio systems
g. Rendering & Visualization
- render.py, gaussian_renderer/network_gui.py → final images / paths

##### Data & Control Flow
- Input: Raw images → Preprocessor → {masks, coarse poses, downsampled images}
- Preprocessed data → Visual Hull Module → initial PLY
- PLY + data → Coarse 3DGS Trainer → coarse Gaussians
- Coarse model → Leave‐One‐Out Module → refined coarse model
- Refined model + text prompts → LoRA Trainer → LoRA weights for repair
- Coarse model + LoRA weights → Gaussian Repair Trainer → final 3DGS (.ply)
- Final .ply + model → Renderer → output images/videos

##### Architectural Patterns & Design Principles
- Staged Pipeline: clearly separated optimization and learning phases
- Modular Design: each stage lives in its own script (visual_hull.py, train_gs.py, etc.) and package (cldm, dust3r, ldm, threestudio)
- Extensibility Points: support COLMAP-free flow via flags and alternate scripts (pred_poses.py vs. pred_poses_mast3r.py)
- Reuse of external libraries via submodules (ControlNet, segment-anything, pytorch3d)

##### Essential Elements Checklist
- Clear component labels (script name + function)
- Annotation of major file artifacts and config files

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Stage 1: Data Preparation & Preprocessing
    subgraph "1. Data Preparation & Preprocessing"
        direction TB
        RawImages["Raw Images"]:::data
        SACloud(["Segment-Anything"]):::external
        SAMask["segment_anything.ipynb"]:::script
        SAMod["submodules/segment-anything/"]:::external
        Annotator["annotator/util.py"]:::script
        Downsample["preprocess/downsample.py"]:::script
        Monodepth["preprocess/pred_monodepth.py"]:::script
        Poses1["pred_poses.py"]:::script
        Poses2["pred_poses_mast3r.py"]:::script
        Dust3R["dust3r/cloud_opt/"]:::external
        MAS3R["mast3r/cloud_opt/"]:::external
    end

    %% Stage 2: Visual Hull Initialization
    subgraph "2. Visual Hull Initialization"
        direction TB
        VHScript["visual_hull.py"]:::script
        COLLoader["scene/colmap_loader.py"]:::script
        InitPLY(["Initial Point Cloud (.ply)"]):::data
    end

    %% Stage 3: Coarse 3DGS Training
    subgraph "3. Coarse 3DGS Training"
        direction TB
        TrainGS["train_gs.py"]:::script
        CLDM(["cldm"]):::external
        LDM(["ldm"]):::external
        GRInit["gaussian_renderer/__init__.py"]:::script
        TSSystem["threestudio/systems/gaussian_object_system.py"]:::script
        CoarseModel(["Coarse Model (coarse_model.pth)"]):::data
    end

    %% Stage 4: Leave-One-Out Analysis
    subgraph "4. Leave-One-Out Analysis"
        direction TB
        LOO1["leave_one_out_stage1.py"]:::script
        LOO2["leave_one_out_stage2.py"]:::script
        LOODS["threestudio/data/loo.py"]:::script
        RefinedModel(["Refined Coarse Model"]):::data
    end

    %% Stage 5: LoRA Fine-Tuning
    subgraph "5. LoRA Fine-Tuning"
        direction TB
        TrainLoRA["train_lora.py"]:::script
        DataLoRA["dataset_lora.py"]:::script
        MinLoRA(["submodules/minLoRA/"]):::external
        CtrlYAML(["models/control_v11f1e_sd15_tile.yaml"]):::config
        LoRAWeights(["LoRA Weights"]):::data
    end

    %% Stage 6: Gaussian Repair Training
    subgraph "6. Gaussian Repair Model Training"
        direction TB
        TrainRepair["train_repair.py"]:::script
        Config1["configs/gaussian-object.yaml"]:::config
        Config2["configs/gaussian-object-colmap-free.yaml"]:::config
        GORSystem["threestudio/systems/gaussian_object_system.py"]:::script
        FinalPLY(["Final 3DGS (.ply)"]):::data
    end

    %% Stage 7: Rendering & Visualization
    subgraph "7. Rendering & Visualization"
        direction TB
        RenderScript["render.py"]:::script
        NetworkGUI["gaussian_renderer/network_gui.py"]:::script
        OutputImgs(["Output Images/Videos"]):::data
    end

    %% Connections and Data Flow
    RawImages -->|"sparse_4.txt"| SAMask
    RawImages -->|"sparse_4.txt"| Poses1
    RawImages -->|"sparse_4.txt"| Monodepth
    SAMask -->|"masks.json"| VHScript
    SAMod --> SAMask
    Annotator --> SAMask
    Downsample -->|"depth_maps"| VHScript
    Monodepth -->|"depth_maps"| VHScript
    Poses1 -->|"coarse_poses.json"| VHScript
    Poses2 -->|"coarse_poses.json"| VHScript
    Dust3R --> Poses2
    MAS3R --> Poses2

    VHScript -->|"initial_ply"| InitPLY
    COLLoader --> InitPLY

    InitPLY -->|"ply + poses"| TrainGS
    TrainGS -->|"coarse_model.pth"| CoarseModel
    CLDM --> TrainGS
    LDM --> TrainGS
    GRInit --> TrainGS
    TSSystem --> TrainGS

    CoarseModel -->|"model"| LOO1
    LOO1 -->|"stage1_model"| LOO2
    LOO2 -->|"refined_model"| RefinedModel
    LOODS --> LOO1

    RefinedModel --> TrainLoRA
    TrainLoRA -->|"LoRA weights"| LoRAWeights
    DataLoRA --> TrainLoRA
    MinLoRA --> TrainLoRA
    CtrlYAML --> TrainLoRA

    CoarseModel --> TrainRepair
    LoRAWeights --> TrainRepair
    TrainRepair -->|"final_ply"| FinalPLY
    Config1 --> TrainRepair
    Config2 --> TrainRepair
    GORSystem --> TrainRepair

    FinalPLY --> RenderScript
    RenderScript -->|"rendered_images"| OutputImgs
    NetworkGUI --> RenderScript

    %% Click Events
    click SAMask "https://github.com/chensjtu/gaussianobject/blob/main/segment_anything.ipynb"
    click SAMod "https://github.com/chensjtu/gaussianobject/tree/main/submodules/segment-anything/"
    click Annotator "https://github.com/chensjtu/gaussianobject/blob/main/annotator/util.py"
    click Downsample "https://github.com/chensjtu/gaussianobject/blob/main/preprocess/downsample.py"
    click Monodepth "https://github.com/chensjtu/gaussianobject/blob/main/preprocess/pred_monodepth.py"
    click Poses1 "https://github.com/chensjtu/gaussianobject/blob/main/pred_poses.py"
    click Poses2 "https://github.com/chensjtu/gaussianobject/blob/main/pred_poses_mast3r.py"
    click Dust3R "https://github.com/chensjtu/gaussianobject/tree/main/dust3r/cloud_opt/"
    click MAS3R "https://github.com/chensjtu/gaussianobject/tree/main/mast3r/cloud_opt/"
    click VHScript "https://github.com/chensjtu/gaussianobject/blob/main/visual_hull.py"
    click COLLoader "https://github.com/chensjtu/gaussianobject/blob/main/scene/colmap_loader.py"
    click TrainGS "https://github.com/chensjtu/gaussianobject/blob/main/train_gs.py"
    click GRInit "https://github.com/chensjtu/gaussianobject/blob/main/gaussian_renderer/__init__.py"
    click TSSystem "https://github.com/chensjtu/gaussianobject/blob/main/threestudio/systems/gaussian_object_system.py"
    click LOO1 "https://github.com/chensjtu/gaussianobject/blob/main/leave_one_out_stage1.py"
    click LOO2 "https://github.com/chensjtu/gaussianobject/blob/main/leave_one_out_stage2.py"
    click LOODS "https://github.com/chensjtu/gaussianobject/blob/main/threestudio/data/loo.py"
    click TrainLoRA "https://github.com/chensjtu/gaussianobject/blob/main/train_lora.py"
    click DataLoRA "https://github.com/chensjtu/gaussianobject/blob/main/dataset_lora.py"
    click MinLoRA "https://github.com/chensjtu/gaussianobject/tree/main/submodules/minLoRA/"
    click CtrlYAML "https://github.com/chensjtu/gaussianobject/blob/main/models/control_v11f1e_sd15_tile.yaml"
    click TrainRepair "https://github.com/chensjtu/gaussianobject/blob/main/train_repair.py"
    click Config1 "https://github.com/chensjtu/gaussianobject/blob/main/configs/gaussian-object.yaml"
    click Config2 "https://github.com/chensjtu/gaussianobject/blob/main/configs/gaussian-object-colmap-free.yaml"
    click GORSystem "https://github.com/chensjtu/gaussianobject/blob/main/threestudio/systems/gaussian_object_system.py"
    click RenderScript "https://github.com/chensjtu/gaussianobject/blob/main/render.py"
    click NetworkGUI "https://github.com/chensjtu/gaussianobject/blob/main/gaussian_renderer/network_gui.py"

    %% Styles
    classDef data fill:#D0E4F5,stroke:#003768,color:#003768
    classDef script fill:#F0F8FF,stroke:#1F4E79,color:#1F4E79
    classDef external fill:#E3FCEF,stroke:#2E7D32,color:#2E7D32
    classDef config fill:#FFF4E5,stroke:#BF360C,color:#BF360C
```
