# boschresearch/open3dsg architecture

> Architecture diagram of the GitHub repository boschresearch/open3dsg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/boschresearch/open3dsg
- Repository: https://github.com/boschresearch/open3dsg

## Overview

##### Project Type and Purpose
- Open3DSG is a research-oriented project for predicting open-vocabulary 3D scene graphs from point clouds. It combines data preprocessing of 3D scans and retrieval of 2D features, with deep network architectures (e.g., PointNet, GNN, BLIP) for learning semantic descriptions.
- The project is both a research prototype and a full training/inference pipeline for multi-modal (3D and 2D) semantic scene understanding.

##### File Structure to Determine Main Components
- The project is organized under a main package directory (“open3dsg”) with subdirectories dedicated to:
- config – Handles configuration settings (parameters and definitions for experiments).
- data – Contains scripts for data preparation, preprocessing, and generation of scene graphs (e.g., gen_scannet_subgraphs, get_object_frame, preprocess_3rscan/preprocess_scannet).
- models – Encompasses implementations of different network architectures (PointNet, GNN, layers, and integration with external vision-language models like BLIP and OpenSeg).
- scripts – Provides runner scripts (run.py, trainer.py, eval.py) that orchestrate training, testing, and feature dumping.
- util – Includes utilities for data loading (e.g., dataLoaderScanNet), plotting, auxiliary processing, and helper functions.
- Additional artifacts like README, LICENSE, requirements.txt, and configuration files (pyproject.toml) support setup and deployment.

a. Main Components:
- Data Input Layer: Shows raw datasets from ScanNet, 3RScan, and 3DSSG (the inputs, plus the external meta files downloaded via helper scripts).
- Configuration Module: A dedicated block indicating the use of “config/config.py” and “config/define.py”. This module feeds parameters to the entire pipeline.
- Model Architectures: Represent the “models” folder components. Separate nodes can be included for:
- Point Cloud Processors (PointNet, PointNet2).
- Graph Processing modules (GNN-based architectures and networks_base).
- Vision-Language Integration (custom_instruct_blip, LLaVA, and BLIP-related modules).
- Training and Evaluation Pipeline: Illustrated via the “scripts” directory. Includes the trainers, evaluators, and runners executing training (using the specified hyperparameters) and inference.
- Utility Services: Encompasses data loaders (e.g., util/dataLoaderScanNet.py), plotting utilities, and miscellaneous helpers that support the main modules.
- External Dependencies: Emphasize connections with external pretrained models (downloaded checkpoints for OpenSeg, BLIP2 embeddings, and pretrained weights for PointNet variants).

b. Relationships and Interactions:
- Configuration flows into every module (data, models, training) to customize behavior and hyperparameters.
- Preprocessed data is then passed to the Model Architectures.
- The Model Architecture module provides outputs (feature embeddings, scene graphs) to the Training/Evaluation Pipeline.
- Utility Services act as supporting sub-modules that many primary components depend on (e.g., data loading, plotting results, logging).

##### Architectural Patterns & Relevant Technologies
- Modularity: Each directory (config, data, models, scripts, util) represents a clearly defined module.
- Pipeline Architecture: Data flows sequentially—from raw input to preprocessing, then feature extraction and model training/evaluation.
- The use of pre-trained models (BLIP, OpenSeg, CLIP) and point cloud-specific models (PointNet, PointNet2) indicates a composite architecture that fuses multi-modal information.
- The project is built with Python, using libraries for deep learning (PyTorch), and is optimized for GPU use with CUDA (version 11.8).

- Clear labels:
- “Data Input & Acquisition” for raw datasets (ScanNet, 3RScan, 3DSSG).
- “Preprocessing & Graph Generation” (from scripts under open3dsg/data).
- “Configuration Management” (open3dsg/config).
- “Model Architectures” broken into sub-components: “Point Cloud Processing (PointNet/PointNet2)”, “Graph Modeling (GNN)”, and “Vision-Language Modules (BLIP, OpenSeg, CLIP integration)”.
- “Training & Evaluation Pipeline” (scripts like run.py, trainer.py, eval.py).
- “Utility & Support Functions” (open3dsg/util).
- “External Dependencies” for pre-trained weights and checkpoints.
- Data flow: e.g., from raw data –> preprocessing –> models –> training output.
- Dependency relationships, such as configuration feeding parameters into data, model, and training modules.
- Processing nodes (round-edged rectangles),
- Be detailed in capturing each of these divisions; a more modular representation (as many components as possible) is ideal to capture system nuance.

##### Final Recommendation
- That each block is self-explanatory with clear labels. Provide annotations where necessary (e.g., “Precomputed 2D features dump (optional step)” for the 2D feature precomputation).
- That the configuration module centrally defines data paths, hyperparameters, and integration points with external resources, making it a critical part of the overall design.

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% Data Input & Acquisition
    A["Data Input & Acquisition (ScanNet, 3RScan, 3DSSG)"]:::data

    %% Preprocessing & Graph Generation
    subgraph "Preprocessing & Graph Generation"
        DPGroup["Data Preprocessing Module"]:::module
        DP1["gen_scannet_subgraphs.py"]:::file
        DP2["get_object_frame.py"]:::file
        DP3["preprocess_3rscan.py"]:::file
        DP4["preprocess_scannet.py"]:::file
        DP5["open_dataset.py"]:::file
        DPGroup --> DP1
        DPGroup --> DP2
        DPGroup --> DP3
        DPGroup --> DP4
        DPGroup --> DP5
    end

    %% Configuration Management
    subgraph "Configuration Management"
        CFGGroup["Configuration Module"]:::module
        CFG1["config.py"]:::file
        CFG2["define.py"]:::file
        CFGGroup --> CFG1
        CFGGroup --> CFG2
    end

    %% Model Architectures
    subgraph "Model Architectures"
        %% Point Cloud Processing
        subgraph "Point Cloud Processing"
            PNP1["network_PointNet.py"]:::file
            PNP2["pointnet.py"]:::file
            PNP3["pointnet2.py"]:::file
            PNP4["pointnet2_utils.py"]:::file
        end
        %% Graph Modeling
        subgraph "Graph Modeling"
            GM1["network_GNN.py"]:::file
            GM2["networks_base.py"]:::file
            GM3["network_util.py"]:::file
            GM4["sgpn.py"]:::file
        end
        %% Vision-Language Integration
        subgraph "Vision-Language Integration"
            VLM1["custom_instruct_blip.py"]:::file
            VLM2["blip_patch_projector.py"]:::file
            VLM3["LLaVA (directory)"]:::dir
        end
    end

    %% Training & Evaluation Pipeline
    subgraph "Training & Evaluation Pipeline"
        TEGroup["Training & Evaluation Scripts"]:::module
        TE1["run.py"]:::file
        TE2["trainer.py"]:::file
        TE3["eval.py"]:::file
        TEGroup --> TE1
        TEGroup --> TE2
        TEGroup --> TE3
    end

    %% Utility & Support Functions
    subgraph "Utility & Support Functions"
        UTILGroup["Utility Services"]:::module
        UTIL1["dataLoaderScanNet.py"]:::file
        UTIL2["plotting_utils.py"]:::file
        UTILGroup --> UTIL1
        UTILGroup --> UTIL2
    end

    %% External Dependencies & Helpers
    subgraph "External Dependencies"
        EXTGroup["External Dependencies & Helpers"]:::module
        EXT1["download_3rscan_meta.sh"]:::file
        EXT2["download_scannet_meta.sh"]:::file
        EXTGroup --> EXT1
        EXTGroup --> EXT2
    end

    %% Connections between components
    A -->|"data_flow"| DPGroup
    CFGGroup -->|"feeds"| DPGroup
    CFGGroup -->|"feeds"| PNP1
    CFGGroup -->|"feeds"| GM1
    CFGGroup -->|"feeds"| VLM1
    DPGroup -->|"processed_data"| PNP1
    DPGroup -->|"processed_data"| GM1
    DPGroup -->|"processed_data"| VLM1
    PNP1 -->|"inference"| TEGroup
    GM1 -->|"inference"| TEGroup
    VLM1 -->|"inference"| TEGroup
    UTILGroup -.->|"supports"| DPGroup
    UTILGroup -.->|"supports"| TEGroup
    EXTGroup -->|"external_weights"| PNP1
    EXTGroup -->|"external_weights"| GM1
    EXTGroup -->|"external_weights"| VLM1

    %% Class definitions for coloring and styles
    classDef data fill:#cff,stroke:#333,stroke-width:2px;
    classDef module fill:#cfc,stroke:#333,stroke-width:2px;
    classDef file fill:#f9f,stroke:#333,stroke-width:1px;
    classDef dir fill:#ccf,stroke:#333,stroke-width:1px;
    
    %% Click Events for Data Preprocessing Module
    click DPGroup "https://github.com/boschresearch/open3dsg/tree/main/open3dsg/data"
    click DP1 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/data/gen_scannet_subgraphs.py"
    click DP2 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/data/get_object_frame.py"
    click DP3 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/data/preprocess_3rscan.py"
    click DP4 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/data/preprocess_scannet.py"
    click DP5 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/data/open_dataset.py"
    
    %% Click Events for Configuration Module
    click CFGGroup "https://github.com/boschresearch/open3dsg/tree/main/open3dsg/config"
    click CFG1 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/config/config.py"
    click CFG2 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/config/define.py"

    %% Click Events for Point Cloud Processing Modules
    click PNP1 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/models/network_PointNet.py"
    click PNP2 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/models/pointnet.py"
    click PNP3 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/models/pointnet2.py"
    click PNP4 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/models/pointnet2_utils.py"
    
    %% Click Events for Graph Modeling Modules
    click GM1 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/models/network_GNN.py"
    click GM2 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/models/networks_base.py"
    click GM3 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/models/network_util.py"
    click GM4 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/models/sgpn.py"
    
    %% Click Events for Vision-Language Integration Modules
    click VLM1 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/models/custom_instruct_blip.py"
    click VLM2 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/models/blip_patch_projector.py"
    click VLM3 "https://github.com/boschresearch/open3dsg/tree/main/open3dsg/models/LLaVA"
    
    %% Click Events for Training & Evaluation Pipeline
    click TE1 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/scripts/run.py"
    click TE2 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/scripts/trainer.py"
    click TE3 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/scripts/eval.py"
    
    %% Click Events for Utility Services
    click UTIL1 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/util/dataLoaderScanNet.py"
    click UTIL2 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/util/plotting_utils.py"
    
    %% Click Events for External Dependencies
    click EXT1 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/helpers/download_3rscan_meta.sh"
    click EXT2 "https://github.com/boschresearch/open3dsg/blob/main/open3dsg/helpers/download_scannet_meta.sh"
```
