How boschresearch/open3dsg works
Project Type and Purpose
• Open3DSG is a research-oriented project for predicting open-vocabulary 3D scene graphs from point clouds. It combines data preprocessing of 3D scans and retrieval of 2D features, with deep network architectures (e.g., PointNet, GNN, BLIP) for learning semantic descriptions.
• The project is both a research prototype and a full training/inference pipeline for multi-modal (3D and 2D) semantic scene understanding.
File Structure to Determine Main Components
• The project is organized under a main package directory (“open3dsg”) with subdirectories dedicated to:
• config – Handles configuration settings (parameters and definitions for experiments).
• data – Contains scripts for data preparation, preprocessing, and generation of scene graphs (e.g., gen_scannet_subgraphs, get_object_frame, preprocess_3rscan/preprocess_scannet).
• models – Encompasses implementations of different network architectures (PointNet, GNN, layers, and integration with external vision-language models like BLIP and OpenSeg).
• scripts – Provides runner scripts (run.py, trainer.py, eval.py) that orchestrate training, testing, and feature dumping.
• util – Includes utilities for data loading (e.g., dataLoaderScanNet), plotting, auxiliary processing, and helper functions.
• Additional artifacts like README, LICENSE, requirements.txt, and configuration files (pyproject.toml) support setup and deployment.
a. Main Components:
• Data Input Layer: Shows raw datasets from ScanNet, 3RScan, and 3DSSG (the inputs, plus the external meta files downloaded via helper scripts).
• Configuration Module: A dedicated block indicating the use of “config/config.py” and “config/define.py”. This module feeds parameters to the entire pipeline.
• Model Architectures: Represent the “models” folder components. Separate nodes can be included for:
• Point Cloud Processors (PointNet, PointNet2).
• Graph Processing modules (GNN-based architectures and networks_base).
• Vision-Language Integration (custom_instruct_blip, LLaVA, and BLIP-related modules).
• Training and Evaluation Pipeline: Illustrated via the “scripts” directory. Includes the trainers, evaluators, and runners executing training (using the specified hyperparameters) and inference.
• Utility Services: Encompasses data loaders (e.g., util/dataLoaderScanNet.py), plotting utilities, and miscellaneous helpers that support the main modules.
• External Dependencies: Emphasize connections with external pretrained models (downloaded checkpoints for OpenSeg, BLIP2 embeddings, and pretrained weights for PointNet variants).
b. Relationships and Interactions:
• Configuration flows into every module (data, models, training) to customize behavior and hyperparameters.
• Preprocessed data is then passed to the Model Architectures.
• The Model Architecture module provides outputs (feature embeddings, scene graphs) to the Training/Evaluation Pipeline.
• Utility Services act as supporting sub-modules that many primary components depend on (e.g., data loading, plotting results, logging).
Architectural Patterns & Relevant Technologies
• Modularity: Each directory (config, data, models, scripts, util) represents a clearly defined module.
• Pipeline Architecture: Data flows sequentially—from raw input to preprocessing, then feature extraction and model training/evaluation.
• The use of pre-trained models (BLIP, OpenSeg, CLIP) and point cloud-specific models (PointNet, PointNet2) indicates a composite architecture that fuses multi-modal information.
• The project is built with Python, using libraries for deep learning (PyTorch), and is optimized for GPU use with CUDA (version 11.8).
• Clear labels:
• “Data Input & Acquisition” for raw datasets (ScanNet, 3RScan, 3DSSG).
• “Preprocessing & Graph Generation” (from scripts under open3dsg/data).
• “Configuration Management” (open3dsg/config).
• “Model Architectures” broken into sub-components: “Point Cloud Processing (PointNet/PointNet2)”, “Graph Modeling (GNN)”, and “Vision-Language Modules (BLIP, OpenSeg, CLIP integration)”.
• “Training & Evaluation Pipeline” (scripts like run.py, trainer.py, eval.py).
• “Utility & Support Functions” (open3dsg/util).
• “External Dependencies” for pre-trained weights and checkpoints.
• Data flow: e.g., from raw data –> preprocessing –> models –> training output.
• Dependency relationships, such as configuration feeding parameters into data, model, and training modules.
• Processing nodes (round-edged rectangles),
• Be detailed in capturing each of these divisions; a more modular representation (as many components as possible) is ideal to capture system nuance.
Final Recommendation
• That each block is self-explanatory with clear labels. Provide annotations where necessary (e.g., “Precomputed 2D features dump (optional step)” for the 2D feature precomputation).
• That the configuration module centrally defines data paths, hyperparameters, and integration points with external resources, making it a critical part of the overall design.