How jonasschult/mask3d works
Main Components
• Data & Preprocessing Layer:
• Raw data and processed data (as seen in the README “data” folder, even if not directly in the repo, the preprocessing scripts in the datasets/preprocessing directory and dedicated dataset scripts indicate this layer).
• Preprocessing scripts for ScanNet, S3DIS, STPLS3D, and others under datasets/preprocessing.
• Configuration & Parameter Management:
• The “conf” folder contains Hydra configuration files for augmentation, callbacks, data loaders, datasets, loss functions, optimizers, schedulers, trainers, etc.
• Model Architecture:
• The “models” directory houses the network components (mask transformer modules) including criterion, model wrappers, backbone networks (e.g., resnet, res16unet, resunet, mask3d modules).
• Also reference “models/modules” for helper functions (e.g., 3detr_helpers, senet_block, etc.) and “models/position_embedding.py”.
• Training and Validation:
• The “trainer” directory includes the train loop (trainer.py) which coordinates the training process.
• The “main_instance_segmentation.py” serves as the entry point for training/inference.
• Benchmark and Evaluation:
• The “benchmark” folder contains scripts for evaluating the semantic instance segmentations and helper utilities for evaluation.
• Numerous scripts under the “scripts” directory orchestrate training, validation, and benchmarking for different datasets.
• External Dependencies & Utilities:
• The “third_party” directory houses external modules like PointNet2 and MinkowskiEngine integration.
• “utils” folder provides additional helper functions (for point cloud visualization, gradient flow checks, k-fold validation, etc.) and also point operations in “utils/pointops2”.
Relationships and Interactions
• Data flows from raw input (acquired externally) → preprocessing (datasets/preprocessing) → processed data storage → data loader (configured via “conf/data” and “conf/data/datasets”).
• The configuration layer (via Hydra and the “conf” folder) dynamically sets up training parameters for the model, datasets, data augmentation, callback functions, optimizers, and schedulers.
• The main training routine in “main_instance_segmentation.py” kicks off by reading configuration files, loading datasets (using custom dataset loaders), instantiating the model architecture (from “models”), and then invoking the trainer loop (from “trainer/trainer.py”).
• Evaluation components in “benchmark” and “scripts” take pretrained checkpoints and run inference, after which metrics (configured in “conf/metrics” and computed in “models/metrics”) are collected.
• External libraries (in “third_party”) are integrated to extend underlying GPU-efficient operations and specialized network layers (e.g., point cloud operations).
Architectural Patterns and Design Principles
• Modularity: The project design follows clear module separation (configuration, dataset, model, trainer, evaluation) which supports ease of experimentation.
• Pipeline Architecture: There is a clear step-by-step pipeline from data ingestion and preprocessing to model training and evaluation.
• Configurability & Extensibility: Use of Hydra configurations throughout enables rapid testing of different architectures, datasets, and training parameters.
• Integration with External Tools: Leveraging third-party libraries (MinkowskiEngine, PointNet2) to boost performance and cover specific use cases.
Relevant Technologies and Frameworks
• PyTorch & PyTorch Lightning: Fundamental frameworks for building and training the deep learning models.
• Hydra: For configuration management.
• CUDA: Ensures GPU acceleration.
• External Toolkits: MinkowskiEngine (for sparse convolution operations) and PointNet2 for specialized point cloud processing.