# jonasschult/mask3d architecture

> Architecture diagram of the GitHub repository jonasschult/mask3d, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jonasschult/mask3d
- Repository: https://github.com/jonasschult/mask3d

## Overview

##### Main Components
- Data & Preprocessing Layer:
- Raw data and processed data (as seen in the README “data” folder, even if not directly in the repo, the preprocessing scripts in the datasets/preprocessing directory and dedicated dataset scripts indicate this layer).
- Preprocessing scripts for ScanNet, S3DIS, STPLS3D, and others under datasets/preprocessing.
- Configuration & Parameter Management:
- The “conf” folder contains Hydra configuration files for augmentation, callbacks, data loaders, datasets, loss functions, optimizers, schedulers, trainers, etc.
- Model Architecture:
- The “models” directory houses the network components (mask transformer modules) including criterion, model wrappers, backbone networks (e.g., resnet, res16unet, resunet, mask3d modules).
- Also reference “models/modules” for helper functions (e.g., 3detr_helpers, senet_block, etc.) and “models/position_embedding.py”.
- Training and Validation:
- The “trainer” directory includes the train loop (trainer.py) which coordinates the training process.
- The “main_instance_segmentation.py” serves as the entry point for training/inference.
- Benchmark and Evaluation:
- The “benchmark” folder contains scripts for evaluating the semantic instance segmentations and helper utilities for evaluation.
- Numerous scripts under the “scripts” directory orchestrate training, validation, and benchmarking for different datasets.
- External Dependencies & Utilities:
- The “third_party” directory houses external modules like PointNet2 and MinkowskiEngine integration.
- “utils” folder provides additional helper functions (for point cloud visualization, gradient flow checks, k-fold validation, etc.) and also point operations in “utils/pointops2”.

##### Relationships and Interactions
- Data flows from raw input (acquired externally) → preprocessing (datasets/preprocessing) → processed data storage → data loader (configured via “conf/data” and “conf/data/datasets”).
- The configuration layer (via Hydra and the “conf” folder) dynamically sets up training parameters for the model, datasets, data augmentation, callback functions, optimizers, and schedulers.
- The main training routine in “main_instance_segmentation.py” kicks off by reading configuration files, loading datasets (using custom dataset loaders), instantiating the model architecture (from “models”), and then invoking the trainer loop (from “trainer/trainer.py”).
- Evaluation components in “benchmark” and “scripts” take pretrained checkpoints and run inference, after which metrics (configured in “conf/metrics” and computed in “models/metrics”) are collected.
- External libraries (in “third_party”) are integrated to extend underlying GPU-efficient operations and specialized network layers (e.g., point cloud operations).

##### Architectural Patterns and Design Principles
- Modularity: The project design follows clear module separation (configuration, dataset, model, trainer, evaluation) which supports ease of experimentation.
- Pipeline Architecture: There is a clear step-by-step pipeline from data ingestion and preprocessing to model training and evaluation.
- Configurability & Extensibility: Use of Hydra configurations throughout enables rapid testing of different architectures, datasets, and training parameters.
- Integration with External Tools: Leveraging third-party libraries (MinkowskiEngine, PointNet2) to boost performance and cover specific use cases.

##### Relevant Technologies and Frameworks
- PyTorch & PyTorch Lightning: Fundamental frameworks for building and training the deep learning models.
- Hydra: For configuration management.
- CUDA: Ensures GPU acceleration.
- External Toolkits: MinkowskiEngine (for sparse convolution operations) and PointNet2 for specialized point cloud processing.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing Layer
    subgraph "Data Ingestion & Preprocessing"
        D["Data & Preprocessing"]:::data
    end

    %% Configuration Layer
    subgraph "Configuration Layer"
        C["Configuration (Hydra)"]:::config
    end

    %% Modeling Layer with subcomponents
    subgraph "Modeling Layer"
        M["Model Architecture"]:::model
        subgraph "Model Components"
            M1["Mask3D Module"]:::model_sub
            M2["Criterion"]:::model_sub
            M3["Helper Modules"]:::model_sub
            M4["Position Embedding"]:::model_sub
        end
    end

    %% Training & Validation
    subgraph "Training & Validation"
        T1["Main Instance Segmentation"]:::training
        T2["Trainer Loop"]:::training
    end

    %% Evaluation & Benchmarking
    subgraph "Evaluation & Benchmarking"
        E1["Benchmark"]:::eval
        E2["Scripts"]:::eval
    end

    %% External Dependencies
    subgraph "External Integrations"
        X["Third-Party Integrations"]:::external
    end

    %% Utilities & Helpers
    subgraph "Utilities"
        U["Utilities & Helpers"]:::utils
    end

    %% Flow Connections
    D --> C
    C --> M
    M --> M1
    M --> M2
    M --> M3
    M --> M4
    C --> T1
    M --> T1
    T1 --> T2
    T2 --> E1
    T2 --> E2
    X --> M
    X --> T1
    X --> T2
    U --> T1
    U --> E1
    U --> E2

    %% Click Events
    click D "https://github.com/jonasschult/mask3d/tree/main/datasets/preprocessing"
    click C "https://github.com/jonasschult/mask3d/tree/main/conf"
    click M "https://github.com/jonasschult/mask3d/tree/main/models"
    click M1 "https://github.com/jonasschult/mask3d/blob/main/models/mask3d.py"
    click M2 "https://github.com/jonasschult/mask3d/blob/main/models/criterion.py"
    click M3 "https://github.com/jonasschult/mask3d/tree/main/models/modules"
    click M4 "https://github.com/jonasschult/mask3d/blob/main/models/position_embedding.py"
    click T1 "https://github.com/jonasschult/mask3d/blob/main/main_instance_segmentation.py"
    click T2 "https://github.com/jonasschult/mask3d/blob/main/trainer/trainer.py"
    click E1 "https://github.com/jonasschult/mask3d/tree/main/benchmark"
    click E2 "https://github.com/jonasschult/mask3d/tree/main/scripts"
    click X "https://github.com/jonasschult/mask3d/tree/main/third_party"
    click U "https://github.com/jonasschult/mask3d/tree/main/utils"

    %% Styles
    classDef data fill:#f9e79f,stroke:#d35400,stroke-width:2px;
    classDef config fill:#d6eaf8,stroke:#2e86c1,stroke-width:2px;
    classDef model fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
    classDef model_sub fill:#ebf5fb,stroke:#2980b9,stroke-width:1px;
    classDef training fill:#f5cba7,stroke:#e67e22,stroke-width:2px;
    classDef eval fill:#f6e3e1,stroke:#c0392b,stroke-width:2px;
    classDef external fill:#eadff7,stroke:#8e44ad,stroke-width:2px;
    classDef utils fill:#fdebd0,stroke:#d35400,stroke-width:2px;
```
