# yilmazkadir/mask4former architecture

> Architecture diagram of the GitHub repository yilmazkadir/mask4former, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yilmazkadir/mask4former
- Repository: https://github.com/yilmazkadir/mask4former

## Overview

##### Project type and purpose
- Mask4Former is a specialized deep‐learning research codebase implementing a transformer-based model for 4D panoptic segmentation on LiDAR data (SemanticKITTI).
- It’s not a web app or microservice suite but a full ML pipeline: data preprocessing → training → inference/evaluation.

##### File structure
- Top-level directories:
- conf/      → Hydra YAML configs for every component (data, model, optimizer, scheduler, callbacks, logging, trainer, etc.)
- datasets/  → preprocessing logic & dataset definitions (SemanticKITTI)
- data/      → preprocessed/serialized data storage
- models/    → backbone (ResNet/UNet), transformer heads, criterion, matcher, metrics
- trainer/   → PyTorch Lightning trainer wrapper
- scripts/   → shell scripts for train, test, val
- main_panoptic.py → entrypoint for training & inference
- Notice use of PyTorch Lightning (lightning.Trainer), Hydra (conf/), and custom modules under utils/.

##### Key README insights
- Relies on Hydra for configuration, PyTorch Lightning for training orchestration.
- External dependencies: PyTorch, MinkowskiEngine, PyTorch3D.
- Two main workflows:
- Data preprocessing (`datasets.preprocessing.semantic_kitti_preprocessing`)
- Training/inference via `main_panoptic.py` with Hydra overrides.

##### Components
a. Main components
- Raw Data Storage (SemanticKITTI)
- Data Preprocessor (datasets.preprocessing)
- Preprocessed Data (data/)
- Data Pipeline & Augmentation (datasets/lidar.py + conf/data/* + volumentations)
- Model Definition (models/… ; backbone + transformer heads + criterion + matcher)
- Config Manager (Hydra reading conf/*)
- Training Orchestrator (PyTorch Lightning / trainer/trainer.py)
- Checkpoint & Logging (conf/logging + wandb or file sink)
- Inference & Post-processing (main_panoptic.py with optional DBSCAN)
- External Libraries (PyTorch, MinkowskiEngine, PyTorch3D)

b. Relationships & interactions
- Config Manager feeds settings to Preprocessor, DataLoader, Model, Trainer
- Model outputs pass to Criterion & Matcher → Loss → Lightning Trainer → Optimizer & Scheduler
- Trainer writes checkpoints & metrics to disk/W&B
- Inference path: Checkpoint → Model → Predictions → (Optional) DBSCAN → Evaluation Metrics

c. Architectural patterns & design principles
- Modular design: separate preprocessing, modeling, training, config
- Config-driven via Hydra for reproducibility and easy hyperparameter sweeps
- Use of PyTorch Lightning to abstract the training loop and callbacks
- Plug-and-play components: different backbones, matchers, augmentations, metrics

d. Key technologies/frameworks
- Hydra (config management)
- PyTorch Lightning (training orchestration)
- PyTorch + MinkowskiEngine + PyTorch3D (3D sparse convs, point cloud ops)
- YAML configs under conf/ for every functional area

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data
    RawData["Raw Data Storage<br/>(SemanticKITTI)"]:::external

    %% Data Preprocessing
    DataPreprocessor["Data Preprocessor"]:::compute
    click DataPreprocessor "https://github.com/yilmazkadir/mask4former/blob/main/datasets/preprocessing/semantic_kitti_preprocessing.py"

    RawData -->|"load raw scans"| DataPreprocessor

    %% Preprocessed Data
    PreprocessedData["Preprocessed Data Storage"]:::data
    click PreprocessedData "https://github.com/yilmazkadir/mask4former/tree/main/data/"

    DataPreprocessor -->|"save serialized data"| PreprocessedData

    %% Data Pipeline & Augmentation
    subgraph "Data Pipeline & Augmentation"
        direction TB
        LidarLoader["LIDAR Loader"]:::compute
        click LidarLoader "https://github.com/yilmazkadir/mask4former/blob/main/datasets/lidar.py"
        AugConfig["Augmentation Config"]:::config
        click AugConfig "https://github.com/yilmazkadir/mask4former/blob/main/conf/augmentation/volumentations_aug.yaml"
        LoaderConfig["DataLoader Config"]:::config
        click LoaderConfig "https://github.com/yilmazkadir/mask4former/blob/main/conf/data/data_loaders/simple_loader.yaml"
        DatasetConfig["Dataset Config"]:::config
        click DatasetConfig "https://github.com/yilmazkadir/mask4former/blob/main/conf/data/datasets/semantic_kitti.yaml"

        PreprocessedData --> LidarLoader
        AugConfig --- LidarLoader
        LoaderConfig --- LidarLoader
        DatasetConfig --- LidarLoader
        LidarLoader -->|"augmented batches"| ModelInput
    end

    %% Model Definition
    subgraph "Model Definition"
        direction TB
        Mask4Former["Mask4Former Model"]:::compute
        click Mask4Former "https://github.com/yilmazkadir/mask4former/blob/main/models/mask4former.py"
        CoreModel["Core Model"]:::compute
        click CoreModel "https://github.com/yilmazkadir/mask4former/blob/main/models/model.py"
        subgraph BackbonesAndHeads
            direction TB
            ResNet["ResNet Backbone"]:::compute
            click ResNet "https://github.com/yilmazkadir/mask4former/blob/main/models/resnet.py"
            ResUNet["ResUNet Backbone"]:::compute
            click ResUNet "https://github.com/yilmazkadir/mask4former/blob/main/models/resunet.py"
            Res16UNet["Res16UNet Backbone"]:::compute
            click Res16UNet "https://github.com/yilmazkadir/mask4former/blob/main/models/res16unet.py"
            Attention["Attention Module"]:::compute
            click Attention "https://github.com/yilmazkadir/mask4former/blob/main/models/modules/attention.py"
            ResBlock["ResNet Block"]:::compute
            click ResBlock "https://github.com/yilmazkadir/mask4former/blob/main/models/modules/resnet_block.py"
            CommonMods["Common Utils"]:::compute
            click CommonMods "https://github.com/yilmazkadir/mask4former/blob/main/models/modules/common.py"
            Helpers3DETR["3DETR Helpers"]:::compute
            click Helpers3DETR "https://github.com/yilmazkadir/mask4former/blob/main/models/modules/helpers_3detr.py"
            PosEmbed["Position Embedding"]:::compute
            click PosEmbed "https://github.com/yilmazkadir/mask4former/blob/main/models/position_embedding.py"
        end
        CoreModel --> Mask4Former
        BackbonesAndHeads --> CoreModel
    end

    ModelInput["Batched Model Input"]:::data -->|"forward()"| Mask4Former

    %% Criterion & Matcher
    subgraph "Criterion & Matcher"
        direction TB
        Criterion["Criterion"]:::compute
        click Criterion "https://github.com/yilmazkadir/mask4former/blob/main/models/criterion.py"
        Matcher["Matcher"]:::compute
        click Matcher "https://github.com/yilmazkadir/mask4former/blob/main/models/matcher.py"
        LossConfig["Criterion Config"]:::config
        click LossConfig "https://github.com/yilmazkadir/mask4former/blob/main/conf/loss/set_criterion.yaml"
        MatcherConfig["Matcher Config"]:::config
        click MatcherConfig "https://github.com/yilmazkadir/mask4former/blob/main/conf/matcher/hungarian_matcher.yaml"

        Mask4Former --> Criterion
        Mask4Former --> Matcher
        LossConfig --- Criterion
        MatcherConfig --- Matcher
        Criterion -->|"compute loss"| LossVal
        Matcher -->|"assign targets"| LossVal
    end

    LossVal["Loss Value"]:::data

    %% Metrics
    subgraph "Metrics"
        direction TB
        PanQuality["Panoptic Quality"]:::compute
        click PanQuality "https://github.com/yilmazkadir/mask4former/blob/main/models/metrics/panoptic_quality.py"
        LSTQConfig["LSTQ Config"]:::config
        click LSTQConfig "https://github.com/yilmazkadir/mask4former/blob/main/conf/metric/lstq.yaml"
        PanQuality --- LSTQConfig
        LossVal --> PanQuality
    end

    %% Config Manager
    ConfigManager["Config Manager<br/>(Hydra)"]:::config
    click ConfigManager "https://github.com/yilmazkadir/mask4former/blob/main/conf/config_panoptic_4d.yaml"
    click ConfigManager "https://github.com/yilmazkadir/mask4former/blob/main/conf/model/mask4former.yaml"
    click ConfigManager "https://github.com/yilmazkadir/mask4former/blob/main/conf/optimizer/adamw.yaml"
    click ConfigManager "https://github.com/yilmazkadir/mask4former/blob/main/conf/scheduler/onecyclelr.yaml"
    click ConfigManager "https://github.com/yilmazkadir/mask4former/blob/main/conf/trainer/trainer30.yaml"
    click ConfigManager "https://github.com/yilmazkadir/mask4former/blob/main/conf/logging/full.yaml"
    click ConfigManager "https://github.com/yilmazkadir/mask4former/blob/main/conf/callbacks/callbacks_panoptic.yaml"
    click ConfigManager "https://github.com/yilmazkadir/mask4former/blob/main/conf/data/collation_functions/voxelize_collate.yaml"
    click ConfigManager "https://github.com/yilmazkadir/mask4former/blob/main/conf/data/kitti.yaml"
    click ConfigManager "https://github.com/yilmazkadir/mask4former/blob/main/conf/semantic-kitti.yaml"

    ConfigManager --- DataPreprocessor
    ConfigManager --- LidarLoader
    ConfigManager --- Mask4Former
    ConfigManager --- Criterion
    ConfigManager --- Matcher
    ConfigManager --- PanQuality
    ConfigManager --- Trainer
    ConfigManager --- Entry

    %% Training Orchestrator
    Trainer["Training Orchestrator"]:::compute
    click Trainer "https://github.com/yilmazkadir/mask4former/blob/main/trainer/trainer.py"
    Mask4Former --> Trainer
    Criterion --> Trainer
    Matcher --> Trainer
    Trainer -->|"optimizer & scheduler"| OptimStep
    OptimStep["Optimization Step"]:::compute
    Trainer -->|"write checkpoints & metrics"| Logging

    %% Checkpoint & Logging
    Logging["Checkpoint & Logging"]:::data
    click Logging "https://github.com/yilmazkadir/mask4former/blob/main/conf/logging/full.yaml"
    click Logging "https://github.com/yilmazkadir/mask4former/blob/main/conf/callbacks/callbacks_panoptic.yaml"

    %% Entrypoint
    Entry["Entrypoint<br/>(main_panoptic.py)"]:::compute
    click Entry "https://github.com/yilmazkadir/mask4former/blob/main/main_panoptic.py"
    Logging --> Entry

    %% Inference & Post-processing
    Entry --> Inference["Inference & Post-processing"]:::compute
    DBSCAN["DBSCAN Post-processing"]:::compute
    Inference --> DBSCAN
    DBSCAN --> Evaluation["Evaluation Metrics"]:::compute

    %% External Libraries
    subgraph "External Libraries"
        direction TB
        PyTorch["PyTorch"]:::external
        Minkowski["MinkowskiEngine"]:::external
        PyTorch3D["PyTorch3D"]:::external
    end
    PyTorch --> Trainer
    Minkowski --> Mask4Former
    PyTorch3D --> Mask4Former

    classDef external fill:#D3D3D3,stroke:#000,shape:cloud;
    classDef data fill:#FFF2CC,stroke:#000,shape:cylinder;
    classDef compute fill:#CCE5FF,stroke:#000,shape:rect;
    classDef config fill:#E8F0FE,stroke:#000,shape:diamond;
```
