# vios-s/gmt-leaf-ins-seg architecture

> Architecture diagram of the GitHub repository vios-s/gmt-leaf-ins-seg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vios-s/gmt-leaf-ins-seg
- Repository: https://github.com/vios-s/gmt-leaf-ins-seg

## Overview

##### Project type and purpose
- This is an ML research codebase (“gmt-leaf-ins-seg”) implementing “GMT: Guided Mask Transformer” for leaf instance segmentation. It’s built on top of Facebook’s Mask2Former and a Harmonic Embeddings library. The main use cases are training and evaluating guided mask transformer models on various plant datasets (CVPPP, Komatsuna, MSU PID).

##### File structure
- Top-level entries:
- configs/            ← YAML files controlling experiments per dataset
- harmonic/           ← code for generating and loading guide embeddings
- mask2former/        ← core model architectures, data loaders, criterion, decoders
- guide_train_net.py  ← training orchestration script
- submission/         ← serialized results for paper
- tools/              ← helper scripts (conversion, evaluation)
- utils/              ← dataset registrations, evaluation metrics
- Patterns:
- “mask2former” follows a modular meta-architecture: backbone → pixel_decoder → transformer_decoder → head
- “harmonic” is a separate module that preprocesses or augments input with learned embeddings
- Config-driven: every experiment is wired up via YAML in configs/
- Scripts in tools/ and utils/ provide dataset registration, model conversion, evaluation

##### README for additional insights
- Models live in mask2former/
- Guide functions (embeddings) generated via harmonic/get_embeddings.py and stored in harmonic/guide_functions/
- Training entrypoint is guide_train_net.py
- Dependencies are managed by environment.yml (PyTorch, detectron2, etc.)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global External Datasets
    subgraph "Data Layer"
        direction TB
        Datasets(("External Datasets")):::data
        RegDatasets["Dataset Registration"]:::data
        click RegDatasets "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/utils/register_datasets.py"
        Mappers["Dataset Mappers"]:::data
        click Mappers "https://github.com/vios-s/gmt-leaf-ins-seg/tree/main/mask2former/data/dataset_mappers/"
        DatasetDefs["Dataset Definitions"]:::data
        click DatasetDefs "https://github.com/vios-s/gmt-leaf-ins-seg/tree/main/mask2former/data/datasets/"
    end

    subgraph "Configuration Manager"
        direction TB
        Configs["YAML Configs"]:::data
        click Configs "https://github.com/vios-s/gmt-leaf-ins-seg/tree/main/configs/guide_exp/"
    end

    subgraph "Embedding Module"
        direction TB
        EmbedGen["Embedding Generator"]:::embed
        click EmbedGen "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/harmonic/get_embeddings.py"
        PrecompGuides["Precomputed Guide Functions"]:::embed
        click PrecompGuides "https://github.com/vios-s/gmt-leaf-ins-seg/tree/main/harmonic/guide_functions/"
        HarmonicLib["Harmonic Embedding Library"]:::embed
        click HarmonicLib "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/harmonic/harmonic/embeddings.py"
    end

    subgraph "Model Module"
        direction TB
        subgraph Backbone
            Swin["Swin Transformer"]:::model
            click Swin "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/mask2former/modeling/backbone/swin.py"
        end
        subgraph "Pixel Decoder"
            FPN["Feature Pyramid Network"]:::model
            click FPN "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/mask2former/modeling/pixel_decoder/fpn.py"
            DefAttn["Deformable Attention"]:::model
            click DefAttn "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/mask2former/modeling/pixel_decoder/msdeformattn.py"
            GuideDefAttn["Guided Deformable Attention"]:::model
            click GuideDefAttn "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/mask2former/modeling/pixel_decoder/guide_msdeformattn.py"
            CUDAOps["Custom C/CUDA Ops"]:::model
            click CUDAOps "https://github.com/vios-s/gmt-leaf-ins-seg/tree/main/mask2former/modeling/pixel_decoder/ops/"
        end
        subgraph "Transformer Decoder"
            StdTransDec["Standard Transformer Decoder"]:::model
            click StdTransDec "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/mask2former/modeling/transformer_decoder/mask2former_transformer_decoder.py"
            GuideTransDec["Guided Transformer Decoder"]:::model
            click GuideTransDec "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/mask2former/modeling/transformer_decoder/guide_mask2former_transformer_decoder.py"
        end
        subgraph "Heads & Criteria"
            GuideHead["Guided Mask Forming Head"]:::model
            click GuideHead "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/mask2former/modeling/meta_arch/guide_mask_former_head.py"
            MaskFormerHead["MaskFormer Head"]:::model
            click MaskFormerHead "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/mask2former/modeling/meta_arch/mask_former_head.py"
            GuideLoss["Guided Loss Criterion"]:::model
            click GuideLoss "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/mask2former/modeling/guide_criterion.py"
        end
    end

    subgraph "Training Orchestrator"
        direction TB
        Trainer["Training Script"]:::train
        click Trainer "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/guide_train_net.py"
    end

    subgraph "Evaluation & Results"
        direction TB
        EvalUtils["Evaluation Utilities"]:::util
        click EvalUtils "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/utils/custom_coco_eval.py"
        click EvalUtils "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/utils/cvppp_evaluation.py"
        click EvalUtils "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/utils/lsc_eval_tools.py"
        EvalTools["Evaluation Tools"]:::util
        click EvalTools "https://github.com/vios-s/gmt-leaf-ins-seg/blob/main/tools/evaluate_coco_boundary_ap.py"
        Results["Results Storage"]:::util
        click Results "https://github.com/vios-s/gmt-leaf-ins-seg/tree/main/submission/results_in_paper/"
    end

    %% External Dependencies
    ExtLibs(("PyTorch / Detectron2 / Mask2Former")):::external
    ExtHarmonic(("Harmonic Embeddings Lib")):::external

    %% Connections
    Configs -->|loads YAML config| Trainer
    Configs -->|parameters| EmbedGen
    Datasets --> RegDatasets
    RegDatasets --> Mappers
    Mappers -->|image/mask batches| Trainer
    EmbedGen --> PrecompGuides
    PrecompGuides --> Trainer
    HarmonicLib --> EmbedGen
    Trainer -->|instantiate model| Swin
    Trainer -->|instantiate model| FPN
    Trainer -->|instantiate model| DefAttn
    Trainer -->|instantiate model| GuideDefAttn
    Trainer -->|instantiate model| CUDAOps
    Trainer -->|instantiate model| StdTransDec
    Trainer -->|instantiate model| GuideTransDec
    Trainer -->|instantiate model| GuideHead
    Trainer -->|instantiate model| MaskFormerHead
    Trainer -->|instantiate model| GuideLoss
    Trainer -->|forward pass: images+embeddings| Swin
    Swin --> FPN
    FPN --> DefAttn
    DefAttn --> GuideDefAttn
    GuideDefAttn --> StdTransDec
    StdTransDec --> GuideTransDec
    GuideTransDec --> GuideHead
    GuideHead --> GuideLoss
    GuideLoss --> Trainer
    Trainer -->|optimizer step| Trainer
    Trainer -->|run evaluation| EvalUtils
    EvalUtils --> EvalTools
    EvalTools --> Results

    %% Dependencies
    ExtLibs --> Trainer
    ExtLibs --> Swin
    ExtLibs --> FPN
    ExtLibs --> StdTransDec
    ExtHarmonic --> EmbedGen

    %% Styles
    classDef data fill:#e3fcec,stroke:#2ecc71,stroke-width:1px;
    classDef embed fill:#f3e8ff,stroke:#9b59b6,stroke-width:1px;
    classDef model fill:#e8f4fc,stroke:#2980b9,stroke-width:1px;
    classDef train fill:#fff0e6,stroke:#e67e22,stroke-width:1px;
    classDef util fill:#f2f2f2,stroke:#7f8c8d,stroke-width:1px;
    classDef external fill:#fcf3cf,stroke:#d35400,stroke-width:1px;
```
