# nmndeep/robust-segmentation architecture

> Architecture diagram of the GitHub repository nmndeep/robust-segmentation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nmndeep/robust-segmentation
- Repository: https://github.com/nmndeep/robust-segmentation

## Overview

1. **Overall shape**
   - This is a **config-driven semantic segmentation research repo** focused on **robustness under adversarial attack**.
   - The top-level flow is: **YAML config → dataset/model construction → training or SEA-based evaluation → metrics/checkpoints**.

2. **Entry points**
   - `run_train.sh` and `run_train_slurm.sh` launch distributed training.
   - `run_infer.sh` launches adversarial evaluation/inference.
   - The Python entry scripts in `tools/` are the main executables:
     - `tools/train_rob_seg.py` for robust/adversarial training
     - `tools/train_segmenter.py` for Segmenter-style training
     - `tools/infer.py` for evaluation/inference
     - `tools/worse_only.py` for selecting worst-case results

3. **Configuration boundary**
   - The repo is organized around YAML configs in `configs/`.
   - These configs encode the important architecture choices: dataset, backbone, segmentation head, checkpoint paths, and whether training is adversarial or clean.
   - Examples in the repo target **PASCAL VOC**, **ADE20K**, **UperNet + ConvNeXt**, and **Segmenter + ViT-S**.

4. **Data layer**
   - `semseg/datasets/` contains dataset loaders and wrappers for:
     - `pascal_voc.py`, `pascal_aug.py`
     - `ade.py`, `ade20k_new.py`
   - `builder.py` likely centralizes dataset creation from config.
   - `transform_util.py`, `dataset_wrappers.py`, and `distributed_sampler.py` support preprocessing and multi-GPU batching.
   - `assets/data_ade20k.py` appears to hold ADE20K metadata or label mappings.

5. **Preprocessing and augmentation**
   - `semseg/augmentations.py` and `semseg/datasets/transform_util.py` define the image/label transformations used before model input.
   - These are part of the training/evaluation boundary because the adversarial attacks operate on the transformed input space.

6. **Model zoo**
   - `semseg/models/` is split into reusable subparts:
     - `backbones/`: ConvNeXt, PVT, ResNet/DDCAT, ViT encoder, and image encoder utilities
     - `heads/`: FCN, FPN, UPerNet, and Segmenter decoder heads
     - `modules/`: supporting blocks such as PPM and PSA
     - `layers/`: common layers and initialization helpers
   - Higher-level model wrappers include `segmenter.py`, `uperforseg.py`, and `ddcat_psp.py`, which assemble backbone + head into end-to-end segmentation models.

7. **Robustness-specific components**
   - `semseg/attacker.py` is a key boundary: it implements adversarial perturbation generation and evaluation logic used during PIR-AT training and SEA inference.
   - The README describes SEA as an ensemble of three attacks:
     - Mask-Cross-Entropy-Balanced
     - Mask-Cross-Entropy
     - Jensen-Shannon Divergence
   - This means evaluation is not just standard accuracy/mIoU; it explicitly includes **worst-case adversarial scoring**.

8. **Training loop support**
   - `semseg/losses.py`, `semseg/metrics.py`, `semseg/optimizers.py`, `semseg/schedulers.py`, and `semseg/layer_decay.py` provide the optimization stack.
   - These modules separate:
     - prediction loss computation
     - segmentation metrics
     - optimizer and LR schedule setup
     - layer-wise decay policies, especially relevant for transformer backbones

9. **Evaluation outputs**
   - `semseg/utils/segmenter_eval.py` and `semseg/val.py` handle validation/evaluation orchestration.
   - `semseg/utils/visualize.py` likely supports qualitative output or debug visualization.
   - The repo’s main reported metrics are **clean mIoU**, **adversarial mIoU**, and **aACC** under SEA.

10. **External model and backbone provenance**
   - The README notes that some robust pretrained backbones come from the **Revisiting-AT** project.
   - The architecture therefore combines this repo’s segmentation heads and attack/eval logic with externally sourced robust backbone checkpoints.

11. **Multi-GPU / distributed execution**
   - `semseg/datasets/distributed_sampler.py` and the SLURM/non-SLURM launcher scripts indicate the repo is designed for **distributed training and evaluation**.
   - This matters architecturally because the training pipeline is not single-process; data loading and iteration are coordinated across GPUs.

12. **What to draw in an architecture diagram**
   - A useful diagram would show:
     - **Config files** at the top
     - branching into **training** and **SEA inference/evaluation**
     - shared **dataset/augmentation** and **model zoo** layers underneath
     - a separate **attacker/metric** path for robust evaluation
     - **optimizer/scheduler/loss** only on the training path
     - **checkpoints** feeding both evaluation and resumed training

Main entry points: [`tools/train_rob_seg.py`](https://github.com/nmndeep/robust-segmentation/blob/main/tools/train_rob_seg.py) (Rob seg train), [`semseg/datasets/builder.py`](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/datasets/builder.py) (Dataset builder), [`tools/infer.py`](https://github.com/nmndeep/robust-segmentation/blob/main/tools/infer.py) (Inference), [`semseg/models/segmenter.py`](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/models/segmenter.py) (Segmenter)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_exec["Execution"]
  node_run_train["Train script<br/>launcher<br/>[run_train.sh]"]
  node_run_train_slurm["SLURM script<br/>launcher<br/>[run_train_slurm.sh]"]
  node_run_infer["Infer script<br/>launcher<br/>[run_infer.sh]"]
  node_train_rob_seg["Rob seg train<br/>entrypoint<br/>[train_rob_seg.py]"]
  node_train_segmenter["Segmenter train<br/>entrypoint<br/>[train_segmenter.py]"]
  node_infer_tool["Inference<br/>entrypoint<br/>[infer.py]"]
  node_worse_only["Worst-case select<br/>entrypoint<br/>[worse_only.py]"]
end

subgraph group_data["Data"]
  node_builder["Dataset builder<br/>factory<br/>[builder.py]"]
  node_datasets[("Dataset loaders<br/>datasets")]
  node_transforms["Transforms<br/>augmentation<br/>[augmentations.py]"]
  node_sampler["Distributed sampler<br/>batching"]
end

subgraph group_model["Models"]
  node_backbones{{"Backbones<br/>feature extractors"}}
  node_heads{{"Seg heads<br/>decoders"}}
  node_segmenter["Segmenter<br/>model wrapper<br/>[segmenter.py]"]
  node_uperforseg["UPerForSeg<br/>model wrapper<br/>[uperforseg.py]"]
  node_ddcat_psp["DDCAT PSP<br/>model wrapper<br/>[ddcat_psp.py]"]
end

subgraph group_learn["Learn & Eval"]
  node_attacker{{"Attacker<br/>robust eval<br/>[attacker.py]"}}
  node_losses["Losses<br/>objective<br/>[losses.py]"]
  node_optim["Optim stack<br/>optimization<br/>[optimizers.py]"]
  node_metrics["Metrics<br/>evaluation<br/>[metrics.py]"]
  node_val["Validation<br/>eval loop<br/>[val.py]"]
  node_segmenter_eval["SEA eval<br/>robust eval<br/>[segmenter_eval.py]"]
  node_visualize["Visualize<br/>output<br/>[visualize.py]"]
end

node_configs["YAML configs<br/>config set"]

node_run_train -->|"launches"| node_train_rob_seg
node_run_train_slurm -->|"launches"| node_train_rob_seg
node_run_infer -->|"launches"| node_infer_tool
node_configs -->|"drives"| node_builder
node_configs -->|"selects"| node_segmenter
node_configs -->|"selects"| node_uperforseg
node_configs -->|"selects"| node_ddcat_psp
node_builder -->|"creates"| node_datasets
node_builder -->|"applies"| node_transforms
node_datasets -->|"batches"| node_sampler
node_transforms -.->|"input space"| node_attacker
node_train_rob_seg -->|"loads data"| node_builder
node_train_rob_seg -->|"builds model"| node_segmenter
node_train_rob_seg -->|"adversarial train"| node_attacker
node_train_rob_seg -->|"optimizes"| node_losses
node_train_rob_seg -->|"updates"| node_optim
node_train_segmenter -->|"loads data"| node_builder
node_train_segmenter -->|"builds model"| node_segmenter
node_train_segmenter -->|"optimizes"| node_losses
node_train_segmenter -->|"updates"| node_optim
node_infer_tool -->|"loads data"| node_builder
node_infer_tool -->|"SEA attacks"| node_attacker
node_infer_tool -->|"scores"| node_segmenter_eval
node_infer_tool -->|"reports"| node_metrics
node_val -->|"computes"| node_metrics
node_val -.->|"renders"| node_visualize
node_segmenter_eval -->|"worst-case score"| node_metrics
node_worse_only -->|"selects"| node_segmenter_eval
node_segmenter -->|"assembles"| node_backbones
node_segmenter -->|"assembles"| node_heads
node_uperforseg -->|"assembles"| node_backbones
node_uperforseg -->|"assembles"| node_heads
node_ddcat_psp -->|"assembles"| node_backbones
node_ddcat_psp -->|"assembles"| node_heads

click node_run_train "https://github.com/nmndeep/robust-segmentation/blob/main/run_train.sh"
click node_run_train_slurm "https://github.com/nmndeep/robust-segmentation/blob/main/run_train_slurm.sh"
click node_run_infer "https://github.com/nmndeep/robust-segmentation/blob/main/run_infer.sh"
click node_train_rob_seg "https://github.com/nmndeep/robust-segmentation/blob/main/tools/train_rob_seg.py"
click node_train_segmenter "https://github.com/nmndeep/robust-segmentation/blob/main/tools/train_segmenter.py"
click node_infer_tool "https://github.com/nmndeep/robust-segmentation/blob/main/tools/infer.py"
click node_worse_only "https://github.com/nmndeep/robust-segmentation/blob/main/tools/worse_only.py"
click node_builder "https://github.com/nmndeep/robust-segmentation/blob/main/semseg/datasets/builder.py"
click node_transforms "https://github.com/nmndeep/robust-segmentation/blob/main/semseg/augmentations.py"
click node_sampler "https://github.com/nmndeep/robust-segmentation/blob/main/semseg/datasets/distributed_sampler.py"
click node_segmenter "https://github.com/nmndeep/robust-segmentation/blob/main/semseg/models/segmenter.py"
click node_uperforseg "https://github.com/nmndeep/robust-segmentation/blob/main/semseg/models/uperforseg.py"
click node_ddcat_psp "https://github.com/nmndeep/robust-segmentation/blob/main/semseg/models/ddcat_psp.py"
click node_attacker "https://github.com/nmndeep/robust-segmentation/blob/main/semseg/attacker.py"
click node_losses "https://github.com/nmndeep/robust-segmentation/blob/main/semseg/losses.py"
click node_optim "https://github.com/nmndeep/robust-segmentation/blob/main/semseg/optimizers.py"
click node_metrics "https://github.com/nmndeep/robust-segmentation/blob/main/semseg/metrics.py"
click node_val "https://github.com/nmndeep/robust-segmentation/blob/main/semseg/val.py"
click node_segmenter_eval "https://github.com/nmndeep/robust-segmentation/blob/main/semseg/utils/segmenter_eval.py"
click node_visualize "https://github.com/nmndeep/robust-segmentation/blob/main/semseg/utils/visualize.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_run_train,node_run_train_slurm,node_run_infer,node_train_rob_seg,node_train_segmenter,node_infer_tool,node_worse_only toneBlue
class node_builder,node_datasets,node_transforms,node_sampler toneAmber
class node_backbones,node_heads,node_segmenter,node_uperforseg,node_ddcat_psp toneMint
class node_attacker,node_losses,node_optim,node_metrics,node_val,node_segmenter_eval,node_visualize toneRose
class node_configs toneNeutral
```

## Components

### Execution

Launch scripts and Python entrypoints.

- [Train script](https://github.com/nmndeep/robust-segmentation/blob/main/run_train.sh): launcher · Shell entry for distributed training. · `run_train.sh`
- [SLURM script](https://github.com/nmndeep/robust-segmentation/blob/main/run_train_slurm.sh): launcher · Cluster launch wrapper for training. · `run_train_slurm.sh`
- [Infer script](https://github.com/nmndeep/robust-segmentation/blob/main/run_infer.sh): launcher · Shell entry for adversarial evaluation. · `run_infer.sh`
- [Rob seg train](https://github.com/nmndeep/robust-segmentation/blob/main/tools/train_rob_seg.py): entrypoint · Main adversarial/robust segmentation trainer. · `tools/train_rob_seg.py`
- [Segmenter train](https://github.com/nmndeep/robust-segmentation/blob/main/tools/train_segmenter.py): entrypoint · Training entry for Segmenter-style models. · `tools/train_segmenter.py`
- [Inference](https://github.com/nmndeep/robust-segmentation/blob/main/tools/infer.py): entrypoint · Evaluation and adversarial inference driver. · `tools/infer.py`
- [Worst-case select](https://github.com/nmndeep/robust-segmentation/blob/main/tools/worse_only.py): entrypoint · Selects worst-case results for SEA-style scoring. · `tools/worse_only.py`

### Data

Dataset loading, transforms, and batching.

- [Dataset builder](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/datasets/builder.py): factory · Constructs datasets and loaders from config. · `semseg/datasets/builder.py`
- **Dataset loaders**: datasets · PASCAL VOC and ADE20K dataset implementations.
- [Transforms](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/augmentations.py): augmentation · Preprocessing, augmentation, and label/image transforms. · `semseg/augmentations.py`
- [Distributed sampler](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/datasets/distributed_sampler.py): batching · Multi-GPU data partitioning and iteration. · `semseg/datasets/distributed_sampler.py`

### Models

Backbones, heads, and assembled segmentation models.

- **Backbones**: feature extractors · ConvNeXt, PVT, ResNet/DDCAT, and ViT encoders.
- **Seg heads**: decoders · FCN, FPN, UPerNet, and Segmenter decoder heads.
- [Segmenter](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/models/segmenter.py): model wrapper · End-to-end Segmenter assembly. · `semseg/models/segmenter.py`
- [UPerForSeg](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/models/uperforseg.py): model wrapper · UPerNet-style segmentation assembly. · `semseg/models/uperforseg.py`
- [DDCAT PSP](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/models/ddcat_psp.py): model wrapper · Robust segmentation model with PSP-style head. · `semseg/models/ddcat_psp.py`

### Learn & Eval

Training, adversarial attack, metrics, and outputs.

- [Attacker](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/attacker.py): robust eval · Generates adversarial perturbations for training and SEA evaluation. · `semseg/attacker.py`
- [Losses](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/losses.py): objective · Segmentation loss computation. · `semseg/losses.py`
- [Optim stack](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/optimizers.py): optimization · Optimizers, schedulers, and layer decay policies. · `semseg/optimizers.py`
- [Metrics](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/metrics.py): evaluation · mIoU and related segmentation metrics. · `semseg/metrics.py`
- [Validation](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/val.py): eval loop · Validation and evaluation orchestration. · `semseg/val.py`
- [SEA eval](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/utils/segmenter_eval.py): robust eval · Segmenter evaluation utilities used for SEA and adversarial scoring. · `semseg/utils/segmenter_eval.py`
- [Visualize](https://github.com/nmndeep/robust-segmentation/blob/main/semseg/utils/visualize.py): output · Qualitative result rendering and debugging views. · `semseg/utils/visualize.py`

### Other components

- **YAML configs**: config set · Defines dataset, backbone, head, checkpoints, and robust/clean mode.

## Connections

- Train script → Rob seg train: launches
- SLURM script → Rob seg train: launches
- Infer script → Inference: launches
- YAML configs → Dataset builder: drives
- YAML configs → Segmenter: selects
- YAML configs → UPerForSeg: selects
- YAML configs → DDCAT PSP: selects
- Dataset builder → Dataset loaders: creates
- Dataset builder → Transforms: applies
- Dataset loaders → Distributed sampler: batches
- Transforms → Attacker: input space
- Rob seg train → Dataset builder: loads data
- Rob seg train → Segmenter: builds model
- Rob seg train → Attacker: adversarial train
- Rob seg train → Losses: optimizes
- Rob seg train → Optim stack: updates
- Segmenter train → Dataset builder: loads data
- Segmenter train → Segmenter: builds model
- Segmenter train → Losses: optimizes
- Segmenter train → Optim stack: updates
- Inference → Dataset builder: loads data
- Inference → Attacker: SEA attacks
- Inference → SEA eval: scores
- Inference → Metrics: reports
- Validation → Metrics: computes
- Validation → Visualize: renders
- SEA eval → Metrics: worst-case score
- Worst-case select → SEA eval: selects
- Segmenter → Backbones: assembles
- Segmenter → Seg heads: assembles
- UPerForSeg → Backbones: assembles
- UPerForSeg → Seg heads: assembles
- DDCAT PSP → Backbones: assembles
- DDCAT PSP → Seg heads: assembles
