How nmndeep/robust-segmentation works
1. Overall shape
- This is a config-driven semantic segmentation research repo focused on robustness under adversarial attack.
- The top-level flow is: YAML config → dataset/model construction → training or SEA-based evaluation → metrics/checkpoints.
2. Entry points
- run_train.sh and run_train_slurm.sh launch distributed training.
- run_infer.sh launches adversarial evaluation/inference.
- The Python entry scripts in tools/ are the main executables:
- tools/train_rob_seg.py for robust/adversarial training
- tools/train_segmenter.py for Segmenter-style training
- tools/infer.py for evaluation/inference
- tools/worse_only.py for selecting worst-case results
3. Configuration boundary
- The repo is organized around YAML configs in configs/.
- These configs encode the important architecture choices: dataset, backbone, segmentation head, checkpoint paths, and whether training is adversarial or clean.
- Examples in the repo target PASCAL VOC, ADE20K, UperNet + ConvNeXt, and Segmenter + ViT-S.
4. Data layer
- semseg/datasets/ contains dataset loaders and wrappers for:
- pascal_voc.py, pascal_aug.py
- ade.py, ade20k_new.py
- builder.py likely centralizes dataset creation from config.
- transform_util.py, dataset_wrappers.py, and distributed_sampler.py support preprocessing and multi-GPU batching.
- assets/data_ade20k.py appears to hold ADE20K metadata or label mappings.
5. Preprocessing and augmentation
- semseg/augmentations.py and semseg/datasets/transform_util.py define the image/label transformations used before model input.
- These are part of the training/evaluation boundary because the adversarial attacks operate on the transformed input space.
6. Model zoo
- semseg/models/ is split into reusable subparts:
- backbones/: ConvNeXt, PVT, ResNet/DDCAT, ViT encoder, and image encoder utilities
- heads/: FCN, FPN, UPerNet, and Segmenter decoder heads
- modules/: supporting blocks such as PPM and PSA
- layers/: common layers and initialization helpers
- Higher-level model wrappers include segmenter.py, uperforseg.py, and ddcat_psp.py, which assemble backbone + head into end-to-end segmentation models.
7. Robustness-specific components
- semseg/attacker.py is a key boundary: it implements adversarial perturbation generation and evaluation logic used during PIR-AT training and SEA inference.
- The README describes SEA as an ensemble of three attacks:
- Mask-Cross-Entropy-Balanced
- Mask-Cross-Entropy
- Jensen-Shannon Divergence
- This means evaluation is not just standard accuracy/mIoU; it explicitly includes worst-case adversarial scoring.
8. Training loop support
- semseg/losses.py, semseg/metrics.py, semseg/optimizers.py, semseg/schedulers.py, and semseg/layer_decay.py provide the optimization stack.
- These modules separate:
- prediction loss computation
- segmentation metrics
- optimizer and LR schedule setup
- layer-wise decay policies, especially relevant for transformer backbones
9. Evaluation outputs
- semseg/utils/segmenter_eval.py and semseg/val.py handle validation/evaluation orchestration.
- semseg/utils/visualize.py likely supports qualitative output or debug visualization.
- The repo’s main reported metrics are clean mIoU, adversarial mIoU, and aACC under SEA.
10. External model and backbone provenance
- The README notes that some robust pretrained backbones come from the Revisiting-AT project.
- The architecture therefore combines this repo’s segmentation heads and attack/eval logic with externally sourced robust backbone checkpoints.
11. Multi-GPU / distributed execution
- semseg/datasets/distributed_sampler.py and the SLURM/non-SLURM launcher scripts indicate the repo is designed for distributed training and evaluation.
- This matters architecturally because the training pipeline is not single-process; data loading and iteration are coordinated across GPUs.
12. What to draw in an architecture diagram
- A useful diagram would show:
- Config files at the top
- branching into training and SEA inference/evaluation
- shared dataset/augmentation and model zoo layers underneath
- a separate attacker/metric path for robust evaluation
- optimizer/scheduler/loss only on the training path
- checkpoints feeding both evaluation and resumed training
Main entry points: train_rob_seg.py Rob seg train · builder.py Dataset builder · infer.py Inference · segmenter.py Segmenter
Components 24
Execution
Launch scripts and Python entrypoints.
- Train script · Shell entry for distributed training.
run_train.sh - SLURM script · Cluster launch wrapper for training.
run_train_slurm.sh - Infer script · Shell entry for adversarial evaluation.
run_infer.sh - Rob seg train · Main adversarial/robust segmentation trainer.
tools/train_rob_seg.py - Segmenter train · Training entry for Segmenter-style models.
tools/train_segmenter.py - Inference · Evaluation and adversarial inference driver.
tools/infer.py - Worst-case select · Selects worst-case results for SEA-style scoring.
tools/worse_only.py
Data
Dataset loading, transforms, and batching.
- Dataset builder · Constructs datasets and loaders from config.
semseg/datasets/builder.py - Dataset loaders · PASCAL VOC and ADE20K dataset implementations.
- Transforms · Preprocessing, augmentation, and label/image transforms.
semseg/augmentations.py - Distributed sampler · Multi-GPU data partitioning and iteration.
semseg/datasets/distributed_sampler.py
Models
Backbones, heads, and assembled segmentation models.
- Backbones · ConvNeXt, PVT, ResNet/DDCAT, and ViT encoders.
- Seg heads · FCN, FPN, UPerNet, and Segmenter decoder heads.
- Segmenter · End-to-end Segmenter assembly.
semseg/models/segmenter.py - UPerForSeg · UPerNet-style segmentation assembly.
semseg/models/uperforseg.py - DDCAT PSP · Robust segmentation model with PSP-style head.
semseg/models/ddcat_psp.py
Learn & Eval
Training, adversarial attack, metrics, and outputs.
- Attacker · Generates adversarial perturbations for training and SEA evaluation.
semseg/attacker.py - Losses · Segmentation loss computation.
semseg/losses.py - Optim stack · Optimizers, schedulers, and layer decay policies.
semseg/optimizers.py - Metrics · mIoU and related segmentation metrics.
semseg/metrics.py - Validation · Validation and evaluation orchestration.
semseg/val.py - SEA eval · Segmenter evaluation utilities used for SEA and adversarial scoring.
semseg/utils/segmenter_eval.py - Visualize · Qualitative result rendering and debugging views.
semseg/utils/visualize.py
Other components
- YAML configs · Defines dataset, backbone, head, checkpoints, and robust/clean mode.
Connections 34
- Train script → Rob seg train: launches
- SLURM script → Rob seg train: launches
- Infer script → Inference: launches
- YAML configs → Dataset builder: drives
- YAML configs → Segmenter: selects
- YAML configs → UPerForSeg: selects
- YAML configs → DDCAT PSP: selects
- Dataset builder → Dataset loaders: creates
- Dataset builder → Transforms: applies
- Dataset loaders → Distributed sampler: batches
- Transforms → Attacker: input space
- Rob seg train → Dataset builder: loads data
- Rob seg train → Segmenter: builds model
- Rob seg train → Attacker: adversarial train
- Rob seg train → Losses: optimizes
- Rob seg train → Optim stack: updates
- Segmenter train → Dataset builder: loads data
- Segmenter train → Segmenter: builds model
- Segmenter train → Losses: optimizes
- Segmenter train → Optim stack: updates
- Inference → Dataset builder: loads data
- Inference → Attacker: SEA attacks
- Inference → SEA eval: scores
- Inference → Metrics: reports
- Validation → Metrics: computes
- Validation → Visualize: renders
- SEA eval → Metrics: worst-case score
- Worst-case select → SEA eval: selects
- Segmenter → Backbones: assembles
- Segmenter → Seg heads: assembles
- UPerForSeg → Backbones: assembles
- UPerForSeg → Seg heads: assembles
- DDCAT PSP → Backbones: assembles
- DDCAT PSP → Seg heads: assembles