# csailvision/semantic-segmentation-pytorch architecture

> Architecture diagram of the GitHub repository csailvision/semantic-segmentation-pytorch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/csailvision/semantic-segmentation-pytorch
- Repository: https://github.com/csailvision/semantic-segmentation-pytorch

## Overview

1. **Overall purpose**
This repository is a PyTorch semantic-segmentation pipeline for the ADE20K scene-parsing dataset. The core runtime path is: load image/label data, build an encoder-decoder segmentation model, train across one or more GPUs, then evaluate or run inference to produce per-pixel class maps and optionally colorized visualizations.

2. **Top-level execution entry points**
The main orchestration lives in `train.py`, `eval.py`, `eval_multipro.py`, and `test.py`. These scripts are the boundaries between the framework and the model/data internals: they parse config, assemble datasets and networks, restore checkpoints, and drive training, validation, or single-image/batch inference. `demo_test.sh` is a convenience wrapper for a preconfigured inference demo.

3. **Configuration subsystem**
Configuration is centralized in `config/` and `mit_semseg/config/defaults.py`, with YAML presets such as `ade20k-resnet50-upernet.yaml` and `ade20k-hrnetv2.yaml`. This is the main architecture control plane: model choice, dataset paths, training hyperparameters, evaluation behavior, GPU selection, and visualization options are set here rather than hardcoded in the scripts.

4. **Dataset and file-format boundary**
`mit_semseg/dataset.py` and the dataset utilities under `mit_semseg/lib/utils/data/` define how ADE20K assets are read and batched. The repo expects ADE20K metadata files like `training.odgt`, `validation.odgt`, `object150_info.csv`, and `color150.mat`, which support sample indexing, class metadata, and label-to-color rendering. The data boundary is image files plus segmentation labels, translated into tensors for the model.

5. **Model family and architecture split**
`mit_semseg/models/` contains the backbone and decoder implementations. The repo explicitly separates **encoders** and **decoders**: encoders include MobileNetV2, ResNet variants, ResNeXt, and HRNet; decoders include C1, PPM, and UPerNet variants with optional deep supervision. `models.py` likely acts as the factory/assembly layer that combines these pieces into a full segmentation network.

6. **Segmentation computation path**
The forward path is the typical semantic-segmentation flow: backbone extracts multi-scale feature maps, decoder aggregates them into class logits, and the result is upsampled to image resolution. The README’s supported variants show two major architectural styles: pyramid pooling–based decoders for global context and UPerNet-style FPN/PPM fusion for multi-scale features. Deep supervision is an optional training-time branch that adds auxiliary losses.

7. **Training and evaluation loops**
Training is handled as a multi-GPU job with checkpointing, while evaluation computes metrics such as mean IoU and pixel accuracy against ADE20K validation labels. `eval_multipro.py` is the multi-device validation path, and `test.py` is the inference path for one image or a folder. The outputs are segmentation masks, with optional visualization overlays saved to disk.

8. **Parallelism and distributed execution**
A notable architectural feature is custom multi-GPU support in `mit_semseg/lib/nn/parallel/` and `mit_semseg/lib/utils/data/distributed.py`. The project reimplements `DataParallel` and adapts the dataloader so each GPU can receive differently sized images, preserving aspect ratio during training. This is not just an optimization; it is a structural choice that influences batching, worker seeding, and how training data is partitioned.

9. **Normalization infrastructure**
`mit_semseg/lib/nn/modules/batchnorm.py` implements synchronized batch normalization across devices. This matters because segmentation often uses small per-GPU batch sizes, so batch statistics must be aggregated to keep training stable. The presence of custom batchnorm tests under `mit_semseg/lib/nn/modules/tests/` indicates this is a first-class infrastructure component, not a model detail.

10. **Utility layer**
`mit_semseg/lib/utils/th.py` and related helpers provide glue code for PyTorch-specific tasks, likely including tensor/device handling, checkpoint serialization, and image preprocessing/postprocessing. `mit_semseg/utils.py` sits near the top of the package and likely contains shared routines used by scripts, evaluation, and visualization.

11. **Packaging and integration boundary**
`setup.py` and the `mit_semseg` package layout make the code installable as a library, not just runnable scripts. The README shows the intended programmatic integration surface: `cfg`, `TestDataset`, `ModelBuilder`, and `SegmentationModule`. That means the repo has both a CLI-oriented workflow and a reusable library boundary.

12. **Notebooks and demo assets**
`notebooks/` provides a self-contained interactive path for experimentation, including a demo segmenter notebook and copied package/data resources. This is a separate UX layer for exploration and presentation rather than a different runtime architecture. `teaser/` and the notebook assets support result visualization and documentation.

13. **External dependencies and operational assumptions**
The architecture depends on PyTorch, CUDA-enabled multi-GPU training, and common scientific Python packages. The README emphasizes 4+ GPUs for training and a single GPU for inference, which reflects the repo’s design point: high-throughput segmentation research and benchmarking rather than lightweight CPU deployment.

14. **End-to-end data flow**
At a high level: ADE20K metadata and image/label files are loaded through the dataset layer, config selects a backbone/decoder pair, the model runs on one or more GPUs with synchronized normalization and custom batching, checkpoints are saved during training, and evaluation/inference scripts reload those checkpoints to produce class masks, metrics, and optional colored outputs.

Main entry points: [`train.py`](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/train.py) (train.py), [`eval.py`](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/eval.py) (eval.py), [`mit_semseg/dataset.py`](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/dataset.py) (dataset.py), [`mit_semseg/models/models.py`](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/models/models.py) (models.py)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry points"]
  node_train_py["train.py<br/>training CLI<br/>[train.py]"]
  node_eval_py["eval.py<br/>eval CLI<br/>[eval.py]"]
  node_eval_mp_py["eval_multipro.py<br/>multi-GPU eval<br/>[eval_multipro.py]"]
  node_test_py["test.py<br/>inference CLI<br/>[test.py]"]
  node_demo_sh["demo_test.sh<br/>demo wrapper<br/>[demo_test.sh]"]
end

subgraph group_cfg["Configuration"]
  node_defaults_py["defaults.py<br/>config defaults<br/>[defaults.py]"]
  node_yaml_cfgs["YAML presets<br/>experiment configs"]
end

subgraph group_data["Data boundary"]
  node_dataset_py[("dataset.py<br/>dataset adapter<br/>[dataset.py]")]
  node_data_utils[("data utils<br/>loader utils<br/>[dataloader.py]")]
  node_ade20k_meta["ADE20K meta<br/>dataset metadata<br/>[object150_info.csv]"]
end

subgraph group_model["Model stack"]
  node_model_factory["models.py<br/>model factory<br/>[models.py]"]
  node_backbones["Backbones<br/>encoders<br/>[resnet.py]"]
  node_decoders["Decoders<br/>[utils.py]"]
end

subgraph group_runtime["Runtime infra"]
  node_syncbn["SyncBN<br/>normalization<br/>[batchnorm.py]"]
  node_data_parallel["DataParallel<br/>multi-gpu runtime<br/>[data_parallel.py]"]
  node_distributed["distributed.py<br/>distributed data<br/>[distributed.py]"]
  node_th_utils["th.py<br/>torch utils<br/>[th.py]"]
end

subgraph group_api["Library API"]
  node_package_api["mit_semseg<br/>library package<br/>[__init__.py]"]
end

subgraph group_demo["Demo assets"]
  node_nb_demo["Demo notebook<br/>notebook demo"]
  node_teaser_assets["Teaser assets<br/>visualization assets"]
end

node_train_py -->|"loads config"| node_defaults_py
node_train_py -->|"overrides"| node_yaml_cfgs
node_eval_py -->|"loads config"| node_defaults_py
node_eval_mp_py -->|"loads config"| node_defaults_py
node_test_py -->|"loads config"| node_defaults_py
node_train_py -->|"builds data"| node_dataset_py
node_eval_py -->|"builds data"| node_dataset_py
node_test_py -->|"builds data"| node_dataset_py
node_dataset_py -->|"reads metadata"| node_ade20k_meta
node_dataset_py -->|"uses loaders"| node_data_utils
node_data_utils -->|"partitions work"| node_distributed
node_train_py -->|"builds model"| node_model_factory
node_eval_py -->|"loads model"| node_model_factory
node_test_py -->|"loads model"| node_model_factory
node_model_factory -->|"selects encoder"| node_backbones
node_model_factory -->|"selects decoder"| node_decoders
node_backbones -->|"normalizes"| node_syncbn
node_train_py -->|"runs on GPUs"| node_data_parallel
node_eval_mp_py -->|"runs on GPUs"| node_data_parallel
node_train_py -->|"uses glue"| node_th_utils
node_eval_py -->|"uses glue"| node_th_utils
node_test_py -->|"uses glue"| node_th_utils
node_train_py -->|"imports package"| node_package_api
node_eval_py -->|"imports package"| node_package_api
node_nb_demo -->|"explores API"| node_package_api
node_nb_demo -->|"renders examples"| node_teaser_assets

click node_train_py "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/train.py"
click node_eval_py "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/eval.py"
click node_eval_mp_py "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/eval_multipro.py"
click node_test_py "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/test.py"
click node_demo_sh "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/demo_test.sh"
click node_defaults_py "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/config/defaults.py"
click node_yaml_cfgs "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/config/ade20k-resnet50-upernet.yaml"
click node_dataset_py "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/dataset.py"
click node_data_utils "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/lib/utils/data/dataloader.py"
click node_ade20k_meta "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/data/object150_info.csv"
click node_model_factory "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/models/models.py"
click node_backbones "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/models/resnet.py"
click node_decoders "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/models/utils.py"
click node_syncbn "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/lib/nn/modules/batchnorm.py"
click node_data_parallel "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/lib/nn/parallel/data_parallel.py"
click node_distributed "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/lib/utils/data/distributed.py"
click node_th_utils "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/lib/utils/th.py"
click node_package_api "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/__init__.py"
click node_nb_demo "https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/notebooks/DemoSegmenter.ipynb"
click node_teaser_assets "https://github.com/csailvision/semantic-segmentation-pytorch/tree/master/teaser"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_train_py,node_eval_py,node_eval_mp_py,node_test_py,node_demo_sh,node_nb_demo,node_teaser_assets toneBlue
class node_defaults_py,node_yaml_cfgs toneAmber
class node_dataset_py,node_data_utils,node_ade20k_meta toneMint
class node_model_factory,node_backbones,node_decoders toneRose
class node_syncbn,node_data_parallel,node_distributed,node_th_utils toneIndigo
class node_package_api toneTeal
```

## Components

### Entry points

CLI scripts that start training, evaluation, and demo inference.

- [train.py](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/train.py): training CLI · Starts distributed training, checkpointing, and validation. · `train.py`
- [eval.py](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/eval.py): eval CLI · Runs single-process evaluation against validation data. · `eval.py`
- [eval_multipro.py](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/eval_multipro.py): multi-GPU eval · Validation path for multi-device inference and metrics. · `eval_multipro.py`
- [test.py](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/test.py): inference CLI · Runs single-image or folder inference and writes masks. · `test.py`
- [demo_test.sh](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/demo_test.sh): demo wrapper · Convenience shell wrapper for a preconfigured demo run. · `demo_test.sh`

### Configuration

Central config and YAML presets that select datasets, models, and runtime options.

- [defaults.py](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/config/defaults.py): config defaults · Defines the base configuration schema and defaults. · `mit_semseg/config/defaults.py`
- [YAML presets](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/config/ade20k-resnet50-upernet.yaml): experiment configs · Dataset- and model-specific overrides such as ResNet, HRNet, and UPerNet. · `config/ade20k-resnet50-upernet.yaml`

### Data boundary

ADE20K assets and loaders that turn files into tensors and labels.

- [dataset.py](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/dataset.py): dataset adapter · Top-level dataset interface for ADE20K loading and batching. · `mit_semseg/dataset.py`
- [data utils](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/lib/utils/data/dataloader.py): loader utils · Dataset, dataloader, sampler, and distributed helpers. · `mit_semseg/lib/utils/data/dataloader.py`
- [ADE20K meta](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/data/object150_info.csv): dataset metadata · Class names, color map, and split index files. · `data/object150_info.csv`

### Model stack

Backbones, decoders, and assembly code for semantic segmentation networks.

- [models.py](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/models/models.py): model factory · Builds complete segmentation networks from encoder and decoder parts. · `mit_semseg/models/models.py`
- [Backbones](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/models/resnet.py): encoders · Feature extractors including ResNet, ResNeXt, MobileNetV2, and HRNet. · `mit_semseg/models/resnet.py`
- [Decoders](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/models/utils.py): decoders · Segmentation heads such as C1, PPM, and UPerNet. · `mit_semseg/models/utils.py`

### Runtime infra

Custom parallelism, distributed loading, and normalization support.

- [SyncBN](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/lib/nn/modules/batchnorm.py): normalization · Synchronized batch normalization across devices. · `mit_semseg/lib/nn/modules/batchnorm.py`
- [DataParallel](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/lib/nn/parallel/data_parallel.py): multi-gpu runtime · Custom parallel execution for segmentation training and inference. · `mit_semseg/lib/nn/parallel/data_parallel.py`
- [distributed.py](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/lib/utils/data/distributed.py): distributed data · Multi-process data partitioning and worker coordination. · `mit_semseg/lib/utils/data/distributed.py`
- [th.py](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/lib/utils/th.py): torch utils · Shared PyTorch glue for tensors, checkpoints, and preprocessing. · `mit_semseg/lib/utils/th.py`

### Library API

Reusable package surface for external integration.

- [mit_semseg](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/mit_semseg/__init__.py): library package · Installable package boundary for scripts and external code. · `mit_semseg/__init__.py`

### Demo assets

Notebook and presentation assets for interactive exploration.

- [Demo notebook](https://github.com/csailvision/semantic-segmentation-pytorch/blob/master/notebooks/DemoSegmenter.ipynb): notebook demo · Interactive segmentation demo and experimentation workflow. · `notebooks/DemoSegmenter.ipynb`
- [Teaser assets](https://github.com/csailvision/semantic-segmentation-pytorch/tree/master/teaser): visualization assets · Presentation and result imagery used for documentation and demos. · `teaser`

## Connections

- train.py → defaults.py: loads config · Training starts from the shared configuration schema.
- train.py → YAML presets: overrides · Experiment YAML selects model, dataset, and hyperparameters.
- eval.py → defaults.py: loads config · Evaluation uses the same config control plane.
- eval_multipro.py → defaults.py: loads config · Multi-process evaluation reads the shared defaults.
- test.py → defaults.py: loads config · Inference resolves runtime options from config.
- train.py → dataset.py: builds data · Training constructs ADE20K loaders through the dataset boundary.
- eval.py → dataset.py: builds data · Evaluation reuses the same dataset adapter.
- test.py → dataset.py: builds data · Inference loads images through the dataset adapter.
- dataset.py → ADE20K meta: reads metadata · Dataset loading depends on split files and class metadata.
- dataset.py → data utils: uses loaders · Batching and sampling are delegated to utility loaders.
- data utils → distributed.py: partitions work · Distributed helpers split data across workers and GPUs.
- train.py → models.py: builds model · Training assembles the segmentation network from components.
- eval.py → models.py: loads model · Evaluation restores the same assembled network.
- test.py → models.py: loads model · Inference instantiates the segmentation model for prediction.
- models.py → Backbones: selects encoder · Factory chooses the requested feature extractor.
- models.py → Decoders: selects decoder · Factory attaches the appropriate segmentation head.
- Backbones → SyncBN: normalizes · Backbones rely on synchronized batch norm during small-batch training.
- train.py → DataParallel: runs on GPUs · Multi-GPU execution wraps the model in custom parallelism.
- eval_multipro.py → DataParallel: runs on GPUs · Multi-device validation uses the parallel runtime.
- train.py → th.py: uses glue · Training shares tensor and checkpoint helpers.
- eval.py → th.py: uses glue · Evaluation relies on the same utility layer.
- test.py → th.py: uses glue · Inference uses shared preprocessing and postprocessing helpers.
- train.py → mit_semseg: imports package · Scripts depend on the installable package boundary.
- eval.py → mit_semseg: imports package · Evaluation uses package modules rather than local copies.
- Demo notebook → mit_semseg: explores API · Notebook demonstrates the reusable library surface.
- Demo notebook → Teaser assets: renders examples · Demo notebook is paired with presentation assets.
