How csailvision/semantic-segmentation-pytorch works
1. Overall purpose
This repository is a PyTorch semantic-segmentation pipeline for the ADE20K scene-parsing dataset. The core runtime path is: load image/label data, build an encoder-decoder segmentation model, train across one or more GPUs, then evaluate or run inference to produce per-pixel class maps and optionally colorized visualizations.
2. Top-level execution entry points
The main orchestration lives in train.py, eval.py, eval_multipro.py, and test.py. These scripts are the boundaries between the framework and the model/data internals: they parse config, assemble datasets and networks, restore checkpoints, and drive training, validation, or single-image/batch inference. demo_test.sh is a convenience wrapper for a preconfigured inference demo.
3. Configuration subsystem
Configuration is centralized in config/ and mit_semseg/config/defaults.py, with YAML presets such as ade20k-resnet50-upernet.yaml and ade20k-hrnetv2.yaml. This is the main architecture control plane: model choice, dataset paths, training hyperparameters, evaluation behavior, GPU selection, and visualization options are set here rather than hardcoded in the scripts.
4. Dataset and file-format boundary
mit_semseg/dataset.py and the dataset utilities under mit_semseg/lib/utils/data/ define how ADE20K assets are read and batched. The repo expects ADE20K metadata files like training.odgt, validation.odgt, object150_info.csv, and color150.mat, which support sample indexing, class metadata, and label-to-color rendering. The data boundary is image files plus segmentation labels, translated into tensors for the model.
5. Model family and architecture split
mit_semseg/models/ contains the backbone and decoder implementations. The repo explicitly separates encoders and decoders: encoders include MobileNetV2, ResNet variants, ResNeXt, and HRNet; decoders include C1, PPM, and UPerNet variants with optional deep supervision. models.py likely acts as the factory/assembly layer that combines these pieces into a full segmentation network.
6. Segmentation computation path
The forward path is the typical semantic-segmentation flow: backbone extracts multi-scale feature maps, decoder aggregates them into class logits, and the result is upsampled to image resolution. The README’s supported variants show two major architectural styles: pyramid pooling–based decoders for global context and UPerNet-style FPN/PPM fusion for multi-scale features. Deep supervision is an optional training-time branch that adds auxiliary losses.
7. Training and evaluation loops
Training is handled as a multi-GPU job with checkpointing, while evaluation computes metrics such as mean IoU and pixel accuracy against ADE20K validation labels. eval_multipro.py is the multi-device validation path, and test.py is the inference path for one image or a folder. The outputs are segmentation masks, with optional visualization overlays saved to disk.
8. Parallelism and distributed execution
A notable architectural feature is custom multi-GPU support in mit_semseg/lib/nn/parallel/ and mit_semseg/lib/utils/data/distributed.py. The project reimplements DataParallel and adapts the dataloader so each GPU can receive differently sized images, preserving aspect ratio during training. This is not just an optimization; it is a structural choice that influences batching, worker seeding, and how training data is partitioned.
9. Normalization infrastructure
mit_semseg/lib/nn/modules/batchnorm.py implements synchronized batch normalization across devices. This matters because segmentation often uses small per-GPU batch sizes, so batch statistics must be aggregated to keep training stable. The presence of custom batchnorm tests under mit_semseg/lib/nn/modules/tests/ indicates this is a first-class infrastructure component, not a model detail.
10. Utility layer
mit_semseg/lib/utils/th.py and related helpers provide glue code for PyTorch-specific tasks, likely including tensor/device handling, checkpoint serialization, and image preprocessing/postprocessing. mit_semseg/utils.py sits near the top of the package and likely contains shared routines used by scripts, evaluation, and visualization.
11. Packaging and integration boundary
setup.py and the mit_semseg package layout make the code installable as a library, not just runnable scripts. The README shows the intended programmatic integration surface: cfg, TestDataset, ModelBuilder, and SegmentationModule. That means the repo has both a CLI-oriented workflow and a reusable library boundary.
12. Notebooks and demo assets
notebooks/ provides a self-contained interactive path for experimentation, including a demo segmenter notebook and copied package/data resources. This is a separate UX layer for exploration and presentation rather than a different runtime architecture. teaser/ and the notebook assets support result visualization and documentation.
13. External dependencies and operational assumptions
The architecture depends on PyTorch, CUDA-enabled multi-GPU training, and common scientific Python packages. The README emphasizes 4+ GPUs for training and a single GPU for inference, which reflects the repo’s design point: high-throughput segmentation research and benchmarking rather than lightweight CPU deployment.
14. End-to-end data flow
At a high level: ADE20K metadata and image/label files are loaded through the dataset layer, config selects a backbone/decoder pair, the model runs on one or more GPUs with synchronized normalization and custom batching, checkpoints are saved during training, and evaluation/inference scripts reload those checkpoints to produce class masks, metrics, and optional colored outputs.
Main entry points: train.py train.py · eval.py eval.py · dataset.py dataset.py · models.py models.py
Components 20
Entry points
CLI scripts that start training, evaluation, and demo inference.
- train.py · Starts distributed training, checkpointing, and validation.
train.py - eval.py · Runs single-process evaluation against validation data.
eval.py - eval_multipro.py · Validation path for multi-device inference and metrics.
eval_multipro.py - test.py · Runs single-image or folder inference and writes masks.
test.py - demo_test.sh · Convenience shell wrapper for a preconfigured demo run.
demo_test.sh
Configuration
Central config and YAML presets that select datasets, models, and runtime options.
- defaults.py · Defines the base configuration schema and defaults.
mit_semseg/config/defaults.py - YAML presets · Dataset- and model-specific overrides such as ResNet, HRNet, and UPerNet.
config/ade20k-resnet50-upernet.yaml
Data boundary
ADE20K assets and loaders that turn files into tensors and labels.
- dataset.py · Top-level dataset interface for ADE20K loading and batching.
mit_semseg/dataset.py - data utils · Dataset, dataloader, sampler, and distributed helpers.
mit_semseg/lib/utils/data/dataloader.py - ADE20K meta · Class names, color map, and split index files.
data/object150_info.csv
Model stack
Backbones, decoders, and assembly code for semantic segmentation networks.
- models.py · Builds complete segmentation networks from encoder and decoder parts.
mit_semseg/models/models.py - Backbones · Feature extractors including ResNet, ResNeXt, MobileNetV2, and HRNet.
mit_semseg/models/resnet.py - Decoders · Segmentation heads such as C1, PPM, and UPerNet.
mit_semseg/models/utils.py
Runtime infra
Custom parallelism, distributed loading, and normalization support.
- SyncBN · Synchronized batch normalization across devices.
mit_semseg/lib/nn/modules/batchnorm.py - DataParallel · Custom parallel execution for segmentation training and inference.
mit_semseg/lib/nn/parallel/data_parallel.py - distributed.py · Multi-process data partitioning and worker coordination.
mit_semseg/lib/utils/data/distributed.py - th.py · Shared PyTorch glue for tensors, checkpoints, and preprocessing.
mit_semseg/lib/utils/th.py
Library API
Reusable package surface for external integration.
- mit_semseg · Installable package boundary for scripts and external code.
mit_semseg/__init__.py
Demo assets
Notebook and presentation assets for interactive exploration.
- Demo notebook · Interactive segmentation demo and experimentation workflow.
notebooks/DemoSegmenter.ipynb - Teaser assets · Presentation and result imagery used for documentation and demos.
teaser
Connections 26
- train.py → defaults.py: loads config · Training starts from the shared configuration schema.
- train.py → YAML presets: overrides · Experiment YAML selects model, dataset, and hyperparameters.
- eval.py → defaults.py: loads config · Evaluation uses the same config control plane.
- eval_multipro.py → defaults.py: loads config · Multi-process evaluation reads the shared defaults.
- test.py → defaults.py: loads config · Inference resolves runtime options from config.
- train.py → dataset.py: builds data · Training constructs ADE20K loaders through the dataset boundary.
- eval.py → dataset.py: builds data · Evaluation reuses the same dataset adapter.
- test.py → dataset.py: builds data · Inference loads images through the dataset adapter.
- dataset.py → ADE20K meta: reads metadata · Dataset loading depends on split files and class metadata.
- dataset.py → data utils: uses loaders · Batching and sampling are delegated to utility loaders.
- data utils → distributed.py: partitions work · Distributed helpers split data across workers and GPUs.
- train.py → models.py: builds model · Training assembles the segmentation network from components.
- eval.py → models.py: loads model · Evaluation restores the same assembled network.
- test.py → models.py: loads model · Inference instantiates the segmentation model for prediction.
- models.py → Backbones: selects encoder · Factory chooses the requested feature extractor.
- models.py → Decoders: selects decoder · Factory attaches the appropriate segmentation head.
- Backbones → SyncBN: normalizes · Backbones rely on synchronized batch norm during small-batch training.
- train.py → DataParallel: runs on GPUs · Multi-GPU execution wraps the model in custom parallelism.
- eval_multipro.py → DataParallel: runs on GPUs · Multi-device validation uses the parallel runtime.
- train.py → th.py: uses glue · Training shares tensor and checkpoint helpers.
- eval.py → th.py: uses glue · Evaluation relies on the same utility layer.
- test.py → th.py: uses glue · Inference uses shared preprocessing and postprocessing helpers.
- train.py → mit_semseg: imports package · Scripts depend on the installable package boundary.
- eval.py → mit_semseg: imports package · Evaluation uses package modules rather than local copies.
- Demo notebook → mit_semseg: explores API · Notebook demonstrates the reusable library surface.
- Demo notebook → Teaser assets: renders examples · Demo notebook is paired with presentation assets.