How microsoft/swin-transformer works
Purpose and execution surface
This is a PyTorch training/evaluation repository for ImageNet image classification with Swin-family hierarchical vision backbones. It supports supervised Swin/SwinV2/Swin-MLP training, SimMIM masked-image pretraining and fine-tuning, and distributed Swin-MoE runs. Primary entry points are main.py, main_moe.py, main_simmim_pt.py, and main_simmim_ft.py.
Configuration-driven experiment selection
Runtime behavior is selected from YAML architecture/task presets and merged configuration logic. Configs encode model scale, resolution, dataset/class count, optimization, and training schedule; command-line options can override them.
• config.py
• configs/swin/swin_tiny_patch4_window7_224.yaml
• configs/simmim/simmim_pretrain__swinv2_base__img192_window12__800ep.yaml
Standard supervised training pipeline
main.py orchestrates distributed initialization, config loading, model/dataloader/optimizer/scheduler construction, checkpoint resume or pretrained-weight loading, epoch training, validation, and artifact logging. The main control flow is configuration → dataset loaders → model forward/loss/backpropagation → optimizer/scheduler update → checkpoint and metrics.
• main.py
• optimizer.py
• lr_scheduler.py
Data ingestion boundary
The data layer creates PyTorch datasets and loaders for ImageNet-style folders, including ImageNet-22K label handling and optional zip/cached-image access. Distributed-aware samplers partition training work across processes. This isolates filesystem/archive layout and label mapping from model execution.
• data/build.py
• data/cached_image_folder.py
• data/imagenet22k_dataset.py
Model factory and backbone variants
A model builder dispatches configured model types to Swin V1, Swin V2, or shifted-window MLP implementations. Swin backbones implement hierarchical patch embedding, alternating regular/shifted local-window stages, patch merging, normalization, and classifier head. Model-specific code is separated from training orchestration.
• models/build.py
• models/swin_transformer.py
• models/swin_transformer_v2.py
• models/swin_mlp.py
SimMIM self-supervised branch
SimMIM has dedicated launchers and dataset paths for masking-based pretraining and downstream fine-tuning. The SimMIM wrapper combines a selected Swin/SwinV2 encoder with masked-image reconstruction behavior; pretrained representations then flow into the fine-tuning runner.
• main_simmim_pt.py
• main_simmim_ft.py
• models/simmim.py
• data/data_simmim_pt.py
Mixture-of-Experts branch
Swin-MoE uses a separate runner and model implementation because expert routing and distributed execution differ materially from dense models. It relies on the external Tutel MoE system and supplies MoE-specific runtime utilities and configurations targeting multi-GPU expert layouts.
• main_moe.py
• models/swin_transformer_moe.py
• utils_moe.py
• configs/swinmoe/swin_moe_small_patch4_window12_192_64expert_64gpu_22k.yaml
Runtime, acceleration, and persistence
The material runtime is Python/PyTorch on CUDA-capable GPUs, with distributed multi-process training and optional mixed precision/Apex noted by the project. Standard utilities centralize distributed setup, checkpointing, metric reduction, and logging. An optional compiled CUDA/C++ window-processing extension accelerates window partition/reversal operations at the Python boundary.
• utils.py
• logger.py
• kernels/window_process/window_process.py
• kernels/window_process/swin_window_process_kernel.cu
Main entry points: main.py Supervised runner · models/build.py Model factory · swin_tiny_patch4_window7_224.yaml YAML presets · data/build.py Dataset and loaders
Components 22
Experiment control
- YAML presets · Select task, scale, resolution, and training settings.
configs/swin/swin_tiny_patch4_window7_224.yaml - Config merge · Merges presets with command-line overrides.
config.py - Supervised runner · Runs dense-model training, validation, and resume flows.
main.py - Optimizer builder
optimizer.py - LR scheduler
lr_scheduler.py - SimMIM pretraining
main_simmim_pt.py - SimMIM fine-tuning
main_simmim_ft.py - Swin-MoE runner · Launches expert-routed multi-GPU training.
main_moe.py
Data ingestion
- Dataset and loaders · Builds ImageNet loaders and distributed samplers.
data/build.py - Cached/zip images
data/cached_image_folder.py - ImageNet-22K labels
data/imagenet22k_dataset.py - SimMIM data path
data/data_simmim_pt.py
Model variants
- Model factory · Dispatches configured dense backbone variants.
models/build.py - Swin dense backbones · Hierarchical patch, shifted-window, merging, and classifier stages.
models/swin_transformer.py - Swin V2 backbone
models/swin_transformer_v2.py - Swin-MLP backbone
models/swin_mlp.py - SimMIM wrapper · Wraps a Swin encoder for masked-image reconstruction.
models/simmim.py - Swin-MoE model · Uses external Tutel MoE routing.
models/swin_transformer_moe.py
Distributed runtime
- MoE runtime utilities
utils_moe.py - Runtime utilities · Provides distributed setup, checkpoints, metric reduction, and logging.
utils.py - Artifact logger
logger.py - CUDA window extension · Accelerates window partition and reversal.
kernels/window_process/window_process.py
Connections 22
- YAML presets → Config merge: preset
- Config merge → Supervised runner: runtime config
- Supervised runner → Dataset and loaders: builds loaders
- Dataset and loaders → Cached/zip images: optional backend
- Dataset and loaders → ImageNet-22K labels: 22K support
- Supervised runner → Model factory: builds model
- Model factory → Swin dense backbones: Swin V1
- Model factory → Swin V2 backbone: Swin V2
- Model factory → Swin-MLP backbone: Swin-MLP
- Supervised runner → Optimizer builder: builds
- Supervised runner → LR scheduler: updates
- Supervised runner → Runtime utilities: distributed and checkpoints
- Runtime utilities → Artifact logger: metrics and artifacts
- Swin dense backbones → CUDA window extension: accelerated windows
- SimMIM pretraining → SimMIM data path: masked inputs
- SimMIM pretraining → SimMIM wrapper: trains
- SimMIM wrapper → Swin V2 backbone: encoder option
- SimMIM fine-tuning → SimMIM wrapper: loads representations
- Swin-MoE runner → Swin-MoE model: trains
- Swin-MoE runner → MoE runtime utilities: distributed execution
- YAML presets → SimMIM pretraining: SimMIM preset
- YAML presets → Swin-MoE runner: MoE preset