How sthalles/simclr works
Project type and purpose
• This is a command-line, monolithic Python framework implementing the SimCLR self-supervised learning algorithm in PyTorch.
• It provides (a) data augmentation components, (b) a contrastive learning training pipeline, (c) model definitions (ResNet-based encoder + projection head), and (d) a downstream linear evaluation protocol.
File structure
• Top-level scripts:
• run.py – Entrypoint: parses CLI args, sets up config, launches training or evaluation
• simclr.py – Core training logic for SimCLR (encoder forward passes, loss computation, backprop)
• data_aug/ – Data pipeline modules generating augmented views:
• contrastive_learning_dataset.py – wrapper around torch.utils.data.Dataset that returns two augmented views per sample
• gaussian_blur.py, view_generator.py – individual augmentation transforms
• models/ – Model definitions:
• resnet_simclr.py – Builds a ResNet backbone + projection head
• feature_eval/ – Notebook for linear evaluation of learned representations
• exceptions/ – Custom error classes
• utils.py – Helper routines (e.g., logging, checkpoint I/O)
• env.yml, requirements.txt – Environment and dependencies
README for additional insights
• Confirms PyTorch & AMP for mixed-precision GPU training
• CLI flags for dataset location, dataset name, logging frequency, epochs, disabling CUDA
• Downstream linear evaluation on STL10/CIFAR10
a. Main components:
1. Data Source (raw image dataset on disk, e.g., STL10, CIFAR10)
2. Data Augmentation Pipeline (data_aug package)
3. Data Loader (PyTorch DataLoader)
4. SimCLR Training Engine (simclr.py + run.py orchestrator)
5. Model (ResNet encoder + projection head from models/)
6. Loss Computation (NT-Xent loss in simclr.py)
7. Optimizer & AMP (PyTorch optimizer, optional fp16 AMP)
8. Logging & Checkpointing (utils.py)
9. Feature Evaluation (feature_eval notebook)
10. CLI & Configuration (argparse in run.py, env.yml)
b. Relationships & interactions:
• Data Source → Data Augmentation Pipeline → DataLoader batches two views
• Batches → Training Engine → Model forward pass → Loss Computation → Backprop → Optimizer step
• Training Engine → Logging & Checkpointing → Filesystem
• After training, saved encoder weights → Feature Evaluation Notebook → Linear classifier training & evaluation
c. Architectural patterns & design principles:
• Modular separation: data augmentation is isolated from model and training logic
• Single-process, pipeline pattern: sequential data → model → loss → optimize
• Plug-in points: CLI flags for dataset, device, optimizer, precision
• Extensibility: one can add new augmentations, swap backbones, or evaluation protocols
d. Key technologies:
• Python 3.x, PyTorch (torch, torchvision)
• Conda for environment (env.yml)
• AMP for mixed precision (built-in)
• Jupyter Notebook for evaluation