How buoyancy99/research-template works
Project type and purpose
• This repo is a research-template for machine-learning experiments.
• It uses Hydra for configuration, PyTorch Lightning for model training, and Weights & Biases (wandb) for logging and checkpointing.
• It provides built-in example experiments (classification, “hello world”), and a boilerplate for adding new algorithms, datasets, and experiments, as well as slurm cluster integration.
High-level architectural overview
• CLI & Config Layer
• Entry point: python -m main +...
• Hydra config system: reads YAML files under configurations/ (algorithm/, dataset/, experiment/, cluster/, sweep/, config.yaml).
• Core Orchestration Layer
• main.py: decorated with @hydra.main, parses configs → sets up environment (conda, Python 3.10) → initializes wandb, cluster utilities → dispatches tasks.
• Task orchestration: training, validation, test, or custom tasks defined in experiment classes.
• Experiment Layer (experiments/)
• Contains experiment definitions inheriting from a base (exp_base.py).
• Registers experiments in __init__.py.
• Each experiment uses a selected algorithm and dataset and defines its task sequence.
• Algorithm Layer (algorithms/)
• Common base classes (base_algo, base_pytorch_algo) and common models (CNN, MLP).
• Example implementations (classifier, helloworld).
• Custom algorithms plug in via +algorithm=<yaml> and extend LightningModule for training logic.
• Dataset Layer (datasets/)
• Collections of dataset loaders (e.g., CIFAR-10 example).
• Conforms to a base interface and integrates with DataLoader.
• Utility Layer (utils/)
• ckpt_utils, cluster_utils, distributed_utils, print_utils, wandb_utils.
• Provide checkpoint management, slurm script generation, distributed training helpers, logging wrappers.
• External Services
• wandb.ai: experiment tracking, logging, checkpoint storage.
• Slurm Clusters: auto-generated scripts for MIT, Harvard, etc., via cluster YAMLs.
• Hydra: hierarchical config management and runtime overrides.
• PyTorch Lightning: Trainer abstraction for distributed, low-precision, multi-GPU training.
Main components & their interactions
• Configuration Files → Hydra
• YAML under configurations/ are loaded into the Hydra config tree.
• Overrides from CLI feed into the same config tree.
• main.py
• Reads Hydra config → calls cluster_utils to generate SLURM or local run → sets wandb mode → instantiates the chosen Experiment class.
• Experiment
• Fetches algorithm and dataset modules based on config names.
• Initializes LightningModule (algorithm) and DataModule (dataset).
• Defines task flow (e.g., training → validation → test).
• Lightning Trainer
• Created under the hood by LightningExperiment base, uses PyTorch Lightning Trainer.
• Calls algorithm.training_step(), validation_step(), etc.
• Uses distributed_utils for multi-GPU or multi-node setups.
• Logging & Checkpointing
• wandb_utils sets up wandb run, config logging, and checkpoint saving.
• ckpt_utils handles resume/load logic.
Architectural patterns & design principles
• Plugin-based / modular design: algorithms, datasets, experiments are pluggable via config.
• Inversion of control via Hydra: code binds to configs rather than hard-coded arguments.
• Separation of concerns: config vs. orchestration vs. algorithm vs. data vs. utilities.
• DRY and reuse: common base classes for algorithms; example templates to follow.
• Extensibility: add new slurm clusters, algorithms, datasets by dropping YAML & code files.