How siboehm/shallowspeed works
Project type and purpose
• From the README and file tree, “Shallowspeed” is a proof-of-concept Python library and CLI for distributed training of sequential deep learning models. It implements data-parallel (DP) and pipeline-parallel (PP) schedules (naive, Gpipe, and soon PipeDream) using plain NumPy and mpi4py.
File structure
• Top-level files:
• train.py: entry point for all training modes (sequential, DP, PP, combined).
• download_dataset.py & data/: dataset acquisition & storage.
• environment.yml & setup.py: dependency/configuration.
• scripts/DDP_PyTorch_MNIST.py: reference/example script.
• Package folder shallowspeed/:
• dataset.py: data‐loading abstraction.
• functional.py: tensor ops, activations, loss functions.
• layers.py: layer definitions for MLP blocks.
• optimizer.py: SGD optimizer logic.
• pipe.py: core DP/PP scheduler, communication routines via mpi4py.
• utils.py: miscellaneous helpers (e.g., parameter initialization).
• tests/: unit tests for each core module.
README for additional insights
• Usage patterns (mpirun + flags) reveal how train.py configures DP and/or PP.
• Internals section hints at a Pebble‐graph representation for pipeline scheduling.
Main components
a. Trainer CLI (train.py)
b. Data Loader (shallowspeed.dataset)
c. Model Definition (layers + functional)
d. Optimizer (shallowspeed.optimizer)
e. Scheduler & Communication Engine (shallowspeed.pipe using mpi4py)
f. Underlying numeric backend (NumPy arrays)
g. External Dataset Storage (local filesystem under data/)
h. MPI processes (worker ranks orchestrated by mpirun)
Relationships and interactions
• The CLI instantiates Dataset → Model → Optimizer → Scheduler.
• For each mini-batch:
• Data Loader yields inputs
• Model forward pass (layers→functional)
• Depending on mode:
– Sequential: local backward & apply gradients.
– DP: All-reduce gradients across ranks (mpi4py).
– PP: Partition model stages across ranks; send/recv activations and gradients.
– Combined DP+PP: inner DP all-reduce within pipeline micro‐batches.
• Optimizer updates parameters
• Scheduler orchestrates the above, invoking appropriate communication primitives.
Architectural patterns & design principles
• Modular layering: separation of dataset, model, optimizer, scheduler.
• Strategy pattern: DP vs. PP scheduling algorithms selected at runtime.
• Functional programming: core tensor ops in functional.py.
• Message passing: mpi4py abstracts inter-process comms.