# siboehm/shallowspeed architecture

> Architecture diagram of the GitHub repository siboehm/shallowspeed, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/siboehm/shallowspeed
- Repository: https://github.com/siboehm/shallowspeed

## Overview

##### Project type and purpose
- From the README and file tree, “Shallowspeed” is a proof-of-concept Python library and CLI for distributed training of sequential deep learning models. It implements data-parallel (DP) and pipeline-parallel (PP) schedules (naive, Gpipe, and soon PipeDream) using plain NumPy and mpi4py.

##### File structure
- Top-level files:
- train.py: entry point for all training modes (sequential, DP, PP, combined).
- download_dataset.py & data/: dataset acquisition & storage.
- environment.yml & setup.py: dependency/configuration.
- scripts/DDP_PyTorch_MNIST.py: reference/example script.
- Package folder `shallowspeed/`:
- dataset.py: data‐loading abstraction.
- functional.py: tensor ops, activations, loss functions.
- layers.py: layer definitions for MLP blocks.
- optimizer.py: SGD optimizer logic.
- pipe.py: core DP/PP scheduler, communication routines via mpi4py.
- utils.py: miscellaneous helpers (e.g., parameter initialization).
- tests/: unit tests for each core module.

##### README for additional insights
- Usage patterns (mpirun + flags) reveal how train.py configures DP and/or PP.
- Internals section hints at a Pebble‐graph representation for pipeline scheduling.

##### Main components
a. Trainer CLI (train.py)
b. Data Loader (shallowspeed.dataset)
c. Model Definition (layers + functional)
d. Optimizer (shallowspeed.optimizer)
e. Scheduler & Communication Engine (shallowspeed.pipe using mpi4py)
f. Underlying numeric backend (NumPy arrays)
g. External Dataset Storage (local filesystem under data/)
h. MPI processes (worker ranks orchestrated by mpirun)

##### Relationships and interactions
- The CLI instantiates Dataset → Model → Optimizer → Scheduler.
- For each mini-batch:
- Data Loader yields inputs
- Model forward pass (layers→functional)
- Depending on mode:
– Sequential: local backward & apply gradients.
– DP: All-reduce gradients across ranks (mpi4py).
– PP: Partition model stages across ranks; send/recv activations and gradients.
– Combined DP+PP: inner DP all-reduce within pipeline micro‐batches.
- Optimizer updates parameters
- Scheduler orchestrates the above, invoking appropriate communication primitives.

##### Architectural patterns & design principles
- Modular layering: separation of dataset, model, optimizer, scheduler.
- Strategy pattern: DP vs. PP scheduling algorithms selected at runtime.
- Functional programming: core tensor ops in functional.py.
- Message passing: mpi4py abstracts inter-process comms.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Components
    TrainerCLI["Trainer CLI"]:::compute
    DataStore["External Dataset Storage"]:::data
    Download["Dataset Downloader"]:::data
    DataLoader["Data Loader"]:::data
    subgraph "Model Definition"
        Layers["Layers"]:::compute
        Functional["Functional Ops"]:::compute
    end
    Optimizer["Optimizer"]:::compute
    Scheduler["Scheduler & MPI Engine"]:::comm
    Utils["Utilities"]:::compute
    ExternalLibs["External Libs\n(NumPy, mpi4py)"]:::external
    Example["DDP Example Script"]:::data
    Env["Env Config"]:::external
    Setup["Setup"]:::external

    %% Relationships
    TrainerCLI -->|"initialize pipeline"| DataLoader
    DataStore -->|"provides data"| DataLoader
    Download -->|"fetches into"| DataStore
    DataLoader -->|"yields batch"| Layers
    Layers -->|"compute forward/backward"| Functional
    Functional -->|"tensor ops"| Scheduler
    Scheduler -->|"DP all-reduce"| Functional
    Scheduler -.->|"PP send/recv"| Functional
    Functional -->|"gradients"| Optimizer
    Optimizer -->|"update params"| Layers
    Scheduler -->|"orchestrate tasks"| Utils
    ExternalLibs -->|"compute backend"| Functional
    ExternalLibs -->|"communication backend"| Scheduler
    TrainerCLI --> Example

    %% Click Events
    click TrainerCLI "https://github.com/siboehm/shallowspeed/blob/master/train.py"
    click DataStore "https://github.com/siboehm/shallowspeed/tree/master/data/"
    click Download "https://github.com/siboehm/shallowspeed/blob/master/download_dataset.py"
    click DataLoader "https://github.com/siboehm/shallowspeed/blob/master/shallowspeed/dataset.py"
    click Layers "https://github.com/siboehm/shallowspeed/blob/master/shallowspeed/layers.py"
    click Functional "https://github.com/siboehm/shallowspeed/blob/master/shallowspeed/functional.py"
    click Optimizer "https://github.com/siboehm/shallowspeed/blob/master/shallowspeed/optimizer.py"
    click Scheduler "https://github.com/siboehm/shallowspeed/blob/master/shallowspeed/pipe.py"
    click Utils "https://github.com/siboehm/shallowspeed/blob/master/shallowspeed/utils.py"
    click Example "https://github.com/siboehm/shallowspeed/blob/master/scripts/DDP_PyTorch_MNIST.py"
    click Env "https://github.com/siboehm/shallowspeed/blob/master/environment.yml"
    click Setup "https://github.com/siboehm/shallowspeed/blob/master/setup.py"

    %% Styles
    classDef compute fill:#bbf,stroke:#333,stroke-width:1px
    classDef data fill:#bfb,stroke:#333,stroke-width:1px
    classDef comm fill:#fb7,stroke:#333,stroke-width:1px
    classDef external fill:#ccc,stroke:#333,stroke-width:1px
```
