# buoyancy99/research-template architecture

> Architecture diagram of the GitHub repository buoyancy99/research-template, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/buoyancy99/research-template
- Repository: https://github.com/buoyancy99/research-template

## Overview

##### Project type and purpose
- This repo is a research-template for machine-learning experiments.
- It uses Hydra for configuration, PyTorch Lightning for model training, and Weights & Biases (wandb) for logging and checkpointing.
- It provides built-in example experiments (classification, “hello world”), and a boilerplate for adding new algorithms, datasets, and experiments, as well as slurm cluster integration.

##### High-level architectural overview
- CLI & Config Layer
- Entry point: `python -m main +...`
- Hydra config system: reads YAML files under `configurations/` (algorithm/, dataset/, experiment/, cluster/, sweep/, config.yaml).
- Core Orchestration Layer
- `main.py`: decorated with `@hydra.main`, parses configs → sets up environment (conda, Python 3.10) → initializes wandb, cluster utilities → dispatches tasks.
- Task orchestration: training, validation, test, or custom tasks defined in experiment classes.
- Experiment Layer (`experiments/`)
- Contains experiment definitions inheriting from a base (`exp_base.py`).
- Registers experiments in `__init__.py`.
- Each experiment uses a selected algorithm and dataset and defines its task sequence.
- Algorithm Layer (`algorithms/`)
- Common base classes (base_algo, base_pytorch_algo) and common models (CNN, MLP).
- Example implementations (classifier, helloworld).
- Custom algorithms plug in via +algorithm=<yaml> and extend LightningModule for training logic.
- Dataset Layer (`datasets/`)
- Collections of dataset loaders (e.g., CIFAR-10 example).
- Conforms to a base interface and integrates with DataLoader.
- Utility Layer (`utils/`)
- ckpt_utils, cluster_utils, distributed_utils, print_utils, wandb_utils.
- Provide checkpoint management, slurm script generation, distributed training helpers, logging wrappers.
- External Services
- wandb.ai: experiment tracking, logging, checkpoint storage.
- Slurm Clusters: auto-generated scripts for MIT, Harvard, etc., via cluster YAMLs.
- Hydra: hierarchical config management and runtime overrides.
- PyTorch Lightning: Trainer abstraction for distributed, low-precision, multi-GPU training.

##### Main components & their interactions
- Configuration Files → Hydra
- YAML under `configurations/` are loaded into the Hydra config tree.
- Overrides from CLI feed into the same config tree.
- main.py
- Reads Hydra config → calls cluster_utils to generate SLURM or local run → sets wandb mode → instantiates the chosen Experiment class.
- Experiment
- Fetches algorithm and dataset modules based on config names.
- Initializes LightningModule (algorithm) and DataModule (dataset).
- Defines task flow (e.g., training → validation → test).
- Lightning Trainer
- Created under the hood by LightningExperiment base, uses PyTorch Lightning Trainer.
- Calls algorithm.training_step(), validation_step(), etc.
- Uses distributed_utils for multi-GPU or multi-node setups.
- Logging & Checkpointing
- wandb_utils sets up wandb run, config logging, and checkpoint saving.
- ckpt_utils handles resume/load logic.

##### Architectural patterns & design principles
- Plugin-based / modular design: algorithms, datasets, experiments are pluggable via config.
- Inversion of control via Hydra: code binds to configs rather than hard-coded arguments.
- Separation of concerns: config vs. orchestration vs. algorithm vs. data vs. utilities.
- DRY and reuse: common base classes for algorithms; example templates to follow.
- Extensibility: add new slurm clusters, algorithms, datasets by dropping YAML & code files.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Layer
    subgraph "Configuration Layer"
        CLI["CLI + Hydra Config Loader"]:::config
        Configs["Configurations\n(config.yaml, algorithm/, dataset/, experiment/, cluster/, sweep/)"]:::config
    end

    %% Orchestration Layer
    Orche["main.py Orchestrator"]:::orchestration

    %% Experiment Layer
    subgraph "Experiment Layer"
        Experiments["Experiment Manager\n(experiments/*.py)"]:::experiment
    end

    %% Algorithm Layer
    subgraph "Algorithm Layer"
        Algos["Algorithm Module\n(algorithms/)"]:::algorithm
    end

    %% Dataset Layer
    subgraph "Dataset Layer"
        Data["Dataset Module\n(datasets/)"]:::dataset
    end

    %% Utility Services
    subgraph "Utility Services"
        Ckpt["ckpt_utils"]:::utils
        ClusterU["cluster_utils"]:::utils
        DistU["distributed_utils"]:::utils
        WandbU["wandb_utils"]:::utils
    end

    %% External Services
    subgraph "External Services"
        Slurm["Slurm Scheduler"]:::external
        Trainer["PyTorch Lightning Trainer"]:::external
        Wandb["Weights & Biases"]:::external
    end

    %% Connections
    CLI -->|"config load"| Configs
    CLI -->|"parse CLI"| Orche
    Configs -->|"hydra inject"| Orche
    Orche -->|"generate job"| ClusterU
    ClusterU -->|"submit SLURM job"| Slurm
    Orche -->|"instantiate"| Experiments
    Experiments -->|"use algorithm"| Algos
    Experiments -->|"use dataset"| Data
    Algos -->|"provide LightningModule"| Trainer
    Data -->|"provide DataModule"| Trainer
    Orche -->|"setup wandb"| WandbU
    Trainer -->|"train loop"| WandbU
    Trainer -->|"checkpoint"| Ckpt
    Trainer -->|"distributed hooks"| DistU
    WandbU -->|"log & save"| Wandb

    %% Click Events
    click CLI "https://github.com/buoyancy99/research-template/blob/main/main.py"
    click Orche "https://github.com/buoyancy99/research-template/blob/main/main.py"
    click Configs "https://github.com/buoyancy99/research-template/tree/main/configurations/"
    click Experiments "https://github.com/buoyancy99/research-template/tree/main/experiments/"
    click Algos "https://github.com/buoyancy99/research-template/tree/main/algorithms/"
    click Data "https://github.com/buoyancy99/research-template/tree/main/datasets/"
    click Ckpt "https://github.com/buoyancy99/research-template/blob/main/utils/ckpt_utils.py"
    click ClusterU "https://github.com/buoyancy99/research-template/blob/main/utils/cluster_utils.py"
    click DistU "https://github.com/buoyancy99/research-template/blob/main/utils/distributed_utils.py"
    click WandbU "https://github.com/buoyancy99/research-template/blob/main/utils/wandb_utils.py"

    %% Styles
    classDef config fill:#ADD8E6,stroke:#000;
    classDef orchestration fill:#90EE90,stroke:#000;
    classDef experiment fill:#FFA500,stroke:#000;
    classDef algorithm fill:#FFFF66,stroke:#000;
    classDef dataset fill:#D8BFD8,stroke:#000;
    classDef utils fill:#D3D3D3,stroke:#000;
    classDef external fill:#87CEFA,stroke:#000;
```
