# rose-stl-lab/dyffusion architecture

> Architecture diagram of the GitHub repository rose-stl-lab/dyffusion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rose-stl-lab/dyffusion
- Repository: https://github.com/rose-stl-lab/dyffusion

## Overview

##### Project Type and Purpose
- This is a research-grade machine-learning library implementing the “DYffusion” spatiotemporal forecasting pipeline.
- It is not a web app or microservice; it’s an ML experiment platform built around PyTorch Lightning for training diffusion-based models on physical systems and SST data.
- Hydra handles configuration, PyTorch Lightning handles training loops, and Weights & Biases (W&B) or TensorBoard handle logging/checkpointing.

##### Key Architectural Components
a. CLI & Configuration Layer
- Entry points: `run.py` and `train.py`.
- Hydra (via `pyproject.toml`, `setup.cfg`, `hydra` imports) loads configs from `src/configs/**`.
- All experiment parameters (model, optimizer, scheduler, datamodule, diffusion settings, logging, trainer) are defined as YAML.

b. Experiment Orchestrator
- `src/interface.py` inspects your Hydra config and instantiates:
- A Lightning `DataModule` (from `src/datamodules/…`)
- An “Experiment” object (classes in `src/experiment_types/…`) that wraps model + training logic
- `src/train.py` kicks off the Lightning trainer with callbacks, loggers, schedulers, etc.

c. Data Layer
- Data download scripts in `scripts/` populate `$DATA_DIR`.
- `src/datamodules/abstract_datamodule.py` defines the interface; concrete datasets are implemented in `src/datamodules/datasets/…`.
- Data transforms and batching follow the Lightning DataModule pattern.

d. Model & Diffusion Layer
- Base model classes in `src/models/_base_model.py`.
- Specific architectures in `src/models/unet.py`, `src/models/simple_conv_net.py`, and advanced MCV-D variants in `src/models/mcvd/…`.
- Diffusion schedulers and algorithms in `src/diffusion/` (e.g., `ddpm.py`, `dyffusion.py`, `mcvd.py`).
- Experiment logic (interpolation vs. forecasting) lives in `src/experiment_types/forecasting_multi_horizon.py` and `interpolation.py`.

e. Training & Callbacks
- PyTorch Lightning `Trainer` configured via `src/configs/trainer/*.yaml`.
- Callbacks (early‐stop, checkpointing, custom logging) in `src/configs/callbacks/*.yaml` and implemented under `src/utilities/wandb_callbacks.py`.
- Learning‐rate schedulers under `src/utilities/lr_scheduler.py` and configurable via `src/configs/scheduler/*.yaml`.

f. Logging & Checkpointing
- W&B integration via `src/configs/logger/wandb.yaml` and `src/utilities/wandb_api.py`.
- TensorBoard alternative via `src/configs/logger/tensorboard.yaml`.

##### Relationships & Data Flows
- CLI → Hydra loads config → interface.py builds DataModule + Experiment
- DataModule fetches raw data from file system → preprocessed batches
- Experiment receives batches → model.forward() + diffusion steps → loss backprop
- Trainer invokes callbacks & logger → checkpoints & metrics sent to W&B/TensorBoard
- Optionally, external scripts handle data acquisition before CLI run.

##### Architectural Patterns & Principles
- Dependency‐injection via Hydra: loose coupling between code and configs.
- PyTorch Lightning’s “DataModule + LightningModule” abstraction for clear separation of data and model logic.
- Modular diffusion logic: base classes + pluggable schedules/models (factory pattern).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI & Configuration Layer
    subgraph "Configuration Layer"
        direction TB
        Hydra["Hydra Config Loader"]:::config
        MainYAML["main_config.yaml"]:::config
        CallbackYAML["callbacks/*.yaml"]:::config
        DatamoduleYAML["datamodule/*.yaml"]:::config
        DiffusionYAML["diffusion/*.yaml"]:::config
        ExperimentYAML["experiment/*.yaml"]:::config
        LoggerYAML["logger/*.yaml"]:::config
        ModelYAML["model/*.yaml"]:::config
        OptimizerYAML["optimizer/*.yaml"]:::config
        SchedulerYAML["scheduler/*.yaml"]:::config
        TrainerYAML["trainer/*.yaml"]:::config
        ModeYAML["mode/*.yaml"]:::config
        ModuleYAML["module/*.yaml"]:::config
        LocalYAML["example_local_config.yaml"]:::config
        Hydra --> MainYAML
        Hydra --> CallbackYAML
        Hydra --> DatamoduleYAML
        Hydra --> DiffusionYAML
        Hydra --> ExperimentYAML
        Hydra --> LoggerYAML
        Hydra --> ModelYAML
        Hydra --> OptimizerYAML
        Hydra --> SchedulerYAML
        Hydra --> TrainerYAML
        Hydra --> ModeYAML
        Hydra --> ModuleYAML
        Hydra --> LocalYAML
    end

    runCLI["run.py"]:::cli
    trainCLI["train.py"]:::cli
    runCLI --> Hydra
    trainCLI --> Hydra

    click runCLI "https://github.com/rose-stl-lab/dyffusion/blob/main/run.py"
    click trainCLI "https://github.com/rose-stl-lab/dyffusion/blob/main/src/train.py"
    click MainYAML "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/main_config.yaml"
    click CallbackYAML "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/callbacks/*.yaml"
    click DatamoduleYAML "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/datamodule/*.yaml"
    click DiffusionYAML "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/diffusion/*.yaml"
    click ExperimentYAML "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/experiment/*.yaml"
    click LoggerYAML "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/logger/*.yaml"
    click ModelYAML "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/model/*.yaml"
    click OptimizerYAML "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/optimizer/*.yaml"
    click SchedulerYAML "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/scheduler/*.yaml"
    click TrainerYAML "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/trainer/*.yaml"
    click ModeYAML "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/mode/*.yaml"
    click ModuleYAML "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/module/*.yaml"
    click LocalYAML "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/local/example_local_config.yaml"

    %% Orchestrator
    Interface["Interface (builds DataModule & Experiment)"]:::orchestrator
    Hydra -->|"load_config"| Interface
    click Interface "https://github.com/rose-stl-lab/dyffusion/blob/main/src/interface.py"

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        Scripts["Download Scripts"]:::external
        DataStore[(Local File System)]:::data
        AbstractDM["abstract_datamodule.py"]:::data
        PhysicalDS["physical_systems_benchmark.py"]:::data
        OISST["oisstv2.py"]:::data
        TorchDS["torch_datasets.py"]:::data
        DimUtils["dataset_dimensions.py"]:::data
        Scripts --> DataStore
        AbstractDM --> PhysicalDS
        AbstractDM --> OISST
        AbstractDM --> TorchDS
        AbstractDM --> DimUtils
        DataModule["DataModule"]:::data
        DataModule --> AbstractDM
        DataModule --> DataStore
    end
    click AbstractDM "https://github.com/rose-stl-lab/dyffusion/blob/main/src/datamodules/abstract_datamodule.py"
    click PhysicalDS "https://github.com/rose-stl-lab/dyffusion/blob/main/src/datamodules/datasets/physical_systems_benchmark.py"
    click OISST "https://github.com/rose-stl-lab/dyffusion/blob/main/src/datamodules/oisstv2.py"
    click TorchDS "https://github.com/rose-stl-lab/dyffusion/blob/main/src/datamodules/torch_datasets.py"
    click DimUtils "https://github.com/rose-stl-lab/dyffusion/blob/main/src/datamodules/dataset_dimensions.py"
    click Scripts "https://github.com/rose-stl-lab/dyffusion/blob/main/scripts/download_navier_stokes.sh"
    click Scripts "https://github.com/rose-stl-lab/dyffusion/blob/main/scripts/download_physical_systems_data.sh"
    click Scripts "https://github.com/rose-stl-lab/dyffusion/blob/main/scripts/download_spring_mesh.sh"

    Interface -->|"instantiate"| DataModule

    %% Experiment Logic
    subgraph "Experiment Types"
        direction TB
        BaseExp["_base_experiment.py"]:::orchestrator
        Forecasting["forecasting_multi_horizon.py"]:::orchestrator
        Interpolation["interpolation.py"]:::orchestrator
        BaseExp --> Forecasting
        BaseExp --> Interpolation
    end
    click BaseExp "https://github.com/rose-stl-lab/dyffusion/blob/main/src/experiment_types/_base_experiment.py"
    click Forecasting "https://github.com/rose-stl-lab/dyffusion/blob/main/src/experiment_types/forecasting_multi_horizon.py"
    click Interpolation "https://github.com/rose-stl-lab/dyffusion/blob/main/src/experiment_types/interpolation.py"

    Interface -->|"instantiate"| BaseExp

    %% Model & Diffusion Layer
    subgraph "Model Layer"
        direction TB
        BaseModel["_base_model.py"]:::model
        ConvNet["simple_conv_net.py"]:::model
        UNet["unet.py"]:::model
        UNetSimple["unet_simple.py"]:::model
        MCVdModel["models/mcvd/..."]:::model
        ModelLogic["Model (LightningModule)"]:::model
        BaseModel --> ConvNet
        BaseModel --> UNet
        BaseModel --> UNetSimple
        BaseModel --> MCVdModel
        ModelLogic --> BaseModel
        Diffusion["Diffusion Scheduler"]:::model
        DBase["_base_diffusion.py"]:::model
        DDPM["ddpm.py"]:::model
        Denoise["denoising_diffusion.py"]:::model
        DYff["dyffusion.py"]:::model
        MCVdDiff["mcvd.py"]:::model
        Schedules["schedules.py"]:::model
        DBase --> DDPM
        DBase --> Denoise
        DBase --> DYff
        DBase --> MCVdDiff
        DBase --> Schedules
        Diffusion --> DBase
    end
    click BaseModel "https://github.com/rose-stl-lab/dyffusion/blob/main/src/models/_base_model.py"
    click ConvNet "https://github.com/rose-stl-lab/dyffusion/blob/main/src/models/simple_conv_net.py"
    click UNet "https://github.com/rose-stl-lab/dyffusion/blob/main/src/models/unet.py"
    click UNetSimple "https://github.com/rose-stl-lab/dyffusion/blob/main/src/models/unet_simple.py"
    click MCVdModel "https://github.com/rose-stl-lab/dyffusion/tree/main/src/models/mcvd/"
    click DBase "https://github.com/rose-stl-lab/dyffusion/blob/main/src/diffusion/_base_diffusion.py"
    click DDPM "https://github.com/rose-stl-lab/dyffusion/blob/main/src/diffusion/ddpm.py"
    click Denoise "https://github.com/rose-stl-lab/dyffusion/blob/main/src/diffusion/denoising_diffusion.py"
    click DYff "https://github.com/rose-stl-lab/dyffusion/blob/main/src/diffusion/dyffusion.py"
    click MCVdDiff "https://github.com/rose-stl-lab/dyffusion/blob/main/src/diffusion/mcvd.py"
    click Schedules "https://github.com/rose-stl-lab/dyffusion/blob/main/src/diffusion/schedules.py"

    BaseExp -->|"run batches"| ModelLogic
    ModelLogic -->|"forward"| Diffusion

    %% Training & Callbacks
    subgraph "Training Layer"
        direction TB
        Trainer["PyTorch Lightning Trainer"]:::training
        CallbackImpl["wandb_callbacks.py"]:::training
        LRSched["lr_scheduler.py"]:::training
        Trainer --> CallbackImpl
        Trainer --> LRSched
    end
    click CallbackImpl "https://github.com/rose-stl-lab/dyffusion/blob/main/src/utilities/wandb_callbacks.py"
    click LRSched "https://github.com/rose-stl-lab/dyffusion/blob/main/src/utilities/lr_scheduler.py"

    trainCLI -->|"kickoff"| Trainer
    ModelLogic --> Trainer
    Forecasting --> Trainer
    Interpolation --> Trainer

    %% Logging & Checkpointing
    subgraph "Logging Layer"
        direction TB
        WandB["Weights & Biases"]:::external
        TensorBoard["TensorBoard"]:::external
        API["wandb_api.py"]:::external
        LoggerWandB["wandb.yaml"]:::config
        LoggerTB["tensorboard.yaml"]:::config
        Trainer -->|"log_metrics"| API
        API --> WandB
        Trainer -->|"log_metrics"| TensorBoard
        LoggerWandB --> API
        LoggerTB --> TensorBoard
    end
    click API "https://github.com/rose-stl-lab/dyffusion/blob/main/src/utilities/wandb_api.py"
    click LoggerWandB "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/logger/wandb.yaml"
    click LoggerTB "https://github.com/rose-stl-lab/dyffusion/blob/main/src/configs/logger/tensorboard.yaml"

    %% Legend
    subgraph Legend
        direction LR
        C1["Config & CLI"]:::config
        D1["Data"]:::data
        M1["Model & Diffusion"]:::model
        T1["Training"]:::training
        E1["External"]:::external
    end

    classDef config fill:#D0E6FF,stroke:#0047AB;
    classDef cli fill:#A3CEF1,stroke:#0047AB;
    classDef orchestrator fill:#E8D0FF,stroke:#800080;
    classDef data fill:#D0FFD6,stroke:#006400;
    classDef model fill:#FFE6B3,stroke:#FF8C00;
    classDef training fill:#F3D1FF,stroke:#800080;
    classDef external fill:#E0E0E0,stroke:#808080;
```
