# m2lines/samudra architecture

> Architecture diagram of the GitHub repository m2lines/samudra, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/m2lines/samudra
- Repository: https://github.com/m2lines/samudra

## Overview

##### Project Type and Purpose
- Project type: An AI-driven scientific emulator for global ocean modeling (a specialized ML pipeline).
- Purpose: Emulate 3D ocean dynamics (sea surface height, velocities, temperature, salinity) at high fidelity and extreme speed—150× faster than the original OM4 climate model.

##### File Structure
Top-level directories & files:
- configs/            ← YAML configs for training & rollout
- src/                ← Core Python code for data, model, training, rollout, utilities
- notebooks/          ← Jupyter notebooks for plotting and paper figures
- assets/             ← Visual assets (e.g., globe.gif)
- environment.yml     ← Conda environment specification
Key scripts in src/:
- train.py            ← Orchestrates model training
- rollout.py          ← Orchestrates model inference (rollout)
- stepper.py          ← Advances time steps in the emulator
- datasets.py         ← Data ingestion & preprocessing
- models/             ← Model definitions (activations, blocks, factory, Samudra core)
- utils/              ← Logging, distributed launch, device placement, writers

##### Extract Architectural Insights from the README
- Data layer: OM4 data & precomputed statistics (means/stds) loaded via Zarr.
- Config layer: Two YAML files (train_samudra_om4.yaml, rollout_samudra_om4.yaml) drive all hyperparameters, I/O paths, and sub-experiment names.
- Compute layer: PyTorch + torchrun for distributed training; single-process Python for rollouts.
- Outputs: Model checkpoints, Zarr outputs of emulated ocean trajectories, plots via notebooks.

##### Main Components
a. External Data Sources
– OM4 Zarr datasets (remote storage endpoints)
– OM4 statistics (means, stds)
b. Configuration Management
– configs/*.yaml
– src/config.py (parses & validates)
c. Data Ingestion & Preprocessing
– src/datasets.py
– src/utils/data.py
d. Model Definition
– src/models/activations.py
– src/models/blocks.py
– src/models/factory.py
– src/models/samudra.py
e. Training Orchestration
– src/train.py
– src/stepper.py
– src/utils/train.py
– src/utils/distributed.py
– src/utils/writer.py
f. Rollout / Inference
– src/rollout.py
– stepper (same as training)
g. Utilities
– Logging (src/utils/logging.py)
– Device management (src/utils/device.py)
– Notebook helpers (src/utils/notebook.py)
h. Visualization & Analysis
– notebooks/samudra_plotting.ipynb
– assets/

##### Relationships & Data Flows
- Configuration files → Configuration parser → Training/Rollout scripts
- Remote Zarr data → Data ingestion modules → Normalization (means/stds) → Model inputs
- Model inputs → Samudra model → Predicted next ocean state
- Stepper → Advances through time sequence → Feeds predictions back as inputs
- Training loop → Loss computation → Backpropagation → Checkpoint writer
- Rollout → Checkpoint loader → Emulated trajectory → Zarr output
- Notebooks → Load outputs/checkpoints → Generate figures

##### Architectural Patterns & Principles
- Modular separation: data / model / orchestrator / utils
- Configuration-driven: All hyperparams and I/O controlled by YAML + CLI flags
- Reusable components: stepper used by both train & rollout
- Distributed training support via torchrun + src/utils/distributed.py
- Clear I/O abstraction: data ingestion and writing results decoupled

##### Final Checklist
[ ] All major components present and labeled
[ ] Config layer linked to both training & rollout
[ ] External dependencies (Zarr endpoints, HF weights) shown
[ ] Utility modules positioned per their use in each pipeline stage

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Data Sources
    subgraph "External Data Sources"
        OM4_Zarr["OM4 Zarr Datasets"]:::external
        OM4_Stats["OM4 Statistics"]:::external
    end

    %% Configuration Management
    subgraph "Configuration Management"
        ConfigTrain["train_samudra_om4.yaml"]:::config
        ConfigRollout["rollout_samudra_om4.yaml"]:::config
        ConfigParser["config.py"]:::config
    end

    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        Datasets["datasets.py"]:::data
        DataUtils["utils/data.py"]:::data
        Normalization["Normalization\n(means/stds)"]:::data
    end

    %% Model Definition
    subgraph "Model Definition"
        Activations["activations.py"]:::model
        Blocks["blocks.py"]:::model
        Factory["factory.py"]:::model
        SamudraCore["samudra.py"]:::model
    end

    %% Training Orchestration
    subgraph "Training Orchestration"
        TrainScript["train.py"]:::compute
        StepperTrain["stepper.py"]:::compute
        TrainUtils["utils/train.py"]:::compute
        Dist["utils/distributed.py"]:::compute
        Writer["utils/writer.py"]:::compute
    end

    %% Rollout / Inference
    subgraph "Rollout / Inference"
        RolloutScript["rollout.py"]:::compute
        StepperRollout["stepper.py"]:::compute
    end

    %% Utilities
    subgraph "Utilities"
        Logging["logging.py"]:::utils
        Device["device.py"]:::utils
        NotebookUtil["notebook.py"]:::utils
    end

    %% Visualization & Analysis
    subgraph "Visualization & Analysis"
        Notebook["samudra_plotting.ipynb"]:::viz
        Assets["assets"]:::viz
    end

    %% Connections
    ConfigTrain --> ConfigParser
    ConfigRollout --> ConfigParser
    ConfigParser --> TrainScript
    ConfigParser --> RolloutScript

    OM4_Zarr --> Datasets
    OM4_Stats --> Normalization
    Datasets --> Normalization
    Normalization -->|"PyTorch Tensor"| TrainScript
    Normalization -->|"PyTorch Tensor"| RolloutScript

    TrainScript --> StepperTrain
    RolloutScript --> StepperRollout

    StepperTrain --> SamudraCore
    StepperRollout --> SamudraCore

    Activations --> Factory
    Blocks --> Factory
    Factory --> SamudraCore

    TrainScript --> TrainUtils
    TrainScript --> Dist
    TrainScript --> Writer
    TrainScript --> Logging
    TrainScript --> Device

    RolloutScript --> Logging
    RolloutScript --> Device

    Writer --> Assets
    Notebook --> Assets
    Notebook --> NotebookUtil

    %% Click Events
    click ConfigTrain "https://github.com/m2lines/samudra/blob/main/configs/train_samudra_om4.yaml"
    click ConfigRollout "https://github.com/m2lines/samudra/blob/main/configs/rollout_samudra_om4.yaml"
    click ConfigParser "https://github.com/m2lines/samudra/blob/main/src/config.py"
    click Datasets "https://github.com/m2lines/samudra/blob/main/src/datasets.py"
    click DataUtils "https://github.com/m2lines/samudra/blob/main/src/utils/data.py"
    click Activations "https://github.com/m2lines/samudra/blob/main/src/models/activations.py"
    click Blocks "https://github.com/m2lines/samudra/blob/main/src/models/blocks.py"
    click Factory "https://github.com/m2lines/samudra/blob/main/src/models/factory.py"
    click SamudraCore "https://github.com/m2lines/samudra/blob/main/src/models/samudra.py"
    click TrainScript "https://github.com/m2lines/samudra/blob/main/src/train.py"
    click StepperTrain "https://github.com/m2lines/samudra/blob/main/src/stepper.py"
    click TrainUtils "https://github.com/m2lines/samudra/blob/main/src/utils/train.py"
    click Dist "https://github.com/m2lines/samudra/blob/main/src/utils/distributed.py"
    click Writer "https://github.com/m2lines/samudra/blob/main/src/utils/writer.py"
    click RolloutScript "https://github.com/m2lines/samudra/blob/main/src/rollout.py"
    click Logging "https://github.com/m2lines/samudra/blob/main/src/utils/logging.py"
    click Device "https://github.com/m2lines/samudra/blob/main/src/utils/device.py"
    click NotebookUtil "https://github.com/m2lines/samudra/blob/main/src/utils/notebook.py"
    click Notebook "https://github.com/m2lines/samudra/blob/main/notebooks/samudra_plotting.ipynb"
    click Assets "https://github.com/m2lines/samudra/tree/main/assets/"

    %% Styles
    classDef external fill:#bbdefb,stroke:#333,stroke-width:1px
    classDef config fill:#fff59d,stroke:#333,stroke-width:1px
    classDef data fill:#b3e5fc,stroke:#333,stroke-width:1px
    classDef model fill:#c8e6c9,stroke:#333,stroke-width:1px
    classDef compute fill:#a5d6a7,stroke:#333,stroke-width:1px
    classDef utils fill:#eeeeee,stroke:#333,stroke-width:1px
    classDef viz fill:#ffe0b2,stroke:#333,stroke-width:1px
```
