How m2lines/samudra works
Project Type and Purpose
• Project type: An AI-driven scientific emulator for global ocean modeling (a specialized ML pipeline).
• Purpose: Emulate 3D ocean dynamics (sea surface height, velocities, temperature, salinity) at high fidelity and extreme speed—150× faster than the original OM4 climate model.
File Structure
Top-level directories & files:
• configs/ ← YAML configs for training & rollout
• src/ ← Core Python code for data, model, training, rollout, utilities
• notebooks/ ← Jupyter notebooks for plotting and paper figures
• assets/ ← Visual assets (e.g., globe.gif)
• environment.yml ← Conda environment specification
Key scripts in src/:
• train.py ← Orchestrates model training
• rollout.py ← Orchestrates model inference (rollout)
• stepper.py ← Advances time steps in the emulator
• datasets.py ← Data ingestion & preprocessing
• models/ ← Model definitions (activations, blocks, factory, Samudra core)
• utils/ ← Logging, distributed launch, device placement, writers
Extract Architectural Insights from the README
• Data layer: OM4 data & precomputed statistics (means/stds) loaded via Zarr.
• Config layer: Two YAML files (train_samudra_om4.yaml, rollout_samudra_om4.yaml) drive all hyperparameters, I/O paths, and sub-experiment names.
• Compute layer: PyTorch + torchrun for distributed training; single-process Python for rollouts.
• Outputs: Model checkpoints, Zarr outputs of emulated ocean trajectories, plots via notebooks.
Main Components
a. External Data Sources
– OM4 Zarr datasets (remote storage endpoints)
– OM4 statistics (means, stds)
b. Configuration Management
– configs/*.yaml
– src/config.py (parses & validates)
c. Data Ingestion & Preprocessing
– src/datasets.py
– src/utils/data.py
d. Model Definition
– src/models/activations.py
– src/models/blocks.py
– src/models/factory.py
– src/models/samudra.py
e. Training Orchestration
– src/train.py
– src/stepper.py
– src/utils/train.py
– src/utils/distributed.py
– src/utils/writer.py
f. Rollout / Inference
– src/rollout.py
– stepper (same as training)
g. Utilities
– Logging (src/utils/logging.py)
– Device management (src/utils/device.py)
– Notebook helpers (src/utils/notebook.py)
h. Visualization & Analysis
– notebooks/samudra_plotting.ipynb
– assets/
Relationships & Data Flows
• Configuration files → Configuration parser → Training/Rollout scripts
• Remote Zarr data → Data ingestion modules → Normalization (means/stds) → Model inputs
• Model inputs → Samudra model → Predicted next ocean state
• Stepper → Advances through time sequence → Feeds predictions back as inputs
• Training loop → Loss computation → Backpropagation → Checkpoint writer
• Rollout → Checkpoint loader → Emulated trajectory → Zarr output
• Notebooks → Load outputs/checkpoints → Generate figures
Architectural Patterns & Principles
• Modular separation: data / model / orchestrator / utils
• Configuration-driven: All hyperparams and I/O controlled by YAML + CLI flags
• Reusable components: stepper used by both train & rollout
• Distributed training support via torchrun + src/utils/distributed.py
• Clear I/O abstraction: data ingestion and writing results decoupled
Final Checklist
[ ] All major components present and labeled
[ ] Config layer linked to both training & rollout
[ ] External dependencies (Zarr endpoints, HF weights) shown
[ ] Utility modules positioned per their use in each pipeline stage