How carlosaccardi/psd-downscaling works
Project type and purpose
• This is a research‐oriented deep learning codebase for statistical downscaling of meteorological data (“PSD-Downscaling”).
• It implements multiple models (deterministic U-Net regression, probabilistic UNet with CRPS loss, and CorrDiff conditional diffusion), aiming to translate low-resolution ERA5 data to high-resolution CERRA outputs while preserving PSD characteristics.
File structure
• Top-level scripts:
• main.py (CLI entry point for training/evaluation)
• metrics.py (evaluation metrics)
• requirements.txt (dependencies)
• src/
• config/ (configuration loading & defaults)
• data/ (dataset classes for ERA5→CERRA tiles)
• models/
– unet.py
– diffusion.py
– losses/ (Fourier & CRPS loss implementations)
• training/ (training loop utilities)
• utils/ (plotting, helper functions)
• yaml_configs/ (model/hyperparameter definitions by experiment)
• saved_models/ (generated at runtime for checkpoints)
• CI: .github/workflows/pre-commit.yml
Key insights from README
• Workflow: main.py reads a YAML config → loads datasets → constructs model → runs training or evaluation → logs to W&B → saves checkpoints.
• Branches split CNN-UNet & CorrDiff vs. CRPS U-Net workflows.
• External systems: Weights & Biases for logging; PyTorch Lightning under the hood (implied by checkpoint conventions).
Main components
A. Configuration Layer
• YAML configs (yaml_configs/)
• src/config/base_config.py & legacy_compat.py
B. Data Pipeline
• Dataset loader (src/data/dataset.py)
• Preprocessing utilities (src/utils/utils.py)
C. Model Layer
• UNet (src/models/unet.py)
• Diffusion model (src/models/diffusion.py)
• Loss modules (src/models/losses/)
D. Training Engine
• Training loops & schedulers (src/training/)
• Metrics calculation (metrics.py)
E. Checkpoint & Logging
• Checkpoint IO (PyTorch Lightning .ckpt files in saved_models/)
• Logging service (Weights & Biases)
F. CLI & Entry Point
• main.py orchestrates all layers based on config
Relationships & interactions
• main.py reads a YAML config → instantiates Config object →
• Config drives Data Pipeline (path & batch settings) → Data Loader produces batches →
• Model Layer consumes batches, returns predictions → Loss modules compute loss →
• Training Engine updates model parameters, computes metrics →
• Checkpoints are written to disk and metrics/logs are pushed to W&B →
• Evaluation follows the same flow with eval flag.
Architectural patterns & design principles
• Modular separation of concerns (config, data, models, training, utils).
• Pluggable model definitions via config → supports swapping UNet/CorrDiff/CRPS.
• External configuration driving behavior (YAML‐driven pipelines).
• Clear I/O boundaries for checkpointing and logging.