How donggeun-yoon/dgdm works
Project type and purpose
• This is a research-oriented machine-learning pipeline implementing “Deterministic Guidance Diffusion Model” (DGDM) for probabilistic weather (or video) forecasting.
• It can run on multiple datasets (Moving MNIST, PNW-Typhoon) by swapping configuration files and dataset classes.
File structure
• Top-level script:
• main.py – command-line entrypoint for training and testing
• Register.py – central registry for models, datasets, runners (reflects plugin pattern)
• Configuration:
• configs/*.yaml – dataset-specific hyperparameters (diffusion steps, network depth, data paths)
• Data management:
• data/ – raw data storage and download scripts
• datasets/ – PyTorch-style Dataset subclasses for each data type
• Model definitions:
• model/Unet3D.py – 3D U-Net backbone for spatiotemporal encoding/decoding
• model/BrownianBridgeModel.py – diffusion model wrapper incorporating deterministic guidance
• model/predictor.py, utils.py – helper constructs (schedulers, samplers)
• Training & evaluation orchestration:
• runners/BaseRunner.py – generic training loop (forward, backward, logging)
• runners/BBDMRunner.py – specialized runner tying the diffusion model, EMA, loss functions
• runners/EMA.py – exponential moving average module for model weights
• Utilities:
• utils.py – shared helpers (logging, checkpoint I/O, metric computation)
Key insights from README
• Architecture image located at resources/architecture.png outlines data flow from raw inputs → diffusion backbone → sample generation
• Usage revolves around invoking main.py with a config YAML, triggering dataset loading, model instantiation, and runner execution
Main components
a. Configuration Loader
– Parses YAML, sets hyperparameters, paths, dataset/model/runner types
b. Registry Module (Register.py)
– Maps string identifiers → classes (Dataset, Model, Runner)
c. Data Pipeline
– Raw data on disk → datasets/*.py → batch tensor loader
d. Model Module
– Unet3D encoder/decoder
– BrownianBridgeModel orchestrator (adds noise, applies reverse diffusion with deterministic guidance)
e. Training Runner
– BaseRunner logic: epoch loop, gradient updates
– BBDMRunner specialization: custom loss, sampling steps
– EMA service tracking smoothed weights
f. Checkpointing & Logging
– utils.py for saving model states, metrics, sample outputs
g. Evaluation/Testing
– Reuses runner to generate probabilistic forecasts and compute statistics
Relationships & data flow
• Data Pipeline → minibatch tensors → Training Runner → Model forward/backward → EMA → updated weights
• Training Runner → Checkpoint module (periodic) → disk
• Testing Runner (modeled as a branch) → Model sampling → generated forecast outputs
Architectural patterns & design principles
• Separation of concerns: config vs. data vs. model vs. runner
• Plugin/registry: easily swap datasets or model variants without editing core code
• Modular pipeline: data ingestion, diffusion process, training orchestration
Relevant technologies & frameworks
• Python 3.9
• PyTorch for tensor operations, model definitions, training loops
• NumPy for data handling
• YAML for config files
• Bash scripts for data download