# bbbbihr/nowcasting-of-extreme-precipitation architecture

> Architecture diagram of the GitHub repository bbbbihr/nowcasting-of-extreme-precipitation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bbbbihr/nowcasting-of-extreme-precipitation
- Repository: https://github.com/bbbbihr/nowcasting-of-extreme-precipitation

## Overview

##### Project Type and Purpose
- This is a scientific machine‐learning pipeline for extreme precipitation nowcasting and detection. It bundles:
- A PySTEPS‐based baseline evaluation (Benchmark_test)
- An event‐selection and data‐analysis module
- A two‐stage deep‐learning model (VQGAN + Transformer)
- A shared RainyMotion library for data slicing, metrics, and utilities

- Raw radar/precipitation files (e.g. NetCDF/HDF5)
- Catchment definitions (mask .npy)
- Catchment_slice_RM.py (data slicing)
- metrics.py (evaluation functions)
- models.py (baseline or auxiliary models)
- utils.py (I/O helpers)
- event_selection_extreme_opti.py → computes accumulation & thresholds
- event_selection_function.py → splits into train/val/test sets
- event_analysis.py → basic statistics & visualization
- HPCrunScripts/*.py → scripts for running deterministic & probabilistic nowcasts
- config/ → PySTEPS configuration
- vqgan+transformer/vqgan_training.ipynb → trains the VQGAN encoder/decoder
- notebook or script that takes radar frames + pretrained VQGAN → latent token sequences
- transformer_training.ipynb → trains autoregressive transformer on token sequences
- experiment_nowcasting.ipynb → generates nowcasting outputs
- experiment_extreme_detection.ipynb → detects extreme events
- Results saved as .npy or visualizations
- README.md, docs/ folders, environment.yml, setup.py

1. Data Sources → RainyMotion Library (slicing, masking)
2. → Event Selection & Data Analysis → generates labeled train/val/test datasets
3. → VQGAN Model Training → outputs VQGAN model artifact
4. VQGAN model + Data → Tokenization → token sequences
5. → Transformer Model Training → outputs Transformer model artifact
6. → Experiment & Evaluation → nowcast maps and extreme‐event detections
7. Separately, Data Sources + PySTEPS Benchmark Module → baseline nowcasting results

##### Architectural Patterns & Principles
- Modular pipeline: each stage encapsulated in its own folder/notebook
- Reusable library (RainyMotion) for shared code
- Two‐stage deep‐learning (representation learning + sequence modeling)
- Separation of benchmark (PySTEPS) and research model

## Diagram (Mermaid)

```mermaid
flowchart TD
    DataSources["Data Sources<br>Raw radar files<br>Catchment masks"]:::rawdata
    RainyMotion["RainyMotion Library<br>Catchment_slice_RM.py<br>metrics.py<br>models.py<br>utils.py"]:::library
    EventSel["Event Selection & Analysis<br>event_selection_extreme_opti.py<br>event_selection_function.py<br>event_analysis.py"]:::preprocess
    VQGAN["VQGAN Model Training<br>vqgan_training.ipynb"]:::model
    Token["Tokenization<br>latent token sequences"]:::model
    Transformer["Transformer Model Training<br>transformer_training.ipynb"]:::model
    Exp["Experiment & Evaluation<br>experiment_nowcasting.ipynb<br>experiment_extreme_detection.ipynb"]:::experiment
    PySTEPS["PySTEPS Benchmark Module<br>HPCrunScripts/*.py<br>config/"]:::preprocess
    Docs["Documentation & Environment<br>README.md<br>docs/, env files"]:::docs

    DataSources -->|"raw data"| RainyMotion
    RainyMotion --> EventSel
    EventSel -->|"train/val/test .npy"| VQGAN
    VQGAN -->|"VQGAN.ckpt + data"| Token
    Token -->|"token_seq.npy"| Transformer
    Transformer -->|"Transformer.ckpt"| Exp

    DataSources -->|"raw data"| PySTEPS
    PySTEPS -->|"baseline nowcasts"| Exp

    RainyMotion -.-> VQGAN
    RainyMotion -.-> Transformer

    click DataSources "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/tree/main/rainymotion/data/"
    click RainyMotion "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/rainymotion/rainymotion/Catchment_slice_RM.py"
    click RainyMotion "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/rainymotion/rainymotion/metrics.py"
    click RainyMotion "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/rainymotion/rainymotion/models.py"
    click RainyMotion "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/rainymotion/rainymotion/utils.py"
    click EventSel "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/event selection and data analysis/event_selection_extreme_opti.py"
    click EventSel "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/event selection and data analysis/event_selection_function.py"
    click EventSel "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/event selection and data analysis/event_analysis.py"
    click PySTEPS "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/Benchmark_test/HPCrunScripts/PS_DeterministicNowcast_parallel_24h.py"
    click PySTEPS "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/Benchmark_test/HPCrunScripts/PS_ProbNowcast_parallel_24h.py"
    click PySTEPS "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/tree/main/Benchmark_test/HPCrunScripts/config/"
    click VQGAN "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/vqgan+transformer/vqgan_training.ipynb"
    click Transformer "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/vqgan+transformer/transformer_training.ipynb"
    click Exp "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/vqgan+transformer/experiment_nowcasting.ipynb"
    click Exp "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/vqgan+transformer/experiment_extreme_detection.ipynb"
    click Docs "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/README.md"
    click Docs "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/rainymotion/README.md"
    click Docs "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/Benchmark_test/pysteps/environment.yml"
    click Docs "https://github.com/bbbbihr/nowcasting-of-extreme-precipitation/blob/main/rainymotion/environment.yml"

    classDef rawdata fill:#a8ddb5,stroke:#2b7a0b,color:#000;
    classDef library fill:#deebf7,stroke:#2171b5,color:#000;
    classDef preprocess fill:#9ecae1,stroke:#08519c,color:#000;
    classDef model fill:#fdae6b,stroke:#e6550d,color:#000;
    classDef experiment fill:#bcbddc,stroke:#6a51a3,color:#000;
    classDef docs fill:#d9d9d9,stroke:#636363,color:#000;
```
