How stream-ad/exgan works
Project type and purpose
• ExGAN is a research-oriented machine-learning pipeline (not a web app or microservice) whose goal is to generate “extreme” precipitation samples using Generative Adversarial Networks (GANs).
• It implements a DCGAN baseline, applies a distribution-shifting stage, then trains an ExGAN model that produces realistic yet extreme rainfall fields. It also provides sampling, evaluation, and visualization utilities.
File structure
• Top-level scripts correspond to each pipeline stage:
• PrepareData.py – downloads and preprocesses raw precipitation data from an external HTTP service.
• DCGAN.py – trains the baseline GAN.
• DistributionShifting.py – takes the DCGAN output and shifts its latent distribution toward extremes.
• ExGAN.py – trains the extreme-sample GAN on the shifted data.
• DCGANSampling.py & ExGANSampling.py – generate samples for various “extremeness” probabilities.
• DCGANRecLoss.py & ExGANRecLoss.py – compute reconstruction-loss metrics on held-out tests.
• FID.py – computes Fréchet Inception Distance.
• plot.py – visualizes rainfall maps with Matplotlib Basemap.
• IPython notebooks (EVT Analysis.ipynb, ExGAN-Colab.ipynb) – interactive analyses and demos.
• Data artifacts: real.pt (preprocessed real data), ExGANweights.pt (trained model weights).
README insights
• Data source: precipitation data from water.weather.gov for 2010–2016.
• Tech stack: Python, PyTorch 1.5, CUDA 10.2, TensorBoard for metrics, Matplotlib Basemap for plots, Jupyter notebooks.
• Workflow is strictly sequential: data preparation → DCGAN training → distribution shifting → ExGAN training → sampling → evaluation → visualization.
Main components & relationships
a. External Services & Data Stores
• Preprocessed Dataset (real.pt)
• Model Repositories (DCGAN weights, ExGANweights.pt)
1. Data Ingestion & Preparation (PrepareData.py)
2. Baseline GAN Training (DCGAN.py)
3. Distribution Shifting (DistributionShifting.py)
4. ExGAN Training (ExGAN.py)
5. Sampling / Inference (DCGANSampling.py, ExGANSampling.py)
6. Evaluation Metrics (FID.py, DCGANRecLoss.py, ExGANRecLoss.py)
7. Visualization (plot.py, notebooks)
c. Logging & Monitoring
• TensorBoard (logs from each training module)
– Raw data → PrepareData → real.pt
– real.pt → DCGAN training → DCGAN model → logs → TensorBoard
– DCGAN model → DistributionShifting → shifted dataset
– shifted dataset → ExGAN training → ExGAN model → logs → TensorBoard
– DCGAN/ExGAN models + extremeness parameters → Sampling modules → synthetic samples
– Synthetic samples + real.pt → Evaluation modules → metrics
– Metrics & samples → Visualization module / notebooks
Architectural patterns & design principles
• Pipeline/Batch-processing pattern: distinct sequential stages.
• Separation of concerns: data prep, model training, distribution shift, inference, evaluation, visualization all modularized.
• Reproducibility & logging: TensorBoard integration for experiment tracking.
• Data & model artifact management: persisted to .pt files.