# stream-ad/exgan architecture

> Architecture diagram of the GitHub repository stream-ad/exgan, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stream-ad/exgan
- Repository: https://github.com/stream-ad/exgan

## Overview

##### Project type and purpose
- ExGAN is a research-oriented machine-learning pipeline (not a web app or microservice) whose goal is to generate “extreme” precipitation samples using Generative Adversarial Networks (GANs).
- It implements a DCGAN baseline, applies a distribution-shifting stage, then trains an ExGAN model that produces realistic yet extreme rainfall fields. It also provides sampling, evaluation, and visualization utilities.

##### File structure
- Top-level scripts correspond to each pipeline stage:
- PrepareData.py – downloads and preprocesses raw precipitation data from an external HTTP service.
- DCGAN.py – trains the baseline GAN.
- DistributionShifting.py – takes the DCGAN output and shifts its latent distribution toward extremes.
- ExGAN.py – trains the extreme-sample GAN on the shifted data.
- DCGANSampling.py & ExGANSampling.py – generate samples for various “extremeness” probabilities.
- DCGANRecLoss.py & ExGANRecLoss.py – compute reconstruction-loss metrics on held-out tests.
- FID.py – computes Fréchet Inception Distance.
- plot.py – visualizes rainfall maps with Matplotlib Basemap.
- IPython notebooks (EVT Analysis.ipynb, ExGAN-Colab.ipynb) – interactive analyses and demos.
- Data artifacts: real.pt (preprocessed real data), ExGANweights.pt (trained model weights).

##### README insights
- Data source: precipitation data from water.weather.gov for 2010–2016.
- Tech stack: Python, PyTorch 1.5, CUDA 10.2, TensorBoard for metrics, Matplotlib Basemap for plots, Jupyter notebooks.
- Workflow is strictly sequential: data preparation → DCGAN training → distribution shifting → ExGAN training → sampling → evaluation → visualization.

##### Main components & relationships
a. External Services & Data Stores
- Preprocessed Dataset (real.pt)
- Model Repositories (DCGAN weights, ExGANweights.pt)
1. Data Ingestion & Preparation (PrepareData.py)
2. Baseline GAN Training (DCGAN.py)
3. Distribution Shifting (DistributionShifting.py)
4. ExGAN Training (ExGAN.py)
5. Sampling / Inference (DCGANSampling.py, ExGANSampling.py)
6. Evaluation Metrics (FID.py, DCGANRecLoss.py, ExGANRecLoss.py)
7. Visualization (plot.py, notebooks)
c. Logging & Monitoring
- TensorBoard (logs from each training module)

– Raw data → PrepareData → real.pt
– real.pt → DCGAN training → DCGAN model → logs → TensorBoard
– DCGAN model → DistributionShifting → shifted dataset
– shifted dataset → ExGAN training → ExGAN model → logs → TensorBoard
– DCGAN/ExGAN models + extremeness parameters → Sampling modules → synthetic samples
– Synthetic samples + real.pt → Evaluation modules → metrics
– Metrics & samples → Visualization module / notebooks

##### Architectural patterns & design principles
- Pipeline/Batch-processing pattern: distinct sequential stages.
- Separation of concerns: data prep, model training, distribution shift, inference, evaluation, visualization all modularized.
- Reproducibility & logging: TensorBoard integration for experiment tracking.
- Data & model artifact management: persisted to .pt files.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Data Layer"
        API["External Precipitation API"]:::cloud
        Prep["PrepareData.py\n(download & preprocess)"]:::compute
        real["real.pt\n(preprocessed dataset)"]:::artifact
    end
    API --> Prep --> real

    subgraph "Modeling Layer"
        DCGAN["DCGAN.py\n(baseline GAN training)"]:::compute
        DCGAN_model["DCGAN Model\n(DCGANweights.pt)"]:::artifact
        Shift["DistributionShifting.py\n(distribution shift)"]:::compute
        shifted["Shifted Dataset"]:::artifact
        ExGAN["ExGAN.py\n(extreme GAN training)"]:::compute
        ExGAN_model["ExGAN Model\n(ExGANweights.pt)"]:::artifact
        TB1["TensorBoard\n(logs)"]:::logs
        TB2["TensorBoard\n(logs)"]:::logs
    end
    real --> DCGAN --> DCGAN_model --> TB1
    DCGAN_model --> Shift --> shifted
    shifted --> ExGAN --> ExGAN_model --> TB2

    subgraph "Inference & Evaluation Layer"
        DCGAN_Sample["DCGANSampling.py\n(generate samples)"]:::compute
        ExGAN_Sample["ExGANSampling.py\n(generate extreme samples)"]:::compute
        samples["Synthetic Samples"]:::artifact
        FID["FID.py\n(FID metric)"]:::compute
        RecDC["DCGANRecLoss.py\n(rec loss DCGAN)"]:::compute
        RecEx["ExGANRecLoss.py\n(rec loss ExGAN)"]:::compute
        metrics["Evaluation Metrics"]:::artifact
    end
    DCGAN_model --> DCGAN_Sample --> samples
    ExGAN_model --> ExGAN_Sample --> samples
    samples --> FID --> metrics
    samples --> RecDC --> metrics
    samples --> RecEx --> metrics
    real --> FID
    real --> RecDC
    real --> RecEx

    subgraph "Visualization Layer"
        plot["plot.py\n(static plots)"]:::compute
        NB1["EVT Analysis.ipynb"]:::notebook
        NB2["ExGAN-Colab.ipynb"]:::notebook
    end
    metrics --> plot --> NB1
    metrics --> NB2
    samples --> plot

    subgraph "Legend"
        direction LR
        C1["Rectangle: Compute Module"]:::compute
        C2["Cylinder: Data/Model Artifact"]:::artifact
        C3["Cloud: External Service"]:::cloud
        C4["Ellipse: Notebook"]:::notebook
        C5["Cylinder: Logs"]:::logs
    end

    classDef compute fill:#f9c784,stroke:#333,stroke-width:1px
    classDef artifact fill:#84c1f9,stroke:#333,stroke-width:1px
    classDef cloud fill:#cccccc,stroke:#333,stroke-width:1px
    classDef notebook fill:#a2f984,stroke:#333,stroke-width:1px
    classDef logs fill:#d4d4d4,stroke:#333,stroke-width:1px

    click Prep "https://github.com/stream-ad/exgan/blob/master/PrepareData.py"
    click DCGAN "https://github.com/stream-ad/exgan/blob/master/DCGAN.py"
    click Shift "https://github.com/stream-ad/exgan/blob/master/DistributionShifting.py"
    click ExGAN "https://github.com/stream-ad/exgan/blob/master/ExGAN.py"
    click DCGAN_Sample "https://github.com/stream-ad/exgan/blob/master/DCGANSampling.py"
    click ExGAN_Sample "https://github.com/stream-ad/exgan/blob/master/ExGANSampling.py"
    click FID "https://github.com/stream-ad/exgan/blob/master/FID.py"
    click RecDC "https://github.com/stream-ad/exgan/blob/master/DCGANRecLoss.py"
    click RecEx "https://github.com/stream-ad/exgan/blob/master/ExGANRecLoss.py"
    click plot "https://github.com/stream-ad/exgan/blob/master/plot.py"
    click NB1 "https://github.com/stream-ad/exgan/blob/master/EVT Analysis.ipynb"
    click NB2 "https://github.com/stream-ad/exgan/blob/master/ExGAN-Colab.ipynb"
    click real "https://github.com/stream-ad/exgan/blob/master/real.pt"
    click ExGAN_model "https://github.com/stream-ad/exgan/blob/master/ExGANweights.pt"
    click README "https://github.com/stream-ad/exgan/blob/master/README.md"
```
