# nickcp39/2025_datageneration architecture

> Architecture diagram of the GitHub repository nickcp39/2025_datageneration, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nickcp39/2025_datageneration
- Repository: https://github.com/nickcp39/2025_datageneration

## Overview

##### Project Type and Purpose
- This is an open-source research pipeline for generative model training and sampling (Diffusion/GAN style) on grayscale images (~8K).
- It provides a minimal reproducible end-to-end flow: data loading → preprocess → custom diffusion engine → UNet ε-prediction model → training with EMA previews → standalone sampling.

##### File Structure Analysis
Top-level directories/files:
- configs/               → YAML files for local vs. cloud hyperparams
- scripts/               → PowerShell scripts to launch training/sampling on various GPUs
- splits/                → train/val/test file lists
- yc_code/               → core Python package
– dataset(.py)       → data loader, crop/resize/normalize/augment
– diffusion/         → self-built DiffusionEngine (β/ᾱ tables, DDPM/DDIM)
– models/            → UNet ε-predictor implementations
– engine.py          → glue for training & sampling loops
– train.py / train_diffusion.py / sample_diffusion.py → CLI entrypoints
– utils.py/asserts   → consistency checks (prediction_type, image_range, channels)
- readme.md              → project overview, quickstart, design constraints
- requirements*.txt      → dependencies (PyTorch cu121)
- setup.sh / train.sh / sample.sh → UNIX helper scripts

##### Key README Insights
- Single Source of Truth: all training & sampling use the custom DiffusionEngine.
- Data range enforced: [0,1] → training [-1,1] → sampling back to [0,1].
- UNet always in_ch == out_ch == channels.
- Automatic pre-training assertions guard consistency.
- CLI flags for mid-bottleneck attention, DDPM/DDIM preview, EMA usage.

##### Main Components & Data Flows
a. Data Ingestion & Preprocessing
- Source: local folder `data2025/` (ignored in repo)
- Split lists: splits/{train,val,test}.txt
- Loader: yc_code/dataset.py (center-crop, resize, normalize to [0,1], optional flip)
b. Model & Engine
- UNet ε-predictor (yc_code/models/unet_eps.py or hf_unet2d.py if HF used)
- DiffusionEngine: β/ᾱ schedules, single‐step DDPM + multi‐step DDIM
c. Training Pipeline
- Entry: train_diffusion.py
- Reads configs (cloud_ddp.yaml / local_3060.yaml) + CLI args
- Builds engine + model + optimizer + EMA
- Logs to .runs/, checkpoints to .runs/.../ckpts
- Periodic EMA sampling (DDPM/DDIM) for previews
d. Sampling Pipeline
- Entry: sample_diffusion.py
- Loads checkpoint → engine → generates N images → denormalizes → saves to out_dir
e. Infrastructure & Utilities
- Shell/PS scripts orchestrate runs on local GPUs or AWS EC2
- configs/*.yaml for hyperparam management

##### Architectural Patterns & Principles
- Modular separation: data loader, model, engine, training loop, sampling loop
- Single Source of Truth engine enforces consistency across training & sampling
- CLI + config files for reproducibility and environment abstraction
- Consistency assertions as hard “guard rails” to catch misconfigurations early

2) Preprocessor output to Training Loop node (train_diffusion.py + engine.py)
5) Sampling Loop connects back to DiffusionEngine + UNet Model for inference, then → Output Images
6) Surround the whole with Scripts & Configs layer to show orchestration (scripts/run_*.ps1, configs/*.yaml)
7) Optionally, annotate that everything runs on GPU nodes (local 3060/4090 or cloud DDP).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Components
    DataSource["Data Source\ndata2025"]:::data
    Splits["Dataset Splits\nsplits/train.txt,\nsplits/val.txt,\nsplits/test.txt"]:::data
    Preprocessor["Preprocessor\nyc_code/dataset.py"]:::code
    click Preprocessor "https://github.com/nickcp39/2025_datageneration/blob/main/yc_code/dataset.py"

    %% Config & CLI
    ConfigCLI["Config & CLI\nconfigs/*.yaml\nCLI Flags"]:::infra
    click ConfigCLI "https://github.com/nickcp39/2025_datageneration/blob/main/configs/cloud_ddp.yaml"
    click ConfigCLI "https://github.com/nickcp39/2025_datageneration/blob/main/configs/local_3060.yaml"
    click ConfigCLI "https://github.com/nickcp39/2025_datageneration/blob/main/configs/cloud_single.yaml"

    %% Orchestration Scripts
    Scripts["Orchestration Scripts\nscripts/*.ps1,\nshell wrappers"]:::infra
    click Scripts "https://github.com/nickcp39/2025_datageneration/blob/main/scripts/run_local_3060.ps1"
    click Scripts "https://github.com/nickcp39/2025_datageneration/blob/main/scripts/run_debug_small.ps1"
    click Scripts "https://github.com/nickcp39/2025_datageneration/blob/main/scripts/gen_sample.ps1"

    %% Training Pipeline
    TrainingLoop["Training Loop\ntrain_diffusion.py + engine.py"]:::runtime
    click TrainingLoop "https://github.com/nickcp39/2025_datageneration/blob/main/yc_code/train_diffusion.py"
    click TrainingLoop "https://github.com/nickcp39/2025_datageneration/blob/main/yc_code/engine.py"

    subgraph "Training Subcomponents"
        direction TB
        DiffEngine["DiffusionEngine\ncore schedules & sampling"]:::code
        click DiffEngine "https://github.com/nickcp39/2025_datageneration/blob/main/yc_code/diffusion/diffusion_engine.py"
        HFEngine["HF DiffusionEngine\n(hf_engine)"]:::code
        click HFEngine "https://github.com/nickcp39/2025_datageneration/blob/main/yc_code/diffusion/hf_engine.py"
        UNet["UNet ε-predictor\nmodel"]:::code
        click UNet "https://github.com/nickcp39/2025_datageneration/blob/main/yc_code/models/unet_eps.py"
        HFUNet["HF UNet\n(hf_unet2d)"]:::code
        click HFUNet "https://github.com/nickcp39/2025_datageneration/blob/main/yc_code/models/hf_unet2d.py"
        EMA["EMA Preview"]:::runtime

        TrainingLoop --> DiffEngine
        TrainingLoop --> UNet
        TrainingLoop --> EMA
        DiffEngine --- HFEngine
        UNet --- HFUNet
    end

    %% Checkpoint Store
    Checkpoint["Checkpoint Store\n.runs/.../ckpts"]:::data

    %% Sampling Pipeline
    SamplingLoop["Sampling Loop\nsample_diffusion.py"]:::runtime
    click SamplingLoop "https://github.com/nickcp39/2025_datageneration/blob/main/yc_code/sample_diffusion.py"

    Output["Output Images\nout_dir"]:::data

    %% Flows
    DataSource -->|"raw images"| Preprocessor
    Splits --> Preprocessor
    Preprocessor -->|"tensors [0,1]"| TrainingLoop
    ConfigCLI -->|"hyperparams & flags"| TrainingLoop
    Scripts --> TrainingLoop
    TrainingLoop -->|"save ckpt"| Checkpoint
    Checkpoint --> SamplingLoop
    SamplingLoop -->|"load ckpt"| DiffEngine
    SamplingLoop -->|"inference"| UNet
    SamplingLoop -->|"tensors [0,1]"| Output
    Scripts --> SamplingLoop
    ConfigCLI --> SamplingLoop

    %% Shell Wrappers
    ShellTrain["train.sh"]:::infra
    click ShellTrain "https://github.com/nickcp39/2025_datageneration/blob/main/train.sh"
    ShellSample["sample.sh"]:::infra
    click ShellSample "https://github.com/nickcp39/2025_datageneration/blob/main/sample.sh"
    ShellTrain --> TrainingLoop
    ShellSample --> SamplingLoop

    %% Styles
    classDef data fill:#DAE8FC,stroke:#6C8EBF,color:#1A237E;
    classDef code fill:#D5E8D4,stroke:#82B366,color:#145A32;
    classDef runtime fill:#FFE6CC,stroke:#D79B00,color:#874E00;
    classDef infra fill:#F2F2F2,stroke:#A6A6A6,color:#333333;
```
