How nickcp39/2025_datageneration works
Project Type and Purpose
• This is an open-source research pipeline for generative model training and sampling (Diffusion/GAN style) on grayscale images (~8K).
• It provides a minimal reproducible end-to-end flow: data loading → preprocess → custom diffusion engine → UNet ε-prediction model → training with EMA previews → standalone sampling.
File Structure Analysis
Top-level directories/files:
• configs/ → YAML files for local vs. cloud hyperparams
• scripts/ → PowerShell scripts to launch training/sampling on various GPUs
• splits/ → train/val/test file lists
• yc_code/ → core Python package
– dataset(.py) → data loader, crop/resize/normalize/augment
– diffusion/ → self-built DiffusionEngine (β/ᾱ tables, DDPM/DDIM)
– models/ → UNet ε-predictor implementations
– engine.py → glue for training & sampling loops
– train.py / train_diffusion.py / sample_diffusion.py → CLI entrypoints
– utils.py/asserts → consistency checks (prediction_type, image_range, channels)
• readme.md → project overview, quickstart, design constraints
• requirements*.txt → dependencies (PyTorch cu121)
• setup.sh / train.sh / sample.sh → UNIX helper scripts
Key README Insights
• Single Source of Truth: all training & sampling use the custom DiffusionEngine.
• Data range enforced: [0,1] → training [-1,1] → sampling back to [0,1].
• UNet always in_ch == out_ch == channels.
• Automatic pre-training assertions guard consistency.
• CLI flags for mid-bottleneck attention, DDPM/DDIM preview, EMA usage.
Main Components & Data Flows
a. Data Ingestion & Preprocessing
• Source: local folder data2025/ (ignored in repo)
• Split lists: splits/{train,val,test}.txt
• Loader: yc_code/dataset.py (center-crop, resize, normalize to [0,1], optional flip)
b. Model & Engine
• UNet ε-predictor (yc_code/models/unet_eps.py or hf_unet2d.py if HF used)
• DiffusionEngine: β/ᾱ schedules, single‐step DDPM + multi‐step DDIM
c. Training Pipeline
• Entry: train_diffusion.py
• Reads configs (cloud_ddp.yaml / local_3060.yaml) + CLI args
• Builds engine + model + optimizer + EMA
• Logs to .runs/, checkpoints to .runs/.../ckpts
• Periodic EMA sampling (DDPM/DDIM) for previews
d. Sampling Pipeline
• Entry: sample_diffusion.py
• Loads checkpoint → engine → generates N images → denormalizes → saves to out_dir
e. Infrastructure & Utilities
• Shell/PS scripts orchestrate runs on local GPUs or AWS EC2
• configs/*.yaml for hyperparam management
Architectural Patterns & Principles
• Modular separation: data loader, model, engine, training loop, sampling loop
• Single Source of Truth engine enforces consistency across training & sampling
• CLI + config files for reproducibility and environment abstraction
• Consistency assertions as hard “guard rails” to catch misconfigurations early
2) Preprocessor output to Training Loop node (train_diffusion.py + engine.py)
5) Sampling Loop connects back to DiffusionEngine + UNet Model for inference, then → Output Images
6) Surround the whole with Scripts & Configs layer to show orchestration (scripts/run_*.ps1, configs/*.yaml)
7) Optionally, annotate that everything runs on GPU nodes (local 3060/4090 or cloud DDP).