How xhkzdepartedream/flow-matching works
Project Type and Purpose
• This is a research‐grade, full‐pipeline machine learning framework for continuous‐time diffusion models (“Flow‐Matching”) built in PyTorch.
• It ingests high-res images (CelebA-HQ), compresses them via a VAE, and trains a lightweight DiT (Diffusion Transformer) in the latent space using flow‐matching schedulers.
High-Level Components
a. Configuration Manager
• Directory: configs/*.yaml
• Loads hyperparameters (model size, data paths, scheduler settings, multi-GPU flags).
b. Data Ingestion & Preprocessing
• Directory: data_processing/
• Scripts fetch/download raw datasets (CelebA-HQ, FFHQ256), build training datasets, extract VGG16 features if needed, and implement dataset utilities (e.g., tree.py).
c. VAE Compression Module
• Files: modules/autoencoderkl.py, modules/vae_modules.py
• Implements AutoencoderKL to compress 256×256 images → 32×32×4 latent codes.
• Used in a two‐stage training: pretrain/fine-tune VAE then freeze encoder for diffusion.
d. Core Diffusion Model
• Directory: models/
• DiT.py: Transformer‐based diffusion architecture operating in latent space.
• Unet.py: alternate U-Net code path if selected via config.
e. Flow-Matching Scheduler & Sampler
• Directory: fm_scheduler/
• OTScheduler & Sampler orchestrate the continuous‐time flow, sampling timesteps, computing vector fields.
f. Training Orchestration
• Directory: trainer/
• Trainer.py: generic loop over batches, optimizer steps, logging.
• AutoencoderKL_trainer.py: specialized VAE training.
g. Execution Scripts
• Directory: scripts/
• scripts/train/train.py & train_finetune_autoencoderkl.py kick off multi-GPU training with torchrun.
• batch_sample.py + variants: use trained weights to generate images.
• compute_fid.py, visualize_latent_space.py, check_recon_effect.py: evaluation and visualization.
h. Utilities & Outputs
• utils.py: helper functions (logging, model I/O, LMDB saving).
• visuals_celebahq/: sample outputs and plots.
i. External Resources
• Multi-GPU cluster (torchrun), external storage for model weights (Google Drive).
Relationships & Data Flows
1. Config Manager → Trainer & Scripts
• YAML → Python dict → model, scheduler, data pipeline initialization.
2. Data Ingestion → Dataset → Trainer
• Raw images → data_processing → torch Dataset → DataLoader in Trainer.
3. VAE Module
• Trainer pre‐trains AutoencoderKL → saves encoder/decoder weights.
4. Diffusion Training
• Frozen encoder produces latents → DiT + Flow-Matching Scheduler → Trainer updates DiT parameters.
5. Checkpoints & Logs
• Trainer writes model weights, training metrics (FID, loss) to disk.
6. Sampling & Evaluation
• batch_sample scripts load DiT + Scheduler → generate latents → decode via VAE decoder → output images.
• compute_fid script compares outputs to ground truth distribution.
Architectural Patterns & Principles
• Modular Design: clear separation of data, models, schedulers, training loops, and scripts.
• Config‐Driven: all hyperparameters, paths, model choices live in YAML.
• Two‐Stage Pipeline: VAE compression stage + diffusion stage.
• Multi-GPU Friendly: use torchrun, distributed DataParallel/Broadcast.