How nftblackmagic/diffusion-tryon-trainer works
Project Type and Purpose
• This is a research‐oriented full-stack ML training & inference pipeline for OOTDiffusion, a diffusion‐based virtual try-on system.
• It extends the original OOTDiffusion code by integrating Huggingface’s Diffusers library and targets the VITON dataset for training and inference.
High-Level File Structure Analysis
• Top‐level scripts & configs:
• train_ootd.sh, inference_test_dataset.sh, environment.yaml, install_conda.sh
• run/ folder: user‐facing inference entrypoints (gradio_ootd.py, run_ootd.py)
• Core application code under ootd/:
• models/: custom modules (ReferenceEncoder, UNet variants)
• pipelines_ootd/: diffusion pipelines (attention_garm, transformer_vton_2d, etc.)
• inference_* .py & train*.py: orchestration scripts for inference and training
• Data preparation:
• data_scripts/: CP (Cloth‐Parsing) dataset extraction
• preprocess/: human parsing, openpose annotation, dataset transforms
• Utilities & dependencies:
• utils/, run/utils_ootd.py for helper functions
• environment.yaml describes Python & library requirements
Key README Insights
• Trains on VITON-HD; requires pre-warped cloth, masks, captions
• Uses Huggingface Diffusers for diffusion model training
• Separate pipelines for high‐res (HD) and garment condition (GARM) variants
• Checkpoints stored under checkpoints/ for inference
Components & Interactions
a. Main Components
1. Data Sources
• VITON-HD raw images & annotations
• Pre-warped cloth & masks
• Cloth captions
2. Preprocessing Layer
• CP dataset extraction (data_scripts/cp_dataset.py)
• Human parsing pipeline (preprocess/humanparsing)
• OpenPose annotation (preprocess/openpose)
3. Training Pipeline
• Dataset loader (ootd/data/dataset.py)
• Model modules (ReferenceEncoder, UNet variants)
• Diffusion pipelines (pipelines_ootd/*.py) using Huggingface Diffusers
• Training scripts (ootd/train.py, train_ootd_hd.py, train_t2g.py)
• Checkpoint manager (Huggingface Trainer/API)
4. Inference Pipeline
• Inference scripts (ootd/inference_ootd*.py)
• Run wrappers (run/run_ootd.py, gradio_ootd.py)
• Reads checkpoints, applies diffusion pipelines, writes output images
5. External Libraries & Services
• Huggingface Diffusers & Transformers
• PyTorch (CUDA)
• Data storage (local FS or network)
• Optional: GPU cluster / scheduler
b. Relationships & Data Flow
• Raw VITON files → Preprocessing layer → Tensor datasets
• Preprocessed tensors → Training pipeline → Model checkpoints
• Checkpoints + new inputs → Inference pipeline → Synthesized try-on images
• CLI scripts or Gradio front-end trigger inference step
c. Architectural Patterns & Principles
• Modular ML pipeline: clear separation between data prep, model definition, training, inference
• Plugin‐style diffusion pipelines: each variant (GARM, VTON) lives in its own module
• Script‐based orchestration: entrypoints wrap Python modules, enabling HPC cluster usage
d. Relevant Technologies & Frameworks
• Python 3.8+, Conda environments
• PyTorch + CUDA
• Huggingface Diffusers library
• Human parsing tools (Detectron2, custom modules)
• OpenPose for keypoint annotation