How yisol/idm-vton works
Project type and purpose
• This is a research-oriented machine-learning project (“IDM-VTON”) that implements a diffusion-based virtual try-on pipeline.
• It includes data preparation, model training (XL diffusion with IP-Adapter), inference (on VITON-HD and DressCode), and a Gradio web demo for DensePose.
File structure
Top-level directories and their roles:
• ckpt – pre-trained checkpoints (DensePose, human parsing, OpenPose, IP-Adapter, image encoder)
• configs – YAML configs for training/inference schedules and model variants
• preprocess – scripts and code for human parsing, DensePose, OpenPose and dataset conversion
• ip_adapter – integration code for IP-Adapter plug-in modules
• src – core try-on pipeline and hacked network architectures (UNet, transformer, attention)
• gradio_demo – standalone DensePose demo (not the main try-on demo)
• train_xl.py / inference.py / inference_dc.py / gradio_demo/app.py – entry-point scripts
• environment.yaml – conda environment specification
README for additional insights
• Uses Hugging Face accelerate, 8-bit Adam, Detectron2, IP-Adapter, custom diffusion UNet forks.
• Two inference modes (VITON-HD and DressCode) and a local Gradio DensePose demo.
• Data expectations: VITON-HD and DressCode folder layouts.
First files called and call hierarchy
A. Training flow
1) Entry: train_xl.py
2) Parses CLI args, loads configs and ckpt/ip_adapter + ckpt/image_encoder
3) Calls core pipeline in src/tryon_pipeline.py
– Instantiates dataset loader (preprocessed VITON-HD or DressCode)
– Builds modified UNet/Transformer/Attention models (src/*_hacked_*.py)
– Wraps IP-Adapter modules (ip_adapter/ip_adapter.py)
– Feeds data into diffusion training loop (accelerate launch)
4) Outputs model weights to --output_dir
B. Inference flow
1) Entry: inference.py or inference_dc.py
2) Loads trained model weights and same IP-Adapter+image encoder ckpt
3) Calls inference branch in src/tryon_pipeline.py (or a dedicated apply_net module)
– Loads test dataset, runs diffusion sampling (guided by prompts or inputs)
– Saves generated images to --output_dir
C. Gradio DensePose demo
1) Entry: gradio_demo/app.py
2) Sets up web UI (Gradio), uses callbacks in gradio_demo/apply_net.py
3) apply_net imports gradio_demo/densepose/* modules:
– Configs, converters, data loaders, engine/trainer, modeling, vis
– Uses Detectron2 under the hood (embedded inside gradio_demo/detectron2)
4) Renders DensePose IUV overlays in browser
Main components
• Data Sources
– VITON-HD dataset (images + JSON tags)
– DressCode dataset (images + captions + DensePose images)
• Preprocessing Modules
– Human parsing (SCHP ONNX in preprocess/humanparsing)
– DensePose & OpenPose (detectron2 under preprocess/openpose & gradio_demo/detectron2)
– Dataset converters (preprocess scripts)
• Model Training Component
– Entry: train_xl.py → tryon_pipeline (src) → networks (src/unet_*, src/transformer_*) → IP-Adapter (ip_adapter)
– Hyperparameters from configs/*.yaml
– Checkpoints from ckpt/*
• Model Inference Component
– Entry: inference.py / inference_dc.py → same pipeline modules as training but inference mode
• Demo Component
– Entry: gradio_demo/app.py → apply_net → densepose modules → web UI
• External Services & Libraries
– PyTorch, Hugging Face Accelerate
– Detectron2 (for DensePose/OpenPose)
– Gradio (for demo)
– ONNX runtime (for human parsing)
Relationships & data flow
• Trained Model → Inference Component → Generated Images
• Generated Images → (optional) Demo or downstream evaluation
• Demo → Live calls to DensePose modules and cached ckpts
Architectural patterns & design principles
• Modular pipeline separation: preprocessing, training, inference, demo
• Config-driven experiments (all hyperparams in YAML)
• Reusable core pipeline (src/tryon_pipeline.py) for both training & inference
Essential elements
• Clear component labels (Data, Preprocessing, Pipeline, Demo)
• Entry-point files at the top of each component
• External dependencies boxed separately