# yisol/idm-vton architecture

> Architecture diagram of the GitHub repository yisol/idm-vton, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yisol/idm-vton
- Repository: https://github.com/yisol/idm-vton

## Overview

##### Project type and purpose
- This is a research-oriented machine-learning project (“IDM-VTON”) that implements a diffusion-based virtual try-on pipeline.
- It includes data preparation, model training (XL diffusion with IP-Adapter), inference (on VITON-HD and DressCode), and a Gradio web demo for DensePose.

##### File structure
Top-level directories and their roles:
- ckpt – pre-trained checkpoints (DensePose, human parsing, OpenPose, IP-Adapter, image encoder)
- configs – YAML configs for training/inference schedules and model variants
- preprocess – scripts and code for human parsing, DensePose, OpenPose and dataset conversion
- ip_adapter – integration code for IP-Adapter plug-in modules
- src – core try-on pipeline and hacked network architectures (UNet, transformer, attention)
- gradio_demo – standalone DensePose demo (not the main try-on demo)
- train_xl.py / inference.py / inference_dc.py / gradio_demo/app.py – entry-point scripts
- environment.yaml – conda environment specification

##### README for additional insights
- Uses Hugging Face accelerate, 8-bit Adam, Detectron2, IP-Adapter, custom diffusion UNet forks.
- Two inference modes (VITON-HD and DressCode) and a local Gradio DensePose demo.
- Data expectations: VITON-HD and DressCode folder layouts.

##### First files called and call hierarchy
A. Training flow
1) Entry: train_xl.py
2) Parses CLI args, loads configs and `ckpt/ip_adapter` + `ckpt/image_encoder`
3) Calls core pipeline in src/tryon_pipeline.py
– Instantiates dataset loader (preprocessed VITON-HD or DressCode)
– Builds modified UNet/Transformer/Attention models (src/*_hacked_*.py)
– Wraps IP-Adapter modules (ip_adapter/ip_adapter.py)
– Feeds data into diffusion training loop (accelerate launch)
4) Outputs model weights to `--output_dir`

B. Inference flow
1) Entry: inference.py or inference_dc.py
2) Loads trained model weights and same IP-Adapter+image encoder ckpt
3) Calls inference branch in src/tryon_pipeline.py (or a dedicated apply_net module)
– Loads test dataset, runs diffusion sampling (guided by prompts or inputs)
– Saves generated images to `--output_dir`

C. Gradio DensePose demo
1) Entry: gradio_demo/app.py
2) Sets up web UI (Gradio), uses callbacks in gradio_demo/apply_net.py
3) apply_net imports `gradio_demo/densepose/*` modules:
– Configs, converters, data loaders, engine/trainer, modeling, vis
– Uses Detectron2 under the hood (embedded inside gradio_demo/detectron2)
4) Renders DensePose IUV overlays in browser

##### Main components
- Data Sources
– VITON-HD dataset (images + JSON tags)
– DressCode dataset (images + captions + DensePose images)
- Preprocessing Modules
– Human parsing (SCHP ONNX in preprocess/humanparsing)
– DensePose & OpenPose (detectron2 under preprocess/openpose & gradio_demo/detectron2)
– Dataset converters (preprocess scripts)
- Model Training Component
– Entry: train_xl.py → tryon_pipeline (src) → networks (src/unet_*, src/transformer_*) → IP-Adapter (ip_adapter)
– Hyperparameters from configs/*.yaml
– Checkpoints from ckpt/*
- Model Inference Component
– Entry: inference.py / inference_dc.py → same pipeline modules as training but inference mode
- Demo Component
– Entry: gradio_demo/app.py → apply_net → densepose modules → web UI
- External Services & Libraries
– PyTorch, Hugging Face Accelerate
– Detectron2 (for DensePose/OpenPose)
– Gradio (for demo)
– ONNX runtime (for human parsing)

##### Relationships & data flow
- Trained Model → Inference Component → Generated Images
- Generated Images → (optional) Demo or downstream evaluation
- Demo → Live calls to DensePose modules and cached ckpts

##### Architectural patterns & design principles
- Modular pipeline separation: preprocessing, training, inference, demo
- Config-driven experiments (all hyperparams in YAML)
- Reusable core pipeline (src/tryon_pipeline.py) for both training & inference

##### Essential elements
- Clear component labels (Data, Preprocessing, Pipeline, Demo)
- Entry-point files at the top of each component
- External dependencies boxed separately

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources Layer
    subgraph "Data Sources" 
        style DataSources fill:#cce4f6,stroke:#333,stroke-width:1px
        DS1["VITON-HD Dataset\n(vitonhd_train_tagged.json,\nvitonhd_test_tagged.json)"] 
        DS2["DressCode Dataset\n(images + captions + DensePose)"]
    end

    %% Preprocessing Layer
    subgraph "Preprocessing Modules"
        style Preprocess fill:#d5e8d4,stroke:#333,stroke-width:1px
        HP["Human Parsing\n(preprocess/humanparsing/)"]
        OP["OpenPose\n(preprocess/openpose/)"]
        DP["DensePose\n(gradio_demo/detectron2/)"]
        DC["Dataset Converters\n(preprocess scripts)"]
    end

    %% Model Pipeline Layer
    subgraph "Model Training & Inference Pipeline"
        style Pipeline fill:#ffe1a8,stroke:#333,stroke-width:1px
        TR["train_xl.py"]
        IF["inference.py\ninference_dc.py"]
        CP["tryon_pipeline.py\n(src/tryon_pipeline.py)"]
        UA["UNet & Transformer\n(src/unet_hacked_tryon.py,\nsrc/unet_block_hacked_tryon.py,\nsrc/transformerhacked_tryon.py,\nsrc/attentionhacked_tryon.py)"]
        IA["IP-Adapter Plugin\n(ip_adapter/ip_adapter.py)"]
        CFG["Configs\n(configs/*.yaml)"]
        CK["Checkpoints\n(ckpt/)"]
    end

    %% Demo Layer
    subgraph "Gradio DensePose Demo"
        style Demo fill:#e1c7ff,stroke:#333,stroke-width:1px
        GA["gradio_demo/app.py"]
        GN["apply_net.py\n(gradio_demo/apply_net.py)"]
        GDM["DensePose Modules\n(gradio_demo/densepose/)"]
        GDT["Detectron2 (embedded)\n(gradio_demo/detectron2/)"]
    end

    %% External Libraries
    subgraph "External Libraries & Services"
        style Ext fill:#dddddd,stroke:#333,stroke-width:1px
        EL1["PyTorch\nHuggingFace Accelerate"]
        EL2["Detectron2"]
        EL3["Gradio"]
        EL4["ONNX Runtime"]
    end

    %% Data Flow
    DS1 -->|provides data to| HP
    DS2 -->|provides data to| OP
    DS2 -->|provides data to| DP
    HP --> DC
    OP --> DC
    DP --> DC
    DC -->|preprocessed data| TR
    DC -->|preprocessed data| IF

    %% Pipeline Connections
    TR -->|"calls"| CP
    IF -->|"calls"| CP
    CP -->|"uses models"| UA
    CP -->|"integrates"| IA
    CP -->|"loads hyperparams"| CFG
    CP -->|"loads ckpts"| CK

    %% Demo Connections
    GA -->|"imports"| GN
    GN -->|"uses modules"| GDM
    GN -->|"calls Detectron2"| GDT

    %% External Dependencies
    CP --> EL1
    HP --> EL4
    GDT --> EL2
    GA --> EL3

    %% Click Events
    click DS1 "https://github.com/yisol/idm-vton/blob/main/vitonhd_train_tagged.json"
    click DS1 "https://github.com/yisol/idm-vton/blob/main/vitonhd_test_tagged.json"
    click CK "https://github.com/yisol/idm-vton/tree/main/ckpt/"
    click CFG "https://github.com/yisol/idm-vton/tree/main/configs/"
    click HP "https://github.com/yisol/idm-vton/tree/main/preprocess/humanparsing/"
    click OP "https://github.com/yisol/idm-vton/tree/main/preprocess/openpose/"
    click IA "https://github.com/yisol/idm-vton/blob/main/ip_adapter/ip_adapter.py"
    click CP "https://github.com/yisol/idm-vton/blob/main/src/tryon_pipeline.py"
    click UA "https://github.com/yisol/idm-vton/blob/main/src/unet_hacked_tryon.py"
    click UA "https://github.com/yisol/idm-vton/blob/main/src/unet_block_hacked_tryon.py"
    click UA "https://github.com/yisol/idm-vton/blob/main/src/transformerhacked_tryon.py"
    click UA "https://github.com/yisol/idm-vton/blob/main/src/attentionhacked_tryon.py"
    click TR "https://github.com/yisol/idm-vton/blob/main/train_xl.py"
    click IF "https://github.com/yisol/idm-vton/blob/main/inference.py"
    click IF "https://github.com/yisol/idm-vton/blob/main/inference_dc.py"
    click GA "https://github.com/yisol/idm-vton/blob/main/gradio_demo/app.py"
    click GN "https://github.com/yisol/idm-vton/blob/main/gradio_demo/apply_net.py"
    click GDM "https://github.com/yisol/idm-vton/tree/main/gradio_demo/densepose/"
    click GDT "https://github.com/yisol/idm-vton/tree/main/gradio_demo/detectron2/"

    %% Styles
    classDef data fill:#cce4f6,stroke:#333;
    classDef preprocess fill:#d5e8d4,stroke:#333;
    classDef pipeline fill:#ffe1a8,stroke:#333;
    classDef demo fill:#e1c7ff,stroke:#333;
    classDef ext fill:#dddddd,stroke:#333;
    class DS1,DS2 data
    class HP,OP,DP,DC preprocess
    class TR,IF,CP,UA,IA,CFG,CK pipeline
    class GA,GN,GDM,GDT demo
    class EL1,EL2,EL3,EL4 ext
```
