# miccunifi/ladi-vton architecture

> Architecture diagram of the GitHub repository miccunifi/ladi-vton, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/miccunifi/ladi-vton
- Repository: https://github.com/miccunifi/ladi-vton

## Overview

##### Project Type and Purpose
- This is a full-stack, research-driven virtual try-on system that processes fashion data (images, masks, keypoints, etc.) and uses a series of models to generate images of a target model wearing a garment.
- The README provides details on datasets (DressCode and VITON-HD), training modules (TPS warping, EMASC for detail preservation, inversion adapter for textual inversion, and the final VTO), as well as inference and evaluation pathways.

##### File Structure
- Top-level files (LICENSE, README.md, environment.yml, hubconf.py) indicate release standards and dependencies.
- The “src” directory is the core of the project, subdivided into:
- "dataset": Data loading and processing scripts (e.g., dresscode.py, vitonhd.py).
- "models": Contains implementations of key architectures like AutoencoderKL, ConvNet_TPS (for warping), UNet components, inversion adapter, VAE, and related parts.
- "vto_pipelines": Houses the try-on pipeline script which likely integrates various model outputs.
- "utils": Provides helper and evaluation functions such as computing CLIP features, pose mapping, seed setup, and computing metrics.
- Training and evaluation scripts (e.g., train_tps.py, train_emasc.py, train_inversion_adapter.py, train_vto.py, inference.py, eval.py) illustrate modular training steps.

##### README for Architectural Insights
- The README clearly lists the sequential training steps, indicating that the system is built modularly:
a. The Warping Module is trained first using TPS.
b. Then the EMASC module is trained for enhanced detail preservation.
c. The inversion adapter is pre-trained to generate CLIP-compatible embeddings for the garments.
d. Finally, the VTO model (with UNet backbone) is trained using outputs from the previous modules.
- It also explains data preparation, inference procedures, metrics computation, and external dependencies (e.g., diffusers, transformers, and CLIP).

- Data Input Layer: Data acquisition from datasets (DressCode, VITON-HD) with pre-processing (masks, keypoints, label maps, etc.).
- Pre-computation/Utility Layer: Scripts for computing CLIP features and data utilities.
- Training Modules:
- Warping Module (TPS) – under "src/models/ConvNet_TPS.py" and "src/train_tps.py".
- EMASC module – implemented in "src/models/emasc.py" with "src/train_emasc.py".
- Inversion Adapter – “src/models/inversion_adapter.py” and its corresponding training script.
- VTO Module – culminating in the final model trained in "src/train_vto.py" and evaluated via "src/eval.py".
- Pipeline Integration: "src/vto_pipelines/tryon_pipe.py" which integrates model outputs for inference.
- Inference & Evaluation: “src/inference.py” and “src/utils/val_metrics.py” handling generation and metric computations.

b. Relationships and Interactions:
- From Data Input (datasets) to pre-processing utilities.
- Flow into the Warping Module, outputs then feed into EMASC.
- Simultaneous extraction of garment features using the inversion adapter.
- The VTO model takes the warped cloth images, enhanced texture information (from EMASC), and textual conditions (from inversion adapter) to generate try-on images.
- An evaluation block assesses model outputs using metrics computed in utility scripts.

c. Architectural Patterns and Design Principles:
- Integration of learning components (autoencoder, UNet, textual inversion, and diffusion model) and how each one aids the final generation task.
- Use of skip connections in the autoencoder module and integration of CLIP feature extraction.

d. Relevant Technologies and Frameworks:
- Python, TensorFlow/PyTorch (implied by use of torch and diffusers).
- Conda for dependency management (environment.yml).
- Specific libraries: diffusers, transformers, accelerate, and xformers memory-efficient attention.
- CLIP for textual inversion.

##### Instructing the Principal Software Engineer
- Clear labels for each component (e.g., “Dataset Loader”, “TPS Warping”, “EMASC”, “Inversion Adapter”, “VTO Pipeline”, “Inference & Evaluation”).
- Be detailed by capturing the architecture into as many distinct components as the project contains:
- Inter-module dependencies, such as the fact that VTO training depends on both the inversion adapter and EMASC outputs.
- Technologies (e.g., diffusion model, CLIP, UNet) where appropriate.
- Block with its role (e.g., “Pre-computation of CLIP features” in the utilities module).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion
    subgraph "Data Ingestion"
        DC["DressCode Loader"]:::dataset
        VH["VITON-HD Loader"]:::dataset
    end

    %% Pre-computation/Utility Layer
    subgraph "Pre-computation/Utility Layer"
        CF["CLIP Features"]:::utils
        TE["Text Embedding"]:::utils
        DU["Data Utils"]:::utils
        PM["Pose Map"]:::utils
        SS["Seed Setup"]:::utils
    end

    %% Training Modules
    subgraph "Training Modules"
        TPS["TPS Warping"]:::module
        EMASC["EMASC Module"]:::module
        InvAdapter["Inversion Adapter"]:::module
        VTO["VTO Module"]:::module
    end

    %% Inference & Evaluation
    subgraph "Inference & Evaluation"
        INF["Inference"]:::infer
        EVAL["Evaluation & Metrics"]:::infer
    end

    %% Data Input to Utilities
    DC --> DU
    VH --> DU

    %% Utilities to other utility components (vertical layout)
    DU --> CF
    CF --> TE
    TE --> PM
    PM --> SS

    %% Utilities feed into Training Modules
    DU --> TPS
    DU --> InvAdapter

    %% Sequential training flow
    TPS --> EMASC
    EMASC --> VTO
    InvAdapter --> VTO

    %% Pipeline for inference and evaluation
    VTO --> INF
    INF --> EVAL

    %% Click Events for Data Ingestion
    click DC "https://github.com/miccunifi/ladi-vton/blob/master/src/dataset/dresscode.py"
    click VH "https://github.com/miccunifi/ladi-vton/blob/master/src/dataset/vitonhd.py"

    %% Click Events for Pre-computation/Utility Layer
    click CF "https://github.com/miccunifi/ladi-vton/blob/master/src/utils/compute_cloth_clip_features.py"
    click TE "https://github.com/miccunifi/ladi-vton/blob/master/src/utils/encode_text_word_embedding.py"
    click DU "https://github.com/miccunifi/ladi-vton/blob/master/src/utils/data_utils.py"
    click PM "https://github.com/miccunifi/ladi-vton/blob/master/src/utils/posemap.py"
    click SS "https://github.com/miccunifi/ladi-vton/blob/master/src/utils/set_seeds.py"

    %% Click Events for Training Modules
    click TPS "https://github.com/miccunifi/ladi-vton/blob/master/src/models/ConvNet_TPS.py"
    click TPS "https://github.com/miccunifi/ladi-vton/blob/master/src/train_tps.py"
    click EMASC "https://github.com/miccunifi/ladi-vton/blob/master/src/models/emasc.py"
    click EMASC "https://github.com/miccunifi/ladi-vton/blob/master/src/train_emasc.py"
    click InvAdapter "https://github.com/miccunifi/ladi-vton/blob/master/src/models/inversion_adapter.py"
    click InvAdapter "https://github.com/miccunifi/ladi-vton/blob/master/src/train_inversion_adapter.py"
    click VTO "https://github.com/miccunifi/ladi-vton/blob/master/src/train_vto.py"
    click VTO "https://github.com/miccunifi/ladi-vton/blob/master/src/models/UNet.py"
    click VTO "https://github.com/miccunifi/ladi-vton/blob/master/src/vto_pipelines/tryon_pipe.py"

    %% Click Events for Inference & Evaluation
    click INF "https://github.com/miccunifi/ladi-vton/blob/master/src/inference.py"
    click EVAL "https://github.com/miccunifi/ladi-vton/blob/master/src/eval.py"
    click EVAL "https://github.com/miccunifi/ladi-vton/blob/master/src/utils/val_metrics.py"

    %% Styles
    classDef dataset fill:#8EC0E4,stroke:#333,stroke-width:2px;
    classDef utils fill:#F4D03F,stroke:#333,stroke-width:2px;
    classDef module fill:#82E0AA,stroke:#333,stroke-width:2px;
    classDef infer fill:#F1948A,stroke:#333,stroke-width:2px;
```
