# nftblackmagic/diffusion-tryon-trainer architecture

> Architecture diagram of the GitHub repository nftblackmagic/diffusion-tryon-trainer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nftblackmagic/diffusion-tryon-trainer
- Repository: https://github.com/nftblackmagic/diffusion-tryon-trainer

## Overview

##### Project Type and Purpose
- This is a research‐oriented full-stack ML training & inference pipeline for OOTDiffusion, a diffusion‐based virtual try-on system.
- It extends the original OOTDiffusion code by integrating Huggingface’s Diffusers library and targets the VITON dataset for training and inference.

##### High-Level File Structure Analysis
- Top‐level scripts & configs:
- train_ootd.sh, inference_test_dataset.sh, environment.yaml, install_conda.sh
- run/ folder: user‐facing inference entrypoints (gradio_ootd.py, run_ootd.py)
- Core application code under ootd/:
- models/: custom modules (ReferenceEncoder, UNet variants)
- pipelines_ootd/: diffusion pipelines (attention_garm, transformer_vton_2d, etc.)
- inference_* .py & train*.py: orchestration scripts for inference and training
- Data preparation:
- data_scripts/: CP (Cloth‐Parsing) dataset extraction
- preprocess/: human parsing, openpose annotation, dataset transforms
- Utilities & dependencies:
- utils/, run/utils_ootd.py for helper functions
- environment.yaml describes Python & library requirements

##### Key README Insights
- Trains on VITON-HD; requires pre-warped cloth, masks, captions
- Uses Huggingface Diffusers for diffusion model training
- Separate pipelines for high‐res (HD) and garment condition (GARM) variants
- Checkpoints stored under checkpoints/ for inference

##### Components & Interactions
a. Main Components
1. Data Sources
- VITON-HD raw images & annotations
- Pre-warped cloth & masks
- Cloth captions
2. Preprocessing Layer
- CP dataset extraction (data_scripts/cp_dataset.py)
- Human parsing pipeline (preprocess/humanparsing)
- OpenPose annotation (preprocess/openpose)
3. Training Pipeline
- Dataset loader (ootd/data/dataset.py)
- Model modules (ReferenceEncoder, UNet variants)
- Diffusion pipelines (pipelines_ootd/*.py) using Huggingface Diffusers
- Training scripts (ootd/train.py, train_ootd_hd.py, train_t2g.py)
- Checkpoint manager (Huggingface Trainer/API)
4. Inference Pipeline
- Inference scripts (ootd/inference_ootd*.py)
- Run wrappers (run/run_ootd.py, gradio_ootd.py)
- Reads checkpoints, applies diffusion pipelines, writes output images
5. External Libraries & Services
- Huggingface Diffusers & Transformers
- PyTorch (CUDA)
- Data storage (local FS or network)
- Optional: GPU cluster / scheduler

b. Relationships & Data Flow
- Raw VITON files → Preprocessing layer → Tensor datasets
- Preprocessed tensors → Training pipeline → Model checkpoints
- Checkpoints + new inputs → Inference pipeline → Synthesized try-on images
- CLI scripts or Gradio front-end trigger inference step

c. Architectural Patterns & Principles
- Modular ML pipeline: clear separation between data prep, model definition, training, inference
- Plugin‐style diffusion pipelines: each variant (GARM, VTON) lives in its own module
- Script‐based orchestration: entrypoints wrap Python modules, enabling HPC cluster usage

d. Relevant Technologies & Frameworks
- Python 3.8+, Conda environments
- PyTorch + CUDA
- Huggingface Diffusers library
- Human parsing tools (Detectron2, custom modules)
- OpenPose for keypoint annotation

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Layer
    RawData["VITON-HD Raw Data"]:::data
    PreWarp["Pre-warped Cloth & Masks"]:::data
    Captions["Cloth Captions"]:::data

    %% Preprocessing Layer
    subgraph "Preprocessing Layer"
        direction TB
        CP["CP Dataset Extraction"]:::compute
        HP["Human Parsing"]:::compute
        OP["OpenPose Annotation"]:::compute
    end

    %% Dataset Loader
    DatasetLoader["Dataset Loader"]:::compute

    %% Models & Pipelines
    subgraph "Models & Pipelines"
        direction TB
        RefEnc["ReferenceEncoder"]:::model
        UNetGBlocks["UNet GARM Blocks"]:::model
        UNetGCond["UNet GARM Condition"]:::model
        UNetVBlocks["UNet VTON Blocks"]:::model
        UNetVCond["UNet VTON Condition"]:::model
        AttGARM["Attention GARM Pipeline"]:::model
        AttVTON["Attention VTON Pipeline"]:::model
        TransGARM["Transformer GARM 2D"]:::model
        TransVTON["Transformer VTON 2D"]:::model
        CorePipe["Core Pipeline Orchestration"]:::model
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        TrainBase["train.py"]:::compute
        TrainHD["train_ootd_hd.py"]:::compute
        TrainT2G["train_t2g.py"]:::compute
        Checkpt["Checkpoint Manager"]:::ext
    end

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        direction TB
        InfBase["inference_ootd.py"]:::compute
        InfDC["inference_ootd_dc.py"]:::compute
        InfHD["inference_ootd_hd.py"]:::compute
        RunStd["run_ootd.py"]:::compute
        RunGradio["gradio_ootd.py"]:::compute
        UtilsRun["utils_ootd.py"]:::compute
    end

    %% External Services
    subgraph "External Libraries & Services"
        direction TB
        HF["Huggingface Diffusers & Transformers"]:::ext
        Pytorch["PyTorch (CUDA)"]:::ext
        Storage["Data Storage"]:::ext
        Cluster["GPU Cluster / Scheduler"]:::ext
    end

    %% Data Flow
    RawData --> CP
    PreWarp --> CP
    Captions --> CP
    CP --> DatasetLoader
    HP --> DatasetLoader
    OP --> DatasetLoader

    DatasetLoader --> CorePipe
    CorePipe --> RefEnc
    CorePipe --> UNetGBlocks
    CorePipe --> UNetGCond
    CorePipe --> UNetVBlocks
    CorePipe --> UNetVCond
    CorePipe --> AttGARM
    CorePipe --> AttVTON
    CorePipe --> TransGARM
    CorePipe --> TransVTON

    RefEnc --> TrainBase
    UNetGBlocks --> TrainHD
    TransGARM --> TrainT2G

    TrainBase --> Checkpt
    TrainHD --> Checkpt
    TrainT2G --> Checkpt

    Checkpt --> InfBase
    Checkpt --> InfDC
    Checkpt --> InfHD

    InfBase --> CorePipe
    InfDC --> CorePipe
    InfHD --> CorePipe

    RunStd --> InfBase
    RunGradio --> InfBase
    UtilsRun --> InfBase

    InfBase --> Storage
    InfDC --> Storage
    InfHD --> Storage

    %% External Dependencies
    HF --> CorePipe
    HF --> TrainBase
    HF --> InfBase
    Pytorch --> CorePipe
    Pytorch --> TrainBase
    Pytorch --> InfBase
    Cluster --> TrainHD
    Storage --> DatasetLoader

    %% Click Events
    click CP "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/data_scripts/cp_dataset.py"
    click HP "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/preprocess/humanparsing/aigc_run_parsing.py"
    click OP "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/preprocess/openpose/run_openpose.py"
    click DatasetLoader "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/data/dataset.py"
    click RefEnc "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/models/ReferenceEncoder.py"
    click AttGARM "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/pipelines_ootd/attention_garm.py"
    click AttVTON "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/pipelines_ootd/attention_vton.py"
    click TransGARM "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/pipelines_ootd/transformer_garm_2d.py"
    click TransVTON "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/pipelines_ootd/transformer_vton_2d.py"
    click UNetGBlocks "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/pipelines_ootd/unet_garm_2d_blocks.py"
    click UNetGCond "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/pipelines_ootd/unet_garm_2d_condition.py"
    click UNetVBlocks "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/pipelines_ootd/unet_vton_2d_blocks.py"
    click UNetVCond "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/pipelines_ootd/unet_vton_2d_condition.py"
    click CorePipe "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/pipelines_ootd/pipeline_ootd.py"
    click TrainBase "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/train.py"
    click TrainHD "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/train_ootd_hd.py"
    click TrainT2G "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/train_t2g.py"
    click InfBase "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/inference_ootd.py"
    click InfDC "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/inference_ootd_dc.py"
    click InfHD "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/ootd/inference_ootd_hd.py"
    click RunStd "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/run/run_ootd.py"
    click RunGradio "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/run/gradio_ootd.py"
    click UtilsRun "https://github.com/nftblackmagic/diffusion-tryon-trainer/blob/main/run/utils_ootd.py"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#333,stroke-width:1px;
    classDef compute fill:#90EE90,stroke:#333,stroke-width:1px;
    classDef model fill:#FFA500,stroke:#333,stroke-width:1px;
    classDef ext fill:#D3D33,stroke:#333,stroke-width:1px;
```
