# zheng-chong/catvton architecture

> Architecture diagram of the GitHub repository zheng-chong/catvton, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zheng-chong/catvton
- Repository: https://github.com/zheng-chong/catvton

## Overview

##### Project type and purpose
- This is a research-grade, full-stack machine-learning pipeline for a virtual try-on system (CatVTON). It comprises:
- Core library modules (data loading, model definition, training/evaluation).
- Command-line scripts for preprocessing, training, inference, evaluation.
- Integration with external services (DensePose, SCHP, HuggingFace Diffusers).
- Two user-facing UIs: a Gradio web app (`app.py`) and a ComfyUI custom-nodes workflow.

##### Top-level components
- **Data Layer**
- Local dataset folders (VITON-HD, DressCode) on disk.
- Preprocessing module (`preprocess_agnostic_mask.py`).
- Data loaders in `densepose/data` and Detectron2’s `data/`.
- **Model Layer**
- Diffusion backbone via HuggingFace Diffusers (inferred in `model/pipeline.py` and `densepose/modeling`).
- Auxiliary networks: DensePose, SCHP for mask and pose estimation (localized in `densepose/` and `model/SCHP`, `model/DensePose`).
- Flux-LoRA modules (`app_flux.py`, `model/flux/`).
- **Engine & Training/Evaluation**
- Training engine in `densepose/engine/trainer.py` (hooks into Detectron2’s `engine/`).
- Evaluation modules in `densepose/evaluation` and top-level `eval.py`.
- **Inference & Serving**
- Inference script (`inference.py`).
- Gradio App (`app.py`) for web UI.
- ComfyUI custom nodes and workflow JSON for visual node-based UI (`app_flux.py`, releases).
- **External Dependencies**
- DensePose & SCHP models (integrated via local copies).
- Detectron2 library.
- HuggingFace Diffusers & Transformers.
- PyTorch.
- **Static Assets**
- Web resources (`resource/js`, `resource/css`, `resource/img`) for the Gradio front end.
- Demo outputs in `resource/demo`.

##### Interactions and data flow
1. **Disk/Storage**: raw images, cloth items, agnostic masks →
2. **Preprocessor** (`preprocess_agnostic_mask.py`) →
3. **Data Loader** (`densepose/data` + Detectron2/data) feeds batches →
4. **Mask & Pose Estimation**: DensePose & SCHP modules generate human parsing & IUV maps →
5. **Diffusion Model Pipeline** (`model/pipeline.py` or `model/flux`) consumes concatenated inputs →
6. **Training Engine** (`densepose/engine/trainer.py`) or **Inference** (`inference.py`) →
7. **Outputs**: synthesized try-on images →
8. **Evaluation** (`eval.py`) or **App Serving** (`app.py` for Gradio, ComfyUI).

##### Architectural patterns & principles
- **Modular ML pipeline**: clear separation of data, model, engine, evaluation, serving.
- **Plugin architecture**: Detectron2 config/registry for different heads and losses.
- **Extensibility points**: `densepose/converters`, `densepose/modeling/predictors`, custom LoRA modules.
- **External-tool integration**: ComfyUI custom nodes packaged separately.

##### Technologies & frameworks
- Python, PyTorch
- Detectron2 (data APIs, training loops)
- HuggingFace Diffusers (inpainting diffusion)
- DensePose & SCHP (human parsing/IUV generation)
- Gradio for web UI
- ComfyUI for node-based UI

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        DS["Local Datasets"]:::storage
        PP["Preprocessor\npreprocess_agnostic_mask.py"]:::compute
        DL["Data Loader\n(densepose/data + detectron2/data)"]:::compute
    end

    DS --> PP --> DL

    %% Model Layer
    subgraph "Model Layer"
        DP["DensePose Module\nmodel/DensePose/"]:::compute
        SH["SCHP Module\nmodel/SCHP/"]:::compute
        DIFF["Diffusion Pipeline\nmodel/pipeline.py"]:::compute
        FLUX["Flux-LoRA\nmodel/flux/"]:::compute
        ATN1["Attention Processor\nmodel/attn_processor.py"]:::compute
        ATN2["Cloth Masker\nmodel/cloth_masker.py"]:::compute
    end

    DL --> DP
    DL --> SH
    DP --> DIFF
    SH --> DIFF
    DIFF --> FLUX

    %% Engine & Evaluation
    subgraph "Engine & Evaluation"
        TR["Training Engine\ndensepose/engine/trainer.py"]:::compute
        EVF1["Eval Modules\ndensepose/evaluation/"]:::compute
        EVALP["Eval Script\neval.py"]:::compute
    end

    DIFF --> TR
    TR --> EVF1
    TR --> EVALP

    %% Serving Layer
    subgraph "Serving Layer"
        INF["Inference Script\ninference.py"]:::compute
        GR["Gradio UI\napp.py"]:::ui
        COMF1["ComfyUI Node\napp_flux.py"]:::ui
        COMF2["ComfyUI P2P\napp_p2p.py"]:::ui
    end

    DIFF --> INF
    FLUX --> INF
    INF --> GR
    INF --> COMF1
    INF --> COMF2

    %% Static Assets
    subgraph "Static Assets"
        CSS["CSS Files\nresource/css/"]:::storage
        JS["JS Files\nresource/js/"]:::storage
        IMG["Images\nresource/img/"]:::storage
        DEMO["Demos\nresource/demo/"]:::storage
    end

    GR --> CSS
    GR --> JS
    GR --> IMG
    GR --> DEMO

    %% External Dependencies
    DET["Detectron2 Library"]:::external
    HF["HuggingFace Diffusers & Transformers"]:::external
    PT["PyTorch"]:::external

    DET --> DL
    DET --> TR
    HF --> DIFF
    PT --> TR
    PT --> DIFF

    %% Utilities
    UTILS["Utilities\nutils.py"]:::compute
    DS --> UTILS
    DL --> UTILS
    DIFF --> UTILS

    %% Click Events
    click PP "https://github.com/zheng-chong/catvton/blob/edited/preprocess_agnostic_mask.py"
    click DL "https://github.com/zheng-chong/catvton/tree/edited/densepose/data/"
    click DP "https://github.com/zheng-chong/catvton/tree/edited/model/DensePose/"
    click SH "https://github.com/zheng-chong/catvton/tree/edited/model/SCHP/"
    click DIFF "https://github.com/zheng-chong/catvton/blob/edited/model/pipeline.py"
    click FLUX "https://github.com/zheng-chong/catvton/tree/edited/model/flux/"
    click ATN1 "https://github.com/zheng-chong/catvton/blob/edited/model/attn_processor.py"
    click ATN2 "https://github.com/zheng-chong/catvton/blob/edited/model/cloth_masker.py"
    click TR "https://github.com/zheng-chong/catvton/blob/edited/densepose/engine/trainer.py"
    click EVF1 "https://github.com/zheng-chong/catvton/tree/edited/densepose/evaluation/"
    click EVALP "https://github.com/zheng-chong/catvton/blob/edited/eval.py"
    click INF "https://github.com/zheng-chong/catvton/blob/edited/inference.py"
    click GR "https://github.com/zheng-chong/catvton/blob/edited/app.py"
    click COMF1 "https://github.com/zheng-chong/catvton/blob/edited/app_flux.py"
    click COMF2 "https://github.com/zheng-chong/catvton/blob/edited/app_p2p.py"
    click DET "https://github.com/zheng-chong/catvton/tree/edited/detectron2/"
    click CSS "https://github.com/zheng-chong/catvton/tree/edited/resource/css/"
    click JS "https://github.com/zheng-chong/catvton/tree/edited/resource/js/"
    click IMG "https://github.com/zheng-chong/catvton/tree/edited/resource/img/"
    click DEMO "https://github.com/zheng-chong/catvton/tree/edited/resource/demo/"
    click UTILS "https://github.com/zheng-chong/catvton/blob/edited/utils.py"

    %% Styles
    classDef storage fill:#FFEEAA,stroke:#CCAA44
    classDef compute fill:#CCEEFF,stroke:#44AACC
    classDef ui fill:#EECFFF,stroke:#AA44CC
    classDef external fill:#EEEEEE,stroke:#999999,stroke-dasharray: 5 5
```
