How zheng-chong/catvton works
Project type and purpose
• This is a research-grade, full-stack machine-learning pipeline for a virtual try-on system (CatVTON). It comprises:
• Core library modules (data loading, model definition, training/evaluation).
• Command-line scripts for preprocessing, training, inference, evaluation.
• Integration with external services (DensePose, SCHP, HuggingFace Diffusers).
• Two user-facing UIs: a Gradio web app (app.py) and a ComfyUI custom-nodes workflow.
Top-level components
• Data Layer
• Local dataset folders (VITON-HD, DressCode) on disk.
• Preprocessing module (preprocess_agnostic_mask.py).
• Data loaders in densepose/data and Detectron2’s data/.
• Model Layer
• Diffusion backbone via HuggingFace Diffusers (inferred in model/pipeline.py and densepose/modeling).
• Auxiliary networks: DensePose, SCHP for mask and pose estimation (localized in densepose/ and model/SCHP, model/DensePose).
• Flux-LoRA modules (app_flux.py, model/flux/).
• Engine & Training/Evaluation
• Training engine in densepose/engine/trainer.py (hooks into Detectron2’s engine/).
• Evaluation modules in densepose/evaluation and top-level eval.py.
• Inference & Serving
• Inference script (inference.py).
• Gradio App (app.py) for web UI.
• ComfyUI custom nodes and workflow JSON for visual node-based UI (app_flux.py, releases).
• External Dependencies
• DensePose & SCHP models (integrated via local copies).
• Detectron2 library.
• HuggingFace Diffusers & Transformers.
• PyTorch.
• Static Assets
• Web resources (resource/js, resource/css, resource/img) for the Gradio front end.
• Demo outputs in resource/demo.
Interactions and data flow
1. Disk/Storage: raw images, cloth items, agnostic masks →
2. Preprocessor (preprocess_agnostic_mask.py) →
3. Data Loader (densepose/data + Detectron2/data) feeds batches →
4. Mask & Pose Estimation: DensePose & SCHP modules generate human parsing & IUV maps →
5. Diffusion Model Pipeline (model/pipeline.py or model/flux) consumes concatenated inputs →
6. Training Engine (densepose/engine/trainer.py) or Inference (inference.py) →
7. Outputs: synthesized try-on images →
8. Evaluation (eval.py) or App Serving (app.py for Gradio, ComfyUI).
Architectural patterns & principles
• Modular ML pipeline: clear separation of data, model, engine, evaluation, serving.
• Plugin architecture: Detectron2 config/registry for different heads and losses.
• Extensibility points: densepose/converters, densepose/modeling/predictors, custom LoRA modules.
• External-tool integration: ComfyUI custom nodes packaged separately.
Technologies & frameworks
• Python, PyTorch
• Detectron2 (data APIs, training loops)
• HuggingFace Diffusers (inpainting diffusion)
• DensePose & SCHP (human parsing/IUV generation)
• Gradio for web UI
• ComfyUI for node-based UI