How lzyhha/visualcloze works
Project Type and Purpose
• VisualCloze is a research-grade machine-learning framework for universal image generation via visual in-context learning.
• It is not a typical full-stack web app; rather, it is a modular ML library/toolkit that supports data preprocessing, model training, and inference (CLI + Gradio UI), along with an optional HuggingFace Diffusers pipeline.
High-Level Architecture Components
a. Data Layer
• Raw data (e.g., Graph200K JSON/images)
• processing.py & data/ modules (data_reader, dataset, degradation_toolkit) for preprocessing and augmentation
• Output: preprocessed dataset JSON + image grids
b. Configuration Layer
• configs/data/visualcloze.yaml
• Command-line arguments (e.g., model_path, resolution)
• Drives both training (train.py / exps/train.sh) and inference (app.py, inference.py, visualcloze.py)
c. Model & Core Library
• models/
– model.py (orchestrator)
– api.py, math.py, util.py
– modules/ subpackage: autoencoder, conditioner, image_embedders, layers, lora
– sampling.py (diffusion samplers)
• transport/
– Implements SDE‐based transport engine: integrators, path definitions, core transport functions
• visualcloze.py
– Standalone pipeline wrapping model + transport + LoRA for custom inference
d. Training Pipeline
• Entry point: train.py or exps/train.sh
• Loads config, dataset, model, transport engine
• Uses PyTorch + FSDP/gradient accumulation (util/parallel.py)
• Outputs model weights (LoRA checkpoint .pth)
e. Inference Pipeline
• Two modes:
1. Custom script: inference.py + visualcloze.py
2. Gradio Web App: app.py (+ supporting examples/gradio_tasks_*.py)
• Or via HuggingFace Diffusers: custom VisualClozePipeline in a diffusers fork
• Inputs: preloaded model weights, YAML config, grids of PIL.Images, task prompt, content prompt
• Outputs: generated/upsampled images
f. CLI & UX
• sample.py / exps/sample.sh for batch sampling on test set
• app.py serves Gradio UI for live demo
• examples/ directory holds sample images & example scripts
Relationships & Data Flow
• Raw data → processing.py → preprocessed JSON/images → data.Dataset → DataLoader → Training
• Config YAML + CLI args → Training Pipeline → Model & Transport modules → Model Weights
• Model Weights + Config + Input Grids → Inference Pipeline → Generated Images
• Gradio (app.py) sits atop the Inference Pipeline for web UI
Architectural Patterns & Principles
• Modular separation:
• Data preprocessing (data/, processing.py)
• Model definition (models/)
• Diffusion/transport engine (transport/)
• Orchestration scripts (train.py, inference.py, app.py)
• Plugin-style extensions:
• Degradation toolkit under data/
• LoRA adapters under models/modules/lora.py
• Config-driven behavior via YAML + CLI
• Reusable pipelines: standalone Python wrapper (visualcloze.py) vs. Diffusers integration
Putting It All Together
• Top-left: Raw Data & Processing → Data Lake / JSON + Images
• Center-left: Configuration (YAML + CLI) feeding both Training and Inference
• Center: Core Model Modules + Transport Engine forming the model (with LoRA adapters)
• Bottom-left: Training Pipeline → Model Weights
• Bottom-right: Inference Pipeline → Generated Images
• Far right: CLI scripts (sample.py, inference.py) and Gradio App (app.py) as user interfaces