# lzyhha/visualcloze architecture

> Architecture diagram of the GitHub repository lzyhha/visualcloze, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lzyhha/visualcloze
- Repository: https://github.com/lzyhha/visualcloze

## Overview

##### Project Type and Purpose
- VisualCloze is a research-grade machine-learning framework for universal image generation via visual in-context learning.
- It is not a typical full-stack web app; rather, it is a modular ML library/toolkit that supports data preprocessing, model training, and inference (CLI + Gradio UI), along with an optional HuggingFace Diffusers pipeline.

##### High-Level Architecture Components
a. Data Layer
- Raw data (e.g., Graph200K JSON/images)
- `processing.py` & `data/` modules (data_reader, dataset, degradation_toolkit) for preprocessing and augmentation
- Output: preprocessed dataset JSON + image grids
b. Configuration Layer
- `configs/data/visualcloze.yaml`
- Command-line arguments (e.g., model_path, resolution)
- Drives both training (`train.py` / `exps/train.sh`) and inference (`app.py`, `inference.py`, `visualcloze.py`)
c. Model & Core Library
- `models/`
– `model.py` (orchestrator)
– `api.py`, `math.py`, `util.py`
– `modules/` subpackage: autoencoder, conditioner, image_embedders, layers, lora
– `sampling.py` (diffusion samplers)
- `transport/`
– Implements SDE‐based transport engine: integrators, path definitions, core transport functions
- `visualcloze.py`
– Standalone pipeline wrapping model + transport + LoRA for custom inference
d. Training Pipeline
- Entry point: `train.py` or `exps/train.sh`
- Loads config, dataset, model, transport engine
- Uses PyTorch + FSDP/gradient accumulation (`util/parallel.py`)
- Outputs model weights (LoRA checkpoint `.pth`)
e. Inference Pipeline
- Two modes:
1. Custom script: `inference.py` + `visualcloze.py`
2. Gradio Web App: `app.py` (+ supporting `examples/gradio_tasks_*.py`)
- Or via HuggingFace Diffusers: custom `VisualClozePipeline` in a diffusers fork
- Inputs: preloaded model weights, YAML config, grids of PIL.Images, task prompt, content prompt
- Outputs: generated/upsampled images
f. CLI & UX
- `sample.py` / `exps/sample.sh` for batch sampling on test set
- `app.py` serves Gradio UI for live demo
- `examples/` directory holds sample images & example scripts

##### Relationships & Data Flow
- Raw data → `processing.py` → preprocessed JSON/images → `data.Dataset` → DataLoader → Training
- Config YAML + CLI args → Training Pipeline → Model & Transport modules → Model Weights
- Model Weights + Config + Input Grids → Inference Pipeline → Generated Images
- Gradio (`app.py`) sits atop the Inference Pipeline for web UI

##### Architectural Patterns & Principles
- Modular separation:
- Data preprocessing (`data/`, `processing.py`)
- Model definition (`models/`)
- Diffusion/transport engine (`transport/`)
- Orchestration scripts (`train.py`, `inference.py`, `app.py`)
- Plugin-style extensions:
- Degradation toolkit under `data/`
- LoRA adapters under `models/modules/lora.py`
- Config-driven behavior via YAML + CLI
- Reusable pipelines: standalone Python wrapper (`visualcloze.py`) vs. Diffusers integration

##### Putting It All Together
- Top-left: Raw Data & Processing → Data Lake / JSON + Images
- Center-left: Configuration (YAML + CLI) feeding both Training and Inference
- Center: Core Model Modules + Transport Engine forming the model (with LoRA adapters)
- Bottom-left: Training Pipeline → Model Weights
- Bottom-right: Inference Pipeline → Generated Images
- Far right: CLI scripts (sample.py, inference.py) and Gradio App (app.py) as user interfaces

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        RawData["Raw Data (JSON/Images)"]:::data
        Proc["processing.py"]:::code
        DR["data/data_reader.py"]:::code
        DS["data/dataset.py"]:::code
        DU["data/data_utils.py"]:::code
        DTK["data/degradation_toolkit/"]:::code
        DU2["data/degradation_utils.py"]:::code
        PI["data/prefix_instruction.py"]:::code
        Preprocessed["Preprocessed Data (JSON + Image Grids)"]:::data
    end
    click Proc "https://github.com/lzyhha/visualcloze/blob/main/processing.py"
    click DR "https://github.com/lzyhha/visualcloze/blob/main/data/data_reader.py"
    click DS "https://github.com/lzyhha/visualcloze/blob/main/data/dataset.py"
    click DU "https://github.com/lzyhha/visualcloze/blob/main/data/data_utils.py"
    click DTK "https://github.com/lzyhha/visualcloze/tree/main/data/degradation_toolkit/"
    click DU2 "https://github.com/lzyhha/visualcloze/blob/main/data/degradation_utils.py"
    click PI "https://github.com/lzyhha/visualcloze/blob/main/data/prefix_instruction.py"

    RawData -->|"read/augment"| Proc
    Proc --> DR
    Proc --> DS
    Proc --> DU
    Proc --> DTK
    Proc --> DU2
    Proc --> PI
    PI --> Preprocessed
    DTK --> Preprocessed

    %% Configuration Layer
    subgraph "Configuration Layer"
        direction TB
        ConfigYAML["configs/data/visualcloze.yaml"]:::code
        CLIArgs["Command-line Args"]:::runtime
    end
    click ConfigYAML "https://github.com/lzyhha/visualcloze/blob/main/configs/data/visualcloze.yaml"

    ConfigYAML --> CLIArgs

    %% Core Model & Transport
    subgraph "Core Model & Transport"
        direction TB
        subgraph "Model Modules"
            direction TB
            ModelOrch["models/model.py"]:::code
            API["models/api.py"]:::code
            Math["models/math.py"]:::code
            Util["models/util.py"]:::code
            subgraph "modules/"
                direction TB
                AE["models/modules/autoencoder.py"]:::code
                Cond["models/modules/conditioner.py"]:::code
                IE["models/modules/image_embedders.py"]:::code
                Layers["models/modules/layers.py"]:::code
                LoRA["models/modules/lora.py"]:::code
            end
            Samp["models/sampling.py"]:::code
        end
        subgraph "Transport Engine"
            direction TB
            Intg["transport/integrators.py"]:::code
            Path["transport/path.py"]:::code
            Trans["transport/transport.py"]:::code
            TUtils["transport/utils.py"]:::code
        end
        Wrapper["visualcloze.py"]:::code
    end
    click ModelOrch "https://github.com/lzyhha/visualcloze/blob/main/models/model.py"
    click API "https://github.com/lzyhha/visualcloze/blob/main/models/api.py"
    click Math "https://github.com/lzyhha/visualcloze/blob/main/models/math.py"
    click Util "https://github.com/lzyhha/visualcloze/blob/main/models/util.py"
    click AE "https://github.com/lzyhha/visualcloze/blob/main/models/modules/autoencoder.py"
    click Cond "https://github.com/lzyhha/visualcloze/blob/main/models/modules/conditioner.py"
    click IE "https://github.com/lzyhha/visualcloze/blob/main/models/modules/image_embedders.py"
    click Layers "https://github.com/lzyhha/visualcloze/blob/main/models/modules/layers.py"
    click LoRA "https://github.com/lzyhha/visualcloze/blob/main/models/modules/lora.py"
    click Samp "https://github.com/lzyhha/visualcloze/blob/main/models/sampling.py"
    click Intg "https://github.com/lzyhha/visualcloze/blob/main/transport/integrators.py"
    click Path "https://github.com/lzyhha/visualcloze/blob/main/transport/path.py"
    click Trans "https://github.com/lzyhha/visualcloze/blob/main/transport/transport.py"
    click TUtils "https://github.com/lzyhha/visualcloze/blob/main/transport/utils.py"
    click Wrapper "https://github.com/lzyhha/visualcloze/blob/main/visualcloze.py"

    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        TrainPy["train.py"]:::runtime
        TrainSh["exps/train.sh"]:::runtime
        Parallel["util/parallel.py"]:::code
        Weights["Model Weights (.pth)"]:::data
    end
    click TrainPy "https://github.com/lzyhha/visualcloze/blob/main/train.py"
    click TrainSh "https://github.com/lzyhha/visualcloze/blob/main/exps/train.sh"
    click Parallel "https://github.com/lzyhha/visualcloze/blob/main/util/parallel.py"

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        direction TB
        InferPy["inference.py"]:::runtime
        VCWrapper["visualcloze.py"]:::code
        Generated["Generated Images"]:::data
    end
    click InferPy "https://github.com/lzyhha/visualcloze/blob/main/inference.py"
    click VCWrapper "https://github.com/lzyhha/visualcloze/blob/main/visualcloze.py"

    %% UI & CLI
    subgraph "CLI & UX"
        direction TB
        SamplePy["sample.py"]:::runtime
        SampleSh["exps/sample.sh"]:::runtime
        App["app.py"]:::runtime
        ExGradio["examples/gradio_tasks*.py"]:::code
        Browser["Web Browser"]:::external
    end
    click SamplePy "https://github.com/lzyhha/visualcloze/blob/main/sample.py"
    click SampleSh "https://github.com/lzyhha/visualcloze/blob/main/exps/sample.sh"
    click App "https://github.com/lzyhha/visualcloze/blob/main/app.py"
    click ExGradio "https://github.com/lzyhha/visualcloze/blob/main/examples/gradio_tasks.py"

    %% External Services
    subgraph "External"
        direction TB
        HFHub["HuggingFace Hub"]:::external
        PyTorch["PyTorch"]:::external
    end

    %% Flows
    Preprocessed -->|"load"| DS -->|"DataLoader"| TrainPy
    CLIArgs -->|drives| TrainPy
    CLIArgs -->|drives| InferPy
    TrainPy --> Parallel
    TrainPy -->|"train on"| PyTorch
    TrainPy -->|"outputs"| Weights

    Weights -->|"load"| InferPy
    Preprocessed -->|"input grids"| InferPy
    ConfigYAML -->|"cfg"| InferPy
    InferPy --> VCWrapper
    VCWrapper --> Generated

    App -->|"calls"| InferPy
    Browser -->|"interacts"| App

    SamplePy -->|"batch sampling"| VCWrapper

    HFHub -.->|"optional pull/push"| TrainPy
    HFHub -.->|"optional pull/push"| InferPy

    %% Styles
    classDef data fill:#D0E6FF,stroke:#004080,color:#003060
    classDef code fill:#E6FFE6,stroke:#008000,color:#006000
    classDef runtime fill:#F3E6FF,stroke:#800080,color:#400040
    classDef external fill:#FFE6CC,stroke:#CC6600,color:#994C00
```
