# zhexinliang/control-color architecture

> Architecture diagram of the GitHub repository zhexinliang/control-color, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zhexinliang/control-color
- Repository: https://github.com/zhexinliang/control-color

## Overview

##### Project type and purpose
- This is an open-source, full-pipeline image colorization tool built on a latent diffusion framework.
- It provides an interactive Gradio demo (“test.py”) that lets users colorize grayscale images via region selection, stroke painting, or text prompts.
- Under the hood it leverages:
– A pretrained VQGAN/VAE (“taming” modules) for image encoding/decoding
– A latent diffusion model (LDM) core (“ldm” modules) for sampling denoised latents
– A ControlLDM extension (“cldm” modules) that injects user controls into the diffusion process
– Annotator utilities (“annotator”) to parse user input (masks, strokes)

##### Top-level components (per file tree)
- Config & model definitions
– CtrlColor_environ.yaml
– config.py
– models/*.yaml (model hyperparameters)
- Pretrained model storage: pretrained_models/
- Annotator: annotator/util.py
- Autoencoder (taming): taming/modules/autoencoder & taming/models/vqgan.py
- Diffusion engine (LDM core): ldm/models/diffusion & ldm/modules
- Control extension: cldm/cldm.py & auxiliary sampling hacks
- Orchestration & UI: share.py, test.py

##### Key architectural patterns
- Layered pipeline:
1) Input handling (Gradio UI)
2) Pre-processing & annotation (masks, strokes)
3) Latent encoding (taming autoencoder)
4) Controlled diffusion sampling (LDM + ControlLDM)
5) Latent decoding back to image
- Configuration-driven instantiation: YAML files + config.py
- Modular separation:
– “annotator” is orthogonal to diffusion core
– “taming” provides VAE
– “ldm” implements diffusion backbone
– “cldm” injects control into the diffusion loop

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% UI Layer
    subgraph "User Interface (UI Layer)"
        UI_Test["Gradio Demo (test.py)"]:::ui
        UI_Share["Share Endpoint (share.py)"]:::ui
    end

    %% Preprocessing & Annotation
    subgraph "Preprocessing & Annotation"
        Annotator["Annotator Module\nannotator/util.py"]:::annot
    end

    %% Configuration & Model Loading
    subgraph "Configuration & Model Loader"
        Config["Config Loader\nconfig.py\nCtrlColor_environ.yaml\nmodels/*.yaml\npretrained_models/"]:::config
    end

    %% Autoencoder (VQGAN/VAE)
    subgraph "Autoencoder (VAE)"
        VAE_Enc["VAE Encoder\ntaming/modules/autoencoder"]:::autoencoder
        VQGAN["VQGAN Model\ntaming/models/vqgan.py"]:::autoencoder
        VAE_Dec["VAE Decoder\ntaming/modules/autoencoder"]:::autoencoder
    end

    %% Diffusion Core
    subgraph "Diffusion Core (LDM)"
        LDM["LDM Core\nldm/models/diffusion/\nldm/modules/\nldm/util.py"]:::diffusion
    end

    %% Control Extension
    subgraph "Control Extension (ControlLDM)"
        CLDM_Main["ControlLDM\ncldm/cldm.py"]:::control
        CLDM_Hack1["DDIM SAG Step\ncldm/ddim_haced_sag_step.py"]:::control
        CLDM_Hack2["Hacked DDIM\ncldm/ddim_hacked_sag.py"]:::control
        CLDM_Hacks["Sampling Hacks\ncldm/hack.py"]:::control
        CLDM_Model["Control Model\ntemplates/cldm/model.py"]:::control
    end

    %% Output
    subgraph "Output & Display"
        Output["Post-processor & Display\n(test.py, share.py)"]:::ui
    end

    %% Data Flow
    UI_Test -->|"user input\nupload/paint/text"| Annotator
    UI_Test -->|"invoke configs"| Config
    UI_Test -->|"provide image"| VAE_Enc
    Annotator -->|"control signals"| CLDM_Main
    Config -->|"load checkpoints\n& params"| VAE_Enc
    Config -->|"load checkpoints\n& params"| LDM
    Config -->|"load configs"| CLDM_Main
    VAE_Enc -->|"latent z"| LDM
    LDM -->|"denoised latent"| CLDM_Main
    CLDM_Main -->|"refined latent z'"| VAE_Dec
    CLDM_Hack1 --> CLDM_Main
    CLDM_Hack2 --> CLDM_Main
    CLDM_Hacks --> CLDM_Main
    CLDM_Model --> CLDM_Main
    VQGAN --> VAE_Enc
    VAE_Dec -->|"reconstruct image"| Output
    UI_Share --> Output

    %% Click Events
    click UI_Test "https://github.com/zhexinliang/control-color/blob/main/test.py"
    click UI_Share "https://github.com/zhexinliang/control-color/blob/main/share.py"
    click Annotator "https://github.com/zhexinliang/control-color/blob/main/annotator/util.py"
    click Config "https://github.com/zhexinliang/control-color/blob/main/config.py"
    click Config "https://github.com/zhexinliang/control-color/blob/main/CtrlColor_environ.yaml"
    click Config "https://github.com/zhexinliang/control-color/tree/main/models/"
    click Config "https://github.com/zhexinliang/control-color/tree/main/pretrained_models/"
    click VAE_Enc "https://github.com/zhexinliang/control-color/tree/main/taming/modules/autoencoder/"
    click VQGAN "https://github.com/zhexinliang/control-color/blob/main/taming/models/vqgan.py"
    click VAE_Dec "https://github.com/zhexinliang/control-color/tree/main/taming/modules/autoencoder/"
    click LDM "https://github.com/zhexinliang/control-color/tree/main/ldm/models/diffusion/"
    click LDM "https://github.com/zhexinliang/control-color/tree/main/ldm/modules/"
    click LDM "https://github.com/zhexinliang/control-color/blob/main/ldm/util.py"
    click CLDM_Main "https://github.com/zhexinliang/control-color/blob/main/cldm/cldm.py"
    click CLDM_Hack1 "https://github.com/zhexinliang/control-color/blob/main/cldm/ddim_haced_sag_step.py"
    click CLDM_Hack2 "https://github.com/zhexinliang/control-color/blob/main/cldm/ddim_hacked_sag.py"
    click CLDM_Hacks "https://github.com/zhexinliang/control-color/blob/main/cldm/hack.py"
    click CLDM_Model "https://github.com/zhexinliang/control-color/blob/main/cldm/model.py"

    %% Styles
    classDef ui fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef annot fill:#90EE90,stroke:#333,stroke-width:1px
    classDef config fill:#FFA500,stroke:#333,stroke-width:1px
    classDef autoencoder fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef diffusion fill:#FF6347,stroke:#333,stroke-width:1px
    classDef control fill:#FF4500,stroke:#333,stroke-width:1px
```
