# lllyasviel/controlnet architecture

> Architecture diagram of the GitHub repository lllyasviel/controlnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lllyasviel/controlnet
- Repository: https://github.com/lllyasviel/controlnet

## Overview

ControlNet is a Stable Diffusion conditioning system: a user supplies a prompt and an image or control map, a Gradio app obtains or accepts the control, and the ControlNet-conditioned diffusion model generates an image. The graph retains documented control modes and the main model, sampling, and latent-image layers. Some app wiring and detector implementations are not excerpted; README-supported links are cited there, and uncertain internal wiring is omitted.

Main entry points: [`gradio_canny2image.py`](https://github.com/lllyasviel/controlnet/blob/main/gradio_canny2image.py) (Gradio apps), [`gradio_annotator.py`](https://github.com/lllyasviel/controlnet/blob/main/gradio_annotator.py) (Annotator UI), [`cldm/cldm.py`](https://github.com/lllyasviel/controlnet/blob/main/cldm/cldm.py) (ControlLDM), [`ldm/models/diffusion/ddpm.py`](https://github.com/lllyasviel/controlnet/blob/main/ldm/models/diffusion/ddpm.py) (Latent diffusion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User interfaces"]
  node_apps["Gradio apps"]
  node_annotator_ui["Annotator UI"]
end

subgraph group_controls["Control preparation"]
  node_canny["Canny edges<br/>[__init__.py]"]
  node_hed["HED boundaries<br/>[__init__.py]"]
  node_mlsd["M-LSD lines<br/>[__init__.py]"]
  node_midas["Depth maps<br/>[api.py]"]
  node_pose["Human pose<br/>[body.py]"]
  node_segmentation["Segmentation<br/>[inference.py]"]
end

subgraph group_conditioning["ControlNet conditioning"]
  node_control_ldm["ControlLDM<br/>[cldm.py]"]
  node_control_net["ControlNet<br/>[cldm.py]"]
end

subgraph group_diffusion["Diffusion and latent image"]
  node_diffusion_model["Latent diffusion<br/>[ddpm.py]"]
  node_ddim["DDIM sampler<br/>[ddim.py]"]
  node_hacked_ddim["Control sampler<br/>[ddim_hacked.py]"]
  node_autoencoder["Latent autoencoder<br/>[autoencoder.py]"]
  node_diffusion_blocks["Diffusion blocks<br/>[model.py]"]
  node_attention["Attention layers<br/>[attention.py]"]
  node_unet["UNet layers<br/>[openaimodel.py]"]
end

subgraph group_support["Model support"]
  node_distributions["Latent distributions<br/>[distributions.py]"]
  node_ema["EMA weights<br/>[ema.py]"]
  node_degradation["Image degradation<br/>[utils_image.py]"]
end

node_user(("User"))
node_output["Generated image"]

node_user -->|"provides prompt"| node_apps
node_user -->|"submits image"| node_annotator_ui
node_annotator_ui -->|"runs detector"| node_canny
node_annotator_ui -->|"runs detector"| node_hed
node_annotator_ui -->|"runs detector"| node_mlsd
node_annotator_ui -->|"runs detector"| node_midas
node_annotator_ui -->|"runs detector"| node_pose
node_annotator_ui -->|"runs detector"| node_segmentation
node_apps -.->|"uses control"| node_canny
node_apps -.->|"uses control"| node_hed
node_apps -.->|"uses control"| node_mlsd
node_apps -.->|"uses control"| node_midas
node_apps -.->|"uses control"| node_pose
node_apps -.->|"uses control"| node_segmentation
node_control_ldm -->|"instantiates"| node_control_net
node_control_ldm -->|"extends"| node_diffusion_model
node_diffusion_model -->|"uses latent model"| node_autoencoder
node_diffusion_model -->|"uses EMA"| node_ema
node_autoencoder -->|"uses encoder decoder"| node_diffusion_blocks
node_autoencoder -->|"uses distributions"| node_distributions
node_diffusion_model -->|"supports sampling"| node_ddim
node_control_ldm -.->|"sampled by"| node_hacked_ddim
node_control_net -->|"reuses blocks"| node_unet
node_unet -->|"uses attention"| node_attention
node_diffusion_blocks -->|"uses attention"| node_attention
node_apps -.->|"generates with"| node_control_ldm
node_control_ldm -.->|"produces"| node_output
node_degradation -.->|"prepares training images"| node_output

click node_apps "https://github.com/lllyasviel/controlnet/blob/main/gradio_canny2image.py"
click node_annotator_ui "https://github.com/lllyasviel/controlnet/blob/main/gradio_annotator.py"
click node_canny "https://github.com/lllyasviel/controlnet/blob/main/annotator/canny/__init__.py"
click node_hed "https://github.com/lllyasviel/controlnet/blob/main/annotator/hed/__init__.py"
click node_mlsd "https://github.com/lllyasviel/controlnet/blob/main/annotator/mlsd/__init__.py"
click node_midas "https://github.com/lllyasviel/controlnet/blob/main/annotator/midas/api.py"
click node_pose "https://github.com/lllyasviel/controlnet/blob/main/annotator/openpose/body.py"
click node_segmentation "https://github.com/lllyasviel/controlnet/blob/main/annotator/uniformer/mmseg/apis/inference.py"
click node_control_ldm "https://github.com/lllyasviel/controlnet/blob/main/cldm/cldm.py"
click node_control_net "https://github.com/lllyasviel/controlnet/blob/main/cldm/cldm.py"
click node_diffusion_model "https://github.com/lllyasviel/controlnet/blob/main/ldm/models/diffusion/ddpm.py"
click node_ddim "https://github.com/lllyasviel/controlnet/blob/main/ldm/models/diffusion/ddim.py"
click node_hacked_ddim "https://github.com/lllyasviel/controlnet/blob/main/cldm/ddim_hacked.py"
click node_autoencoder "https://github.com/lllyasviel/controlnet/blob/main/ldm/models/autoencoder.py"
click node_diffusion_blocks "https://github.com/lllyasviel/controlnet/blob/main/ldm/modules/diffusionmodules/model.py"
click node_attention "https://github.com/lllyasviel/controlnet/blob/main/ldm/modules/attention.py"
click node_unet "https://github.com/lllyasviel/controlnet/blob/main/ldm/modules/diffusionmodules/openaimodel.py"
click node_distributions "https://github.com/lllyasviel/controlnet/blob/main/ldm/modules/distributions/distributions.py"
click node_ema "https://github.com/lllyasviel/controlnet/blob/main/ldm/modules/ema.py"
click node_degradation "https://github.com/lllyasviel/controlnet/blob/main/ldm/modules/image_degradation/utils_image.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_apps,node_annotator_ui,node_user toneBlue
class node_canny,node_hed,node_mlsd,node_midas,node_pose,node_segmentation toneAmber
class node_control_ldm,node_control_net toneMint
class node_diffusion_model,node_ddim,node_hacked_ddim,node_autoencoder,node_diffusion_blocks,node_attention,node_unet toneRose
class node_distributions,node_ema,node_degradation,node_output toneIndigo
```

## Components

### User interfaces

- [Gradio apps](https://github.com/lllyasviel/controlnet/blob/main/gradio_canny2image.py): `gradio_canny2image.py`
- [Annotator UI](https://github.com/lllyasviel/controlnet/blob/main/gradio_annotator.py): `gradio_annotator.py`

### Control preparation

- [Canny edges](https://github.com/lllyasviel/controlnet/blob/main/annotator/canny/__init__.py): `annotator/canny/__init__.py`
- [HED boundaries](https://github.com/lllyasviel/controlnet/blob/main/annotator/hed/__init__.py): `annotator/hed/__init__.py`
- [M-LSD lines](https://github.com/lllyasviel/controlnet/blob/main/annotator/mlsd/__init__.py): `annotator/mlsd/__init__.py`
- [Depth maps](https://github.com/lllyasviel/controlnet/blob/main/annotator/midas/api.py): `annotator/midas/api.py`
- [Human pose](https://github.com/lllyasviel/controlnet/blob/main/annotator/openpose/body.py): `annotator/openpose/body.py`
- [Segmentation](https://github.com/lllyasviel/controlnet/blob/main/annotator/uniformer/mmseg/apis/inference.py): `annotator/uniformer/mmseg/apis/inference.py`

### ControlNet conditioning

- [ControlLDM](https://github.com/lllyasviel/controlnet/blob/main/cldm/cldm.py): `cldm/cldm.py`
- [ControlNet](https://github.com/lllyasviel/controlnet/blob/main/cldm/cldm.py): `cldm/cldm.py`

### Diffusion and latent image

- [Latent diffusion](https://github.com/lllyasviel/controlnet/blob/main/ldm/models/diffusion/ddpm.py): `ldm/models/diffusion/ddpm.py`
- [DDIM sampler](https://github.com/lllyasviel/controlnet/blob/main/ldm/models/diffusion/ddim.py): `ldm/models/diffusion/ddim.py`
- [Control sampler](https://github.com/lllyasviel/controlnet/blob/main/cldm/ddim_hacked.py): `cldm/ddim_hacked.py`
- [Latent autoencoder](https://github.com/lllyasviel/controlnet/blob/main/ldm/models/autoencoder.py): `ldm/models/autoencoder.py`
- [Diffusion blocks](https://github.com/lllyasviel/controlnet/blob/main/ldm/modules/diffusionmodules/model.py): `ldm/modules/diffusionmodules/model.py`
- [Attention layers](https://github.com/lllyasviel/controlnet/blob/main/ldm/modules/attention.py): `ldm/modules/attention.py`
- [UNet layers](https://github.com/lllyasviel/controlnet/blob/main/ldm/modules/diffusionmodules/openaimodel.py): `ldm/modules/diffusionmodules/openaimodel.py`

### Model support

- [Latent distributions](https://github.com/lllyasviel/controlnet/blob/main/ldm/modules/distributions/distributions.py): `ldm/modules/distributions/distributions.py`
- [EMA weights](https://github.com/lllyasviel/controlnet/blob/main/ldm/modules/ema.py): `ldm/modules/ema.py`
- [Image degradation](https://github.com/lllyasviel/controlnet/blob/main/ldm/modules/image_degradation/utils_image.py): `ldm/modules/image_degradation/utils_image.py`

### Other components

- **User**
- **Generated image**

## Connections

- User → Gradio apps: provides prompt (evidence: `README.md`)
- User → Annotator UI: submits image (evidence: `gradio_annotator.py`)
- Annotator UI → Canny edges: runs detector (evidence: `gradio_annotator.py`)
- Annotator UI → HED boundaries: runs detector (evidence: `gradio_annotator.py`)
- Annotator UI → M-LSD lines: runs detector (evidence: `gradio_annotator.py`)
- Annotator UI → Depth maps: runs detector (evidence: `gradio_annotator.py`)
- Annotator UI → Human pose: runs detector (evidence: `gradio_annotator.py`)
- Annotator UI → Segmentation: runs detector (evidence: `gradio_annotator.py`)
- Gradio apps → Canny edges: uses control
- Gradio apps → HED boundaries: uses control
- Gradio apps → M-LSD lines: uses control
- Gradio apps → Depth maps: uses control
- Gradio apps → Human pose: uses control
- Gradio apps → Segmentation: uses control
- ControlLDM → ControlNet: instantiates (evidence: `cldm/cldm.py`)
- ControlLDM → Latent diffusion: extends (evidence: `cldm/cldm.py`)
- Latent diffusion → Latent autoencoder: uses latent model (evidence: `ldm/models/diffusion/ddpm.py`)
- Latent diffusion → EMA weights: uses EMA (evidence: `ldm/models/diffusion/ddpm.py`)
- Latent autoencoder → Diffusion blocks: uses encoder decoder (evidence: `ldm/models/autoencoder.py`)
- Latent autoencoder → Latent distributions: uses distributions (evidence: `ldm/models/autoencoder.py`)
- Latent diffusion → DDIM sampler: supports sampling (evidence: `ldm/models/diffusion/ddpm.py`)
- ControlLDM → Control sampler: sampled by
- ControlNet → UNet layers: reuses blocks (evidence: `cldm/cldm.py`)
- UNet layers → Attention layers: uses attention
- Diffusion blocks → Attention layers: uses attention (evidence: `ldm/modules/diffusionmodules/model.py`)
- Gradio apps → ControlLDM: generates with
- ControlLDM → Generated image: produces
- Image degradation → Generated image: prepares training images
