How lllyasviel/controlnet works
ControlNet is a Stable Diffusion conditioning system: a user supplies a prompt and an image or control map, a Gradio app obtains or accepts the control, and the ControlNet-conditioned diffusion model generates an image. The graph retains documented control modes and the main model, sampling, and latent-image layers. Some app wiring and detector implementations are not excerpted; README-supported links are cited there, and uncertain internal wiring is omitted.
Main entry points: gradio_canny2image.py Gradio apps · gradio_annotator.py Annotator UI · cldm.py ControlLDM · ddpm.py Latent diffusion
Components 22
User interfaces
- Gradio apps
gradio_canny2image.py - Annotator UI
gradio_annotator.py
Control preparation
- Canny edges
annotator/canny/__init__.py - HED boundaries
annotator/hed/__init__.py - M-LSD lines
annotator/mlsd/__init__.py - Depth maps
annotator/midas/api.py - Human pose
annotator/openpose/body.py - Segmentation
annotator/uniformer/mmseg/apis/inference.py
ControlNet conditioning
- ControlLDM
cldm/cldm.py - ControlNet
cldm/cldm.py
Diffusion and latent image
- Latent diffusion
ldm/models/diffusion/ddpm.py - DDIM sampler
ldm/models/diffusion/ddim.py - Control sampler
cldm/ddim_hacked.py - Latent autoencoder
ldm/models/autoencoder.py - Diffusion blocks
ldm/modules/diffusionmodules/model.py - Attention layers
ldm/modules/attention.py - UNet layers
ldm/modules/diffusionmodules/openaimodel.py
Model support
- Latent distributions
ldm/modules/distributions/distributions.py - EMA weights
ldm/modules/ema.py - Image degradation
ldm/modules/image_degradation/utils_image.py
Other components
- User
- Generated image
Connections 28
- User → Gradio apps: provides prompt
README.md - User → Annotator UI: submits image
gradio_annotator.py - Annotator UI → Canny edges: runs detector
gradio_annotator.py - Annotator UI → HED boundaries: runs detector
gradio_annotator.py - Annotator UI → M-LSD lines: runs detector
gradio_annotator.py - Annotator UI → Depth maps: runs detector
gradio_annotator.py - Annotator UI → Human pose: runs detector
gradio_annotator.py - Annotator UI → Segmentation: runs detector
gradio_annotator.py - Gradio apps → Canny edges: uses control
- Gradio apps → HED boundaries: uses control
- Gradio apps → M-LSD lines: uses control
- Gradio apps → Depth maps: uses control
- Gradio apps → Human pose: uses control
- Gradio apps → Segmentation: uses control
- ControlLDM → ControlNet: instantiates
cldm/cldm.py - ControlLDM → Latent diffusion: extends
cldm/cldm.py - Latent diffusion → Latent autoencoder: uses latent model
ldm/models/diffusion/ddpm.py - Latent diffusion → EMA weights: uses EMA
ldm/models/diffusion/ddpm.py - Latent autoencoder → Diffusion blocks: uses encoder decoder
ldm/models/autoencoder.py - Latent autoencoder → Latent distributions: uses distributions
ldm/models/autoencoder.py - Latent diffusion → DDIM sampler: supports sampling
ldm/models/diffusion/ddpm.py - ControlLDM → Control sampler: sampled by
- ControlNet → UNet layers: reuses blocks
cldm/cldm.py - UNet layers → Attention layers: uses attention
- Diffusion blocks → Attention layers: uses attention
ldm/modules/diffusionmodules/model.py - Gradio apps → ControlLDM: generates with
- ControlLDM → Generated image: produces
- Image degradation → Generated image: prepares training images