# ltdrdata/comfyui architecture

> Architecture diagram of the GitHub repository ltdrdata/comfyui, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ltdrdata/comfyui
- Repository: https://github.com/ltdrdata/comfyui

## Overview

1. **System shape**
   - This repo is a Python backend for ComfyUI, a graph-based generative AI engine. The UI is now mostly shipped as compiled frontend assets from a separate frontend repo, while this repository owns the runtime, APIs, execution engine, model loading, and bundled nodes.

2. **Primary entrypoints**
   - `main.py` is the usual launcher.
   - `server.py` and `api_server/` expose HTTP/WebSocket endpoints.
   - `execution.py` and `comfy_execution/` drive the workflow queue and graph execution.
   - `protocol.py` and `latent_preview.py` support runtime communication and previews.

3. **Workflow execution core**
   - The heart of the system is the node graph executor in `comfy_execution/`.
   - Workflows are validated, partially re-executed, cached, and scheduled as jobs rather than run monolithically.
   - This engine is what gives ComfyUI its “only recompute changed parts” behavior.

4. **Node ecosystem**
   - `nodes.py`, `comfy_extras/`, and `custom_nodes/` define the node catalog.
   - Built-in nodes cover image, video, audio, 3D, sampler, control, and utility operations.
   - `custom_nodes/` is the extension boundary for user-installed packages; these are loaded dynamically and can participate in the same graph system.

5. **Model/runtime management**
   - `comfy/model_management.py`, `comfy/model_patcher.py`, `comfy/memory_management.py`, and related modules manage model loading, patching, precision, and offloading.
   - `folder_paths.py` and `extra_model_paths.yaml.example` define the model-directory contract.
   - The repo supports many backends and hardware targets, with runtime-specific launch scripts under `.ci/` for NVIDIA, AMD, Intel, CPU, and nightly builds.

6. **Model family implementations**
   - The `comfy/ldm/` tree contains concrete model architectures and their helpers: diffusion models, VAEs, text encoders, audio encoders, video models, and specialized variants.
   - These modules are the low-level inference implementations used by nodes and loaders, not end-user workflow logic.

7. **Frontend boundary**
   - The live UI is not authored here anymore; the repository serves compiled frontend assets and the backend API that the frontend consumes.
   - The README notes that newer frontend builds come from the separate `ComfyUI_frontend` repository, with version pinning controlled by launch flags.

8. **Asset management subsystem**
   - `app/assets/` is a distinct business subsystem for managing file assets, metadata, tags, references, hashing, ingest, scanning, and bulk import.
   - It has its own API layer under `app/assets/api/`, service layer under `app/assets/services/`, database models/queries under `app/assets/database/`, and seeding/scanning utilities.
   - This is separate from the generative workflow engine; it manages repository-local asset records and filesystem synchronization.

9. **Database and migrations**
   - `app/database/` and `alembic_db/` represent the application persistence layer and schema migrations.
   - The migrations include asset/reference related schema evolution.
   - Tests indicate database-backed behaviors for assets, users, and other app subsystems.

10. **Internal API and external provider nodes**
   - `comfy_api/` defines the internal public API surface and generated stubs for programmatic graph/control access.
   - `comfy_api_nodes/` is a separate integration layer for third-party model providers such as OpenAI, Gemini, Runway, Stability, Luma, Kling, and others.
   - These nodes are a boundary to external network services and typically differ from local model execution because they depend on remote credentials, uploads, and API contracts.

11. **Security, users, and management**
   - `app/user_manager.py`, `app/custom_node_manager.py`, `app/model_manager.py`, and `app/frontend_management.py` handle higher-level application concerns: user/system identity, node installation, model discovery, and frontend versioning.
   - The ComfyUI-Manager integration is optional and feature-flagged via CLI options.

12. **I/O and file conventions**
   - Standard folders such as `models/`, `input/`, `output/`, and `blueprints/` define the application’s data plane.
   - Inputs may come from uploaded files, dragged workflow images, or filesystem scan results; outputs are written to `output/`.
   - `blueprints/` contains reusable workflow templates and shader-like fragments used by some nodes and examples.

13. **Testing and release pipeline**
   - The repo has extensive unit and integration tests under `tests/` and `tests-unit/`, covering execution, assets, server utilities, config, and API behavior.
   - `.github/workflows/` and `.ci/` show a strong release/packaging pipeline, including Windows-specific portable builds and dependency release jobs.
   - This matters architecturally because the repo is maintained as a distributable runtime, not just a library.

14. **Diagramming summary**
   - If you draw the architecture, the main flow is: frontend/UI or API request → server/API layer → graph validation and execution engine → node implementations → model/runtime loaders and file I/O → outputs, previews, and persisted state.
   - Side subsystems to show separately are assets/database management, custom node/plugin loading, and remote API-node integrations.

Main entry points: [`server.py`](https://github.com/ltdrdata/comfyui/blob/main/server.py) (Server), [`comfy_execution`](https://github.com/ltdrdata/comfyui/tree/main/comfy_execution) (Graph Exec), [`comfy/model_management.py`](https://github.com/ltdrdata/comfyui/blob/main/comfy/model_management.py) (Model Mgmt), [`app/assets`](https://github.com/ltdrdata/comfyui/tree/main/app/assets) (Assets)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_runtime["Runtime & API"]
  node_main_py["Main<br/>launcher<br/>[main.py]"]
  node_server_py["Server<br/>[server.py]"]
  node_api_server["API Server<br/>internal api"]
  node_protocol_py["Protocol<br/>[protocol.py]"]
  node_latent_preview_py(("Previews<br/>preview stream<br/>[latent_preview.py]"))
  node_blueprints["Blueprints<br/>workflow templates"]
end

subgraph group_grp_exec["Execution Engine"]
  node_execution_py["Queueing<br/>workflow scheduler<br/>[execution.py]"]
  node_comfy_exec["Graph Exec<br/>executor core"]
end

subgraph group_grp_nodes["Node Ecosystem"]
  node_nodes_py["Built-in Nodes<br/>node catalog<br/>[nodes.py]"]
  node_comfy_extras["Extras<br/>extended nodes"]
  node_custom_nodes["Custom Nodes<br/>plugin boundary"]
end

subgraph group_grp_models["Model Runtime"]
  node_model_mgmt["Model Mgmt<br/>runtime control"]
  node_memory_mgmt["Memory<br/>resource manager"]
  node_model_patcher["Patching<br/>model patcher<br/>[model_patcher.py]"]
  node_ldm_tree["LDM Models<br/>inference architectures"]
  node_folder_paths["Model Paths<br/>path contract<br/>[folder_paths.py]"]
end

subgraph group_grp_assets["Assets & Persistence"]
  node_app_assets["Assets<br/>asset subsystem"]
  node_app_db[("App DB<br/>persistence layer")]
  node_alembic_db["Migrations"]
  node_app_mgmt["App Mgmt<br/>admin services"]
end

subgraph group_grp_integrations["Integrations"]
  node_comfy_api["Comfy API<br/>public api"]
  node_api_nodes["API Nodes<br/>remote providers"]
end

node_main_py -->|"starts"| node_server_py
node_server_py -->|"routes"| node_api_server
node_server_py -->|"speaks"| node_protocol_py
node_server_py -->|"streams"| node_latent_preview_py
node_server_py -->|"submits"| node_execution_py
node_execution_py -->|"drives"| node_comfy_exec
node_comfy_exec -->|"invokes"| node_nodes_py
node_comfy_exec -->|"invokes"| node_comfy_extras
node_comfy_exec -->|"loads"| node_custom_nodes
node_comfy_exec -->|"loads"| node_model_mgmt
node_model_mgmt -->|"coordinates"| node_memory_mgmt
node_model_mgmt -->|"patches"| node_model_patcher
node_model_mgmt -->|"dispatches"| node_ldm_tree
node_model_mgmt -->|"resolves"| node_folder_paths
node_nodes_py -->|"uses"| node_folder_paths
node_comfy_extras -->|"wraps"| node_ldm_tree
node_custom_nodes -.->|"extends"| node_comfy_api
node_app_mgmt -->|"manages"| node_custom_nodes
node_app_mgmt -->|"configures"| node_folder_paths
node_app_assets -->|"persists"| node_app_db
node_app_assets -->|"evolves"| node_alembic_db
node_server_py -->|"exposes"| node_app_assets
node_server_py -->|"publishes"| node_comfy_api
node_server_py -->|"enables"| node_api_nodes
node_blueprints -.->|"feeds"| node_comfy_exec

click node_main_py "https://github.com/ltdrdata/comfyui/blob/main/main.py"
click node_server_py "https://github.com/ltdrdata/comfyui/blob/main/server.py"
click node_api_server "https://github.com/ltdrdata/comfyui/tree/main/api_server"
click node_protocol_py "https://github.com/ltdrdata/comfyui/blob/main/protocol.py"
click node_latent_preview_py "https://github.com/ltdrdata/comfyui/blob/main/latent_preview.py"
click node_execution_py "https://github.com/ltdrdata/comfyui/blob/main/execution.py"
click node_comfy_exec "https://github.com/ltdrdata/comfyui/tree/main/comfy_execution"
click node_nodes_py "https://github.com/ltdrdata/comfyui/blob/main/nodes.py"
click node_comfy_extras "https://github.com/ltdrdata/comfyui/tree/main/comfy_extras"
click node_custom_nodes "https://github.com/ltdrdata/comfyui/tree/main/custom_nodes"
click node_model_mgmt "https://github.com/ltdrdata/comfyui/blob/main/comfy/model_management.py"
click node_memory_mgmt "https://github.com/ltdrdata/comfyui/blob/main/comfy/memory_management.py"
click node_model_patcher "https://github.com/ltdrdata/comfyui/blob/main/comfy/model_patcher.py"
click node_ldm_tree "https://github.com/ltdrdata/comfyui/tree/main/comfy/ldm"
click node_folder_paths "https://github.com/ltdrdata/comfyui/blob/main/folder_paths.py"
click node_app_assets "https://github.com/ltdrdata/comfyui/tree/main/app/assets"
click node_app_db "https://github.com/ltdrdata/comfyui/tree/main/app/database"
click node_alembic_db "https://github.com/ltdrdata/comfyui/tree/main/alembic_db"
click node_app_mgmt "https://github.com/ltdrdata/comfyui/tree/main/app"
click node_comfy_api "https://github.com/ltdrdata/comfyui/tree/main/comfy_api"
click node_api_nodes "https://github.com/ltdrdata/comfyui/tree/main/comfy_api_nodes"
click node_blueprints "https://github.com/ltdrdata/comfyui/tree/main/blueprints"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_py,node_server_py,node_api_server,node_protocol_py,node_latent_preview_py,node_blueprints toneBlue
class node_execution_py,node_comfy_exec toneAmber
class node_nodes_py,node_comfy_extras,node_custom_nodes toneMint
class node_model_mgmt,node_memory_mgmt,node_model_patcher,node_ldm_tree,node_folder_paths toneRose
class node_app_assets,node_app_db,node_alembic_db,node_app_mgmt toneIndigo
class node_comfy_api,node_api_nodes toneTeal
```

## Components

### Runtime & API

Launch, server, protocol, and frontend boundary

- [Main](https://github.com/ltdrdata/comfyui/blob/main/main.py): launcher · Default application entrypoint · `main.py`
- [Server](https://github.com/ltdrdata/comfyui/blob/main/server.py): http/ws server · Core backend server and request handling · `server.py`
- [API Server](https://github.com/ltdrdata/comfyui/tree/main/api_server): internal api · Additional HTTP and internal routes · `api_server`
- [Protocol](https://github.com/ltdrdata/comfyui/blob/main/protocol.py): runtime protocol · Shared request/response and websocket communication types · `protocol.py`
- [Previews](https://github.com/ltdrdata/comfyui/blob/main/latent_preview.py): preview stream · Preview generation and delivery during execution · `latent_preview.py`
- [Blueprints](https://github.com/ltdrdata/comfyui/tree/main/blueprints): workflow templates · Reusable workflow templates and GLSL fragments · `blueprints`

### Execution Engine

Graph validation, scheduling, caching, and job execution

- [Queueing](https://github.com/ltdrdata/comfyui/blob/main/execution.py): workflow scheduler · Prompt queue, dispatch, and execution coordination · `execution.py`
- [Graph Exec](https://github.com/ltdrdata/comfyui/tree/main/comfy_execution): executor core · Validation, caching, jobs, and partial re-execution · `comfy_execution`

### Node Ecosystem

Built-in, extra, and custom nodes

- [Built-in Nodes](https://github.com/ltdrdata/comfyui/blob/main/nodes.py): node catalog · Primary bundled node registrations · `nodes.py`
- [Extras](https://github.com/ltdrdata/comfyui/tree/main/comfy_extras): extended nodes · Bundled specialized node implementations · `comfy_extras`
- [Custom Nodes](https://github.com/ltdrdata/comfyui/tree/main/custom_nodes): plugin boundary · User-installed dynamic node packages · `custom_nodes`

### Model Runtime

Model loading, patching, memory, and architecture-specific inference code

- [Model Mgmt](https://github.com/ltdrdata/comfyui/blob/main/comfy/model_management.py): runtime control · Loading, patching, precision, and offloading · `comfy/model_management.py`
- [Memory](https://github.com/ltdrdata/comfyui/blob/main/comfy/memory_management.py): resource manager · Device and memory orchestration · `comfy/memory_management.py`
- [Patching](https://github.com/ltdrdata/comfyui/blob/main/comfy/model_patcher.py): model patcher · Weight and model mutation support · `comfy/model_patcher.py`
- [LDM Models](https://github.com/ltdrdata/comfyui/tree/main/comfy/ldm): inference architectures · Concrete diffusion, VAE, text, audio, and video model implementations · `comfy/ldm`
- [Model Paths](https://github.com/ltdrdata/comfyui/blob/main/folder_paths.py): path contract · Standard model, input, output, and blueprint directory resolution · `folder_paths.py`

### Assets & Persistence

Asset records, ingestion, metadata, and database-backed state

- [Assets](https://github.com/ltdrdata/comfyui/tree/main/app/assets): asset subsystem · Asset ingest, metadata, references, tags, and filesystem sync · `app/assets`
- [App DB](https://github.com/ltdrdata/comfyui/tree/main/app/database): persistence layer · Application database models and connection management · `app/database`
- [Migrations](https://github.com/ltdrdata/comfyui/tree/main/alembic_db): schema migrations · Database schema evolution for app and assets · `alembic_db`
- [App Mgmt](https://github.com/ltdrdata/comfyui/tree/main/app): admin services · Users, custom nodes, model discovery, and frontend versioning · `app`

### Integrations

Internal API surface and remote provider nodes

- [Comfy API](https://github.com/ltdrdata/comfyui/tree/main/comfy_api): public api · Programmatic internal API surface and generated stubs · `comfy_api`
- [API Nodes](https://github.com/ltdrdata/comfyui/tree/main/comfy_api_nodes): remote providers · Third-party provider integrations backed by network calls · `comfy_api_nodes`

## Connections

- Main → Server: starts · Launcher initializes backend server
- Server → API Server: routes · Server exposes API and internal endpoints
- Server → Protocol: speaks · HTTP and websocket message contracts
- Server → Previews: streams · Server publishes execution previews
- Server → Queueing: submits · Requests become queued workflow runs
- Queueing → Graph Exec: drives · Queue hands workflows to graph executor
- Graph Exec → Built-in Nodes: invokes · Executor runs registered built-in nodes
- Graph Exec → Extras: invokes · Executor uses bundled extension nodes
- Graph Exec → Custom Nodes: loads · Executor discovers user-installed node packages
- Graph Exec → Model Mgmt: loads · Node execution triggers model acquisition
- Model Mgmt → Memory: coordinates · Model residency and offload decisions
- Model Mgmt → Patching: patches · Weights and adapters are applied at runtime
- Model Mgmt → LDM Models: dispatches · Runtime selects concrete model implementations
- Model Mgmt → Model Paths: resolves · Model and asset directories are mapped here
- Built-in Nodes → Model Paths: uses · Built-in nodes locate models and inputs
- Extras → LDM Models: wraps · Extra nodes bind higher-level workflows to model code
- Custom Nodes → Comfy API: extends · Plugins can target the typed internal API surface
- App Mgmt → Custom Nodes: manages · Installation and lifecycle for external nodes
- App Mgmt → Model Paths: configures · Model discovery and frontend/version settings
- Assets → App DB: persists · Asset metadata and references are stored in the app database
- Assets → Migrations: evolves · Schema changes support asset-related storage
- Server → Assets: exposes · Backend serves asset management APIs
- Server → Comfy API: publishes · Backend exposes programmatic graph/control APIs
- Server → API Nodes: enables · Server supports remote provider node integrations
- Blueprints → Graph Exec: feeds · Templates are instantiated into executable workflows