# comfyanonymous/comfyui architecture

> Architecture diagram of the GitHub repository comfyanonymous/comfyui, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/comfyanonymous/comfyui
- Repository: https://github.com/comfyanonymous/comfyui

## Overview

##### Project type and purpose
- ComfyUI is a full-stack, modular, graph-based AI engine and application for designing and executing Stable Diffusion pipelines.
- It provides a local web-based UI (nodes/graphs interface) and a Python backend that loads models, builds execution graphs, runs them on CPU/GPU via PyTorch, and returns images.

##### Top-level architecture components
a. Frontend (UI)
- A browser-based app (JS/Vue) served by the Python server.
- Communicates over HTTP/WebSocket with the backend to send graphs, receive status, display images.
b. Backend server
- Entry points: `main.py` (CLI), `server.py` (WS/HTTP server).
- Hosts API endpoints defined in `api_server/routes` (internal & external).
- Manages services (`api_server/services/terminal_service.py`, `app/frontend_management.py`, `app/user_manager.py`, etc.).
c. Application layer
- `app/` package: global settings (`app_settings.py`), custom node registry, user sessions, model manager.
- Orchestrates loading workflows, dispatching graph executions, caching, logging.
d. Core engine
- `comfy_execution/`: graph representation, validation, caching, execution scheduling.
- `comfy/`: diffusion model wrappers (LDMs, ControlNet, CLIP, samplers, text/image encoders), node definitions, options, patchers.
- `comfy_api/` & `comfy_api_nodes/`: internal Python API for building custom nodes and programmatic graph construction.
e. Persistence & assets
- Filesystem folders under `models/` (checkpoints, VAEs, Loras, embeddings).
- Workflow save/load as JSON and embedded in PNG/WebP.
- Config files: `extra_model_paths.yaml`, `pyproject.toml`, `requirements.txt`.
f. External resources
- GPU/CPU device via PyTorch.
- Optional external frontend repo (`--front-end-version`) and CLI tool (`comfy-cli`).
- CI pipelines (`.github/workflows`, `.ci/`) for builds, tests, releases.

##### Relationships and data flows
1. User interacts with Frontend → builds node graph.
2. Frontend sends graph JSON via WebSocket/HTTP POST to Backend (`server.py` → `api_server/routes/external`).
3. Route handler forwards to service layer (`app/custom_node_manager`, `app/model_manager`).
4. Service constructs an execution graph object in `comfy_execution/graph.py`.
5. Execution engine validates graph, checks cache (`comfy_execution/caching.py`), executes only changed nodes.
6. For each node, core modules in `comfy/` load models or run samplers/encoders on GPU via PyTorch.
7. Outputs (images, tensors) return up the call chain → Backend returns results via WS to Frontend.
8. Frontend displays results.
9. All model & workflow files are read from / written to the filesystem under `models/` and `output/`.

##### Architectural patterns & principles
- Layered MVC-style separation:
- View: Frontend
- Controller: API routes
- Services: app/* modules
- Model/Domain: comfy_execution + comfy libraries
- Plugin/extension pattern: custom nodes (`app/custom_node_manager.py`, `custom_nodes/`, `comfy_extras/`) discovered at runtime.
- Graph execution pipeline: asynchronous queue, incremental re-execution, caching of node outputs.
- Clear separation of concerns: UI vs. session management vs. graph logic vs. ML model code.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Client Layer
    subgraph "UI Layer" 
        direction TB
        Browser(["Browser (Vue.js)"]):::ui
        FrontendMgr(["Frontend Management"]):::ui
    end

    %% API Layer
    subgraph "API Server Layer"
        direction TB
        CLI["CLI Entry (main.py)"]:::api
        Server["WS/HTTP Server (server.py)"]:::api
        InternalRoutes["Internal API Routes"]:::api
    end

    %% Services Layer
    subgraph "Application Services Layer"
        direction TB
        AppSettings["Global Settings"]:::services
        CustomNodeMgr["Custom Node Manager"]:::services
        FrontendSess["Frontend Session Manager"]:::services
        ModelMgr["Model Manager"]:::services
        UserMgr["User Manager"]:::services
        Logger["Logger"]:::services
    end

    %% API Utilities/Services
    subgraph "API Services & Utils"
        direction TB
        TerminalSvc["Terminal Service"]:::apiSvc
        FileOps["File Operations"]:::apiSvc
    end

    %% Core Engine Layer
    subgraph "Core Engine: Graph Execution" 
        direction TB
        GraphBuild["Graph Builder"]:::engine
        GraphVal["Graph Validation"]:::engine
        Caching["Caching"]:::engine
        GraphUtils["Graph Utils"]:::engine
    end

    subgraph "Core Engine: Diffusion & Nodes"
        direction TB
        ModelLoad["Model Management & Loading"]:::engine
        Samplers["Samplers"]:::engine
        ControlNet["ControlNet Integration"]:::engine
        CLIP["CLIP & Vision Models"]:::engine
        NodeDefs["Node Definitions & Options"]:::engine
        TextEnc["Text & Image Encoders"]:::engine
    end

    %% Internal Python API
    subgraph "Internal Python API"
        direction TB
        APIInput["Basic Input Types"]:::engine
        InputImpl["Input Implementations"]:::engine
        APIUtils["API Utilities"]:::engine
        NodesAPI["Nodes API Controllers"]:::engine
    end

    %% Plugins
    subgraph "Plugins & Extensions"
        direction TB
        Extras["Official Extra Nodes"]:::engine
        CustomNodes["User-defined Custom Nodes"]:::engine
    end

    %% Storage & Configuration
    subgraph "Persistence & Config"
        direction TB
        ModelsStore[(Models Folder)]:::storage
        WorkflowFiles[(Workflow JSON/PNG)]:::storage
        OutputStore[(Output Images)]:::storage
        ConfigFiles[(Config & Dependency Files)]:::storage
        CI[(CI Pipelines)]:::storage
    end

    %% Hardware Layer
    subgraph "Hardware"
        direction TB
        GPU["GPU/CPU (PyTorch)"]:::hw
    end

    %% Connections
    Browser -->|"HTTP/WS"| Server
    Server -->|routes to| InternalRoutes
    InternalRoutes -->|uses| TerminalSvc
    InternalRoutes -->|uses| FileOps
    InternalRoutes -->|forwards to| CustomNodeMgr
    InternalRoutes -->|forwards to| ModelMgr
    CLI -->|invokes| Server

    CustomNodeMgr -->|registers| GraphBuild
    ModelMgr -->|constructs| GraphBuild
    FrontendSess -->|manages| Browser

    GraphBuild -->|validates| GraphVal
    GraphVal -->|checks| Caching
    Caching -->|executes| Samplers
    Caching -->|executes| ModelLoad

    Samplers -->|calls| GPU
    ModelLoad -->|loads via PyTorch| GPU
    ControlNet -->|runs on| GPU
    CLIP -->|runs on| GPU
    TextEnc -->|runs on| GPU

    GraphUtils -->|aids| GraphBuild

    APIInput -->|provides to| GraphBuild
    InputImpl -->|extends| APIInput
    APIUtils -->|utility for| NodesAPI
    NodesAPI -->|integrates| GraphBuild

    Extras -->|plugins to| GraphBuild
    CustomNodes -->|plugins to| GraphBuild

    GraphBuild -->|reads/writes| ModelsStore
    GraphBuild -->|saves workflows| WorkflowFiles
    GraphBuild -->|writes images| OutputStore

    Server -->|reads| ConfigFiles
    CI -->|tests| Server

    %% Click Events
    click FrontendMgr "https://github.com/comfyanonymous/comfyui/blob/master/app/frontend_management.py"
    click CLI "https://github.com/comfyanonymous/comfyui/blob/master/main.py"
    click Server "https://github.com/comfyanonymous/comfyui/blob/master/server.py"
    click InternalRoutes "https://github.com/comfyanonymous/comfyui/blob/master/api_server/routes/internal/internal_routes.py"
    click TerminalSvc "https://github.com/comfyanonymous/comfyui/blob/master/api_server/services/terminal_service.py"
    click FileOps "https://github.com/comfyanonymous/comfyui/blob/master/api_server/utils/file_operations.py"
    click AppSettings "https://github.com/comfyanonymous/comfyui/blob/master/app/app_settings.py"
    click CustomNodeMgr "https://github.com/comfyanonymous/comfyui/blob/master/app/custom_node_manager.py"
    click FrontendSess "https://github.com/comfyanonymous/comfyui/blob/master/app/frontend_management.py"
    click ModelMgr "https://github.com/comfyanonymous/comfyui/blob/master/app/model_manager.py"
    click UserMgr "https://github.com/comfyanonymous/comfyui/blob/master/app/user_manager.py"
    click Logger "https://github.com/comfyanonymous/comfyui/blob/master/app/logger.py"
    click GraphBuild "https://github.com/comfyanonymous/comfyui/blob/master/comfy_execution/graph.py"
    click GraphVal "https://github.com/comfyanonymous/comfyui/blob/master/comfy_execution/validation.py"
    click Caching "https://github.com/comfyanonymous/comfyui/blob/master/comfy_execution/caching.py"
    click GraphUtils "https://github.com/comfyanonymous/comfyui/blob/master/comfy_execution/graph_utils.py"
```
