How comfyanonymous/comfyui works
Project type and purpose
• ComfyUI is a full-stack, modular, graph-based AI engine and application for designing and executing Stable Diffusion pipelines.
• It provides a local web-based UI (nodes/graphs interface) and a Python backend that loads models, builds execution graphs, runs them on CPU/GPU via PyTorch, and returns images.
Top-level architecture components
a. Frontend (UI)
• A browser-based app (JS/Vue) served by the Python server.
• Communicates over HTTP/WebSocket with the backend to send graphs, receive status, display images.
b. Backend server
• Entry points: main.py (CLI), server.py (WS/HTTP server).
• Hosts API endpoints defined in api_server/routes (internal & external).
• Manages services (api_server/services/terminal_service.py, app/frontend_management.py, app/user_manager.py, etc.).
c. Application layer
• app/ package: global settings (app_settings.py), custom node registry, user sessions, model manager.
• Orchestrates loading workflows, dispatching graph executions, caching, logging.
d. Core engine
• comfy_execution/: graph representation, validation, caching, execution scheduling.
• comfy/: diffusion model wrappers (LDMs, ControlNet, CLIP, samplers, text/image encoders), node definitions, options, patchers.
• comfy_api/ & comfy_api_nodes/: internal Python API for building custom nodes and programmatic graph construction.
e. Persistence & assets
• Filesystem folders under models/ (checkpoints, VAEs, Loras, embeddings).
• Workflow save/load as JSON and embedded in PNG/WebP.
• Config files: extra_model_paths.yaml, pyproject.toml, requirements.txt.
f. External resources
• GPU/CPU device via PyTorch.
• Optional external frontend repo (--front-end-version) and CLI tool (comfy-cli).
• CI pipelines (.github/workflows, .ci/) for builds, tests, releases.
Relationships and data flows
1. User interacts with Frontend → builds node graph.
2. Frontend sends graph JSON via WebSocket/HTTP POST to Backend (server.py → api_server/routes/external).
3. Route handler forwards to service layer (app/custom_node_manager, app/model_manager).
4. Service constructs an execution graph object in comfy_execution/graph.py.
5. Execution engine validates graph, checks cache (comfy_execution/caching.py), executes only changed nodes.
6. For each node, core modules in comfy/ load models or run samplers/encoders on GPU via PyTorch.
7. Outputs (images, tensors) return up the call chain → Backend returns results via WS to Frontend.
8. Frontend displays results.
9. All model & workflow files are read from / written to the filesystem under models/ and output/.
Architectural patterns & principles
• Layered MVC-style separation:
• View: Frontend
• Controller: API routes
• Services: app/* modules
• Model/Domain: comfy_execution + comfy libraries
• Plugin/extension pattern: custom nodes (app/custom_node_manager.py, custom_nodes/, comfy_extras/) discovered at runtime.
• Graph execution pipeline: asynchronous queue, incremental re-execution, caching of node outputs.
• Clear separation of concerns: UI vs. session management vs. graph logic vs. ML model code.