# qo-qiao/comfyui-omni-llm architecture

> Architecture diagram of the GitHub repository qo-qiao/comfyui-omni-llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/qo-qiao/comfyui-omni-llm
- Repository: https://github.com/qo-qiao/comfyui-omni-llm

## Overview

ComfyUI-omni-llm is a ComfyUI plugin for local llama.cpp multimodal inference, prompt creation, and chat, with optional external API support. Users compose nodes in a ComfyUI workflow: load models and set parameters, provide text or media, then run unified inference or task-specific audio/chat nodes to produce text, analysis, or speech-related outputs. Prompt presets cover image, video, and audio creation; multi-turn chat can use loaded skills. The source excerpts establish plugin registration, selected imports, and API calls; workflow connections beyond those documented or visible are not asserted.

Main entry points: [`__init__.py`](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/__init__.py) (Node registration), [`common.py`](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/common.py) (Shared multimodal helpers), [`nodes/unified_inference.py`](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/unified_inference.py) (Unified inference), [`nodes/multi_image_input.py`](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/multi_image_input.py) (Multi-image input)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_plugin["ComfyUI Plugin"]
  node_plugin_entry["Node registration<br/>[__init__.py]"]
end

subgraph group_inference["Inference and Media"]
  node_model_loader["Model loader<br/>[model_loader.py]"]
  node_parameters["Inference parameters"]
  node_clean_states["Clear state<br/>[clean_states.py]"]
  node_unified["Unified inference"]
  node_common["Shared multimodal helpers<br/>[common.py]"]
  node_multi_image["Multi-image input"]
  node_video["Video loader<br/>[video_loader.py]"]
  node_asr["ASR loader<br/>[asr_loader.py]"]
  node_tts["TTS loader<br/>[tts_loader.py]"]
end

subgraph group_conversation["Chat and Skills"]
  node_realtime["Multi-turn chat<br/>[realtime_chat.py]"]
  node_skill["Skill catalog and loader<br/>[skill_loader.py]"]
  node_skill_files["Moss image skill<br/>[compile_prompt.py]"]
end

subgraph group_prompts["Prompt Presets"]
  node_image_presets["Image prompt presets"]
  node_video_presets["Video prompt presets"]
  node_audio_presets["Audio prompt presets"]
end

subgraph group_api["External API"]
  node_api_config["API configuration<br/>[api_config.py]"]
  node_api_presets["Provider presets"]
end

node_user(("Workflow author"))
node_api_service{{"Remote model APIs"}}

node_user -->|"uses"| node_plugin_entry
node_plugin_entry -->|"registers"| node_model_loader
node_plugin_entry -->|"registers"| node_parameters
node_plugin_entry -->|"registers"| node_clean_states
node_plugin_entry -->|"registers"| node_unified
node_plugin_entry -->|"registers"| node_multi_image
node_plugin_entry -->|"registers"| node_video
node_plugin_entry -->|"registers"| node_asr
node_plugin_entry -->|"registers"| node_tts
node_plugin_entry -->|"registers"| node_realtime
node_plugin_entry -->|"registers"| node_skill
node_plugin_entry -->|"registers"| node_api_config
node_model_loader -->|"uses"| node_common
node_parameters -->|"uses"| node_common
node_clean_states -->|"uses"| node_common
node_unified -->|"uses"| node_common
node_multi_image -->|"uses"| node_common
node_multi_image -->|"uses formulas"| node_video_presets
node_asr -->|"uses"| node_common
node_tts -->|"uses"| node_common
node_realtime -->|"uses"| node_common
node_realtime -->|"uses skills"| node_skill
node_unified -->|"loads presets"| node_image_presets
node_unified -->|"loads presets"| node_video_presets
node_unified -->|"loads presets"| node_audio_presets
node_api_config -->|"reads defaults"| node_api_presets
node_unified -->|"sends requests"| node_api_service
node_api_config -.->|"configures access"| node_api_service

click node_plugin_entry "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/__init__.py"
click node_model_loader "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/model_loader.py"
click node_parameters "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/model_parameters.py"
click node_clean_states "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/clean_states.py"
click node_unified "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/unified_inference.py"
click node_common "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/common.py"
click node_multi_image "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/multi_image_input.py"
click node_video "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/video_loader.py"
click node_asr "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/asr_loader.py"
click node_tts "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/tts_loader.py"
click node_realtime "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/realtime_chat.py"
click node_skill "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/skill_loader.py"
click node_skill_files "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/skills/moss-image/scripts/compile_prompt.py"
click node_image_presets "https://github.com/qo-qiao/comfyui-omni-llm/tree/main/support"
click node_video_presets "https://github.com/qo-qiao/comfyui-omni-llm/tree/main/support"
click node_audio_presets "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/support/multi_speaker_dialogue.py"
click node_api_config "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/api_config.py"
click node_api_presets "https://github.com/qo-qiao/comfyui-omni-llm/blob/main/api/api_provider_presets.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_plugin_entry toneBlue
class node_model_loader,node_parameters,node_clean_states,node_unified,node_common,node_multi_image,node_video,node_asr,node_tts toneAmber
class node_realtime,node_skill,node_skill_files,node_api_service toneMint
class node_image_presets,node_video_presets,node_audio_presets toneRose
class node_api_config,node_api_presets,node_user toneIndigo
```

## Components

### ComfyUI Plugin

- [Node registration](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/__init__.py): `__init__.py`

### Inference and Media

- [Model loader](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/model_loader.py): `nodes/model_loader.py`
- [Inference parameters](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/model_parameters.py): `nodes/model_parameters.py`
- [Clear state](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/clean_states.py): `nodes/clean_states.py`
- [Unified inference](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/unified_inference.py): `nodes/unified_inference.py`
- [Shared multimodal helpers](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/common.py): `common.py`
- [Multi-image input](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/multi_image_input.py): `nodes/multi_image_input.py`
- [Video loader](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/video_loader.py): `nodes/video_loader.py`
- [ASR loader](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/asr_loader.py): `nodes/asr_loader.py`
- [TTS loader](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/tts_loader.py): `nodes/tts_loader.py`

### Chat and Skills

- [Multi-turn chat](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/realtime_chat.py): `nodes/realtime_chat.py`
- [Skill catalog and loader](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/skill_loader.py): `nodes/skill_loader.py`
- [Moss image skill](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/skills/moss-image/scripts/compile_prompt.py): `skills/moss-image/scripts/compile_prompt.py`

### Prompt Presets

- [Image prompt presets](https://github.com/qo-qiao/comfyui-omni-llm/tree/main/support): `support`
- [Video prompt presets](https://github.com/qo-qiao/comfyui-omni-llm/tree/main/support): `support`
- [Audio prompt presets](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/support/multi_speaker_dialogue.py): `support/multi_speaker_dialogue.py`

### External API

- [API configuration](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/nodes/api_config.py): `nodes/api_config.py`
- [Provider presets](https://github.com/qo-qiao/comfyui-omni-llm/blob/main/api/api_provider_presets.py): `api/api_provider_presets.py`

### Other components

- **Workflow author**
- **Remote model APIs**

## Connections

- Workflow author → Node registration: uses (evidence: `README.md`)
- Node registration → Model loader: registers (evidence: `__init__.py`)
- Node registration → Inference parameters: registers (evidence: `__init__.py`)
- Node registration → Clear state: registers (evidence: `__init__.py`)
- Node registration → Unified inference: registers (evidence: `__init__.py`)
- Node registration → Multi-image input: registers (evidence: `__init__.py`)
- Node registration → Video loader: registers (evidence: `__init__.py`)
- Node registration → ASR loader: registers (evidence: `__init__.py`)
- Node registration → TTS loader: registers (evidence: `__init__.py`)
- Node registration → Multi-turn chat: registers (evidence: `__init__.py`)
- Node registration → Skill catalog and loader: registers (evidence: `__init__.py`)
- Node registration → API configuration: registers (evidence: `__init__.py`)
- Model loader → Shared multimodal helpers: uses (evidence: `nodes/model_loader.py`)
- Inference parameters → Shared multimodal helpers: uses (evidence: `nodes/model_parameters.py`)
- Clear state → Shared multimodal helpers: uses (evidence: `nodes/clean_states.py`)
- Unified inference → Shared multimodal helpers: uses (evidence: `nodes/unified_inference.py`)
- Multi-image input → Shared multimodal helpers: uses (evidence: `nodes/multi_image_input.py`)
- Multi-image input → Video prompt presets: uses formulas (evidence: `nodes/multi_image_input.py`)
- ASR loader → Shared multimodal helpers: uses (evidence: `nodes/asr_loader.py`)
- TTS loader → Shared multimodal helpers: uses (evidence: `nodes/tts_loader.py`)
- Multi-turn chat → Shared multimodal helpers: uses (evidence: `nodes/realtime_chat.py`)
- Multi-turn chat → Skill catalog and loader: uses skills (evidence: `nodes/realtime_chat.py`)
- Unified inference → Image prompt presets: loads presets (evidence: `nodes/unified_inference.py`)
- Unified inference → Video prompt presets: loads presets (evidence: `nodes/unified_inference.py`)
- Unified inference → Audio prompt presets: loads presets (evidence: `nodes/unified_inference.py`)
- API configuration → Provider presets: reads defaults (evidence: `nodes/api_config.py`)
- Unified inference → Remote model APIs: sends requests (evidence: `nodes/unified_inference.py`)
- API configuration → Remote model APIs: configures access