How qo-qiao/comfyui-omni-llm works
ComfyUI-omni-llm is a ComfyUI plugin for local llama.cpp multimodal inference, prompt creation, and chat, with optional external API support. Users compose nodes in a ComfyUI workflow: load models and set parameters, provide text or media, then run unified inference or task-specific audio/chat nodes to produce text, analysis, or speech-related outputs. Prompt presets cover image, video, and audio creation; multi-turn chat can use loaded skills. The source excerpts establish plugin registration, selected imports, and API calls; workflow connections beyond those documented or visible are not asserted.
Main entry points: __init__.py Node registration · common.py Shared multimodal helpers · unified_inference.py Unified inference · multi_image_input.py Multi-image input
Components 20
ComfyUI Plugin
- Node registration
__init__.py
Inference and Media
- Model loader
nodes/model_loader.py - Inference parameters
nodes/model_parameters.py - Clear state
nodes/clean_states.py - Unified inference
nodes/unified_inference.py - Shared multimodal helpers
common.py - Multi-image input
nodes/multi_image_input.py - Video loader
nodes/video_loader.py - ASR loader
nodes/asr_loader.py - TTS loader
nodes/tts_loader.py
Chat and Skills
- Multi-turn chat
nodes/realtime_chat.py - Skill catalog and loader
nodes/skill_loader.py - Moss image skill
skills/moss-image/scripts/compile_prompt.py
Prompt Presets
- Image prompt presets
support - Video prompt presets
support - Audio prompt presets
support/multi_speaker_dialogue.py
External API
- API configuration
nodes/api_config.py - Provider presets
api/api_provider_presets.py
Other components
- Workflow author
- Remote model APIs
Connections 28
- Workflow author → Node registration: uses
README.md - Node registration → Model loader: registers
__init__.py - Node registration → Inference parameters: registers
__init__.py - Node registration → Clear state: registers
__init__.py - Node registration → Unified inference: registers
__init__.py - Node registration → Multi-image input: registers
__init__.py - Node registration → Video loader: registers
__init__.py - Node registration → ASR loader: registers
__init__.py - Node registration → TTS loader: registers
__init__.py - Node registration → Multi-turn chat: registers
__init__.py - Node registration → Skill catalog and loader: registers
__init__.py - Node registration → API configuration: registers
__init__.py - Model loader → Shared multimodal helpers: uses
nodes/model_loader.py - Inference parameters → Shared multimodal helpers: uses
nodes/model_parameters.py - Clear state → Shared multimodal helpers: uses
nodes/clean_states.py - Unified inference → Shared multimodal helpers: uses
nodes/unified_inference.py - Multi-image input → Shared multimodal helpers: uses
nodes/multi_image_input.py - Multi-image input → Video prompt presets: uses formulas
nodes/multi_image_input.py - ASR loader → Shared multimodal helpers: uses
nodes/asr_loader.py - TTS loader → Shared multimodal helpers: uses
nodes/tts_loader.py - Multi-turn chat → Shared multimodal helpers: uses
nodes/realtime_chat.py - Multi-turn chat → Skill catalog and loader: uses skills
nodes/realtime_chat.py - Unified inference → Image prompt presets: loads presets
nodes/unified_inference.py - Unified inference → Video prompt presets: loads presets
nodes/unified_inference.py - Unified inference → Audio prompt presets: loads presets
nodes/unified_inference.py - API configuration → Provider presets: reads defaults
nodes/api_config.py - Unified inference → Remote model APIs: sends requests
nodes/unified_inference.py - API configuration → Remote model APIs: configures access