How minuszoneai/comfyui-prompt-mz works
1. What this repo is
This is a ComfyUI custom node package for prompt-generation workflows. It does not define a standalone app; instead it plugs into ComfyUI and adds nodes for prompt beautification, translation, and image interrogation/captioning. The primary output of the package is text that can be fed back into ComfyUI text-conditioning nodes.
2. Top-level integration boundary
The package boundary is the ComfyUI extension layer: __init__.py is the registration entrypoint, while web/prompt_mz.js adds frontend behavior for the custom nodes. The Python side defines node classes; the JS side handles UI wiring in the ComfyUI browser interface.
3. Core node layer
mz_llama_core_nodes.py looks like the shared node implementation layer: common node definitions, parameter schemas, and orchestration for “CLIPTextEncode” style outputs. This is where the repo centralizes prompt assembly and the shared logic used by multiple backends.
4. LLM backend adapters
The repo supports multiple model runtimes through separate adapters:
- mz_llama_cpp.py for llama.cpp / llama-cpp-python
- mz_openaiapi.py for OpenAI-compatible API calls
- mz_transformers.py for local Hugging Face / transformers models
- mz_gen_translate.py for translation-oriented generation
These modules are backend boundaries: they normalize different inference providers into the same ComfyUI node style.
5. Prompt and instruction pipeline
mz_prompts.py and mz_prompt_utils.py provide the reusable prompt templates, instruction text, formatting helpers, and post-processing logic. The README mentions custom instructions, chat format selection, and A1111-style weight scaling, so these utilities are the main place where raw model output is transformed into workflow-ready prompt text.
6. Image interrogation path
Image-to-text functionality is a separate branch of the architecture. Nodes such as LLava, Florence-2, and the “ImageInterrogator” variants take an image input, run caption/interrogation inference, and return prompt text. The outputs can optionally be formatted for Stable Diffusion / A1111-style prompt usage.
7. Florence-2 and model hooks
hook/modeling_florence2.py indicates a targeted runtime hook/override for Florence-2 integration, likely to adapt the upstream model behavior to ComfyUI’s execution model. This is an important boundary because it suggests the repo sometimes patches third-party model code rather than wrapping it only at the node level.
8. Model selection and distribution
configs/model_zoo.json is the model catalog used by the downloader/select nodes. The repo supports both manual model path selection and predefined/downloadable model presets, which means model management is part of the architecture rather than being left entirely to the user.
9. Auxiliary utilities
half_json.py is a resilience utility: it likely repairs or parses partially malformed JSON emitted by LLMs, which matters because prompt-generation models often return non-strict structured output. ADV_CLIP_emb_encode.py suggests compatibility with ComfyUI_ADV_CLIP_emb-style weighted prompt encoding behavior.
10. Versioning and compatibility
The v1/ folder contains older or deprecated node implementations (mz_llama3.py, mz_llava.py, mz_phi3.py, etc.), showing an explicit compatibility layer for legacy workflows. The newer “universal” and manual-select nodes appear to replace those older per-model nodes while preserving existing graphs.
11. Execution flow
Typical data flow is: user prompt or image input → node-specific preprocessing/instruction assembly → backend inference (llama.cpp, OpenAI, transformers, Ollama-like local model selection) → output post-processing → text returned to ComfyUI. For image interrogation, the path starts with an image tensor and ends with generated caption/prompt text; for prompt beautification, it starts with text and returns revised text.
12. Operational dependencies
The architecture materially depends on ComfyUI, llama.cpp / llama-cpp-python, and sometimes OpenAI-compatible APIs or local model runtimes. The repo is designed to be installed under custom_nodes, with model files managed externally rather than bundled into the package.
Main entry points: mz_llama_core_nodes.py Core nodes · mz_prompt_utils.py Prompt utils · mz_deprecated.py Legacy nodes · __init__.py Package init
Components 19
ComfyUI extension
Package entrypoints and frontend wiring for the custom node extension.
- Package init · Registers the custom nodes with ComfyUI.
__init__.py - Web UI · Browser-side wiring for custom node controls and behavior.
web/prompt_mz.js
Prompt core
Shared prompt assembly, formatting, and node orchestration logic.
- Core nodes · Common node definitions, schemas, and orchestration shared across backends.
mz_llama_core_nodes.py - Prompt utils · Reusable formatting, post-processing, and prompt transformation helpers.
mz_prompt_utils.py - Prompt templates · Reusable prompt text, system instructions, and task-specific templates.
mz_prompts.py
Model backends
Adapters that normalize different text-generation runtimes and APIs.
- Llama.cpp · Adapter for llama.cpp / llama-cpp-python style inference.
mz_llama_cpp.py - OpenAI API · Adapter for OpenAI-compatible text generation APIs.
mz_openaiapi.py - Transformers · Adapter for local Hugging Face / transformers model execution.
mz_transformers.py - Translate gen · Generation path specialized for translation-oriented prompt output.
mz_gen_translate.py
Vision path
Image interrogation and captioning nodes and hooks.
- Image nodes · Image-to-text nodes for captioning and prompt generation.
- LLaVA path · Vision-language interrogation flow for captioning and prompt extraction.
v1/mz_llava.py - Florence-2 path · Florence-2 based interrogation flow with model-specific handling.
- Florence hook · Targeted override or adaptation layer for Florence-2 model behavior.
hook/modeling_florence2.py
Compatibility
Legacy and auxiliary modules that preserve older workflows or recover model output.
- Legacy nodes · Older node implementations retained for existing workflows.
v1/mz_deprecated.py - Llama3 legacy · Older model-specific node implementation kept for compatibility.
v1/mz_llama3.py - Phi3 legacy · Older model-specific node implementation kept for compatibility.
v1/mz_phi3.py - JSON repair · Helper for recovering or parsing malformed JSON emitted by models.
half_json.py - CLIP encode · Compatibility support for weighted prompt encoding behavior.
ADV_CLIP_emb_encode.py
Model catalog
Preset model selection and download metadata.
- Model zoo · Preset model catalog used for selection and download workflows.
configs/model_zoo.json
Connections 19
- Package init → Core nodes: registers · Package initialization exposes the core node set to ComfyUI.
- Web UI → Core nodes: drives · Frontend controls feed the shared node implementations.
- Core nodes → Prompt utils: uses · Core nodes rely on shared formatting and post-processing helpers.
- Core nodes → Prompt templates: builds from · Prompt assembly is driven by reusable instruction templates.
- Core nodes → Llama.cpp: dispatches to · Shared node layer can execute through llama.cpp.
- Core nodes → OpenAI API: dispatches to · Shared node layer can execute through OpenAI-compatible APIs.
- Core nodes → Transformers: dispatches to · Shared node layer can execute through local transformers models.
- Core nodes → Translate gen: dispatches to · Shared node layer can route into translation-oriented generation.
- Core nodes → Image nodes: exposes · Core package includes image interrogation node families.
- Image nodes → LLaVA path: includes · One branch uses LLaVA-style vision-language inference.
- Image nodes → Florence-2 path: includes · Another branch uses Florence-2 captioning/interrogation.
- Florence-2 path → Florence hook: adapts via · Florence-2 behavior is adjusted through a model hook.
- Image nodes → Prompt utils: formats with · Caption outputs are normalized into workflow-ready text.
- Prompt utils → JSON repair: recovers · Prompt utilities can depend on JSON repair for model output.
- Prompt utils → CLIP encode: aligns with · Formatting helpers support weighted prompt encoding conventions.
- Core nodes → Model zoo: selects from · Model-aware nodes can use the preset catalog for selection/download.
- Package init → Legacy nodes: keeps legacy · Entry point preserves compatibility with older node sets.
- Legacy nodes → Llama3 legacy: contains · Legacy compatibility includes older Llama3-specific nodes.
- Legacy nodes → Phi3 legacy: contains · Legacy compatibility includes older Phi3-specific nodes.