How ant-research/magicquill works
Project Type and Purpose
• MagicQuill is a full-stack, interactive AI image-editing system.
Directory Structure and Architectural Layers
Top-level
• gradio_run.py – Main entry point for launching the Gradio web server.
• Dockerfile, docker-compose.yaml, linux_setup.sh, windows_setup.bat – Deployment scripts and container config.
• pyproject.toml, requirements.txt – Python packaging and dependencies.
MagicQuill/
• brushnet/ – Code for the segmentation/inpainting network (BrushNet).
• comfy/ – ComfyUI integration and stable-diffusion pipeline wrappers (samplers, modules, controlnet, tokenizers, etc.).
• LLaVA/ – Submodule for language-vision assistant (prompt understanding and suggestion).
• misc modules (magic_utils.py, model_patch.py, pidi.py, latent_preview.py) – Glue code for processing user strokes, patching models, visualizing latents.
A. Client / Frontend
• Browser with Gradio interface (canvas powered by fabric.js).
B. API Layer / Controller
• gradio_run.py + magic_utils.py + model_patch.py: orchestrates calls, maintains session state.
C. BrushNet Module
• Processes brush strokes and original image → generates masks, edge maps, local guidance.
D. Prompt Assistant (LLaVA)
• Uses drawn strokes to guess “what user is drawing” → auto-fills prompt text.
E. Diffusion Pipeline (ComfyUI Wrappers)
• Stable-Diffusion models (sd1, sd2, sdxl, ControlNet, LoRA, etc.).
• Samplers (k_diffusion, extra_samplers), encoders, upscalers.
F. Model Storage and Checkpoints
• Local filesystem (/data/magicquill/models or models.zip unzip).
• External endpoints (HuggingFace, Modelscope).
G. Deployment Environment
• Docker container, Docker Compose orchestration.
• Shell scripts for local or CI/CD installs.
Relationships & Data Flow
1. User draws on canvas → strokes & settings packaged as JSON + image.
2. Gradio server receives input → Controller invokes:
a. BrushNet → mask/edge generation
b. LLaVA → textual prompt suggestion
c. ComfyUI pipeline → diffusion inference with mask & prompt
3. Generated image returned → UI updates canvas
4. User may accept (continue editing) or discard (revert to previous)
Architectural Patterns & Design Principles
• Layered MVC-style separation:
• View (Gradio UI)
• Controller/orchestration (gradio_run.py + utils)
• Model inference (BrushNet, ComfyUI, LLaVA)
• Plugin-based extensibility: ComfyUI custom nodes, BrushNet configs, new samplers.
• Containerized deployment: repeatable, isolated environment.
• Configuration-as-code: JSON model configs, pyproject.toml, requirements.txt.
Technologies & Frameworks
• Python 3.10, PyTorch, diffusers, k-diffusion
• Gradio (web UI), fabric.js (canvas)
• ComfyUI, ControlNet, LoRA, T5, CLIP (vision & text)
• LLaVA submodule for prompt assistance
• Docker, Docker Compose, Conda (env management)