# ant-research/magicquill architecture

> Architecture diagram of the GitHub repository ant-research/magicquill, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ant-research/magicquill
- Repository: https://github.com/ant-research/magicquill

## Overview

##### Project Type and Purpose
- MagicQuill is a full-stack, interactive AI image-editing system.

##### Directory Structure and Architectural Layers
Top-level
- gradio_run.py – Main entry point for launching the Gradio web server.
- Dockerfile, docker-compose.yaml, linux_setup.sh, windows_setup.bat – Deployment scripts and container config.
- pyproject.toml, requirements.txt – Python packaging and dependencies.
MagicQuill/
- brushnet/ – Code for the segmentation/inpainting network (BrushNet).
- comfy/ – ComfyUI integration and stable-diffusion pipeline wrappers (samplers, modules, controlnet, tokenizers, etc.).
- LLaVA/ – Submodule for language-vision assistant (prompt understanding and suggestion).
- misc modules (magic_utils.py, model_patch.py, pidi.py, latent_preview.py) – Glue code for processing user strokes, patching models, visualizing latents.

A. Client / Frontend
- Browser with Gradio interface (canvas powered by fabric.js).
B. API Layer / Controller
- gradio_run.py + magic_utils.py + model_patch.py: orchestrates calls, maintains session state.
C. BrushNet Module
- Processes brush strokes and original image → generates masks, edge maps, local guidance.
D. Prompt Assistant (LLaVA)
- Uses drawn strokes to guess “what user is drawing” → auto-fills prompt text.
E. Diffusion Pipeline (ComfyUI Wrappers)
- Stable-Diffusion models (sd1, sd2, sdxl, ControlNet, LoRA, etc.).
- Samplers (k_diffusion, extra_samplers), encoders, upscalers.
F. Model Storage and Checkpoints
- Local filesystem (/data/magicquill/models or models.zip unzip).
- External endpoints (HuggingFace, Modelscope).
G. Deployment Environment
- Docker container, Docker Compose orchestration.
- Shell scripts for local or CI/CD installs.

##### Relationships & Data Flow
1. User draws on canvas → strokes & settings packaged as JSON + image.
2. Gradio server receives input → Controller invokes:
a. BrushNet → mask/edge generation
b. LLaVA → textual prompt suggestion
c. ComfyUI pipeline → diffusion inference with mask & prompt
3. Generated image returned → UI updates canvas
4. User may accept (continue editing) or discard (revert to previous)

##### Architectural Patterns & Design Principles
- Layered MVC-style separation:
- View (Gradio UI)
- Controller/orchestration (gradio_run.py + utils)
- Model inference (BrushNet, ComfyUI, LLaVA)
- Plugin-based extensibility: ComfyUI custom nodes, BrushNet configs, new samplers.
- Containerized deployment: repeatable, isolated environment.
- Configuration-as-code: JSON model configs, pyproject.toml, requirements.txt.

##### Technologies & Frameworks
- Python 3.10, PyTorch, diffusers, k-diffusion
- Gradio (web UI), fabric.js (canvas)
- ComfyUI, ControlNet, LoRA, T5, CLIP (vision & text)
- LLaVA submodule for prompt assistance
- Docker, Docker Compose, Conda (env management)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top Actor
    User["User (Browser)"]:::external

    %% Frontend Layer
    GradioUI["Gradio UI\n(fabric.js canvas)"]:::frontend

    %% API Layer / Controller
    Controller["Controller\n(gradio_run.py, magic_utils.py,\nmodel_patch.py, pidi.py)"]:::api

    %% Model Inference Layer
    subgraph "Model Inference Services"
        BrushNetService["BrushNet Service"]:::model
        LLavaService["LLaVA Prompt Engine"]:::model
        ComfyUIService["ComfyUI Diffusion Pipeline"]:::model
    end

    %% Storage and External
    StorageLocal["Model Storage\n(/data/magicquill/models)"]:::storage
    ExternalModels["External Models\n(HuggingFace, Modelscope)"]:::external

    %% Deployment Environment
    subgraph "Deployment Environment"
        DockerfileNode["Dockerfile"]:::deploy
        ComposeNode["docker-compose.yaml"]:::deploy
        InstallScripts["docker/install-all.sh"]:::deploy
        LinuxScript["linux_setup.sh"]:::deploy
        WindowsScript["windows_setup.bat"]:::deploy
        CINode[".github/workflows"]:::deploy
    end

    %% Data Flow
    User -->|"draws strokes"| GradioUI
    GradioUI -->|"input JSON & image"| Controller
    Controller -->|"calls"| BrushNetService
    BrushNetService -->|"mask & edges"| Controller
    Controller -->|"prompt suggestion"| LLavaService
    LLavaService -->|"text prompt"| Controller
    Controller -->|"inference"| ComfyUIService
    ComfyUIService -->|"edited image"| Controller
    Controller -->|"return image"| GradioUI

    %% Storage interactions
    Controller -->|"load/save"| StorageLocal
    ComfyUIService -->|"load models"| StorageLocal
    ComfyUIService -->|"fetch models"| ExternalModels

    %% Deployment connections
    Controller -.-> DockerfileNode
    Controller -.-> ComposeNode
    DockerfileNode -.-> InstallScripts
    DockerfileNode -.-> LinuxScript
    DockerfileNode -.-> WindowsScript
    DockerfileNode -.-> CINode

    %% Click Events
    click GradioUI "https://github.com/ant-research/magicquill/blob/main/gradio_run.py"
    click Controller "https://github.com/ant-research/magicquill/blob/main/gradio_run.py"
    click Controller "https://github.com/ant-research/magicquill/blob/main/MagicQuill/magic_utils.py"
    click Controller "https://github.com/ant-research/magicquill/blob/main/MagicQuill/model_patch.py"
    click Controller "https://github.com/ant-research/magicquill/blob/main/MagicQuill/pidi.py"
    click BrushNetService "https://github.com/ant-research/magicquill/blob/main/MagicQuill/brushnet/brushnet.py"
    click BrushNetService "https://github.com/ant-research/magicquill/blob/main/MagicQuill/brushnet/brushnet_ca.py"
    click BrushNetService "https://github.com/ant-research/magicquill/blob/main/MagicQuill/brushnet/powerpaint_utils.py"
    click BrushNetService "https://github.com/ant-research/magicquill/blob/main/MagicQuill/brushnet/unet_2d_blocks.py"
    click BrushNetService "https://github.com/ant-research/magicquill/blob/main/MagicQuill/brushnet/unet_2d_condition.py"
    click BrushNetService "https://github.com/ant-research/magicquill/blob/main/MagicQuill/brushnet_nodes.py"
    click LLavaService "https://github.com/ant-research/magicquill/tree/main/MagicQuill/LLaVA/"
    click LLavaService "https://github.com/ant-research/magicquill/blob/main/llava_new.py"
    click ComfyUIService "https://github.com/ant-research/magicquill/tree/main/MagicQuill/comfy/"
    click ComfyUIService "https://github.com/ant-research/magicquill/blob/main/MagicQuill/comfyui_utils.py"
    click StorageLocal "https://github.com/ant-research/magicquill/blob/main/MagicQuill/folder_paths.py"
    click ComposeNode "https://github.com/ant-research/magicquill/blob/main/docker-compose.yaml"
    click DockerfileNode "https://github.com/ant-research/magicquill/tree/main/Dockerfile"
    click InstallScripts "https://github.com/ant-research/magicquill/blob/main/docker/install-all.sh"
    click LinuxScript "https://github.com/ant-research/magicquill/blob/main/linux_setup.sh"
    click WindowsScript "https://github.com/ant-research/magicquill/blob/main/windows_setup.bat"
    click CINode "https://github.com/ant-research/magicquill/tree/main/.github/workflows"

    %% Styles
    classDef frontend fill:#e3f2fd,stroke:#2196f3,color:#0d47a1;
    classDef api fill:#e8f5e9,stroke:#4caf50,color:#1b5e20;
    classDef model fill:#fff3e0,stroke:#fb8c00,color:#e65100;
    classDef storage fill:#eceff1,stroke:#78909c,color:#37474f;
    classDef deployment fill:#f3e5f5,stroke:#ab47bc,color:#4a148c;
    classDef deploy fill:#f3e5f5,stroke:#ab47bc,color:#4a148c;
    classDef external fill:#fafafa,stroke:#9e9e9e,color:#424242;
```
