How calligrapher2025/calligrapher works
Project type and purpose
• This is an open-source machine-learning inference tool for “freestyle text image customization” called Calligrapher.
• It combines pretrained diffusion models (FLUX.1-Fill-dev), an image encoder (SigLIP), and custom model weights (Calligrapher) into a Python-based CLI and Gradio UI for batch or interactive inpainting of text regions in images.
File structure
Top-level directories and files:
• README.md, env.yml, requirements.txt
• path_dict.json – centralizes all dataset and model paths
• pipeline_calligrapher.py – core inference pipeline
• models/ – modules defining attention, projection, transformer and Calligrapher models
• infer_*.py – CLI scripts for self-reference and cross-reference modes
• gradio_demo*.py – interactive web UI scripts (basic + upload-mask)
• utils.py – shared utilities
• samples/ – example inputs/outputs
• docs/ – landing page, static assets & teaser/framework images
• gradio_save_dir, cli_save_dir, data_dir, gradio_temp_dir – runtime artifacts (configured via path_dict.json)
README for architecture clues
• Describes three customization modes: self-reference, cross-reference, non-text reference
• Emphasizes two interfaces (CLI batch vs. Gradio interactive)
• Shows a training framework image (style injection → diffusion learning)
• Lists external dependencies: PyTorch, CUDA, Huggingface Hub downloads for FLUX.1-Fill-dev, SigLIP, Calligrapher
Main components
a. User Interfaces
• CLI Scripts (infer_calligrapher_self_custom.py, infer_calligrapher_cross_custom.py)
• Gradio Web UI (gradio_demo.py, gradio_demo_upload_mask.py)
b. Inference Pipeline
• pipeline_calligrapher.py (orchestrates preprocessing, model invocation, postprocessing)
• utils.py (common I/O, mask handling)
c. Model Layer
• Base Diffusion Model (FLUX.1-Fill-dev)
• Image Encoder (SigLIP)
• Calligrapher Custom Model (calligrapher.bin loaded via projection_models.py, transformer_flux_inpainting.py, attention_processor.py)
d. Storage & Data
• Local Data Dir (path_dict.json → data_dir for test dataset)
• Model Weights Repositories (Huggingface Hub)
• Output Directories (cli_save_dir, gradio_save_dir, gradio_temp_dir)
• samples/ for demonstration files
e. External Services & Dependencies
• Huggingface Hub API for snapshot_download
• PyTorch runtime (GPU/CUDA)
• Gradio server for web UI
Relationships & data flow
• User → (CLI or Gradio UI) → pipeline_calligrapher
• pipeline_calligrapher invokes:
1. Load paths from path_dict.json
2. Download or load pretrained weights & encoder
3. Preprocess input image, mask & reference
4. Forward through SigLIP encoder → style embeddings
5. Inpainting via FLUX diffusion + style injection (Calligrapher modules)
6. Postprocess & save result
• Outputs written to designated save directories
• Gradio UI also serves frontend assets (docs/static) and may hold temp mask files
Architectural patterns & design principles
• Modular layering: UI layer, pipeline orchestrator, model implementations, utilities
• Separation of concerns: CLI vs. UI vs. core logic
• Config-driven design via path_dict.json
• Extensibility: new modes or models plugged into existing pipeline
Essential details to capture
• Distinct CLI vs. Gradio paths into the same pipeline
• Path configuration driving dataset/model discovery
• Sequence: user input → mask generation → style encoding → diffusion inpainting → saved artifacts
• External model dependencies & download mechanism