How river-zhang/icedit works
Project Type & Purpose
• This is an open‐source research codebase for “In-Context Edit,” a deep-learning image editing system built on a large diffusion transformer (Flux) with LoRA adapters.
• It covers two major subsystems:
a. Training pipeline (under train/) – data preparation, model implementation (Flux + LoRA), training loops, config management, logging.
b. Inference/Serving pipeline (under scripts/) – CLI inference, Gradio demo UI, model loading/offloading.
File Structure & Architectural Patterns
• Top‐level:
• scripts/ – inference (inference.py), demo (gradio_demo.py), static config (config.json), assets/ sample images.
• train/ – training code, data (parquet/), training scripts & configs, source code split into Flux core and training modules.
• requirements.txt + train/requirements.txt – separate dependency sets for inference vs. training.
• Patterns:
• Pipeline pattern for both training and inference.
• Config‐driven workflows: YAML/JSON configs controlling hyperparameters, model paths, CPU/GPU offload flags.
• Modular layering:
– Core model library (train/src/flux/…)
– Training logic (train/src/train/…)
– CLI/API entrypoints (scripts/…)
Key Components
a. Data Preparation & Storage
• Raw training data in train/parquet/
• Preparation script train/parquet/prepare.sh
• Artifact storage: train/runs/… (checkpoints, logs, config snapshots)
b. Model Training Subsystem
• Flux core modules (train/src/flux/) implement diffusion transformer blocks, pipeline tools, LoRA controller.
• Training modules (train/src/train/): data loaders, callbacks, model definition, training loop.
• Config & orchestration (train/train/config/*.yaml, train/train/script/train.sh).
c. Model Artifact Management
• Checkpoints & LoRA weights exported after training.
• Optionally uploaded to Hugging Face Hub (external service).
d. Inference & Serving Subsystem
• CLI inference script (scripts/inference.py), reads image + instruction, loads base Flux model + LoRA weights, runs diffusion, outputs edited image.
• Gradio UI (scripts/gradio_demo.py), provides web interface for live editing, supports CPU offload flags, custom weight paths, quantized GGUF models.
• Config file (scripts/config.json) for default model paths, ports.
e. External Services & Integrations
• Hugging Face model repository & Spaces (hosting demo)
• ComfyUI workflows (external Git repos/install via Git URL)
Relationships & Data Flows
• Training Flow:
[Raw Images + Edit Instructions] → Data Prep → Flux+LoRA Training → Checkpoints & LoRA Weights → Model Registry / Local FS
• Inference Flow:
[User Input (Image + Instruction)] → CLI/Gradio → Load Flux Base Model + LoRA Weights → Diffusion Transformer Inference → Edited Image Output
• Deployment Flow:
Local Gradio or Hugging Face Space serves the inference component.
Architectural Patterns & Design Principles
• Clear separation of concerns: data prep, model core, training loops, inference endpoints.
• Configurable pipelines: externalize parameters in YAML/JSON to enable reproducibility and environment‐specific overrides.
• Modular extensibility:
• New LoRA variants by dropping new checkpoint files.
• Extend inference frontends (ComfyUI, Gradio, CLI) without touching core model code.
Final Checklist
• [ ] Clear, self-contained labels for each module/directory.
• [ ] Callouts for key config files (e.g., .yaml, config.json).
• [ ] Annotation of major dependencies (PyTorch/Hugging Face, Parquet, Gradio).