How hidream-ai/hidream-e1 works
Project Type and Purpose
• This is an open-source, instruction-based image editing tool built on top of a pretrained diffusion backbone (HiDream-I1). It provides CLI scripts (inference.py, instruction_refinement.py) and a Gradio demo (gradio_demo.py) to let end users and developers perform and refine image edits via natural-language instructions.
File-Structure Analysis
• Top level:
• inference.py – main CLI entry point for image editing inference.
• instruction_refinement.py – CLI for improving user instructions via a VLM or OpenAI API.
• pipeline_hidream_image_editing.py – defines the custom Diffusers pipeline that glues together tokenizer, text encoder, image transformer, and LoRA adapters.
• gradio_demo.py – launches an interactive web UI.
• assets/ – sample images and demo visuals.
• requirements.txt – Python dependencies.
• No explicit frontend/backend split—this is a library/tool with two user-facing entrypoints (CLI and web demo) and a core model pipeline.
Key Components
a. User Interfaces
• CLI Inference (inference.py)
• Instruction Refinement CLI (instruction_refinement.py)
• Gradio Web UI (gradio_demo.py)
b. Core Pipeline
• Instruction Refinement Module (optional; calls VLM or OpenAI API)
• HiDream Image Editing Pipeline (inherits Diffusers Pipeline):
• Diffusion backbone model (HiDream-I1 transformer)
• LoRA adapter layers (injected via peft)
• Image preprocessor / postprocessor (PIL-based resize, normalization)
c. External Services & Repositories
• HuggingFace Hub (model checkpoints for Llama and HiDream)
• VLM API or local vLLM (for instruction refinement)
d. Data Flow
1. User submits: source image + raw instruction
2. (Optional) Instruction Refinement Module returns refined instruction
3. Tokenizer & text encoder encode prompt
4. Pipeline drives diffusion steps: editing stage + optional refinement stage
5. Pipeline outputs an edited image tensor → PIL image → saved or returned to UI
Architectural Patterns & Design Principles
• Modular Pipeline Pattern: each stage (tokenization, text encoding, diffusion editing, refinement) is encapsulated.
• Adapter-based extensibility: LoRA adapters let you fine-tune or swap in new editing behaviors without retraining entire model.
• Dual-mode UI: reuse the same core pipeline for CLI and web demo, separating presentation from business logic.