How bidisha1005/prompt_controlled_image_editing works
1. What this repo is
- This is a Python-based RAG image editing system centered on InstructPix2Pix.
- The main design goal is to improve editing quality by combining prompt decomposition, retrieval from a curated example index, and stateful multi-step edit memory.
2. Primary runtime boundary
- The main online path is a FastAPI backend started from main.py.
- The backend serves both the API and the local UI, so the system is effectively a single-machine app with a browser front end and a Python inference service behind it.
3. Frontend layer
- files/frontend/index.html is the main single-page interface for uploading an image, entering an instruction, toggling RAG, and viewing results.
- saved.html and saved1 look like auxiliary/static result pages or experiments, not the main architecture.
4. API orchestration
- backend/server.py is the central request coordinator.
- The important endpoints are:
- /health for liveness
- /decompose to preview prompt splitting
- /edit for the full edit pipeline
- /reset to clear edit memory
- /state to inspect the current state
- Requests carry a base64 image, text instruction, guidance parameters, step count, and a use_rag switch.
5. Offline RAG preparation
- rag_pipeline/build_index.py is a one-time preprocessing job.
- It downloads the MagicBrush dataset from Hugging Face and builds a FAISS vector index over roughly 1200 samples.
- This creates the retrieval backbone used during inference; the online editor depends on this generated index being present.
6. Prompt decomposition subsystem
- rag_pipeline/prompt_decomposer.py splits complex instructions into atomic editing sub-tasks.
- It also tracks “preserve” constraints, which is important for multi-object edits where the model must not drift away from previously specified elements.
7. State memory and constraint propagation
- The prompt decomposition layer is not stateless: it maintains edit history, removed/added objects, and explicit preservation constraints.
- This state is injected into later edits through prompt_reconstructor.py, so each new instruction is interpreted in the context of prior edits.
8. Retrieval path
- The retrieval step is per sub-task, not just once per prompt.
- Each atomic edit is embedded, matched against the FAISS index, and used to pull relevant examples from MagicBrush.
- That retrieved context is then used to improve the prompt fed into the image editor.
9. Image editing core
- backend/editor.py is the main wrapper around InstructPix2Pix.
- It receives the prepared instruction plus the input image and returns the edited image.
- The critical data flow is: browser image + instruction → server orchestration → decomposed/reconstructed prompt + retrieval context → diffusion editor → output image.
10. Optional LLM-assisted reconstruction
- The README indicates optional Groq integration for prompt reconstruction via environment variables.
- That makes the prompt-building step an external LLM-assisted boundary, separate from the local diffusion inference path.
11. Evaluation and ablation
- utils/metrics.py provides CLIP-based evaluation utilities such as CLIPEvaluator, evaluate_edit, and run_ablation.
- This is the experimentation layer used to compare vanilla editing, decomposition-only editing, and full RAG-assisted editing.
12. Alternative backend track
- Files like main_replicate.py, backend/server_replicate.py, and backend/editor_replicate_flux_kontext.py indicate a parallel implementation path.
- Architecturally, this looks like a swap-in backend that likely offloads generation to an external hosted model service instead of the local InstructPix2Pix path.
13. Data and asset boundaries
- files/assets/music and music_library.json, plus modules like backend/music_selector.py, suggest an auxiliary media-selection feature or experiment.
- That asset set is separate from the main RAG image-editing pipeline and should be treated as a secondary subsystem unless the architecture diagram is specifically covering that extension.
14. What to draw in the architecture diagram
- The key boxes are: Browser UI, FastAPI orchestration, Prompt decomposition/state memory, FAISS retrieval index, Prompt reconstruction, Diffusion editor, and Evaluation utilities.
- The main boundary to emphasize is offline index building vs. online edit inference, with optional external LLM/replicate services as pluggable dependencies.
Main entry points: server.py API server · prompt_reconstructor.py Reconstructor · editor.py Pix2Pix editor · build_index.py Retriever
Components 17
Online edit path
Interactive browser-to-editor runtime
- Browser UI · Single-page interface for image upload, instruction entry, RAG toggle, and result viewing
files/frontend/index.html - App entry · Starts the FastAPI application and serves the app
files/main.py - API server · Central request coordinator exposing health, decompose, edit, reset, and state endpoints
files/backend/server.py - Decomposer · Breaks instructions into atomic edit steps and tracks preserve constraints
files/rag_pipeline/prompt_decomposer.py - Edit memory · Maintains edit history, added/removed objects, and preservation constraints across steps
files/backend/prompt_reconstructor.py - Reconstructor · Rebuilds the next-step prompt from decomposition state and prior edits, optionally with LLM help
files/backend/prompt_reconstructor.py - Retriever · Embeds each atomic sub-task and fetches relevant MagicBrush examples from the FAISS index
files/rag_pipeline/build_index.py - Pix2Pix editor · Runs InstructPix2Pix editing on the prepared instruction and input image
files/backend/editor.py - Result pages · Auxiliary static pages and experiments around saved outputs
files/frontend/saved.html - Groq LLM · Optional hosted model for prompt reconstruction and instruction refinement
Offline RAG prep
One-time index construction for retrieval
- Index build · Downloads MagicBrush data and builds the retrieval index
files/rag_pipeline/build_index.py - FAISS index · Curated example index used by retrieval during editing
files/rag_pipeline/build_index.py - MagicBrush data · Training/example corpus used to populate the retrieval index
files/rag_pipeline/build_index.py
Alternative backend
Swap-in hosted generation path
- Replicate path · Hosted-generation implementation that swaps in for the local editor path
files/backend/server_replicate.py - Flux editor · External model wrapper used by the Replicate backend track
files/backend/editor_replicate_flux_kontext.py - Alt entry · Alternative application entry point for the hosted generation path
files/main_replicate.py
Evaluation
Metrics and ablation utilities
- Metrics · CLIP-based evaluation and ablation helpers for comparing editing strategies
files/utils/metrics.py
Connections 20
- App entry → API server: starts · Launches the API service
- Browser UI → API server: calls · Sends edit, decomposition, reset, and state requests
- API server → Decomposer: decompose · Splits instructions into atomic edit steps
- API server → Edit memory: update state · Stores and reads edit history and constraints
- Edit memory → Reconstructor: feeds context · Provides prior edits and preservation constraints
- API server → Reconstructor: build prompt · Reconstructs the next prompt from state and current instruction
- Reconstructor → Groq LLM: optional assist · Can use hosted LLM support for reconstruction
- API server → Retriever: retrieve examples · Runs per-sub-task retrieval against the curated index
- Retriever → FAISS index: searches · Queries vector similarity over indexed examples
- FAISS index → MagicBrush data: built from · Derived from the MagicBrush dataset
- Retriever → Pix2Pix editor: context · Passes retrieved examples into the editing prompt
- Reconstructor → Pix2Pix editor: prepared prompt · Supplies the final instruction used for image editing
- API server → Pix2Pix editor: edit image · Sends base64 image plus guidance parameters to the diffusion editor
- Pix2Pix editor → API server: returns image · Produces the edited result for the client
- Index build → FAISS index: creates · Offline preprocessing job materializes the retrieval store
- API server → Replicate path: swap path · Alternative request flow for hosted generation
- Replicate path → Flux editor: invokes · Delegates generation to the hosted model wrapper
- Alt entry → Replicate path: starts · Launches the alternative backend application
- API server → Metrics: evaluate · Supports ablation and CLIP-based comparison workflows
- Browser UI → Result pages: aux pages · Static saved-result pages and related experiments