# bidisha1005/prompt_controlled_image_editing architecture

> Architecture diagram of the GitHub repository bidisha1005/prompt_controlled_image_editing, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bidisha1005/prompt_controlled_image_editing
- Repository: https://github.com/bidisha1005/prompt_controlled_image_editing

## Overview

1. **What this repo is**
   - This is a Python-based **RAG image editing system** centered on **InstructPix2Pix**.
   - The main design goal is to improve editing quality by combining **prompt decomposition**, **retrieval from a curated example index**, and **stateful multi-step edit memory**.

2. **Primary runtime boundary**
   - The main online path is a **FastAPI backend** started from `main.py`.
   - The backend serves both the API and the local UI, so the system is effectively a single-machine app with a browser front end and a Python inference service behind it.

3. **Frontend layer**
   - `files/frontend/index.html` is the main single-page interface for uploading an image, entering an instruction, toggling RAG, and viewing results.
   - `saved.html` and `saved1` look like auxiliary/static result pages or experiments, not the main architecture.

4. **API orchestration**
   - `backend/server.py` is the central request coordinator.
   - The important endpoints are:
     - `/health` for liveness
     - `/decompose` to preview prompt splitting
     - `/edit` for the full edit pipeline
     - `/reset` to clear edit memory
     - `/state` to inspect the current state
   - Requests carry a base64 image, text instruction, guidance parameters, step count, and a `use_rag` switch.

5. **Offline RAG preparation**
   - `rag_pipeline/build_index.py` is a **one-time preprocessing job**.
   - It downloads the **MagicBrush** dataset from Hugging Face and builds a **FAISS vector index** over roughly 1200 samples.
   - This creates the retrieval backbone used during inference; the online editor depends on this generated index being present.

6. **Prompt decomposition subsystem**
   - `rag_pipeline/prompt_decomposer.py` splits complex instructions into atomic editing sub-tasks.
   - It also tracks “preserve” constraints, which is important for multi-object edits where the model must not drift away from previously specified elements.

7. **State memory and constraint propagation**
   - The prompt decomposition layer is not stateless: it maintains edit history, removed/added objects, and explicit preservation constraints.
   - This state is injected into later edits through `prompt_reconstructor.py`, so each new instruction is interpreted in the context of prior edits.

8. **Retrieval path**
   - The retrieval step is **per sub-task**, not just once per prompt.
   - Each atomic edit is embedded, matched against the FAISS index, and used to pull relevant examples from MagicBrush.
   - That retrieved context is then used to improve the prompt fed into the image editor.

9. **Image editing core**
   - `backend/editor.py` is the main wrapper around **InstructPix2Pix**.
   - It receives the prepared instruction plus the input image and returns the edited image.
   - The critical data flow is: browser image + instruction → server orchestration → decomposed/reconstructed prompt + retrieval context → diffusion editor → output image.

10. **Optional LLM-assisted reconstruction**
   - The README indicates optional **Groq** integration for prompt reconstruction via environment variables.
   - That makes the prompt-building step an external LLM-assisted boundary, separate from the local diffusion inference path.

11. **Evaluation and ablation**
   - `utils/metrics.py` provides CLIP-based evaluation utilities such as `CLIPEvaluator`, `evaluate_edit`, and `run_ablation`.
   - This is the experimentation layer used to compare vanilla editing, decomposition-only editing, and full RAG-assisted editing.

12. **Alternative backend track**
   - Files like `main_replicate.py`, `backend/server_replicate.py`, and `backend/editor_replicate_flux_kontext.py` indicate a parallel implementation path.
   - Architecturally, this looks like a swap-in backend that likely offloads generation to an external hosted model service instead of the local InstructPix2Pix path.

13. **Data and asset boundaries**
   - `files/assets/music` and `music_library.json`, plus modules like `backend/music_selector.py`, suggest an auxiliary media-selection feature or experiment.
   - That asset set is separate from the main RAG image-editing pipeline and should be treated as a secondary subsystem unless the architecture diagram is specifically covering that extension.

14. **What to draw in the architecture diagram**
   - The key boxes are: **Browser UI**, **FastAPI orchestration**, **Prompt decomposition/state memory**, **FAISS retrieval index**, **Prompt reconstruction**, **Diffusion editor**, and **Evaluation utilities**.
   - The main boundary to emphasize is **offline index building vs. online edit inference**, with optional external LLM/replicate services as pluggable dependencies.

Main entry points: [`files/backend/server.py`](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/server.py) (API server), [`files/backend/prompt_reconstructor.py`](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/prompt_reconstructor.py) (Reconstructor), [`files/backend/editor.py`](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/editor.py) (Pix2Pix editor), [`files/rag_pipeline/build_index.py`](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/rag_pipeline/build_index.py) (Retriever)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_online["Online edit path"]
  node_browser_ui["Browser UI<br/>frontend spa<br/>[index.html]"]
  node_main_entry["App entry<br/>launcher<br/>[main.py]"]
  node_server["API server<br/>orchestrator<br/>[server.py]"]
  node_decomposer["Decomposer<br/>prompt split"]
  node_memory["Edit memory<br/>state store"]
  node_reconstructor["Reconstructor<br/>prompt build"]
  node_retrieval[("Retriever<br/>example search<br/>[build_index.py]")]
  node_editor["Pix2Pix editor<br/>diffusion wrapper<br/>[editor.py]"]
  node_frontend_assets["Result pages<br/>static pages<br/>[saved.html]"]
  node_groq_llm{{"Groq LLM<br/>external llm"}}
end

subgraph group_offline["Offline RAG prep"]
  node_index_build[("Index build<br/>offline job<br/>[build_index.py]")]
  node_faiss_index[("FAISS index<br/>vector store<br/>[build_index.py]")]
  node_magicbrush_data[("MagicBrush data<br/>dataset source<br/>[build_index.py]")]
end

subgraph group_alt["Alternative backend"]
  node_replicate_backend["Replicate path<br/>alternate backend"]
  node_replicate_editor["Flux editor<br/>hosted editor"]
  node_alt_entry["Alt entry<br/>launcher<br/>[main_replicate.py]"]
end

subgraph group_eval["Evaluation"]
  node_metrics["Metrics<br/>evaluation<br/>[metrics.py]"]
end

node_main_entry -->|"starts"| node_server
node_browser_ui -->|"calls"| node_server
node_server -->|"decompose"| node_decomposer
node_server -->|"update state"| node_memory
node_memory -->|"feeds context"| node_reconstructor
node_server -->|"build prompt"| node_reconstructor
node_reconstructor -.->|"optional assist"| node_groq_llm
node_server -->|"retrieve examples"| node_retrieval
node_retrieval -->|"searches"| node_faiss_index
node_faiss_index -->|"built from"| node_magicbrush_data
node_retrieval -->|"context"| node_editor
node_reconstructor -->|"prepared prompt"| node_editor
node_server -->|"edit image"| node_editor
node_editor -->|"returns image"| node_server
node_index_build -->|"creates"| node_faiss_index
node_server -.->|"swap path"| node_replicate_backend
node_replicate_backend -->|"invokes"| node_replicate_editor
node_alt_entry -->|"starts"| node_replicate_backend
node_server -.->|"evaluate"| node_metrics
node_browser_ui -.->|"aux pages"| node_frontend_assets

click node_browser_ui "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/frontend/index.html"
click node_main_entry "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/main.py"
click node_server "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/server.py"
click node_decomposer "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/rag_pipeline/prompt_decomposer.py"
click node_memory "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/prompt_reconstructor.py"
click node_reconstructor "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/prompt_reconstructor.py"
click node_retrieval "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/rag_pipeline/build_index.py"
click node_editor "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/editor.py"
click node_frontend_assets "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/frontend/saved.html"
click node_index_build "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/rag_pipeline/build_index.py"
click node_faiss_index "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/rag_pipeline/build_index.py"
click node_magicbrush_data "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/rag_pipeline/build_index.py"
click node_replicate_backend "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/server_replicate.py"
click node_replicate_editor "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/editor_replicate_flux_kontext.py"
click node_alt_entry "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/main_replicate.py"
click node_metrics "https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/utils/metrics.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_browser_ui,node_main_entry,node_server,node_decomposer,node_memory,node_reconstructor,node_retrieval,node_editor,node_frontend_assets,node_groq_llm toneBlue
class node_index_build,node_faiss_index,node_magicbrush_data toneAmber
class node_replicate_backend,node_replicate_editor,node_alt_entry toneMint
class node_metrics toneRose
```

## Components

### Online edit path

Interactive browser-to-editor runtime

- [Browser UI](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/frontend/index.html): frontend spa · Single-page interface for image upload, instruction entry, RAG toggle, and result viewing · `files/frontend/index.html`
- [App entry](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/main.py): launcher · Starts the FastAPI application and serves the app · `files/main.py`
- [API server](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/server.py): orchestrator · Central request coordinator exposing health, decompose, edit, reset, and state endpoints · `files/backend/server.py`
- [Decomposer](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/rag_pipeline/prompt_decomposer.py): prompt split · Breaks instructions into atomic edit steps and tracks preserve constraints · `files/rag_pipeline/prompt_decomposer.py`
- [Edit memory](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/prompt_reconstructor.py): state store · Maintains edit history, added/removed objects, and preservation constraints across steps · `files/backend/prompt_reconstructor.py`
- [Reconstructor](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/prompt_reconstructor.py): prompt build · Rebuilds the next-step prompt from decomposition state and prior edits, optionally with LLM help · `files/backend/prompt_reconstructor.py`
- [Retriever](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/rag_pipeline/build_index.py): example search · Embeds each atomic sub-task and fetches relevant MagicBrush examples from the FAISS index · `files/rag_pipeline/build_index.py`
- [Pix2Pix editor](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/editor.py): diffusion wrapper · Runs InstructPix2Pix editing on the prepared instruction and input image · `files/backend/editor.py`
- [Result pages](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/frontend/saved.html): static pages · Auxiliary static pages and experiments around saved outputs · `files/frontend/saved.html`
- **Groq LLM**: external llm · Optional hosted model for prompt reconstruction and instruction refinement

### Offline RAG prep

One-time index construction for retrieval

- [Index build](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/rag_pipeline/build_index.py): offline job · Downloads MagicBrush data and builds the retrieval index · `files/rag_pipeline/build_index.py`
- [FAISS index](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/rag_pipeline/build_index.py): vector store · Curated example index used by retrieval during editing · `files/rag_pipeline/build_index.py`
- [MagicBrush data](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/rag_pipeline/build_index.py): dataset source · Training/example corpus used to populate the retrieval index · `files/rag_pipeline/build_index.py`

### Alternative backend

Swap-in hosted generation path

- [Replicate path](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/server_replicate.py): alternate backend · Hosted-generation implementation that swaps in for the local editor path · `files/backend/server_replicate.py`
- [Flux editor](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/backend/editor_replicate_flux_kontext.py): hosted editor · External model wrapper used by the Replicate backend track · `files/backend/editor_replicate_flux_kontext.py`
- [Alt entry](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/main_replicate.py): launcher · Alternative application entry point for the hosted generation path · `files/main_replicate.py`

### Evaluation

Metrics and ablation utilities

- [Metrics](https://github.com/bidisha1005/prompt_controlled_image_editing/blob/main/files/utils/metrics.py): evaluation · CLIP-based evaluation and ablation helpers for comparing editing strategies · `files/utils/metrics.py`

## Connections

- App entry → API server: starts · Launches the API service
- Browser UI → API server: calls · Sends edit, decomposition, reset, and state requests
- API server → Decomposer: decompose · Splits instructions into atomic edit steps
- API server → Edit memory: update state · Stores and reads edit history and constraints
- Edit memory → Reconstructor: feeds context · Provides prior edits and preservation constraints
- API server → Reconstructor: build prompt · Reconstructs the next prompt from state and current instruction
- Reconstructor → Groq LLM: optional assist · Can use hosted LLM support for reconstruction
- API server → Retriever: retrieve examples · Runs per-sub-task retrieval against the curated index
- Retriever → FAISS index: searches · Queries vector similarity over indexed examples
- FAISS index → MagicBrush data: built from · Derived from the MagicBrush dataset
- Retriever → Pix2Pix editor: context · Passes retrieved examples into the editing prompt
- Reconstructor → Pix2Pix editor: prepared prompt · Supplies the final instruction used for image editing
- API server → Pix2Pix editor: edit image · Sends base64 image plus guidance parameters to the diffusion editor
- Pix2Pix editor → API server: returns image · Produces the edited result for the client
- Index build → FAISS index: creates · Offline preprocessing job materializes the retrieval store
- API server → Replicate path: swap path · Alternative request flow for hosted generation
- Replicate path → Flux editor: invokes · Delegates generation to the hosted model wrapper
- Alt entry → Replicate path: starts · Launches the alternative backend application
- API server → Metrics: evaluate · Supports ablation and CLIP-based comparison workflows
- Browser UI → Result pages: aux pages · Static saved-result pages and related experiments
