# rahulsajnani/geodiffuser architecture

> Architecture diagram of the GitHub repository rahulsajnani/geodiffuser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rahulsajnani/geodiffuser
- Repository: https://github.com/rahulsajnani/geodiffuser

## Overview

1. **Repo shape**
   - This repository is centered on **GeoDiffuser**, a local, GPU-based image editing system.
   - The root `GeoDiffuser/` directory contains the main product code.
   - The `Evaluation/` directory is a collection of **benchmark and baseline implementations** used to compare GeoDiffuser against related editing methods.

2. **Main entry point and runtime**
   - The primary user-facing entry point is `geo_diff_ui.py`, which launches the editing interface.
   - `large_scale_editor.py` appears to support batch or higher-throughput editing workflows.
   - The project is designed to run locally on CUDA GPUs; the README calls out a **20–24 GB VRAM** requirement and a Conda environment.

3. **Core editing pipeline**
   - The core flow is: **input image + user edit intent → foreground segmentation → depth/geometry estimation → image inversion into diffusion space → geometry-aware diffusion editing → output visualization**.
   - This is consistent with the root utilities such as `editor.py`, `inversion.py`, `depth_predictor.py`, `optimization.py`, `warp_utils.py`, and `softsplat.py`.

4. **Vision preprocessing boundary**
   - Object localization is handled by the vendored **Segment Anything** package under `segment_anything/`.
   - Depth/shape estimation is supported by multiple vendored depth stacks: `depth_anything/`, `dpt/`, `midas_repo/`, and `zoedepth/`.
   - This makes preprocessing a distinct subsystem that feeds geometric signals into the editor rather than directly modifying pixels.

5. **Geometry-aware diffusion subsystem**
   - The `utils/` package contains the diffusion-side editing logic: attention processors, attention sharing, diffusion utilities, optimization loops, and loss functions.
   - `ptp_utils.py` and attention-related helpers suggest the model is manipulated at the **attention layer level**, matching the paper’s geometry-conditioned diffusion idea.
   - `warp_utils.py`, `eq_rotation.py`, and `softsplat.py` provide the spatial transform and reprojection machinery used to move content geometrically.

6. **Inversion and latent-space control**
   - The repository uses inversion utilities to map an input image into the latent/diffusion trajectory before editing.
   - This is a critical boundary: the source image is not edited directly; it is first encoded/inverted, then the denoising process is steered to preserve identity/style while changing geometry.

7. **Visualization and interaction**
   - `ui_utils.py`, `vis_utils.py`, `image_processing.py`, and `pallete.py` handle display, overlays, and edit visualization.
   - The editing experience is therefore interactive and visual-first rather than a backend service architecture.

8. **Model and dependency packaging**
   - Unlike a service-oriented repo, this one is mostly a **self-contained research codebase** with vendored third-party model code.
   - Important external dependencies are pulled in as source trees rather than as opaque binaries, especially SAM, DPT/MiDaS, ZoeDepth, and related transformer backbones.
   - `envs/` defines the expected Python/CUDA environment.

9. **Evaluation is isolated from the core system**
   - `Evaluation/` is not one app; it is a set of **independent reproduction subprojects**.
   - Each subfolder (`DiffusionHandles`, `DragDiffusion`, `DragonDiffusion`, `FreeDrag`, `Lama`, `object-edit`, `zero123`, and others) has its own README, environment files, scripts, and sometimes its own vendored model code.
   - Architecturally, this is a “research workspace” boundary around multiple baselines rather than a shared runtime.

10. **Evaluation pipelines and outputs**
    - The evaluation code includes scripts for generating results, running benchmark tasks, and computing metrics such as FID/IS and perceptual scores.
    - Examples include `Evaluation/scripts/fid_is/`, `Evaluation/GeoDiffuser/evaluation/`, and the various `run_eval*.py` / `predict*.py` / `test_*.py` scripts inside baseline folders.
    - Outputs are written to local directories such as `outputs/`, `lama_outputs/`, or per-experiment folders, with no central database or remote orchestration.

11. **Notable infrastructure choices**
    - The repo relies on **Python + PyTorch + CUDA** as the execution stack.
    - `pytorch3d` is explicitly required, which matters because geometry editing, mesh/rendering, or reprojection are first-class concerns.
    - `gradio` appears in the UI-oriented files, indicating the interface is likely a lightweight local web UI rather than a deployed backend.

12. **How to draw the architecture diagram**
    - At the center: the **GeoDiffuser editing pipeline**.
    - On the left: **user input / source image / edit controls**.
    - Above or beside it: **segmentation + depth estimation** as preprocessing services.
    - In the middle: **inversion + geometry-aware diffusion control**.
    - On the right: **rendered edited image + visualization artifacts**.
    - Around the core, show separate **evaluation/baseline subrepos** as independent experimental branches, each with its own models, scripts, and environment.

Main entry points: [`GeoDiffuser/utils/editor.py`](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/editor.py) (Editor), [`GeoDiffuser/utils/diffusion.py`](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/diffusion.py) (Diffusion control), [`GeoDiffuser/utils/warp_utils.py`](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/warp_utils.py) (Warping), [`geo_diff_ui.py`](https://github.com/rahulsajnani/geodiffuser/blob/main/geo_diff_ui.py) (UI entry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["GeoDiffuser core"]
  node_editor["Editor<br/>pipeline orchestrator<br/>[editor.py]"]
  node_inversion["Inversion<br/>latent control<br/>[inversion.py]"]
  node_diffusion["Diffusion control<br/>attention editing<br/>[diffusion.py]"]
  node_opt["Optimization<br/>loss loop<br/>[optimization.py]"]
  node_warp["Warping<br/>geometry transform<br/>[warp_utils.py]"]
  node_softsplat["Softsplat<br/>reprojection op<br/>[softsplat.py]"]
  node_rotation["Rotation<br/>pose warp<br/>[eq_rotation.py]"]
  node_attn["Attention utils<br/>attention layer control<br/>[attention_utils.py]"]
  node_ptp["PTP utils<br/>prompt guidance<br/>[ptp_utils.py]"]
end

subgraph group_vision["Vision preprocessing"]
  node_seg["SAM<br/>foreground segmentation"]
  node_depth_any["Depth Anything<br/>depth estimator"]
  node_dpt["DPT/MiDaS<br/>depth backbone"]
  node_zoe["ZoeDepth<br/>depth estimator"]
  node_midas["MiDaS repo<br/>depth backbone"]
end

subgraph group_runtime["Interaction runtime"]
  node_geo_ui["UI entry<br/>gradio app<br/>[geo_diff_ui.py]"]
  node_batch_editor["Batch editor<br/>edit runner"]
  node_vis["Visuals<br/>overlay/render<br/>[vis_utils.py]"]
  node_ui_utils["UI helpers<br/>interaction helpers<br/>[ui_utils.py]"]
end

subgraph group_eval["Evaluation workspace"]
  node_eval_root["Eval suite<br/>benchmark workspace"]
  node_eval_baselines["Baselines<br/>research subrepos"]
  node_dragdiff["DragDiffusion<br/>baseline subproject"]
  node_lama["LaMa<br/>inpainting baseline"]
end

node_geo_ui -->|"launches"| node_editor
node_batch_editor -->|"runs"| node_editor
node_editor -->|"masks"| node_seg
node_editor -->|"depth"| node_depth_any
node_editor -.->|"depth fallback"| node_dpt
node_editor -.->|"depth fallback"| node_zoe
node_editor -.->|"depth fallback"| node_midas
node_editor -->|"inverts"| node_inversion
node_editor -->|"edits"| node_diffusion
node_editor -->|"refines"| node_opt
node_diffusion -->|"controls"| node_attn
node_diffusion -->|"guides"| node_ptp
node_diffusion -->|"reprojects"| node_warp
node_warp -->|"uses"| node_softsplat
node_warp -->|"uses"| node_rotation
node_editor -->|"renders"| node_vis
node_geo_ui -->|"uses"| node_ui_utils
node_vis -->|"feeds"| node_ui_utils
node_eval_root -->|"evaluates"| node_editor
node_eval_baselines -->|"contains"| node_dragdiff
node_eval_baselines -->|"contains"| node_lama
node_dragdiff -.->|"compares"| node_geo_ui
node_lama -.->|"compares"| node_eval_root

click node_geo_ui "https://github.com/rahulsajnani/geodiffuser/blob/main/geo_diff_ui.py"
click node_batch_editor "https://github.com/rahulsajnani/geodiffuser/blob/main/large_scale_editor.py"
click node_editor "https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/editor.py"
click node_inversion "https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/inversion.py"
click node_diffusion "https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/diffusion.py"
click node_opt "https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/optimization.py"
click node_warp "https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/warp_utils.py"
click node_softsplat "https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/softsplat.py"
click node_rotation "https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/eq_rotation.py"
click node_attn "https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/attention_utils.py"
click node_ptp "https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/ptp_utils.py"
click node_seg "https://github.com/rahulsajnani/geodiffuser/tree/main/GeoDiffuser/segment_anything"
click node_depth_any "https://github.com/rahulsajnani/geodiffuser/tree/main/GeoDiffuser/depth_anything"
click node_dpt "https://github.com/rahulsajnani/geodiffuser/tree/main/GeoDiffuser/dpt"
click node_zoe "https://github.com/rahulsajnani/geodiffuser/tree/main/GeoDiffuser/zoedepth"
click node_midas "https://github.com/rahulsajnani/geodiffuser/tree/main/GeoDiffuser/midas_repo"
click node_vis "https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/vis_utils.py"
click node_ui_utils "https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/ui_utils.py"
click node_eval_root "https://github.com/rahulsajnani/geodiffuser/tree/main/GeoDiffuser/evaluation"
click node_eval_baselines "https://github.com/rahulsajnani/geodiffuser/tree/main/Evaluation"
click node_dragdiff "https://github.com/rahulsajnani/geodiffuser/tree/main/Evaluation/DragDiffusion"
click node_lama "https://github.com/rahulsajnani/geodiffuser/tree/main/Evaluation/Lama"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_editor,node_inversion,node_diffusion,node_opt,node_warp,node_softsplat,node_rotation,node_attn,node_ptp toneBlue
class node_seg,node_depth_any,node_dpt,node_zoe,node_midas toneAmber
class node_geo_ui,node_batch_editor,node_vis,node_ui_utils toneMint
class node_eval_root,node_eval_baselines,node_dragdiff,node_lama toneRose
```

## Components

### GeoDiffuser core

Main local editing system for geometry-aware image diffusion.

- [Editor](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/editor.py): pipeline orchestrator · Coordinates the main edit flow from inputs to outputs. · `GeoDiffuser/utils/editor.py`
- [Inversion](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/inversion.py): latent control · Maps the source image into diffusion space before editing. · `GeoDiffuser/utils/inversion.py`
- [Diffusion control](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/diffusion.py): attention editing · Geometry-aware denoising and attention manipulation utilities. · `GeoDiffuser/utils/diffusion.py`
- [Optimization](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/optimization.py): loss loop · Iterative guidance, loss computation, and edit refinement. · `GeoDiffuser/utils/optimization.py`
- [Warping](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/warp_utils.py): geometry transform · Spatial reprojection and flow-based content movement. · `GeoDiffuser/utils/warp_utils.py`
- [Softsplat](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/softsplat.py): reprojection op · Soft splatting / resampling used in geometric alignment. · `GeoDiffuser/utils/softsplat.py`
- [Rotation](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/eq_rotation.py): pose warp · Auxiliary geometric transform helpers. · `GeoDiffuser/utils/eq_rotation.py`
- [Attention utils](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/attention_utils.py): attention layer control · Attention processors, sharing, and prompt-to-prompt style control. · `GeoDiffuser/utils/attention_utils.py`
- [PTP utils](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/ptp_utils.py): prompt guidance · Prompt-attention utilities that steer diffusion edits. · `GeoDiffuser/utils/ptp_utils.py`

### Vision preprocessing

Foreground segmentation and depth estimation subsystems feeding geometry signals.

- [SAM](https://github.com/rahulsajnani/geodiffuser/tree/main/GeoDiffuser/segment_anything): foreground segmentation · Object localization and mask generation. · `GeoDiffuser/segment_anything`
- [Depth Anything](https://github.com/rahulsajnani/geodiffuser/tree/main/GeoDiffuser/depth_anything): depth estimator · Monocular depth prediction stack. · `GeoDiffuser/depth_anything`
- [DPT/MiDaS](https://github.com/rahulsajnani/geodiffuser/tree/main/GeoDiffuser/dpt): depth backbone · Alternative depth estimation implementation used by the repo. · `GeoDiffuser/dpt`
- [ZoeDepth](https://github.com/rahulsajnani/geodiffuser/tree/main/GeoDiffuser/zoedepth): depth estimator · Additional depth model family and trainers. · `GeoDiffuser/zoedepth`
- [MiDaS repo](https://github.com/rahulsajnani/geodiffuser/tree/main/GeoDiffuser/midas_repo): depth backbone · Vendored MiDaS depth stack and backbones. · `GeoDiffuser/midas_repo`

### Interaction runtime

User-facing entry points and visualization helpers.

- [UI entry](https://github.com/rahulsajnani/geodiffuser/blob/main/geo_diff_ui.py): gradio app · Launches the interactive editing interface. · `geo_diff_ui.py`
- [Batch editor](https://github.com/rahulsajnani/geodiffuser/blob/main/large_scale_editor.py): edit runner · Higher-throughput or large-scale editing workflow. · `large_scale_editor.py`
- [Visuals](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/vis_utils.py): overlay/render · Image overlays, palette handling, and result visualization. · `GeoDiffuser/utils/vis_utils.py`
- [UI helpers](https://github.com/rahulsajnani/geodiffuser/blob/main/GeoDiffuser/utils/ui_utils.py): interaction helpers · Local UI support for editor controls and display. · `GeoDiffuser/utils/ui_utils.py`

### Evaluation workspace

Independent baseline and benchmark subprojects used for comparison.

- [Eval suite](https://github.com/rahulsajnani/geodiffuser/tree/main/GeoDiffuser/evaluation): benchmark workspace · Repository-local evaluation scripts and study generation. · `GeoDiffuser/evaluation`
- [Baselines](https://github.com/rahulsajnani/geodiffuser/tree/main/Evaluation): research subrepos · Independent reproduction projects for related editing methods. · `Evaluation`
- [DragDiffusion](https://github.com/rahulsajnani/geodiffuser/tree/main/Evaluation/DragDiffusion): baseline subproject · Drag-based diffusion editing baseline with its own UI and evals. · `Evaluation/DragDiffusion`
- [LaMa](https://github.com/rahulsajnani/geodiffuser/tree/main/Evaluation/Lama): inpainting baseline · Inpainting baseline with training, prediction, and metric tooling. · `Evaluation/Lama`

## Connections

- UI entry → Editor: launches · Interactive UI invokes the main editor pipeline.
- Batch editor → Editor: runs · Batch workflow reuses the same editing core.
- Editor → SAM: masks · Foreground segmentation identifies editable regions.
- Editor → Depth Anything: depth · Depth estimates provide geometry priors.
- Editor → DPT/MiDaS: depth fallback · Alternative depth implementation feeds the pipeline.
- Editor → ZoeDepth: depth fallback · Additional depth stack can supply geometric signals.
- Editor → MiDaS repo: depth fallback · MiDaS-compatible depth code is vendored for preprocessing.
- Editor → Inversion: inverts · Source image is encoded into latent diffusion space.
- Editor → Diffusion control: edits · Geometry-aware denoising applies the edit intent.
- Editor → Optimization: refines · Optimization loops tune the edit outcome.
- Diffusion control → Attention utils: controls · Attention-level manipulation steers content preservation and change.
- Diffusion control → PTP utils: guides · Prompt-guided attention utilities shape diffusion behavior.
- Diffusion control → Warping: reprojects · Spatial transforms align geometry during editing.
- Warping → Softsplat: uses · Soft splatting implements differentiable resampling.
- Warping → Rotation: uses · Rotation helpers support pose/geometry transforms.
- Editor → Visuals: renders · Visualization layer assembles outputs and overlays.
- UI entry → UI helpers: uses · UI support code handles interaction and display.
- Visuals → UI helpers: feeds · Rendered artifacts are shown through UI helpers.
- Eval suite → Editor: evaluates · Study scripts and fidelity metrics assess GeoDiffuser.
- Baselines → DragDiffusion: contains · Evaluation workspace includes baseline subprojects.
- Baselines → LaMa: contains · Evaluation workspace includes inpainting baselines.
- DragDiffusion → UI entry: compares · Baseline UI/pipeline parallels the main interactive experience.
- LaMa → Eval suite: compares · Baseline outputs are scored against GeoDiffuser studies.
