How rahulsajnani/geodiffuser works
1. Repo shape
- This repository is centered on GeoDiffuser, a local, GPU-based image editing system.
- The root GeoDiffuser/ directory contains the main product code.
- The Evaluation/ directory is a collection of benchmark and baseline implementations used to compare GeoDiffuser against related editing methods.
2. Main entry point and runtime
- The primary user-facing entry point is geo_diff_ui.py, which launches the editing interface.
- large_scale_editor.py appears to support batch or higher-throughput editing workflows.
- The project is designed to run locally on CUDA GPUs; the README calls out a 20–24 GB VRAM requirement and a Conda environment.
3. Core editing pipeline
- The core flow is: input image + user edit intent → foreground segmentation → depth/geometry estimation → image inversion into diffusion space → geometry-aware diffusion editing → output visualization.
- This is consistent with the root utilities such as editor.py, inversion.py, depth_predictor.py, optimization.py, warp_utils.py, and softsplat.py.
4. Vision preprocessing boundary
- Object localization is handled by the vendored Segment Anything package under segment_anything/.
- Depth/shape estimation is supported by multiple vendored depth stacks: depth_anything/, dpt/, midas_repo/, and zoedepth/.
- This makes preprocessing a distinct subsystem that feeds geometric signals into the editor rather than directly modifying pixels.
5. Geometry-aware diffusion subsystem
- The utils/ package contains the diffusion-side editing logic: attention processors, attention sharing, diffusion utilities, optimization loops, and loss functions.
- ptp_utils.py and attention-related helpers suggest the model is manipulated at the attention layer level, matching the paper’s geometry-conditioned diffusion idea.
- warp_utils.py, eq_rotation.py, and softsplat.py provide the spatial transform and reprojection machinery used to move content geometrically.
6. Inversion and latent-space control
- The repository uses inversion utilities to map an input image into the latent/diffusion trajectory before editing.
- This is a critical boundary: the source image is not edited directly; it is first encoded/inverted, then the denoising process is steered to preserve identity/style while changing geometry.
7. Visualization and interaction
- ui_utils.py, vis_utils.py, image_processing.py, and pallete.py handle display, overlays, and edit visualization.
- The editing experience is therefore interactive and visual-first rather than a backend service architecture.
8. Model and dependency packaging
- Unlike a service-oriented repo, this one is mostly a self-contained research codebase with vendored third-party model code.
- Important external dependencies are pulled in as source trees rather than as opaque binaries, especially SAM, DPT/MiDaS, ZoeDepth, and related transformer backbones.
- envs/ defines the expected Python/CUDA environment.
9. Evaluation is isolated from the core system
- Evaluation/ is not one app; it is a set of independent reproduction subprojects.
- Each subfolder (DiffusionHandles, DragDiffusion, DragonDiffusion, FreeDrag, Lama, object-edit, zero123, and others) has its own README, environment files, scripts, and sometimes its own vendored model code.
- Architecturally, this is a “research workspace” boundary around multiple baselines rather than a shared runtime.
10. Evaluation pipelines and outputs
- The evaluation code includes scripts for generating results, running benchmark tasks, and computing metrics such as FID/IS and perceptual scores.
- Examples include Evaluation/scripts/fid_is/, Evaluation/GeoDiffuser/evaluation/, and the various run_eval*.py / predict*.py / test_*.py scripts inside baseline folders.
- Outputs are written to local directories such as outputs/, lama_outputs/, or per-experiment folders, with no central database or remote orchestration.
11. Notable infrastructure choices
- The repo relies on Python + PyTorch + CUDA as the execution stack.
- pytorch3d is explicitly required, which matters because geometry editing, mesh/rendering, or reprojection are first-class concerns.
- gradio appears in the UI-oriented files, indicating the interface is likely a lightweight local web UI rather than a deployed backend.
12. How to draw the architecture diagram
- At the center: the GeoDiffuser editing pipeline.
- On the left: user input / source image / edit controls.
- Above or beside it: segmentation + depth estimation as preprocessing services.
- In the middle: inversion + geometry-aware diffusion control.
- On the right: rendered edited image + visualization artifacts.
- Around the core, show separate evaluation/baseline subrepos as independent experimental branches, each with its own models, scripts, and environment.
Main entry points: editor.py Editor · diffusion.py Diffusion control · warp_utils.py Warping · geo_diff_ui.py UI entry
Components 22
GeoDiffuser core
Main local editing system for geometry-aware image diffusion.
- Editor · Coordinates the main edit flow from inputs to outputs.
GeoDiffuser/utils/editor.py - Inversion · Maps the source image into diffusion space before editing.
GeoDiffuser/utils/inversion.py - Diffusion control · Geometry-aware denoising and attention manipulation utilities.
GeoDiffuser/utils/diffusion.py - Optimization · Iterative guidance, loss computation, and edit refinement.
GeoDiffuser/utils/optimization.py - Warping · Spatial reprojection and flow-based content movement.
GeoDiffuser/utils/warp_utils.py - Softsplat · Soft splatting / resampling used in geometric alignment.
GeoDiffuser/utils/softsplat.py - Rotation · Auxiliary geometric transform helpers.
GeoDiffuser/utils/eq_rotation.py - Attention utils · Attention processors, sharing, and prompt-to-prompt style control.
GeoDiffuser/utils/attention_utils.py - PTP utils · Prompt-attention utilities that steer diffusion edits.
GeoDiffuser/utils/ptp_utils.py
Vision preprocessing
Foreground segmentation and depth estimation subsystems feeding geometry signals.
- SAM · Object localization and mask generation.
GeoDiffuser/segment_anything - Depth Anything · Monocular depth prediction stack.
GeoDiffuser/depth_anything - DPT/MiDaS · Alternative depth estimation implementation used by the repo.
GeoDiffuser/dpt - ZoeDepth · Additional depth model family and trainers.
GeoDiffuser/zoedepth - MiDaS repo · Vendored MiDaS depth stack and backbones.
GeoDiffuser/midas_repo
Interaction runtime
User-facing entry points and visualization helpers.
- UI entry · Launches the interactive editing interface.
geo_diff_ui.py - Batch editor · Higher-throughput or large-scale editing workflow.
large_scale_editor.py - Visuals · Image overlays, palette handling, and result visualization.
GeoDiffuser/utils/vis_utils.py - UI helpers · Local UI support for editor controls and display.
GeoDiffuser/utils/ui_utils.py
Evaluation workspace
Independent baseline and benchmark subprojects used for comparison.
- Eval suite · Repository-local evaluation scripts and study generation.
GeoDiffuser/evaluation - Baselines · Independent reproduction projects for related editing methods.
Evaluation - DragDiffusion · Drag-based diffusion editing baseline with its own UI and evals.
Evaluation/DragDiffusion - LaMa · Inpainting baseline with training, prediction, and metric tooling.
Evaluation/Lama
Connections 23
- UI entry → Editor: launches · Interactive UI invokes the main editor pipeline.
- Batch editor → Editor: runs · Batch workflow reuses the same editing core.
- Editor → SAM: masks · Foreground segmentation identifies editable regions.
- Editor → Depth Anything: depth · Depth estimates provide geometry priors.
- Editor → DPT/MiDaS: depth fallback · Alternative depth implementation feeds the pipeline.
- Editor → ZoeDepth: depth fallback · Additional depth stack can supply geometric signals.
- Editor → MiDaS repo: depth fallback · MiDaS-compatible depth code is vendored for preprocessing.
- Editor → Inversion: inverts · Source image is encoded into latent diffusion space.
- Editor → Diffusion control: edits · Geometry-aware denoising applies the edit intent.
- Editor → Optimization: refines · Optimization loops tune the edit outcome.
- Diffusion control → Attention utils: controls · Attention-level manipulation steers content preservation and change.
- Diffusion control → PTP utils: guides · Prompt-guided attention utilities shape diffusion behavior.
- Diffusion control → Warping: reprojects · Spatial transforms align geometry during editing.
- Warping → Softsplat: uses · Soft splatting implements differentiable resampling.
- Warping → Rotation: uses · Rotation helpers support pose/geometry transforms.
- Editor → Visuals: renders · Visualization layer assembles outputs and overlays.
- UI entry → UI helpers: uses · UI support code handles interaction and display.
- Visuals → UI helpers: feeds · Rendered artifacts are shown through UI helpers.
- Eval suite → Editor: evaluates · Study scripts and fidelity metrics assess GeoDiffuser.
- Baselines → DragDiffusion: contains · Evaluation workspace includes baseline subprojects.
- Baselines → LaMa: contains · Evaluation workspace includes inpainting baselines.
- DragDiffusion → UI entry: compares · Baseline UI/pipeline parallels the main interactive experience.
- LaMa → Eval suite: compares · Baseline outputs are scored against GeoDiffuser studies.