# walbouss/gem architecture

> Architecture diagram of the GitHub repository walbouss/gem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/walbouss/gem
- Repository: https://github.com/walbouss/gem

## Overview

GEM is a training-free adaptation of pretrained CLIP-style vision-language models for open-vocabulary image localization and segmentation. The principal workflow preprocesses an image, compares its visual patch features with text prompts, and returns per-prompt heatmaps; callers can visualize these or derive segmentation maps. Model creation loads a pretrained OpenCLIP model and wraps its visual transformer with GEM attention. The README also documents batched inference and a Gradio demo. The graph reflects the README and sampled source; dataset-level segmentation postprocessing is documented as caller-side work, not a separate library subsystem.

Main entry points: [`gem/gem_wrapper.py`](https://github.com/walbouss/gem/blob/main/gem/gem_wrapper.py) (Single-image inference), [`gem/gem.py`](https://github.com/walbouss/gem/blob/main/gem/gem.py) (Model factory), [`app.py`](https://github.com/walbouss/gem/blob/main/app.py) (Gradio demo), [`gem/gem_utils.py`](https://github.com/walbouss/gem/blob/main/gem/gem_utils.py) (GEM attention)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Public API"]
  node_gem_api["Model and utilities API<br/>[gem.py]"]
  node_image_transform["Image preprocessing<br/>[gem.py]"]
end

subgraph group_model["GEM Model"]
  node_model_factory["Model factory<br/>[gem.py]"]
  node_open_clip["OpenCLIP model loader"]
  node_wrapper["GEM wrapper<br/>[gem_wrapper.py]"]
  node_attention["GEM attention<br/>[gem_utils.py]"]
  node_vision_transformer["Vision transformer"]
  node_tokenizer["Text tokenizer"]
end

subgraph group_inference["Inference Results"]
  node_single_inference["Single-image inference<br/>[gem_wrapper.py]"]
  node_batch_inference["Batched inference<br/>[gem_wrapper.py]"]
  node_heatmaps["Prompt heatmaps<br/>[gem_wrapper.py]"]
  node_visualization["Heatmap visualization<br/>[gem.py]"]
  node_segmentation["Segmentation postprocessing"]
end

subgraph group_demo["Interactive Demo"]
  node_demo_app["Gradio demo<br/>[app.py]"]
end

node_user(("Library caller"))
node_images["Images and prompts"]
node_demo_user(("Demo user"))

node_user -->|"calls"| node_gem_api
node_images -->|"preprocesses"| node_image_transform
node_user -->|"requests transform"| node_image_transform
node_gem_api -->|"creates"| node_image_transform
node_user -->|"requests model"| node_model_factory
node_gem_api -->|"exposes"| node_model_factory
node_model_factory -->|"loads model"| node_open_clip
node_model_factory -->|"gets tokenizer"| node_tokenizer
node_model_factory -->|"wraps model"| node_wrapper
node_wrapper -->|"installs attention"| node_attention
node_wrapper -->|"modifies forward"| node_vision_transformer
node_user -->|"calls"| node_single_inference
node_user -->|"calls"| node_batch_inference
node_images -->|"provides inputs"| node_single_inference
node_images -->|"provides batches"| node_batch_inference
node_wrapper -->|"runs"| node_single_inference
node_wrapper -->|"runs"| node_batch_inference
node_single_inference -->|"encodes image"| node_vision_transformer
node_batch_inference -->|"encodes images"| node_vision_transformer
node_single_inference -->|"tokenizes prompts"| node_tokenizer
node_batch_inference -->|"tokenizes prompts"| node_tokenizer
node_single_inference -->|"returns logits"| node_heatmaps
node_batch_inference -->|"returns logits"| node_heatmaps
node_user -->|"requests plots"| node_visualization
node_gem_api -->|"exposes"| node_visualization
node_heatmaps -->|"supplies logits"| node_visualization
node_heatmaps -.->|"supports prediction"| node_segmentation
node_demo_user -->|"submits inputs"| node_demo_app
node_demo_app -->|"loads weights"| node_model_factory
node_demo_app -->|"preprocesses image"| node_image_transform
node_demo_app -->|"runs inference"| node_single_inference
node_demo_app -->|"shows heatmap"| node_demo_user

click node_gem_api "https://github.com/walbouss/gem/blob/main/gem/gem.py"
click node_image_transform "https://github.com/walbouss/gem/blob/main/gem/gem.py"
click node_model_factory "https://github.com/walbouss/gem/blob/main/gem/gem.py"
click node_wrapper "https://github.com/walbouss/gem/blob/main/gem/gem_wrapper.py"
click node_attention "https://github.com/walbouss/gem/blob/main/gem/gem_utils.py"
click node_single_inference "https://github.com/walbouss/gem/blob/main/gem/gem_wrapper.py"
click node_batch_inference "https://github.com/walbouss/gem/blob/main/gem/gem_wrapper.py"
click node_heatmaps "https://github.com/walbouss/gem/blob/main/gem/gem_wrapper.py"
click node_visualization "https://github.com/walbouss/gem/blob/main/gem/gem.py"
click node_demo_app "https://github.com/walbouss/gem/blob/main/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gem_api,node_image_transform,node_demo_user toneBlue
class node_model_factory,node_open_clip,node_wrapper,node_attention,node_vision_transformer,node_tokenizer toneAmber
class node_single_inference,node_batch_inference,node_heatmaps,node_visualization,node_segmentation toneMint
class node_demo_app toneRose
class node_user,node_images toneIndigo
```

## Components

### Public API

- [Model and utilities API](https://github.com/walbouss/gem/blob/main/gem/gem.py): `gem/gem.py`
- [Image preprocessing](https://github.com/walbouss/gem/blob/main/gem/gem.py): `gem/gem.py`

### GEM Model

- [Model factory](https://github.com/walbouss/gem/blob/main/gem/gem.py): `gem/gem.py`
- **OpenCLIP model loader**
- [GEM wrapper](https://github.com/walbouss/gem/blob/main/gem/gem_wrapper.py): `gem/gem_wrapper.py`
- [GEM attention](https://github.com/walbouss/gem/blob/main/gem/gem_utils.py): `gem/gem_utils.py`
- **Vision transformer**
- **Text tokenizer**

### Inference Results

- [Single-image inference](https://github.com/walbouss/gem/blob/main/gem/gem_wrapper.py): `gem/gem_wrapper.py`
- [Batched inference](https://github.com/walbouss/gem/blob/main/gem/gem_wrapper.py): `gem/gem_wrapper.py`
- [Prompt heatmaps](https://github.com/walbouss/gem/blob/main/gem/gem_wrapper.py): `gem/gem_wrapper.py`
- [Heatmap visualization](https://github.com/walbouss/gem/blob/main/gem/gem.py): `gem/gem.py`
- **Segmentation postprocessing**

### Interactive Demo

- [Gradio demo](https://github.com/walbouss/gem/blob/main/app.py): `app.py`

### Other components

- **Library caller**
- **Images and prompts**
- **Demo user**

## Connections

- Library caller → Model and utilities API: calls
- Images and prompts → Image preprocessing: preprocesses
- Library caller → Image preprocessing: requests transform
- Model and utilities API → Image preprocessing: creates
- Library caller → Model factory: requests model
- Model and utilities API → Model factory: exposes
- Model factory → OpenCLIP model loader: loads model
- Model factory → Text tokenizer: gets tokenizer
- Model factory → GEM wrapper: wraps model
- GEM wrapper → GEM attention: installs attention
- GEM wrapper → Vision transformer: modifies forward
- Library caller → Single-image inference: calls
- Library caller → Batched inference: calls
- Images and prompts → Single-image inference: provides inputs
- Images and prompts → Batched inference: provides batches
- GEM wrapper → Single-image inference: runs
- GEM wrapper → Batched inference: runs
- Single-image inference → Vision transformer: encodes image
- Batched inference → Vision transformer: encodes images
- Single-image inference → Text tokenizer: tokenizes prompts
- Batched inference → Text tokenizer: tokenizes prompts
- Single-image inference → Prompt heatmaps: returns logits
- Batched inference → Prompt heatmaps: returns logits
- Library caller → Heatmap visualization: requests plots
- Model and utilities API → Heatmap visualization: exposes
- Prompt heatmaps → Heatmap visualization: supplies logits
- Prompt heatmaps → Segmentation postprocessing: supports prediction
- Demo user → Gradio demo: submits inputs
- Gradio demo → Model factory: loads weights
- Gradio demo → Image preprocessing: preprocesses image
- Gradio demo → Single-image inference: runs inference
- Gradio demo → Demo user: shows heatmap