How walbouss/gem works
GEM is a training-free adaptation of pretrained CLIP-style vision-language models for open-vocabulary image localization and segmentation. The principal workflow preprocesses an image, compares its visual patch features with text prompts, and returns per-prompt heatmaps; callers can visualize these or derive segmentation maps. Model creation loads a pretrained OpenCLIP model and wraps its visual transformer with GEM attention. The README also documents batched inference and a Gradio demo. The graph reflects the README and sampled source; dataset-level segmentation postprocessing is documented as caller-side work, not a separate library subsystem.
Main entry points: gem_wrapper.py Single-image inference · gem.py Model factory · app.py Gradio demo · gem_utils.py GEM attention
Components 17
Public API
- Model and utilities API
gem/gem.py - Image preprocessing
gem/gem.py
GEM Model
- Model factory
gem/gem.py - OpenCLIP model loader
- GEM wrapper
gem/gem_wrapper.py - GEM attention
gem/gem_utils.py - Vision transformer
- Text tokenizer
Inference Results
- Single-image inference
gem/gem_wrapper.py - Batched inference
gem/gem_wrapper.py - Prompt heatmaps
gem/gem_wrapper.py - Heatmap visualization
gem/gem.py - Segmentation postprocessing
Interactive Demo
- Gradio demo
app.py
Other components
- Library caller
- Images and prompts
- Demo user
Connections 32
- Library caller → Model and utilities API: calls
- Images and prompts → Image preprocessing: preprocesses
- Library caller → Image preprocessing: requests transform
- Model and utilities API → Image preprocessing: creates
- Library caller → Model factory: requests model
- Model and utilities API → Model factory: exposes
- Model factory → OpenCLIP model loader: loads model
- Model factory → Text tokenizer: gets tokenizer
- Model factory → GEM wrapper: wraps model
- GEM wrapper → GEM attention: installs attention
- GEM wrapper → Vision transformer: modifies forward
- Library caller → Single-image inference: calls
- Library caller → Batched inference: calls
- Images and prompts → Single-image inference: provides inputs
- Images and prompts → Batched inference: provides batches
- GEM wrapper → Single-image inference: runs
- GEM wrapper → Batched inference: runs
- Single-image inference → Vision transformer: encodes image
- Batched inference → Vision transformer: encodes images
- Single-image inference → Text tokenizer: tokenizes prompts
- Batched inference → Text tokenizer: tokenizes prompts
- Single-image inference → Prompt heatmaps: returns logits
- Batched inference → Prompt heatmaps: returns logits
- Library caller → Heatmap visualization: requests plots
- Model and utilities API → Heatmap visualization: exposes
- Prompt heatmaps → Heatmap visualization: supplies logits
- Prompt heatmaps → Segmentation postprocessing: supports prediction
- Demo user → Gradio demo: submits inputs
- Gradio demo → Model factory: loads weights
- Gradio demo → Image preprocessing: preprocesses image
- Gradio demo → Single-image inference: runs inference
- Gradio demo → Demo user: shows heatmap