How esraa-hassan0/paligemma works
This repository implements single-image PaliGemma inference for captioning, object detection, and instance segmentation. The main workflow is: a user supplies an image and task prompt, inference loads the checkpoint and tokenizer, the processor prepares image and text inputs, and the multimodal model generates tokens. Detection coordinates and segmentation tokens are represented in the output; segmentation masks can be decoded with the OID VQ-VAE and rendered over the image. The README documents the product flow; some inference call sites and rendering connections were not included in the excerpts.
Main entry points: inference.py Inference runner · utils.py Checkpoint loader · processing_paligemma.py Image-text processor · modeling_gemma.py Multimodal model
Components 13
Inference
- Inference runner
inference.py - Checkpoint loader
utils.py
Input preparation
- Image-text processor
processing_paligemma.py - Tokenizer
Vision-language model
- Multimodal model
modeling_gemma.py - SigLIP encoder
modeling_siglip.py - Token generation
modeling_gemma.py
Task outputs
- Task tokens
processing_paligemma.py - Boxes and overlays
inference.py - OID VQ-VAE decoder
variational_autoencoder.py - Generated result
inference.py
Other components
- User
- Model checkpoint
Connections 14
- User → Inference runner: runs with inputs
README.md - Inference runner → Checkpoint loader: loads model
inference.py - Checkpoint loader → Model checkpoint: reads weights
utils.py - Checkpoint loader → Tokenizer: loads tokenizer
utils.py - Inference runner → Image-text processor: prepares inputs
inference.py - Image-text processor → Tokenizer: tokenizes prompt
processing_paligemma.py - Inference runner → Multimodal model: runs generation
inference.py - Multimodal model → SigLIP encoder: encodes image
README.md - Multimodal model → Token generation: generates tokens
modeling_gemma.py - Token generation → Generated result: returns tokens
inference.py - Inference runner → OID VQ-VAE decoder: decodes mask tokens
inference.py - Image-text processor → Task tokens: adds task tokens
processing_paligemma.py - Inference runner → Boxes and overlays: renders detections
inference.py - Inference runner → Generated result: produces task output
README.md