How sarapieri/panorama_grounding works
PANORAMA is a vision-language model for panoptic grounded captioning: it generates scene text with phrase markers, then uses each phrase representation to condition mask proposals and select phrase-aligned regions. The repository also provides training datasets and evaluation paths for PanoCaps, referring-expression segmentation, and other grounding benchmarks. The graph shows documented model and dataset relationships; some evaluation and training wiring is inferred only from the README and is marked dashed where source evidence is unavailable.
Main entry points: panorama_2b.py Model configurations · panorama_base.py Shared model plumbing · modeling_panorama_qwen.py Hugging Face inference · build_image.py SAM 3 image builder
Components 18
Data preparation
- Training datasets
data - Tokenization and batching
data/data_utils.py
Captioning model
- Model configurations
src/configs/panorama_2b.py - Qwen3-VL backbone
src/models/qwen3vl.py - Shared model plumbing
src/models/panorama_base.py - PANORAMA grounding
src/models/panorama.py - Hugging Face inference
src/hf/models_qwen3vl_sam3_concept/modeling_panorama_qwen.py
Mask proposal system
- SAM 3 runner
src/models/sam3_concept_train.py - SAM 3 image builder
third_parts/sam3_concept/build_image.py - SAM 3 image model
third_parts/sam3_concept/model/sam3_image.py - Proposal decoder
third_parts/sam3_concept/model/decoder.py - Geometry encoders
third_parts/sam3_concept/model/geometry_encoders.py
Evaluation
- PanoCaps evaluation
eval/panocaps_eval.py - Grounding metrics
eval/utils/vl_metrics.py - Referring segmentation evaluation
eval/refcoco_eval.py - Referring evaluation data
eval/utils/res_dataset.py
Other components
- Researcher
- Grounded captions and masks
Connections 18
- Researcher → Model configurations: configures
- Model configurations → Training datasets: selects datasets
src/configs/panorama_2b.py - Training datasets → Tokenization and batching: provides samples
data/base.py - Model configurations → PANORAMA grounding: builds model
- PANORAMA grounding → Shared model plumbing: inherits from
src/models/panorama.py - Shared model plumbing → Qwen3-VL backbone: builds backbone
- Shared model plumbing → SAM 3 runner: builds segmenter
- SAM 3 runner → SAM 3 image builder: constructs model
- SAM 3 image builder → Proposal decoder: uses decoder
third_parts/sam3_concept/build_image.py - SAM 3 image builder → Geometry encoders: uses encoders
third_parts/sam3_concept/build_image.py - SAM 3 image model → Geometry encoders: uses encoders
third_parts/sam3_concept/model/sam3_image.py - Researcher → Hugging Face inference: runs inference
- Hugging Face inference → SAM 3 runner: conditions proposals
src/hf/models_qwen3vl_sam3_concept/modeling_panorama_qwen.py - Hugging Face inference → Grounded captions and masks: returns predictions
- Researcher → PanoCaps evaluation: evaluates benchmark
- PanoCaps evaluation → Grounding metrics: scores predictions
- Researcher → Referring segmentation evaluation: evaluates referring masks
- Referring segmentation evaluation → Referring evaluation data: loads evaluation data
eval/refcoco_eval.py