# sarapieri/panorama_grounding architecture

> Architecture diagram of the GitHub repository sarapieri/panorama_grounding, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sarapieri/panorama_grounding
- Repository: https://github.com/sarapieri/panorama_grounding

## Overview

PANORAMA is a vision-language model for panoptic grounded captioning: it generates scene text with phrase markers, then uses each phrase representation to condition mask proposals and select phrase-aligned regions. The repository also provides training datasets and evaluation paths for PanoCaps, referring-expression segmentation, and other grounding benchmarks. The graph shows documented model and dataset relationships; some evaluation and training wiring is inferred only from the README and is marked dashed where source evidence is unavailable.

Main entry points: [`src/configs/panorama_2b.py`](https://github.com/sarapieri/panorama_grounding/blob/main/src/configs/panorama_2b.py) (Model configurations), [`src/models/panorama_base.py`](https://github.com/sarapieri/panorama_grounding/blob/main/src/models/panorama_base.py) (Shared model plumbing), [`src/hf/models_qwen3vl_sam3_concept/modeling_panorama_qwen.py`](https://github.com/sarapieri/panorama_grounding/blob/main/src/hf/models_qwen3vl_sam3_concept/modeling_panorama_qwen.py) (Hugging Face inference), [`third_parts/sam3_concept/build_image.py`](https://github.com/sarapieri/panorama_grounding/blob/main/third_parts/sam3_concept/build_image.py) (SAM 3 image builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_datasets["Training datasets"]
  node_batching["Tokenization and batching<br/>[data_utils.py]"]
end

subgraph group_model["Captioning model"]
  node_training_config["Model configurations<br/>[panorama_2b.py]"]
  node_qwen["Qwen3-VL backbone<br/>[qwen3vl.py]"]
  node_panorama_base["Shared model plumbing<br/>[panorama_base.py]"]
  node_panorama["PANORAMA grounding<br/>[panorama.py]"]
  node_hf_model["Hugging Face inference"]
end

subgraph group_segmenter["Mask proposal system"]
  node_sam_runner["SAM 3 runner"]
  node_sam_builder["SAM 3 image builder<br/>[build_image.py]"]
  node_sam_image["SAM 3 image model<br/>[sam3_image.py]"]
  node_sam_decoder["Proposal decoder<br/>[decoder.py]"]
  node_sam_geometry["Geometry encoders"]
end

subgraph group_evaluation["Evaluation"]
  node_panocaps_eval["PanoCaps evaluation<br/>[panocaps_eval.py]"]
  node_grounding_eval["Grounding metrics<br/>[vl_metrics.py]"]
  node_refseg_eval["Referring segmentation evaluation<br/>[refcoco_eval.py]"]
  node_res_dataset["Referring evaluation data<br/>[res_dataset.py]"]
end

node_user(("Researcher"))
node_result["Grounded captions and masks"]

node_user -.->|"configures"| node_training_config
node_training_config -->|"selects datasets"| node_datasets
node_datasets -.->|"provides samples"| node_batching
node_training_config -.->|"builds model"| node_panorama
node_panorama -->|"inherits from"| node_panorama_base
node_panorama_base -.->|"builds backbone"| node_qwen
node_panorama_base -.->|"builds segmenter"| node_sam_runner
node_sam_runner -.->|"constructs model"| node_sam_builder
node_sam_builder -->|"uses decoder"| node_sam_decoder
node_sam_builder -->|"uses encoders"| node_sam_geometry
node_sam_image -->|"uses encoders"| node_sam_geometry
node_user -.->|"runs inference"| node_hf_model
node_hf_model -->|"conditions proposals"| node_sam_runner
node_hf_model -.->|"returns predictions"| node_result
node_user -.->|"evaluates benchmark"| node_panocaps_eval
node_panocaps_eval -.->|"scores predictions"| node_grounding_eval
node_user -.->|"evaluates referring masks"| node_refseg_eval
node_refseg_eval -->|"loads evaluation data"| node_res_dataset

click node_datasets "https://github.com/sarapieri/panorama_grounding/tree/main/data"
click node_batching "https://github.com/sarapieri/panorama_grounding/blob/main/data/data_utils.py"
click node_training_config "https://github.com/sarapieri/panorama_grounding/blob/main/src/configs/panorama_2b.py"
click node_qwen "https://github.com/sarapieri/panorama_grounding/blob/main/src/models/qwen3vl.py"
click node_panorama_base "https://github.com/sarapieri/panorama_grounding/blob/main/src/models/panorama_base.py"
click node_panorama "https://github.com/sarapieri/panorama_grounding/blob/main/src/models/panorama.py"
click node_hf_model "https://github.com/sarapieri/panorama_grounding/blob/main/src/hf/models_qwen3vl_sam3_concept/modeling_panorama_qwen.py"
click node_sam_runner "https://github.com/sarapieri/panorama_grounding/blob/main/src/models/sam3_concept_train.py"
click node_sam_builder "https://github.com/sarapieri/panorama_grounding/blob/main/third_parts/sam3_concept/build_image.py"
click node_sam_image "https://github.com/sarapieri/panorama_grounding/blob/main/third_parts/sam3_concept/model/sam3_image.py"
click node_sam_decoder "https://github.com/sarapieri/panorama_grounding/blob/main/third_parts/sam3_concept/model/decoder.py"
click node_sam_geometry "https://github.com/sarapieri/panorama_grounding/blob/main/third_parts/sam3_concept/model/geometry_encoders.py"
click node_panocaps_eval "https://github.com/sarapieri/panorama_grounding/blob/main/eval/panocaps_eval.py"
click node_grounding_eval "https://github.com/sarapieri/panorama_grounding/blob/main/eval/utils/vl_metrics.py"
click node_refseg_eval "https://github.com/sarapieri/panorama_grounding/blob/main/eval/refcoco_eval.py"
click node_res_dataset "https://github.com/sarapieri/panorama_grounding/blob/main/eval/utils/res_dataset.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_datasets,node_batching toneBlue
class node_training_config,node_qwen,node_panorama_base,node_panorama,node_hf_model toneAmber
class node_sam_runner,node_sam_builder,node_sam_image,node_sam_decoder,node_sam_geometry toneMint
class node_panocaps_eval,node_grounding_eval,node_refseg_eval,node_res_dataset toneRose
class node_user,node_result toneIndigo
```

## Components

### Data preparation

- [Training datasets](https://github.com/sarapieri/panorama_grounding/tree/main/data): `data`
- [Tokenization and batching](https://github.com/sarapieri/panorama_grounding/blob/main/data/data_utils.py): `data/data_utils.py`

### Captioning model

- [Model configurations](https://github.com/sarapieri/panorama_grounding/blob/main/src/configs/panorama_2b.py): `src/configs/panorama_2b.py`
- [Qwen3-VL backbone](https://github.com/sarapieri/panorama_grounding/blob/main/src/models/qwen3vl.py): `src/models/qwen3vl.py`
- [Shared model plumbing](https://github.com/sarapieri/panorama_grounding/blob/main/src/models/panorama_base.py): `src/models/panorama_base.py`
- [PANORAMA grounding](https://github.com/sarapieri/panorama_grounding/blob/main/src/models/panorama.py): `src/models/panorama.py`
- [Hugging Face inference](https://github.com/sarapieri/panorama_grounding/blob/main/src/hf/models_qwen3vl_sam3_concept/modeling_panorama_qwen.py): `src/hf/models_qwen3vl_sam3_concept/modeling_panorama_qwen.py`

### Mask proposal system

- [SAM 3 runner](https://github.com/sarapieri/panorama_grounding/blob/main/src/models/sam3_concept_train.py): `src/models/sam3_concept_train.py`
- [SAM 3 image builder](https://github.com/sarapieri/panorama_grounding/blob/main/third_parts/sam3_concept/build_image.py): `third_parts/sam3_concept/build_image.py`
- [SAM 3 image model](https://github.com/sarapieri/panorama_grounding/blob/main/third_parts/sam3_concept/model/sam3_image.py): `third_parts/sam3_concept/model/sam3_image.py`
- [Proposal decoder](https://github.com/sarapieri/panorama_grounding/blob/main/third_parts/sam3_concept/model/decoder.py): `third_parts/sam3_concept/model/decoder.py`
- [Geometry encoders](https://github.com/sarapieri/panorama_grounding/blob/main/third_parts/sam3_concept/model/geometry_encoders.py): `third_parts/sam3_concept/model/geometry_encoders.py`

### Evaluation

- [PanoCaps evaluation](https://github.com/sarapieri/panorama_grounding/blob/main/eval/panocaps_eval.py): `eval/panocaps_eval.py`
- [Grounding metrics](https://github.com/sarapieri/panorama_grounding/blob/main/eval/utils/vl_metrics.py): `eval/utils/vl_metrics.py`
- [Referring segmentation evaluation](https://github.com/sarapieri/panorama_grounding/blob/main/eval/refcoco_eval.py): `eval/refcoco_eval.py`
- [Referring evaluation data](https://github.com/sarapieri/panorama_grounding/blob/main/eval/utils/res_dataset.py): `eval/utils/res_dataset.py`

### Other components

- **Researcher**
- **Grounded captions and masks**

## Connections

- Researcher → Model configurations: configures
- Model configurations → Training datasets: selects datasets (evidence: `src/configs/panorama_2b.py`)
- Training datasets → Tokenization and batching: provides samples (evidence: `data/base.py`)
- Model configurations → PANORAMA grounding: builds model
- PANORAMA grounding → Shared model plumbing: inherits from (evidence: `src/models/panorama.py`)
- Shared model plumbing → Qwen3-VL backbone: builds backbone
- Shared model plumbing → SAM 3 runner: builds segmenter
- SAM 3 runner → SAM 3 image builder: constructs model
- SAM 3 image builder → Proposal decoder: uses decoder (evidence: `third_parts/sam3_concept/build_image.py`)
- SAM 3 image builder → Geometry encoders: uses encoders (evidence: `third_parts/sam3_concept/build_image.py`)
- SAM 3 image model → Geometry encoders: uses encoders (evidence: `third_parts/sam3_concept/model/sam3_image.py`)
- Researcher → Hugging Face inference: runs inference
- Hugging Face inference → SAM 3 runner: conditions proposals (evidence: `src/hf/models_qwen3vl_sam3_concept/modeling_panorama_qwen.py`)
- Hugging Face inference → Grounded captions and masks: returns predictions
- Researcher → PanoCaps evaluation: evaluates benchmark
- PanoCaps evaluation → Grounding metrics: scores predictions
- Researcher → Referring segmentation evaluation: evaluates referring masks
- Referring segmentation evaluation → Referring evaluation data: loads evaluation data (evidence: `eval/refcoco_eval.py`)