# isee-laboratory/referdino architecture

> Architecture diagram of the GitHub repository isee-laboratory/referdino, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/isee-laboratory/referdino
- Repository: https://github.com/isee-laboratory/referdino

## Overview

ReferDINO is a referring video object segmentation system: users provide a video and referring expression, and the model predicts object masks across frames. Its main runtime branches are dataset-based pretraining/fine-tuning and dataset-specific inference, with a separate video demo. The README documents COCO pretraining and YouTube-VOS, MeViS, DAVIS, A2D-Sentences, and JHMDB-Sentences workflows. Some subsystem links are based on documented capabilities and entry-point imports; unsampled implementation details are not inferred.

Main entry points: [`eval`](https://github.com/isee-laboratory/referdino/tree/main/eval) (Dataset inference), [`trainer.py`](https://github.com/isee-laboratory/referdino/blob/main/trainer.py) (Fine-tuning), [`pretrainer.py`](https://github.com/isee-laboratory/referdino/blob/main/pretrainer.py) (COCO pretraining), [`models/__init__.py`](https://github.com/isee-laboratory/referdino/blob/main/models/__init__.py) (Model construction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime Entry Points"]
  node_main["Train/eval CLI<br/>[main.py]"]
  node_demo["Video demo<br/>[demo_video.py]"]
  node_infer["Dataset inference"]
end

subgraph group_data["Video Data"]
  node_refdata["Referring datasets"]
  node_transforms["Video transforms<br/>[transform_video.py]"]
  node_sampler["Dataset sampler<br/>[sampler.py]"]
end

subgraph group_model["Grounding Model"]
  node_modelbuild["Model construction<br/>[__init__.py]"]
  node_grounding["GroundingDINO<br/>[groundingdino.py]"]
  node_backbone["Swin backbone"]
  node_temporal["Temporal modeling"]
  node_segmentation["Mask prediction<br/>[segmentation.py]"]
end

subgraph group_learning["Training and Evaluation"]
  node_trainer["Fine-tuning<br/>[trainer.py]"]
  node_pretrainer["COCO pretraining<br/>[pretrainer.py]"]
  node_criterion["Training losses<br/>[criterion.py]"]
end

subgraph group_results["Inference Results"]
  node_postprocess["Mask postprocessing<br/>[postprocessing.py]"]
  node_metrics["Dataset evaluation"]
end

node_user(("Researcher"))
node_video["Video and expression"]
node_annotations["Mask annotations"]

node_user -->|"configures run"| node_main
node_user -->|"submits video"| node_demo
node_user -->|"starts inference"| node_infer
node_video -->|"provides input"| node_demo
node_main -->|"starts training"| node_trainer
node_main -->|"starts pretraining"| node_pretrainer
node_trainer -->|"loads samples"| node_refdata
node_pretrainer -->|"loads samples"| node_refdata
node_trainer -->|"selects batches"| node_sampler
node_pretrainer -->|"selects batches"| node_sampler
node_refdata -->|"augments clips"| node_transforms
node_trainer -->|"builds model"| node_modelbuild
node_pretrainer -->|"builds model"| node_modelbuild
node_trainer -->|"computes losses"| node_criterion
node_pretrainer -->|"computes losses"| node_criterion
node_modelbuild -->|"constructs"| node_grounding
node_grounding -->|"extracts features"| node_backbone
node_grounding -->|"models video"| node_temporal
node_grounding -->|"predicts masks"| node_segmentation
node_infer -->|"builds model"| node_modelbuild
node_infer -->|"formats predictions"| node_postprocess
node_infer -->|"evaluates outputs"| node_metrics
node_infer -->|"writes masks"| node_annotations
node_demo -->|"loads model"| node_modelbuild
node_demo -->|"returns predictions"| node_annotations

click node_main "https://github.com/isee-laboratory/referdino/blob/main/main.py"
click node_demo "https://github.com/isee-laboratory/referdino/blob/main/demo_video.py"
click node_infer "https://github.com/isee-laboratory/referdino/tree/main/eval"
click node_refdata "https://github.com/isee-laboratory/referdino/tree/main/datasets"
click node_transforms "https://github.com/isee-laboratory/referdino/blob/main/datasets/transform_video.py"
click node_sampler "https://github.com/isee-laboratory/referdino/blob/main/datasets/sampler.py"
click node_modelbuild "https://github.com/isee-laboratory/referdino/blob/main/models/__init__.py"
click node_grounding "https://github.com/isee-laboratory/referdino/blob/main/models/GroundingDINO/groundingdino.py"
click node_backbone "https://github.com/isee-laboratory/referdino/blob/main/models/GroundingDINO/backbone/swin_transformer.py"
click node_temporal "https://github.com/isee-laboratory/referdino/blob/main/models/GroundingDINO/temporal_modules.py"
click node_segmentation "https://github.com/isee-laboratory/referdino/blob/main/models/GroundingDINO/segmentation.py"
click node_trainer "https://github.com/isee-laboratory/referdino/blob/main/trainer.py"
click node_pretrainer "https://github.com/isee-laboratory/referdino/blob/main/pretrainer.py"
click node_criterion "https://github.com/isee-laboratory/referdino/blob/main/models/criterion.py"
click node_postprocess "https://github.com/isee-laboratory/referdino/blob/main/models/postprocessing.py"
click node_metrics "https://github.com/isee-laboratory/referdino/tree/main/eval"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_demo,node_infer toneBlue
class node_refdata,node_transforms,node_sampler toneAmber
class node_modelbuild,node_grounding,node_backbone,node_temporal,node_segmentation toneMint
class node_trainer,node_pretrainer,node_criterion toneRose
class node_postprocess,node_metrics,node_user,node_video,node_annotations toneIndigo
```

## Components

### Runtime Entry Points

- [Train/eval CLI](https://github.com/isee-laboratory/referdino/blob/main/main.py): `main.py`
- [Video demo](https://github.com/isee-laboratory/referdino/blob/main/demo_video.py): `demo_video.py`
- [Dataset inference](https://github.com/isee-laboratory/referdino/tree/main/eval): `eval`

### Video Data

- [Referring datasets](https://github.com/isee-laboratory/referdino/tree/main/datasets): `datasets`
- [Video transforms](https://github.com/isee-laboratory/referdino/blob/main/datasets/transform_video.py): `datasets/transform_video.py`
- [Dataset sampler](https://github.com/isee-laboratory/referdino/blob/main/datasets/sampler.py): `datasets/sampler.py`

### Grounding Model

- [Model construction](https://github.com/isee-laboratory/referdino/blob/main/models/__init__.py): `models/__init__.py`
- [GroundingDINO](https://github.com/isee-laboratory/referdino/blob/main/models/GroundingDINO/groundingdino.py): `models/GroundingDINO/groundingdino.py`
- [Swin backbone](https://github.com/isee-laboratory/referdino/blob/main/models/GroundingDINO/backbone/swin_transformer.py): `models/GroundingDINO/backbone/swin_transformer.py`
- [Temporal modeling](https://github.com/isee-laboratory/referdino/blob/main/models/GroundingDINO/temporal_modules.py): `models/GroundingDINO/temporal_modules.py`
- [Mask prediction](https://github.com/isee-laboratory/referdino/blob/main/models/GroundingDINO/segmentation.py): `models/GroundingDINO/segmentation.py`

### Training and Evaluation

- [Fine-tuning](https://github.com/isee-laboratory/referdino/blob/main/trainer.py): `trainer.py`
- [COCO pretraining](https://github.com/isee-laboratory/referdino/blob/main/pretrainer.py): `pretrainer.py`
- [Training losses](https://github.com/isee-laboratory/referdino/blob/main/models/criterion.py): `models/criterion.py`

### Inference Results

- [Mask postprocessing](https://github.com/isee-laboratory/referdino/blob/main/models/postprocessing.py): `models/postprocessing.py`
- [Dataset evaluation](https://github.com/isee-laboratory/referdino/tree/main/eval): `eval`

### Other components

- **Researcher**
- **Video and expression**
- **Mask annotations**

## Connections

- Researcher → Train/eval CLI: configures run
- Researcher → Video demo: submits video
- Researcher → Dataset inference: starts inference
- Video and expression → Video demo: provides input
- Train/eval CLI → Fine-tuning: starts training
- Train/eval CLI → COCO pretraining: starts pretraining
- Fine-tuning → Referring datasets: loads samples
- COCO pretraining → Referring datasets: loads samples
- Fine-tuning → Dataset sampler: selects batches
- COCO pretraining → Dataset sampler: selects batches
- Referring datasets → Video transforms: augments clips
- Fine-tuning → Model construction: builds model
- COCO pretraining → Model construction: builds model
- Fine-tuning → Training losses: computes losses
- COCO pretraining → Training losses: computes losses
- Model construction → GroundingDINO: constructs
- GroundingDINO → Swin backbone: extracts features
- GroundingDINO → Temporal modeling: models video
- GroundingDINO → Mask prediction: predicts masks
- Dataset inference → Model construction: builds model
- Dataset inference → Mask postprocessing: formats predictions
- Dataset inference → Dataset evaluation: evaluates outputs
- Dataset inference → Mask annotations: writes masks
- Video demo → Model construction: loads model
- Video demo → Mask annotations: returns predictions