# zhang-tao-whu/dvis architecture

> Architecture diagram of the GitHub repository zhang-tao-whu/dvis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zhang-tao-whu/dvis
- Repository: https://github.com/zhang-tao-whu/dvis

## Overview

DVIS is a universal video-segmentation framework for video instance, panoptic, and semantic segmentation, with online and offline modes. Its principal workflow prepares sampled video frames and annotations, runs a video segmentation model, and produces task-specific predictions; training adds matching and loss computation, while dataset evaluators score outputs. The repository also includes the Mask2Former image-model foundation and a video demo/presentation layer. Edges are limited to README statements and relationships visible in the supplied excerpts or SOURCE INDEX; unsampled wiring is not asserted.

Main entry points: [`dvis/meta_architecture.py`](https://github.com/zhang-tao-whu/dvis/blob/main/dvis/meta_architecture.py) (DVIS model), [`mask2former_video/video_maskformer_model.py`](https://github.com/zhang-tao-whu/dvis/blob/main/mask2former_video/video_maskformer_model.py) (Video MaskFormer), [`demo_video/predictor.py`](https://github.com/zhang-tao-whu/dvis/blob/main/demo_video/predictor.py) (Video predictor), [`mask2former_video/modeling/criterion.py`](https://github.com/zhang-tao-whu/dvis/blob/main/mask2former_video/modeling/criterion.py) (Video criterion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Video data"]
  node_ytvis_data["YTVIS dataset<br/>[ytvis.py]"]
  node_ytvis_api["YTVIS API<br/>[ytvos.py]"]
  node_vps_data["VPS dataset<br/>[vps.py]"]
  node_vss_data["VSS dataset<br/>[vss.py]"]
  node_mapper["Frame mapper<br/>[dataset_mapper.py]"]
  node_augment["Video augmentation<br/>[augmentation.py]"]
end

subgraph group_models["Segmentation models"]
  node_dvis_model["DVIS model"]
  node_dvis_modules["Temporal modules"]
  node_video_decoder["DVIS decoder"]
  node_video_maskformer["Video MaskFormer"]
  node_video_decoder_m2f["Video decoder"]
  node_image_model["Mask2Former model"]
  node_backbone["Swin backbone<br/>[swin.py]"]
end

subgraph group_training["Training internals"]
  node_criterion["Video criterion<br/>[criterion.py]"]
  node_matcher["Video matcher<br/>[matcher.py]"]
  node_memory["OOM recovery<br/>[memory.py]"]
end

subgraph group_results["Prediction and evaluation"]
  node_predictor["Video predictor<br/>[predictor.py]"]
  node_visualizer["Video visualizer<br/>[visualizer.py]"]
  node_ytvis_eval["YTVIS evaluator<br/>[ytvis_eval.py]"]
  node_vps_eval["VPS evaluator<br/>[vps_eval.py]"]
  node_vss_eval["VSS evaluator<br/>[vss_eval.py]"]
end

node_user(("Video user"))
node_output["Segmentation output"]

node_user -.->|"requests prediction"| node_predictor
node_ytvis_data -->|"uses API"| node_ytvis_api
node_mapper -.->|"applies transforms"| node_augment
node_dvis_model -->|"computes losses"| node_criterion
node_dvis_model -->|"matches targets"| node_matcher
node_dvis_model -->|"retries OOM"| node_memory
node_dvis_model -.->|"uses temporal modules"| node_dvis_modules
node_dvis_modules -->|"uses decoder"| node_video_decoder
node_video_maskformer -->|"computes losses"| node_criterion
node_video_maskformer -->|"matches targets"| node_matcher
node_video_maskformer -->|"retries OOM"| node_memory
node_video_maskformer -.->|"decodes video"| node_video_decoder_m2f
node_image_model -.->|"computes losses"| node_criterion
node_image_model -.->|"extracts features"| node_backbone
node_predictor -.->|"runs model"| node_dvis_model
node_predictor -->|"uses visualization"| node_visualizer
node_dvis_model -.->|"produces predictions"| node_output
node_video_maskformer -.->|"produces predictions"| node_output
node_ytvis_eval -.->|"scores predictions"| node_output
node_vps_eval -.->|"scores predictions"| node_output
node_vss_eval -.->|"scores predictions"| node_output

click node_ytvis_data "https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/datasets/ytvis.py"
click node_ytvis_api "https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/datasets/ytvis_api/ytvos.py"
click node_vps_data "https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/datasets/vps.py"
click node_vss_data "https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/datasets/vss.py"
click node_mapper "https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/dataset_mapper.py"
click node_augment "https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/augmentation.py"
click node_dvis_model "https://github.com/zhang-tao-whu/dvis/blob/main/dvis/meta_architecture.py"
click node_dvis_modules "https://github.com/zhang-tao-whu/dvis/blob/main/dvis/video_dvis_modules.py"
click node_video_decoder "https://github.com/zhang-tao-whu/dvis/blob/main/dvis/video_mask2former_transformer_decoder.py"
click node_video_maskformer "https://github.com/zhang-tao-whu/dvis/blob/main/mask2former_video/video_maskformer_model.py"
click node_video_decoder_m2f "https://github.com/zhang-tao-whu/dvis/blob/main/mask2former_video/modeling/transformer_decoder/video_mask2former_transformer_decoder.py"
click node_image_model "https://github.com/zhang-tao-whu/dvis/blob/main/mask2former/maskformer_model.py"
click node_backbone "https://github.com/zhang-tao-whu/dvis/blob/main/mask2former/modeling/backbone/swin.py"
click node_criterion "https://github.com/zhang-tao-whu/dvis/blob/main/mask2former_video/modeling/criterion.py"
click node_matcher "https://github.com/zhang-tao-whu/dvis/blob/main/mask2former_video/modeling/matcher.py"
click node_memory "https://github.com/zhang-tao-whu/dvis/blob/main/mask2former_video/utils/memory.py"
click node_predictor "https://github.com/zhang-tao-whu/dvis/blob/main/demo_video/predictor.py"
click node_visualizer "https://github.com/zhang-tao-whu/dvis/blob/main/demo_video/visualizer.py"
click node_ytvis_eval "https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/ytvis_eval.py"
click node_vps_eval "https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/vps_eval.py"
click node_vss_eval "https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/vss_eval.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ytvis_data,node_ytvis_api,node_vps_data,node_vss_data,node_mapper,node_augment,node_user toneBlue
class node_dvis_model,node_dvis_modules,node_video_decoder,node_video_maskformer,node_video_decoder_m2f,node_image_model,node_backbone toneAmber
class node_criterion,node_matcher,node_memory toneMint
class node_predictor,node_visualizer,node_ytvis_eval,node_vps_eval,node_vss_eval toneRose
class node_output toneIndigo
```

## Components

### Video data

- [YTVIS dataset](https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/datasets/ytvis.py): `dvis/data_video/datasets/ytvis.py`
- [YTVIS API](https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/datasets/ytvis_api/ytvos.py): `dvis/data_video/datasets/ytvis_api/ytvos.py`
- [VPS dataset](https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/datasets/vps.py): `dvis/data_video/datasets/vps.py`
- [VSS dataset](https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/datasets/vss.py): `dvis/data_video/datasets/vss.py`
- [Frame mapper](https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/dataset_mapper.py): `dvis/data_video/dataset_mapper.py`
- [Video augmentation](https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/augmentation.py): `dvis/data_video/augmentation.py`

### Segmentation models

- [DVIS model](https://github.com/zhang-tao-whu/dvis/blob/main/dvis/meta_architecture.py): `dvis/meta_architecture.py`
- [Temporal modules](https://github.com/zhang-tao-whu/dvis/blob/main/dvis/video_dvis_modules.py): `dvis/video_dvis_modules.py`
- [DVIS decoder](https://github.com/zhang-tao-whu/dvis/blob/main/dvis/video_mask2former_transformer_decoder.py): `dvis/video_mask2former_transformer_decoder.py`
- [Video MaskFormer](https://github.com/zhang-tao-whu/dvis/blob/main/mask2former_video/video_maskformer_model.py): `mask2former_video/video_maskformer_model.py`
- [Video decoder](https://github.com/zhang-tao-whu/dvis/blob/main/mask2former_video/modeling/transformer_decoder/video_mask2former_transformer_decoder.py): `mask2former_video/modeling/transformer_decoder/video_mask2former_transformer_decoder.py`
- [Mask2Former model](https://github.com/zhang-tao-whu/dvis/blob/main/mask2former/maskformer_model.py): `mask2former/maskformer_model.py`
- [Swin backbone](https://github.com/zhang-tao-whu/dvis/blob/main/mask2former/modeling/backbone/swin.py): `mask2former/modeling/backbone/swin.py`

### Training internals

- [Video criterion](https://github.com/zhang-tao-whu/dvis/blob/main/mask2former_video/modeling/criterion.py): `mask2former_video/modeling/criterion.py`
- [Video matcher](https://github.com/zhang-tao-whu/dvis/blob/main/mask2former_video/modeling/matcher.py): `mask2former_video/modeling/matcher.py`
- [OOM recovery](https://github.com/zhang-tao-whu/dvis/blob/main/mask2former_video/utils/memory.py): `mask2former_video/utils/memory.py`

### Prediction and evaluation

- [Video predictor](https://github.com/zhang-tao-whu/dvis/blob/main/demo_video/predictor.py): `demo_video/predictor.py`
- [Video visualizer](https://github.com/zhang-tao-whu/dvis/blob/main/demo_video/visualizer.py): `demo_video/visualizer.py`
- [YTVIS evaluator](https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/ytvis_eval.py): `dvis/data_video/ytvis_eval.py`
- [VPS evaluator](https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/vps_eval.py): `dvis/data_video/vps_eval.py`
- [VSS evaluator](https://github.com/zhang-tao-whu/dvis/blob/main/dvis/data_video/vss_eval.py): `dvis/data_video/vss_eval.py`

### Other components

- **Video user**
- **Segmentation output**

## Connections

- Video user → Video predictor: requests prediction
- YTVIS dataset → YTVIS API: uses API (evidence: `dvis/data_video/datasets/ytvis.py`)
- Frame mapper → Video augmentation: applies transforms
- DVIS model → Video criterion: computes losses (evidence: `dvis/meta_architecture.py`)
- DVIS model → Video matcher: matches targets (evidence: `dvis/meta_architecture.py`)
- DVIS model → OOM recovery: retries OOM (evidence: `dvis/meta_architecture.py`)
- DVIS model → Temporal modules: uses temporal modules
- Temporal modules → DVIS decoder: uses decoder (evidence: `dvis/video_dvis_modules.py`)
- Video MaskFormer → Video criterion: computes losses (evidence: `mask2former_video/video_maskformer_model.py`)
- Video MaskFormer → Video matcher: matches targets (evidence: `mask2former_video/video_maskformer_model.py`)
- Video MaskFormer → OOM recovery: retries OOM (evidence: `mask2former_video/video_maskformer_model.py`)
- Video MaskFormer → Video decoder: decodes video
- Mask2Former model → Video criterion: computes losses
- Mask2Former model → Swin backbone: extracts features
- Video predictor → DVIS model: runs model
- Video predictor → Video visualizer: uses visualization (evidence: `demo_video/predictor.py`)
- DVIS model → Segmentation output: produces predictions
- Video MaskFormer → Segmentation output: produces predictions
- YTVIS evaluator → Segmentation output: scores predictions
- VPS evaluator → Segmentation output: scores predictions
- VSS evaluator → Segmentation output: scores predictions