How zhang-tao-whu/dvis works
DVIS is a universal video-segmentation framework for video instance, panoptic, and semantic segmentation, with online and offline modes. Its principal workflow prepares sampled video frames and annotations, runs a video segmentation model, and produces task-specific predictions; training adds matching and loss computation, while dataset evaluators score outputs. The repository also includes the Mask2Former image-model foundation and a video demo/presentation layer. Edges are limited to README statements and relationships visible in the supplied excerpts or SOURCE INDEX; unsampled wiring is not asserted.
Main entry points: meta_architecture.py DVIS model · video_maskformer_model.py Video MaskFormer · predictor.py Video predictor · criterion.py Video criterion
Components 23
Video data
- YTVIS dataset
dvis/data_video/datasets/ytvis.py - YTVIS API
dvis/data_video/datasets/ytvis_api/ytvos.py - VPS dataset
dvis/data_video/datasets/vps.py - VSS dataset
dvis/data_video/datasets/vss.py - Frame mapper
dvis/data_video/dataset_mapper.py - Video augmentation
dvis/data_video/augmentation.py
Segmentation models
- DVIS model
dvis/meta_architecture.py - Temporal modules
dvis/video_dvis_modules.py - DVIS decoder
dvis/video_mask2former_transformer_decoder.py - Video MaskFormer
mask2former_video/video_maskformer_model.py - Video decoder
mask2former_video/modeling/transformer_decoder/video_mask2former_transformer_decoder.py - Mask2Former model
mask2former/maskformer_model.py - Swin backbone
mask2former/modeling/backbone/swin.py
Training internals
- Video criterion
mask2former_video/modeling/criterion.py - Video matcher
mask2former_video/modeling/matcher.py - OOM recovery
mask2former_video/utils/memory.py
Prediction and evaluation
- Video predictor
demo_video/predictor.py - Video visualizer
demo_video/visualizer.py - YTVIS evaluator
dvis/data_video/ytvis_eval.py - VPS evaluator
dvis/data_video/vps_eval.py - VSS evaluator
dvis/data_video/vss_eval.py
Other components
- Video user
- Segmentation output
Connections 21
- Video user → Video predictor: requests prediction
- YTVIS dataset → YTVIS API: uses API
dvis/data_video/datasets/ytvis.py - Frame mapper → Video augmentation: applies transforms
- DVIS model → Video criterion: computes losses
dvis/meta_architecture.py - DVIS model → Video matcher: matches targets
dvis/meta_architecture.py - DVIS model → OOM recovery: retries OOM
dvis/meta_architecture.py - DVIS model → Temporal modules: uses temporal modules
- Temporal modules → DVIS decoder: uses decoder
dvis/video_dvis_modules.py - Video MaskFormer → Video criterion: computes losses
mask2former_video/video_maskformer_model.py - Video MaskFormer → Video matcher: matches targets
mask2former_video/video_maskformer_model.py - Video MaskFormer → OOM recovery: retries OOM
mask2former_video/video_maskformer_model.py - Video MaskFormer → Video decoder: decodes video
- Mask2Former model → Video criterion: computes losses
- Mask2Former model → Swin backbone: extracts features
- Video predictor → DVIS model: runs model
- Video predictor → Video visualizer: uses visualization
demo_video/predictor.py - DVIS model → Segmentation output: produces predictions
- Video MaskFormer → Segmentation output: produces predictions
- YTVIS evaluator → Segmentation output: scores predictions
- VPS evaluator → Segmentation output: scores predictions
- VSS evaluator → Segmentation output: scores predictions