How sitonggong/vrs-hq works
Project Type and Purpose
• VRS-HQ is a research codebase for “High Quality Video Reasoning Segmentation” (CVPR 2025). It’s a deep-learning pipeline combining a Multimodal Large Language Model (Chat-UniVi), a temporal token aggregation mechanism, and the Segment Anything Model v2 (SAM2) for single-stage video object segmentation with reasoning.
File Structure Analysis
• Top-level directories
• dataset – data loaders, video frame extractors, custom dataset classes for ReVOS, MeVis, VQA, etc.
• model – core implementations subdivided into:
– llava – wrappers around MLLM (Chat-UniVi) plus model builders, adapters, delta-finetuning, training scripts.
– segment_anything – SAM2 code: image encoder, prompt encoder, mask decoder, utilities, mask generator.
– univi – Chat-UniVi integration: configuration, multimodal encoder, evaluation benchmarks, training utilities.
– tf – TensorFlow interoperability outputs (less prominent).
• sam2 – a parallel copy of SAM2 code customized for video inference (sam2_image_predictor, sam2_video_predictor).
• checkpoints – shell scripts to download pretrained weights (SAM2 hiera_large, Chat-UniVi).
• evaluation_multiseg.py – top-level inference/evaluation script using DeepSpeed.
• Key patterns:
– Pipeline style: data → MLLM→ temporal aggregator → keyframe selector → SAM2 → output masks → evaluation.
– Modularity: separate subpackages for model backbones (MLLM vs SAM2) and dataset utilities.
– External dependencies & build scripts: deepspeed, conda environment, pip install, pretrained weights downloads.
README Insights
• Describes 4 architectural stages:
a. MLLM-based Temporal Token Encoding (<SEG> & <TAK> tokens)
b. Temporal Dynamic Aggregation (TDA)
c. Token-driven Keyframe Selection (TKS)
d. Mask Decoding & Propagation (MDP) via SAM2’s memory mechanism
• Lists external weight sources (Huggingface, Baidu Drive), datasets, training‐inference commands, and evaluation metrics.