# sitonggong/vrs-hq architecture

> Architecture diagram of the GitHub repository sitonggong/vrs-hq, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sitonggong/vrs-hq
- Repository: https://github.com/sitonggong/vrs-hq

## Overview

##### Project Type and Purpose
- VRS-HQ is a research codebase for “High Quality Video Reasoning Segmentation” (CVPR 2025). It’s a deep-learning pipeline combining a Multimodal Large Language Model (Chat-UniVi), a temporal token aggregation mechanism, and the Segment Anything Model v2 (SAM2) for single-stage video object segmentation with reasoning.

##### File Structure Analysis
- **Top-level directories**
- dataset – data loaders, video frame extractors, custom dataset classes for ReVOS, MeVis, VQA, etc.
- model – core implementations subdivided into:
– llava – wrappers around MLLM (Chat-UniVi) plus model builders, adapters, delta-finetuning, training scripts.
– segment_anything – SAM2 code: image encoder, prompt encoder, mask decoder, utilities, mask generator.
– univi – Chat-UniVi integration: configuration, multimodal encoder, evaluation benchmarks, training utilities.
– tf – TensorFlow interoperability outputs (less prominent).
- sam2 – a parallel copy of SAM2 code customized for video inference (sam2_image_predictor, sam2_video_predictor).
- checkpoints – shell scripts to download pretrained weights (SAM2 hiera_large, Chat-UniVi).
- evaluation_multiseg.py – top-level inference/evaluation script using DeepSpeed.
- **Key patterns**:
– Pipeline style: data → MLLM→ temporal aggregator → keyframe selector → SAM2 → output masks → evaluation.
– Modularity: separate subpackages for model backbones (MLLM vs SAM2) and dataset utilities.
– External dependencies & build scripts: deepspeed, conda environment, pip install, pretrained weights downloads.

##### README Insights
- Describes 4 architectural stages:
a. MLLM-based Temporal Token Encoding (<SEG> & <TAK> tokens)
b. Temporal Dynamic Aggregation (TDA)
c. Token-driven Keyframe Selection (TKS)
d. Mask Decoding & Propagation (MDP) via SAM2’s memory mechanism
- Lists external weight sources (Huggingface, Baidu Drive), datasets, training‐inference commands, and evaluation metrics.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        DL["Video Dataset Loader"]:::dataset
        FE["Frame Extractor"]:::preprocess
    end

    %% MLLM Module
    subgraph "MLLM Module"
        LLAVA["model/llava/"]:::core
        CLIP_ENC["clip_encoder.py"]:::core
        UNIVI["model/univi/"]:::core
    end

    %% Temporal Dynamic Aggregation
    TDA["TDA (cluster.py)"]:::core

    %% Token-driven Keyframe Selector
    TKS["TKS (utils.py)"]:::core

    %% SAM2 Mask Decoder & Propagation
    subgraph "SAM2 Module"
        IMG_ENC["image_encoder.py"]:::core
        PROMPT_ENC["prompt_encoder.py"]:::core
        MASK_DEC["mask_decoder.py"]:::core
        SAM2_VID["sam2_video_predictor.py"]:::core
    end

    %% Output & Evaluation
    subgraph "Output & Evaluation"
        EVAL_SCRIPT["evaluation_multiseg.py"]:::eval
        EVAL_BENCH["model/univi/eval/"]:::eval
        METRICS["Compute J&F Metrics"]:::eval
    end

    %% Training Loop
    subgraph "Training Loop"
        LLAVAT1["train.py (llava)"]:::train
        LLAVAT2["train_mem.py (llava)"]:::train
        UNIVIT1["train.py (univi)"]:::train
        UNIVIT2["train_mem.py (univi)"]:::train
        CKPT_SAVE["Checkpoint Save"]:::train
    end

    %% External Weights
    HF["Huggingface"]:::external
    BD["Baidu Drive"]:::external
    DOWNLOAD["download_ckpts.sh"]:::external

    %% Data Flow
    DL --> FE --> LLAVA
    LLAVA --> CLIP_ENC
    LLAVA --> UNIVI
    CLIP_ENC --> TDA
    UNIVI --> TDA
    TDA --> TKS
    TKS --> IMG_ENC
    IMG_ENC --> PROMPT_ENC --> MASK_DEC --> SAM2_VID
    SAM2_VID --> METRICS
    METRICS --> EVAL_SCRIPT --> EVAL_BENCH

    %% Control & Config Flows
    DOWNLOAD -.-> LLAVA
    DOWNLOAD -.-> IMG_ENC
    HF -.-> LLAVA
    HF -.-> IMG_ENC
    BD -.-> LLAVA
    BD -.-> IMG_ENC

    %% Training Loop Flows
    LLAVA --> LLAVAT1 --> CKPT_SAVE
    LLAVA --> LLAVAT2 --> CKPT_SAVE
    UNIVI --> UNIVIT1 --> CKPT_SAVE
    UNIVI --> UNIVIT2 --> CKPT_SAVE
    CKPT_SAVE --> DOWNLOAD

    %% Click Events
    click DL "https://github.com/sitonggong/vrs-hq/tree/main/dataset/"
    click FE "https://github.com/sitonggong/vrs-hq/blob/main/dataset/extract_video_frames.py"
    click LLAVA "https://github.com/sitonggong/vrs-hq/tree/main/model/llava/"
    click CLIP_ENC "https://github.com/sitonggong/vrs-hq/blob/main/model/llava/model/multimodal_encoder/clip_encoder.py"
    click UNIVI "https://github.com/sitonggong/vrs-hq/tree/main/model/univi/"
    click TDA "https://github.com/sitonggong/vrs-hq/blob/main/model/univi/model/cluster.py"
    click TKS "https://github.com/sitonggong/vrs-hq/blob/main/model/univi/utils.py"
    click IMG_ENC "https://github.com/sitonggong/vrs-hq/blob/main/model/segment_anything/modeling/image_encoder.py"
    click PROMPT_ENC "https://github.com/sitonggong/vrs-hq/blob/main/model/segment_anything/modeling/prompt_encoder.py"
    click MASK_DEC "https://github.com/sitonggong/vrs-hq/blob/main/model/segment_anything/modeling/mask_decoder.py"
    click SAM2_VID "https://github.com/sitonggong/vrs-hq/blob/main/sam2/sam2_video_predictor.py"
    click EVAL_SCRIPT "https://github.com/sitonggong/vrs-hq/blob/main/evaluation_multiseg.py"
    click EVAL_BENCH "https://github.com/sitonggong/vrs-hq/tree/main/model/univi/eval/"
    click LLAVAT1 "https://github.com/sitonggong/vrs-hq/blob/main/model/llava/train/train.py"
    click LLAVAT2 "https://github.com/sitonggong/vrs-hq/blob/main/model/llava/train/train_mem.py"
    click UNIVIT1 "https://github.com/sitonggong/vrs-hq/blob/main/model/univi/train/train.py"
    click UNIVIT2 "https://github.com/sitonggong/vrs-hq/blob/main/model/univi/train/train_mem.py"
    click DOWNLOAD "https://github.com/sitonggong/vrs-hq/blob/main/checkpoints/download_ckpts.sh"

    %% Styles
    classDef dataset fill:#ccffcc,stroke:#2d862d,color:#000;
    classDef preprocess fill:#ffffcc,stroke:#a88f0f,color:#000;
    classDef core fill:#cce5ff,stroke:#2f75b5,color:#000;
    classDef eval fill:#e5ccff,stroke:#7a2fa5,color:#000;
    classDef train fill:#ffcccc,stroke:#b52d2d,color:#000;
    classDef external fill:#ffe5cc,stroke:#b56b2f,color:#000;
```
