How starsuzi/videorag works
Major subsystems and their roles
• Data & Preprocessing:
– Represent the datasets, including QA, synthetic, and retrieval data stored under “datasets” and the “assets” directory.
– Include preprocessing scripts where text and video are prepared (e.g., datasets/scripts/asr and datasets/scripts/original).
– Show how textual information, such as subtitles or transcribed ASR results, is extracted to accompany video frames.
• Retrieval Module:
– Describe the “retrieval” directory which contains modules for feature extraction and inference.
– Within this, highlight the “retrieval/models/InternVideo2” component that implements the video retrieval backbone.
– Show that this module includes:
▪ Dataset handling and dataloader components (the “dataset” subdirectory inside InternVideo2).
▪ Model definitions and backbones (including sub-models from BEATs, BERT-based modules, and InternVideo2 “mobileclip” support).
▪ Pretraining and evaluation scripts (under “scripts”, “tasks”, and “tests”).
▪ Utility functions for configuration, logging, distributed computing, and optimizers.
– Illustrate that the retrieval module takes in a query (or video sample) and outputs a ranked set of video (or clip) features.
• Generation Module:
– Represent the “generation” directory which contains:
▪ Inference and evaluation scripts.
▪ Different models – such as InternVL, LLaVA_NeXT, and QwenVL – which fuse the multimodal information.
▪ Supporting utilities to handle prompt formulation, data I/O, etc.
– This module receives the query and the retrieved video representations as context and generates a final response.
• Integration & Results:
– Show a component that integrates the outputs from the retrieval module with the generation module.
– Indicate that the “results” directory stores final outputs for both retrieval and generation, including JSON files with predictions.
Data Flow in the System
– Start with user input or an application query.
• In one path, the query triggers the retrieval module.
• In the other, the query is later forwarded to the generation module.
1. Data extraction & feature computation (covered by retrieval/extract_features.py).
2. InternVideo2 model processing the video corpus for multimodal feature extraction – show its submodules (backbones, datasets, utils).
3. A ranking or selection process that outputs top candidate videos (and key frames) – optionally annotate the frame selection mechanism here.
– In the generation path, illustrate:
1. The generative models (InternVL, LLaVA_NeXT, QwenVL) that consume the query along with the retrieved video’s visual and textual representations.
2. The prompt formulation and multimodal fusion logic, possibly using the “generation/utils” scripts.
3. The generation of a final text response.
Architectural Patterns and Technologies
• Modular design – separate modules for retrieval, generation, preprocessing, and evaluation.
• Design principles such as reusability (via utility modules and common config files) and extensibility (multiple model backbones under generation and retrieval).
• Technologies: Python 3.8+ is used; the repository likely leverages deep learning frameworks (e.g., PyTorch) for the model implementations.