# starsuzi/videorag architecture

> Architecture diagram of the GitHub repository starsuzi/videorag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/starsuzi/videorag
- Repository: https://github.com/starsuzi/videorag

## Overview

##### Major subsystems and their roles

- Data & Preprocessing:
– Represent the datasets, including QA, synthetic, and retrieval data stored under “datasets” and the “assets” directory.
– Include preprocessing scripts where text and video are prepared (e.g., datasets/scripts/asr and datasets/scripts/original).
– Show how textual information, such as subtitles or transcribed ASR results, is extracted to accompany video frames.

- Retrieval Module:
– Describe the “retrieval” directory which contains modules for feature extraction and inference.
– Within this, highlight the “retrieval/models/InternVideo2” component that implements the video retrieval backbone.
– Show that this module includes:
▪ Dataset handling and dataloader components (the “dataset” subdirectory inside InternVideo2).
▪ Model definitions and backbones (including sub-models from BEATs, BERT-based modules, and InternVideo2 “mobileclip” support).
▪ Pretraining and evaluation scripts (under “scripts”, “tasks”, and “tests”).
▪ Utility functions for configuration, logging, distributed computing, and optimizers.
– Illustrate that the retrieval module takes in a query (or video sample) and outputs a ranked set of video (or clip) features.

- Generation Module:
– Represent the “generation” directory which contains:
▪ Inference and evaluation scripts.
▪ Different models – such as InternVL, LLaVA_NeXT, and QwenVL – which fuse the multimodal information.
▪ Supporting utilities to handle prompt formulation, data I/O, etc.
– This module receives the query and the retrieved video representations as context and generates a final response.

- Integration & Results:
– Show a component that integrates the outputs from the retrieval module with the generation module.
– Indicate that the “results” directory stores final outputs for both retrieval and generation, including JSON files with predictions.

##### Data Flow in the System

– Start with user input or an application query.
- In one path, the query triggers the retrieval module.
- In the other, the query is later forwarded to the generation module.
1. Data extraction & feature computation (covered by retrieval/extract_features.py).
2. InternVideo2 model processing the video corpus for multimodal feature extraction – show its submodules (backbones, datasets, utils).
3. A ranking or selection process that outputs top candidate videos (and key frames) – optionally annotate the frame selection mechanism here.
– In the generation path, illustrate:
1. The generative models (InternVL, LLaVA_NeXT, QwenVL) that consume the query along with the retrieved video’s visual and textual representations.
2. The prompt formulation and multimodal fusion logic, possibly using the “generation/utils” scripts.
3. The generation of a final text response.

##### Architectural Patterns and Technologies

- Modular design – separate modules for retrieval, generation, preprocessing, and evaluation.
- Design principles such as reusability (via utility modules and common config files) and extensibility (multiple model backbones under generation and retrieval).
- Technologies: Python 3.8+ is used; the repository likely leverages deep learning frameworks (e.g., PyTorch) for the model implementations.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Top Level: Input Query
    input["Input Query"]:::input

    %% Data & Preprocessing Subgraph
    subgraph "Datasets & Preprocessing"
        ds["Datasets"]:::dataproc
        p1["Preprocessing (ASR)"]:::dataproc
        p2["Preprocessing (Original)"]:::dataproc
    end

    %% Retrieval Module Subgraph
    subgraph "Retrieval Module (InternVideo2)"
        r1["Extract Features"]:::retrieval
        r2["InternVideo2 Model<br/>(Frame Selection & Multimodal Extraction)"]:::retrieval
        r3["Retrieval Inference"]:::retrieval
    end

    %% Generation Module Subgraph
    subgraph "Generation Module (InternVL, LLaVA_NeXT, QwenVL)"
        g1["Generation Inference"]:::generation
        g2["Generation Evaluation"]:::generation
        g3["Generation Models<br/>(InternVL, LLaVA_NeXT, QwenVL)"]:::generation
        g4["Generation Utils"]:::generation
    end

    %% Results Subgraph
    subgraph "Output/Results"
        res1["Retrieval Results"]:::results
        res2["Generation Results"]:::results
    end

    %% Data Flow Connections
    input -->|"sends query"| ds
    input -->|"sends query"| p1
    input -->|"sends query"| p2
    input -->|"sends query"| g1

    ds -->|"provides raw data"| r1
    p1 -->|"text extraction"| r1
    p2 -->|"text extraction"| r1

    r1 -->|"computes features"| r2
    r2 -->|"processes video corpus"| r3

    r3 -->|"RetrievedVideos"| res1
    r3 -->|"VideoFeatures"| g1

    g1 -->|"generates response"| g2
    g2 -->|"finalizes output"| res2

    %% Click Events
    click ds "https://github.com/starsuzi/videorag/tree/main/datasets/"
    click p1 "https://github.com/starsuzi/videorag/tree/main/datasets/scripts/asr/"
    click p2 "https://github.com/starsuzi/videorag/tree/main/datasets/scripts/original/"
    click r1 "https://github.com/starsuzi/videorag/blob/main/retrieval/extract_features.py"
    click r3 "https://github.com/starsuzi/videorag/blob/main/retrieval/inference.py"
    click r2 "https://github.com/starsuzi/videorag/tree/main/retrieval/models/InternVideo2/"
    click g1 "https://github.com/starsuzi/videorag/blob/main/generation/inference.py"
    click g2 "https://github.com/starsuzi/videorag/blob/main/generation/evaluation.py"
    click g3 "https://github.com/starsuzi/videorag/tree/main/generation/models/"
    click g4 "https://github.com/starsuzi/videorag/tree/main/generation/utils/"
    click res1 "https://github.com/starsuzi/videorag/tree/main/results/retrieval/"
    click res2 "https://github.com/starsuzi/videorag/tree/main/results/generation/"

    %% Styles
    classDef input fill:#FFEB99,stroke:#333,stroke-width:2px;
    classDef dataproc fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef retrieval fill:#A9DFBF,stroke:#1E8449,stroke-width:2px;
    classDef generation fill:#F5CBA7,stroke:#943126,stroke-width:2px;
    classDef results fill:#D7BDE2,stroke:#6C3483,stroke-width:2px;
    
    %% Assign classes
    class input input;
    class ds,p1,p2 dataproc;
    class r1,r2,r3 retrieval;
    class g1,g2,g3,g4 generation;
    class res1,res2 results;
```
