# vectorspacelab/video-xl architecture

> Architecture diagram of the GitHub repository vectorspacelab/video-xl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vectorspacelab/video-xl
- Repository: https://github.com/vectorspacelab/video-xl

## Overview

##### Project type and purpose
- This is a research codebase (“Video-XL Family”) for developing and evaluating very long-sequence Video-Language Models (VLMs) (Video-XL, Video-XL-Pro, Video-XL-2).
- It includes data processing, model definition, training pipelines (DeepSpeed, TRL), and a comprehensive evaluation suite (LMMs-Eval / LVU).

##### File structure
- Top-level modules:
- Video-XL, Video-XL-Pro, Video-XL-2 — each has subfolders for data_processing, trl (train-loop), model definitions, scripts.
- eval — contains evaluation framework (lvu, lmms_eval) with API, filters, models, tasks, loggers.
- train — wrapper for training Video-XL-2 (deepspeed configurations, longva).
- scripts & demo.py — orchestration scripts for pretrain/finetune/train/eval.
- requirements.txt / pyproject.toml — dependency declarations.
- Patterns:
- Modular separation: data_processing → model → train → eval → tools.
- Use of TRL for reinforcement-style fine-tuning, DeepSpeed for large-scale training.

##### Extract README insights
- Purpose: “Efficient VLMs for extremely long video understanding.”
- Dependencies: PyTorch, DeepSpeed, TRL, HuggingFace tokenizers, WandB for logging, LMMs-Eval for evaluation.

##### Main components and interactions (
a. Components:
1. External Data Stores
– Raw video datasets on disk or HuggingFace Datasets
– Configuration files for tasks (YAML under eval/lvu/lmms_eval/tasks)
2. Data Processing Layer
– video-xl/data_processing/utils.py
– videoxlpro/data_processing/utils.py
– train/longva/data_processing/utils.py
3. Model Definition Layer
– multimodal encoder, projector, resampler (model/multimodal_*/)
– Builder, Consolidate, MakeDelta modules
– Language model adapters (LLAMA, QWen, etc.) under model/language_model
4. Training Pipeline
– Scripts: pretrain.sh, finetune.sh, dpo.sh, zero*.json
– DeepSpeed configs (train/deepspeed/*.json)
– TRL core and trainer modules
– train/longva/longva/train/train.py, train_dpo.py
5. Evaluation Pipeline
– eval/lvu/lmms_eval/__main__.py → orchestrates tasks
– API modules (filter.py, metrics.py, task.py)
– Task definitions (YAML under tasks/)
– Loggers (WandB, evaluation_tracker)
6. Tools & Utilities
– tools/live_bench, tools/lite (embedder, shrinker)
– notebooks for data generation / summary
7. Orchestration & Scripts
– demo.py for quick tests
– top-level scripts in Video-XL, Video-XL-Pro, Video-XL-2

b. Relationships & data flow:
1. External Data → Data Processing → Model Input Tensors
2. Model Inputs → Model Definition → Forward Pass → Loss
3. Loss → Training Pipeline (DeepSpeed / TRL) → Checkpoint Storage
4. Checkpoints → Evaluation Pipeline → Metrics, Logs
5. Results → WandB / Filesystem → Analysis

c. Architectural patterns & principles:
- Layered architecture: clear separation between preprocessing, modeling, training, and evaluation.
- Plug-in modules for different language backbones, encoders, tasks.
- Configuration-driven: YAML and JSON files control behavior without code changes.
- Reuse of external frameworks (DeepSpeed, TRL, LMMs-Eval)

d. Technologies & frameworks:
- Python 3.9+, PyTorch, DeepSpeed, HuggingFace Transformers/Tokenizers
- TRL (for reinforcement-style training)
- WandB (logging), LMMs-Eval (evaluation), FastAPI or CLI for orchestration

##### Essential details
- Sub-folder as a component if it encapsulates a cohesive function (e.g., trl/, data_processing/, eval/lvu/).
- Plugin points (language_model adapters, task YAMLs, filter modules) where new functionality can drop in.
- Don’t overload with file names—group related files into their functional module blocks.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data
    RawVideo["Raw Video Datasets"]:::external
    HFDatasets["HuggingFace Datasets"]:::external
    TaskYAMLs["Task Configurations (YAML)"]:::config
    click TaskYAMLs "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL-2/eval/lvu/lmms_eval/tasks"

    %% Data Processing Layer
    subgraph "Data Processing Layer"
        direction TB
        VideoXL_DP["Video-XL Data Processing"]:::code
        VideoXLPro_DP["Video-XL-Pro Data Processing"]:::code
        LongVA_DP["Video-XL-2/LongVA Data Processing"]:::code
        EvalLongVA_DP["Eval LongVA Data Processing"]:::code
    end
    click VideoXL_DP "https://github.com/vectorspacelab/video-xl/blob/main/Video-XL/videoxl/videoxl/data_processing/utils.py"
    click VideoXLPro_DP "https://github.com/vectorspacelab/video-xl/blob/main/Video-XL-Pro/videoxlpro/data_processing/utils.py"
    click LongVA_DP "https://github.com/vectorspacelab/video-xl/blob/main/Video-XL-2/train/longva/data_processing/utils.py"
    click EvalLongVA_DP "https://github.com/vectorspacelab/video-xl/blob/main/Video-XL-2/eval/lvu/longva/data_processing/utils.py"

    %% Model Definition Layer
    subgraph "Model Definition Layer"
        direction TB
        VideoXL_Model["Video-XL Model Definitions"]:::code
        VideoXLPro_Model["Video-XL-Pro Model Definitions"]:::code
        LongVA_Model["LongVA Model Definitions"]:::code
    end
    click VideoXL_Model "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL/videoxl/videoxl/model"
    click VideoXLPro_Model "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL-Pro/videoxlpro/videoxlpro/model"
    click LongVA_Model "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL-2/eval/lvu/longva/longva/model"

    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        DeepSpeedCFG["DeepSpeed Configs (JSON)"]:::config
        TRL_Scripts["TRL & DPO Trainer"]:::code
        VideoXL_Scripts["Video-XL Scripts"]:::code
        VideoXLPro_Scripts["Video-XL-Pro Scripts"]:::code
    end
    click DeepSpeedCFG "https://github.com/vectorspacelab/video-xl/blob/main/Video-XL-2/train/deepspeed/*.json"
    click TRL_Scripts "https://github.com/vectorspacelab/video-xl/blob/main/Video-XL-2/train/longva/longva/train/train.py"
    click VideoXL_Scripts "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL/scripts"
    click VideoXL_Scripts "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL/videoxl/trl"
    click VideoXL_Scripts "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL/videoxl/scripts"
    click VideoXLPro_Scripts "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL-Pro/videoxlpro/scripts"
    click VideoXLPro_Scripts "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL-Pro/videoxlpro/trl"

    %% Checkpoint Storage
    Checkpoints["Model Checkpoints"]:::storage

    %% Evaluation Pipeline
    subgraph "Evaluation Pipeline"
        direction TB
        EvalEntry["Orchestrator Entry (__main__.py)"]:::code
        EvalAPI["LMMS-Eval API"]:::code
        Filters["Filters Module"]:::code
        EvalModels["Evaluation Models"]:::code
        Loggers["Loggers (WandB/EvalTracker)"]:::code
    end
    click EvalEntry "https://github.com/vectorspacelab/video-xl/blob/main/Video-XL-2/eval/lvu/lmms_eval/__main__.py"
    click EvalAPI "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL-2/eval/lvu/lmms_eval/api"
    click Filters "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL-2/eval/lvu/lmms_eval/filters"
    click EvalModels "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL-2/eval/lvu/lmms_eval/models"
    click Loggers "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL-2/eval/lvu/lmms_eval/loggers"

    %% Results
    WandB["WandB Service"]:::external
    FSResults["Results Storage"]:::storage

    %% Tools & Utilities
    LiteTools["Lite Embedder/Shrinker"]:::code
    LiveBench["Live Bench Tools"]:::code
    click LiteTools "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL-2/eval/lvu/tools/lite"
    click LiveBench "https://github.com/vectorspacelab/video-xl/tree/main/Video-XL-2/eval/lvu/tools/live_bench"

    %% Orchestration & Demos
    subgraph "Orchestration & Demos"
        direction TB
        Demo_VXL["Video-XL Quick Test"]:::code
        Demo_VXLPro["Video-XL-Pro Quick Test"]:::code
        Demo_Eval["Eval CLI Demo"]:::code
    end
    click Demo_VXL "https://github.com/vectorspacelab/video-xl/blob/main/Video-XL/demo.py"
    click Demo_VXLPro "https://github.com/vectorspacelab/video-xl/blob/main/Video-XL-Pro/demo.py"
    click Demo_Eval "https://github.com/vectorspacelab/video-xl/blob/main/Video-XL-2/eval/lvu/demo.py"

    %% Data Flow
    RawVideo --> VideoXL_DP
    HFDatasets --> VideoXL_DP
    TaskYAMLs --> EvalEntry
    VideoXL_DP --> VideoXL_Model
    VideoXLPro_DP --> VideoXLPro_Model
    LongVA_DP --> LongVA_Model
    EvalLongVA_DP --> LongVA_Model
    VideoXL_Model --> TRL_Scripts
    VideoXLPro_Model --> VideoXLPro_Scripts
    LongVA_Model --> TRL_Scripts
    TRL_Scripts --> DeepSpeedCFG
    DeepSpeedCFG --> Checkpoints
    VideoXL_Scripts --> Checkpoints
    VideoXLPro_Scripts --> Checkpoints

    Checkpoints --> EvalEntry
    EvalEntry --> EvalAPI
    EvalAPI --> Filters
    Filters --> EvalModels
    EvalModels --> Loggers
    Loggers --> WandB
    Loggers --> FSResults

    %% Demos Control Flow
    Demo_VXL --> VideoXL_DP
    Demo_VXLPro --> VideoXLPro_DP
    Demo_Eval --> EvalEntry

    %% Tools Integration
    LiteTools --> VideoXL_DP
    LiveBench --> VideoXL_DP
    LiteTools --> EvalEntry
    LiveBench --> EvalEntry

    %% Styles
    classDef code fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef config fill:#90EE90,stroke:#333,stroke-width:1px
    classDef storage fill:#FFA500,stroke:#333,stroke-width:1px
    classDef external fill:#DDA0DD,stroke:#333,stroke-width:1px
```
