How vectorspacelab/video-xl works
Project type and purpose
• This is a research codebase (“Video-XL Family”) for developing and evaluating very long-sequence Video-Language Models (VLMs) (Video-XL, Video-XL-Pro, Video-XL-2).
• It includes data processing, model definition, training pipelines (DeepSpeed, TRL), and a comprehensive evaluation suite (LMMs-Eval / LVU).
File structure
• Top-level modules:
• Video-XL, Video-XL-Pro, Video-XL-2 — each has subfolders for data_processing, trl (train-loop), model definitions, scripts.
• eval — contains evaluation framework (lvu, lmms_eval) with API, filters, models, tasks, loggers.
• train — wrapper for training Video-XL-2 (deepspeed configurations, longva).
• scripts & demo.py — orchestration scripts for pretrain/finetune/train/eval.
• requirements.txt / pyproject.toml — dependency declarations.
• Patterns:
• Modular separation: data_processing → model → train → eval → tools.
• Use of TRL for reinforcement-style fine-tuning, DeepSpeed for large-scale training.
Extract README insights
• Purpose: “Efficient VLMs for extremely long video understanding.”
• Dependencies: PyTorch, DeepSpeed, TRL, HuggingFace tokenizers, WandB for logging, LMMs-Eval for evaluation.
Main components and interactions (
a. Components:
1. External Data Stores
– Raw video datasets on disk or HuggingFace Datasets
– Configuration files for tasks (YAML under eval/lvu/lmms_eval/tasks)
2. Data Processing Layer
– video-xl/data_processing/utils.py
– videoxlpro/data_processing/utils.py
– train/longva/data_processing/utils.py
3. Model Definition Layer
– multimodal encoder, projector, resampler (model/multimodal_*/)
– Builder, Consolidate, MakeDelta modules
– Language model adapters (LLAMA, QWen, etc.) under model/language_model
4. Training Pipeline
– Scripts: pretrain.sh, finetune.sh, dpo.sh, zero*.json
– DeepSpeed configs (train/deepspeed/*.json)
– TRL core and trainer modules
– train/longva/longva/train/train.py, train_dpo.py
5. Evaluation Pipeline
– eval/lvu/lmms_eval/__main__.py → orchestrates tasks
– API modules (filter.py, metrics.py, task.py)
– Task definitions (YAML under tasks/)
– Loggers (WandB, evaluation_tracker)
6. Tools & Utilities
– tools/live_bench, tools/lite (embedder, shrinker)
– notebooks for data generation / summary
7. Orchestration & Scripts
– demo.py for quick tests
– top-level scripts in Video-XL, Video-XL-Pro, Video-XL-2
b. Relationships & data flow:
1. External Data → Data Processing → Model Input Tensors
2. Model Inputs → Model Definition → Forward Pass → Loss
3. Loss → Training Pipeline (DeepSpeed / TRL) → Checkpoint Storage
4. Checkpoints → Evaluation Pipeline → Metrics, Logs
5. Results → WandB / Filesystem → Analysis
c. Architectural patterns & principles:
• Layered architecture: clear separation between preprocessing, modeling, training, and evaluation.
• Plug-in modules for different language backbones, encoders, tasks.
• Configuration-driven: YAML and JSON files control behavior without code changes.
• Reuse of external frameworks (DeepSpeed, TRL, LMMs-Eval)
d. Technologies & frameworks:
• Python 3.9+, PyTorch, DeepSpeed, HuggingFace Transformers/Tokenizers
• TRL (for reinforcement-style training)
• WandB (logging), LMMs-Eval (evaluation), FastAPI or CLI for orchestration
Essential details
• Sub-folder as a component if it encapsulates a cohesive function (e.g., trl/, data_processing/, eval/lvu/).
• Plugin points (language_model adapters, task YAMLs, filter modules) where new functionality can drop in.
• Don’t overload with file names—group related files into their functional module blocks.