How pku-yuangroup/video-llava works
Project type and purpose
• Video-LLaVA is an open-source research framework and toolkit for multi-modal video (and image) question answering and reasoning. It extends LLaVA and LanguageBind to videos by unifying visual representations before projection into an LLM.
• It provides:
• Data conversion and evaluation scripts (in scripts/)
• A core Python library (videollava/) for model building, training, evaluation, and serving
• Pre-training, fine-tuning, and multi-benchmark evaluation pipelines
• CLI and Gradio web UI for inference
File structure
• Top-level directories:
• scripts/ – data converters, finetune/pretrain wrappers, JSON configs
• videollava/ – core package subdivided into
– constants.py, conversation.py, mm_utils.py, utils.py
– eval/ – benchmark-specific evaluation code, web page generator
– model/ – builder, architecture definitions, delta apply,
• language_model/ – LLaMA & MPT wrappers
• multimodal_encoder/ – Clip, image, video, audio, depth, thermal encoders
• multimodal_projector/ – projection layers
– serve/ – CLI, controller, Gradio server, model workers
– train/ – trainer scripts, monkey patches for attention libs
• assets/ – demo videos, images, README assets
• pyproject.toml – packaging and dependencies
README for additional insights
• Purpose: “Learning United Visual Representation by Alignment Before Projection” (paper link)
• Capabilities: zero-shot video QA, multi-benchmark evaluation
• Entry points:
• python –m videollava.serve.gradio_web_server
• python –m videollava.serve.cli
• Training & validation instructions in TRAIN_AND_VALIDATE.md
Detail without over-splitting
– Capture each logical component (e.g., “Multimodal Encoder” rather than each languagebind/* folder)
– Show both shell scripts and Python modules as entrypoints
– Ensure any important adapter or delta-weight component is included (LoRA, QLoRA)
– Represent benchmarks or result outputs collectively under “Evaluation” rather than individually