# pku-yuangroup/video-llava architecture

> Architecture diagram of the GitHub repository pku-yuangroup/video-llava, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pku-yuangroup/video-llava
- Repository: https://github.com/pku-yuangroup/video-llava

## Overview

##### Project type and purpose
- Video-LLaVA is an open-source research framework and toolkit for multi-modal video (and image) question answering and reasoning. It extends LLaVA and LanguageBind to videos by unifying visual representations before projection into an LLM.
- It provides:
- Data conversion and evaluation scripts (in scripts/)
- A core Python library (videollava/) for model building, training, evaluation, and serving
- Pre-training, fine-tuning, and multi-benchmark evaluation pipelines
- CLI and Gradio web UI for inference

##### File structure
- Top-level directories:
- scripts/ – data converters, finetune/pretrain wrappers, JSON configs
- videollava/ – core package subdivided into
– constants.py, conversation.py, mm_utils.py, utils.py
– eval/ – benchmark-specific evaluation code, web page generator
– model/ – builder, architecture definitions, delta apply,
- language_model/ – LLaMA & MPT wrappers
- multimodal_encoder/ – Clip, image, video, audio, depth, thermal encoders
- multimodal_projector/ – projection layers
– serve/ – CLI, controller, Gradio server, model workers
– train/ – trainer scripts, monkey patches for attention libs
- assets/ – demo videos, images, README assets
- pyproject.toml – packaging and dependencies

##### README for additional insights
- Purpose: “Learning United Visual Representation by Alignment Before Projection” (paper link)
- Capabilities: zero-shot video QA, multi-benchmark evaluation
- Entry points:
- python –m videollava.serve.gradio_web_server
- python –m videollava.serve.cli
- Training & validation instructions in TRAIN_AND_VALIDATE.md

##### Detail without over-splitting
– Capture each logical component (e.g., “Multimodal Encoder” rather than each languagebind/* folder)
– Show both shell scripts and Python modules as entrypoints
– Ensure any important adapter or delta-weight component is included (LoRA, QLoRA)
– Represent benchmarks or result outputs collectively under “Evaluation” rather than individually

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Flow Start
    RawData(("Raw Benchmark Data")):::external
    RawData -->|"raw data"| DataPrep

    subgraph "Data Conversion & Preparation"
        direction TB
        DataPrep["scripts/"]:::dataprep
    end

    DataPrep -->|"preprocessed inputs"| PreprocessedInputs(("Preprocessed Inputs")):::artifact

    PreprocessedInputs --> Training

    subgraph "Training Pipeline"
        direction TB
        FinetuneSH["finetune.sh"]:::training
        FinetuneFullSH["finetune_full_schedule.sh"]:::training
        FinetuneLoRA["finetune_lora.sh"]:::training
        FinetuneQLORA["finetune_qlora.sh"]:::training
        PretrainSH["pretrain.sh"]:::training

        TrainerLLava["llava_trainer.py"]:::training
        Train["train.py"]:::training
        TrainMem["train_mem.py"]:::training
        TrainXformers["train_xformers.py"]:::training
        FlashPatch["llama_flash_attn_monkey_patch.py"]:::training
        XformersPatch["llama_xformers_attn_monkey_patch.py"]:::training
    end

    Training -->|"checkpoints"| Checkpoints(("Trained Model Checkpoints")):::artifact

    Checkpoints -->|load_pretrained_model| CoreModel

    subgraph "Core Model Library"
        direction TB
        BuilderCore["builder.py"]:::core
        ApplyDelta["apply_delta.py"]:::core
        MakeDelta["make_delta.py"]:::core
        Consolidate["consolidate.py"]:::core

        subgraph "Language Model Backbones"
            direction TB
            LLAMA["llava_llama.py"]:::core
            MPT["llava_mpt.py"]:::core
        end

        subgraph "Multimodal Encoder"
            direction TB
            MMBuilder["builder.py"]:::core
            ClipEnc["clip_encoder.py"]:::core
        end

        subgraph "Multimodal Projector"
            direction TB
            MMProjBuilder["builder.py"]:::core
        end
    end

    CoreModel --> Evaluation
    CoreModel --> Serving
    CoreModel -.->|depends on| ExternalDeps

    subgraph "Evaluation Pipeline"
        direction TB
        EvalScriptsDir["videollava/eval/"]:::evaluation
        WebGen["webpage/index.html & assets"]:::evaluation
        LegacyScripts["scripts/v1_5/eval/"]:::evaluation
    end

    Evaluation -->|"results + assets"| BenchmarkResults(("Benchmark Results & Web Assets")):::artifact

    subgraph "Serving Layer"
        direction TB
        CLI["cli.py"]:::serving
        Controller["controller.py"]:::serving
        RegisterWorker["register_worker.py"]:::serving
        GradioServer["gradio_web_server.py"]:::serving
        GradioUtils["gradio_utils.py"]:::serving
        ModelWorker["model_worker.py"]:::serving
        ServeUtils["utils.py"]:::serving
        ServeExamples["examples/"]:::serving
    end

    Serving -->|"UI endpoints"| User(("User (CLI/Web)")):::user
    User -->|"inference outputs"| Inference(("Inference Outputs")):::artifact

    subgraph "Utilities & Constants"
        direction TB
        Constants["constants.py"]:::util
        MMUtils["mm_utils.py"]:::util
        Utils["utils.py"]:::util
    end

    Training -.->|uses| ExternalDeps
    Evaluation -.->|uses| ExternalDeps
    Serving -.->|uses| ExternalDeps

    ExternalDeps(("External Libraries")):::external

    %% Click Events
    click DataPrep "https://github.com/pku-yuangroup/video-llava/tree/main/scripts/"
    click FinetuneSH "https://github.com/pku-yuangroup/video-llava/blob/main/scripts/finetune.sh"
    click FinetuneFullSH "https://github.com/pku-yuangroup/video-llava/blob/main/scripts/finetune_full_schedule.sh"
    click FinetuneLoRA "https://github.com/pku-yuangroup/video-llava/blob/main/scripts/finetune_lora.sh"
    click FinetuneQLORA "https://github.com/pku-yuangroup/video-llava/blob/main/scripts/finetune_qlora.sh"
    click PretrainSH "https://github.com/pku-yuangroup/video-llava/blob/main/scripts/pretrain.sh"
    click TrainerLLava "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/train/llava_trainer.py"
    click Train "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/train/train.py"
    click TrainMem "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/train/train_mem.py"
    click TrainXformers "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/train/train_xformers.py"
    click FlashPatch "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/train/llama_flash_attn_monkey_patch.py"
    click XformersPatch "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/train/llama_xformers_attn_monkey_patch.py"
    click BuilderCore "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/model/builder.py"
    click ApplyDelta "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/model/apply_delta.py"
    click MakeDelta "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/model/make_delta.py"
    click Consolidate "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/model/consolidate.py"
    click LLAMA "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/model/language_model/llava_llama.py"
    click MPT "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/model/language_model/llava_mpt.py"
    click MMBuilder "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/model/multimodal_encoder/builder.py"
    click ClipEnc "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/model/multimodal_encoder/clip_encoder.py"
    click MMProjBuilder "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/model/multimodal_projector/builder.py"
    click EvalScriptsDir "https://github.com/pku-yuangroup/video-llava/tree/main/videollava/eval/"
    click WebGen "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/eval/webpage/index.html"
    click LegacyScripts "https://github.com/pku-yuangroup/video-llava/tree/main/scripts/v1_5/eval/"
    click CLI "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/serve/cli.py"
    click Controller "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/serve/controller.py"
    click RegisterWorker "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/serve/register_worker.py"
    click GradioServer "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/serve/gradio_web_server.py"
    click GradioUtils "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/serve/gradio_utils.py"
    click ModelWorker "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/serve/model_worker.py"
    click ServeUtils "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/serve/utils.py"
    click ServeExamples "https://github.com/pku-yuangroup/video-llava/tree/main/videollava/serve/examples/"
    click Constants "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/constants.py"
    click MMUtils "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/mm_utils.py"
    click Utils "https://github.com/pku-yuangroup/video-llava/blob/main/videollava/utils.py"

    %% Styles
    classDef dataprep fill:#ADD8E6,stroke:#000
    classDef training fill:#90EE90,stroke:#000
    classDef core fill:#00008B,stroke:#FFF,color:#FFF
    classDef evaluation fill:#FFA500,stroke:#000
    classDef serving fill:#800080,stroke:#FFF,color:#FFF
    classDef util fill:#D3D3D3,stroke:#000
    classDef external fill:#CCCCCC,stroke:#000
    classDef artifact fill:#FFFFFF,stroke:#000
    classDef user fill:#FFFFE0,stroke:#000
```
