# damo-nlp-sg/video-llama architecture

> Architecture diagram of the GitHub repository damo-nlp-sg/video-llama, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/damo-nlp-sg/video-llama
- Repository: https://github.com/damo-nlp-sg/video-llama

## Overview

##### Project type and purpose
- This is a full-stack machine-learning system for audio-visual language understanding. It provides:
- Two cross-modal pretrain & fine-tune pipelines (Vision-Language and Audio-Language branches)
- An inference/demo layer for interactive video-grounded chat
- Configurable scripts (train.py, demo_audiovideo.py, demo_video.py) driven by YAML configs
- Integration of multiple third-party models: BLIP-2, MiniGPT-4, ImageBind, LLaMA/Vicuna decoders

##### File structure
- Top-level directories & files:
- train_configs/, eval_configs/: YAML configs for pre-training, fine-tuning, evaluation
- video_llama/: core Python package, subdivided into common, configs, datasets, models, processors, runners, tasks
- demo_*.py, train.py, setup.py, environment.yml, requirement.txt
- examples/: sample videos/audio; prompts/: text prompts
- Patterns & architectural cues:
- “datasets/builders” & “datasets/datasets”: data ingestion and loader utilities
- “models”: wrappers for encoders (Vision Q-Former, Audio Q-Former), language decoder interfaces
- “processors”: video/audio/frame transformations and augmentations
- “runners” & “tasks”: higher-level orchestrators for training/inference loops

##### README for additional insights
- Two-stage training: pre-training on WebVid & CC3M, then instruction-tuning on MiniGPT-4, LLaVA, VideoChat
- Demo setup: configuration → demo script → GPU inference

##### Elements
- Clear labels for each component (e.g., “Vision Q-Former”, “Audio Segment Embedder”)
- Blue for data related (datasets, dataloaders)
- Green for model components (encoders, Q-Formers, decoder)
- Orange for orchestration (scripts, runners, tasks)
- Grey for utilities/config/common modules
- Rectangles for processing modules
- Trapezoids or parallelograms for scripts/CLI

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Entrypoints
    subgraph "CLI Entrypoints"
        train["train.py"]:::script
        demo_video["demo_video.py"]:::script
        demo_av["demo_audiovideo.py"]:::script
    end

    %% Data Layer
    subgraph "Data Layer"
        ext1["WebVid, CC3M"]:::data
        ext2["LLaVA-Instruct, VideoChat"]:::data
        builders["Dataset Builders"]:::data
        datasets["Dataset Definitions & Loaders"]:::data
    end

    %% Preprocessing
    subgraph "Preprocessing Layer"
        transforms["Video/Audio/Frame Transforms"]:::proc
    end

    %% Vision-Language Branch
    subgraph "Vision-Language Branch"
        vit["ViT Backbone (EVA-ViT)"]:::model
        blip2["BLIP-2 Visual Encoder"]:::model
        qformer["Cross-Modal Q-Formers"]:::model
    end

    %% Audio-Language Branch
    subgraph "Audio-Language Branch"
        ibind["ImageBind Audio Encoder"]:::model
        qformer2["Cross-Modal Q-Formers"]:::model
    end

    %% Fusion & Decoder
    fusion["Fusion & Cross-Modal Adapter"]:::model
    decoder["Language Decoder Interface"]:::model

    %% Orchestration & Utilities
    subgraph "Utilities & Core"
        configmgmt["Configuration Management"]:::util
        registry["Registry Pattern"]:::util
        dist["Distributed Utilities & Process Setup"]:::util
        logger["Logging Utilities"]:::util
        runnerbase["Runner Base (training/inference loops)"]:::util
        basetask["Base Task Abstraction"]:::util
    end

    subgraph "Tasks"
        task1["Video-Text Pretrain Task"]:::util
        task2["Image-Text Pretrain Task"]:::util
    end

    %% Sample Data & Prompts
    samples["Sample Videos/Audio & Prompts"]:::data

    %% Connections
    ext1 --> builders
    ext2 --> builders
    builders --> datasets
    datasets --> transforms
    transforms --> vit
    transforms --> ibind
    vit --> blip2 --> qformer
    ibind --> qformer2
    qformer --> fusion
    qformer2 --> fusion
    fusion --> decoder
    train -->|loads configs| configmgmt
    demo_video -->|loads configs| configmgmt
    demo_av -->|loads configs| configmgmt
    configmgmt --> registry
    registry --> runnerbase
    runnerbase --> task1
    runnerbase --> task2
    task1 --> datasets
    task2 --> datasets
    task1 --> transforms
    task2 --> transforms
    decoder --> samples

    %% Click Events
    click builders "https://github.com/damo-nlp-sg/video-llama/tree/main/video_llama/datasets/builders/"
    click datasets "https://github.com/damo-nlp-sg/video-llama/tree/main/video_llama/datasets/datasets/"
    click transforms "https://github.com/damo-nlp-sg/video-llama/tree/main/video_llama/processors/"
    click blip2 "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/models/blip2.py"
    click vit "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/models/eva_vit.py"
    click ibind "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/models/ImageBind/models/imagebind_model.py"
    click qformer "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/models/Qformer.py"
    click fusion "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/models/video_llama.py"
    click decoder "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/models/modeling_llama.py"
    click configmgmt "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/common/config.py"
    click registry "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/common/registry.py"
    click dist "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/common/dist_utils.py"
    click logger "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/common/logger.py"
    click runnerbase "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/runners/runner_base.py"
    click basetask "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/tasks/base_task.py"
    click task1 "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/tasks/video_text_pretrain.py"
    click task2 "https://github.com/damo-nlp-sg/video-llama/blob/main/video_llama/tasks/image_text_pretrain.py"
    click train "https://github.com/damo-nlp-sg/video-llama/blob/main/train.py"
    click demo_video "https://github.com/damo-nlp-sg/video-llama/blob/main/demo_video.py"
    click demo_av "https://github.com/damo-nlp-sg/video-llama/blob/main/demo_audiovideo.py"

    %% Styles
    classDef data fill:#D0E7FF,stroke:#0366d6,color:#0366d6,stroke-width:1px
    classDef model fill:#DFF5D0,stroke:#228B22,color:#228B22,stroke-width:1px
    classDef script fill:#FFE7C0,stroke:#FF8C00,color:#FF8C00,stroke-width:1px
    classDef proc fill:#FFF3D0,stroke:#DAA520,color:#DAA520,stroke-width:1px
    classDef util fill:#EEEEEE,stroke:#666666,color:#666666,stroke-width:1px
```
