# fantasyele/llava-kd architecture

> Architecture diagram of the GitHub repository fantasyele/llava-kd, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fantasyele/llava-kd
- Repository: https://github.com/fantasyele/llava-kd

## Overview

The project is a research framework for distilling multimodal large language models (MLLM) into smaller models. In other words, it is designed to transfer the knowledge from a larger, more computationally intensive model (l‑MLLM) into a smaller, resource-friendly model (s‑MLLM) while preserving as much capability as possible. The goal is to support tasks that involve both visual and textual inputs by aligning visual-textual representations and then refining the model with fine-tuning.

##### Main Components
- Data Processing & Preprocessing Module:
– Modules under llavakd/data handle dataset preparation, image preprocessing, text preprocessing, and template formatting. They define how input images and text queries are turned into model-ready data.
- Model Components:
– The “llavakd/model” directory is subdivided into several key parts:
▪ LLM Modules: The “llavakd/model/llm” folder includes implementations of various language models (e.g., Qwen, tinyllama, phi, etc.).
▪ Vision Tower Modules: Under “llavakd/model/vision_tower,” you have different visual feature extractors (e.g., clip, dinov2, openclip).
▪ Connector Modules: The “llavakd/model/connector” folder includes various connectors (e.g., linear, mlp, qformer) which are essential for bridging the representations between vision and language components.
▪ Configuration & Loading: Files like configuration_tinyllava.py and load_model.py configure and initialize the models.
- Distillation and Training Pipeline:
– Although the training code is planned (listed in the TODO), the repository includes evaluation scripts and utilities for distillation (e.g., methods for Multimodal Distillation (MDist) and Relation Distillation (RDist) in the evaluation and utils modules).
- Evaluation and Benchmarking:
– The “llavakd/eval” directory and “scripts/eval” folder contain multiple evaluation scripts for different benchmarks (e.g., for textVQA, science QA, etc.).
– There are also conversion scripts in the “scripts” directory for adapting data to various benchmark formats.
- Supporting Utilities:
– The “llavakd/utils” folder offers helper functions, logging, argument parsing, and methods for data and evaluation utilities.
- External Dependencies:
– Dependencies such as bitsandbytes, flash-attention, TorchVision (and PyTorch) are used as indicated in the README, supporting efficient computing and model inference.

##### Relationships and Interactions
- Flow of Data:
– The system accepts inputs (typically an image and associated text query) which are preprocessed by components in the “data” modules.
– Preprocessed data is fed into the Visual Tower and LLM components via connector modules that align the visual and text representations.
– Distillation methods compare the outputs of the large (teacher) and small (student) models thereby guiding the training (or fine-tuning) process.
- Inference & Evaluation:
– The quick_inference.py entry point represents the inference pipeline where a pre-trained distilled model is loaded and used to answer queries based on input images.
– Multiple评 scripts and evaluation modules maintain consistency in benchmarking across different evaluation datasets.
- External Tools Integration:

##### Architectural Patterns and Design Principles
- Modularity: The clear separation into modules for data management, model architecture (LLM, vision tower, connectors), evaluation, and utilities.
- Separation of Concerns: Distinct directories take care of preprocessing, model configuration, evaluation, and conversion scripts highlights purposeful decoupling.
- Extensibility: The design allows for adding new evaluation benchmarks, modifying distillation techniques, or swapping underlying model architectures without disrupting the entire pipeline.

##### Technologies, Frameworks, and Libraries
- Programming Framework: PyTorch (with TorchVision) for model implementation.
- External Libraries: Bitsandbytes and Flash-Attention to accelerate and compress model computations.
- Evaluation: Shell scripts in the “scripts/eval” folder and Python-based evaluation in the “llavakd/eval” directory.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Nodes definitions
    Input["Input: Image & Text Query"]:::input
    DataPrep["Data Preprocessing"]:::dataPreproc
    Vision["Vision Tower Modules"]:::model
    LLM["LLM Modules"]:::model
    Connector["Connector Modules"]:::connector
    Distill["Distillation & Training Pipeline"]:::train
    Inference["Quick Inference"]:::train
    EvalGroup["Evaluation & Benchmarking"]:::eval
    EvalScripts["Evaluation Scripts"]:::eval
    BenchScripts["Benchmark Scripts"]:::eval
    Utilities["Supporting Utilities"]:::util
    ExtDep["External Dependencies"]:::extdep

    %% Subgraph for Model Configuration
    subgraph "Model Configuration"
        Config["Config (tinyllava)"]:::config
        LoadMod["Load Model"]:::config
        ModelDef["Modeling (LLaVA KD)"]:::config
    end

    %% Primary data flow
    Input --> DataPrep
    DataPrep --> Vision
    DataPrep --> LLM
    Vision --> Connector
    LLM --> Connector

    %% Configuration integration (dotted arrows indicate initialization dependency)
    Config --> LoadMod
    LoadMod --> ModelDef
    ModelDef --- LLM
    ModelDef --- Vision

    Connector --> Distill
    Distill --> Inference
    Inference --> EvalGroup

    %% Evaluation subgraph split
    EvalGroup --> EvalScripts
    EvalGroup --> BenchScripts

    %% Utilities and External Dependencies connections (dotted lines)
    Distill --- Utilities
    LLM --- ExtDep
    Vision --- ExtDep

    %% Click events
    click DataPrep "https://github.com/fantasyele/llava-kd/tree/main/llavakd/data"
    click Vision "https://github.com/fantasyele/llava-kd/tree/main/llavakd/model/vision_tower"
    click LLM "https://github.com/fantasyele/llava-kd/tree/main/llavakd/model/llm"
    click Connector "https://github.com/fantasyele/llava-kd/tree/main/llavakd/model/connector"
    click Config "https://github.com/fantasyele/llava-kd/blob/main/llavakd/model/configuration_tinyllava.py"
    click LoadMod "https://github.com/fantasyele/llava-kd/blob/main/llavakd/model/load_model.py"
    click ModelDef "https://github.com/fantasyele/llava-kd/blob/main/llavakd/model/modeling_LLaVA_KD.py"
    click Distill "https://github.com/fantasyele/llava-kd/blob/main/llavakd/utils/distill_loss_utils.py"
    click Inference "https://github.com/fantasyele/llava-kd/blob/main/quick_inference.py"
    click EvalScripts "https://github.com/fantasyele/llava-kd/tree/main/llavakd/eval"
    click BenchScripts "https://github.com/fantasyele/llava-kd/tree/main/scripts/eval"
    click Utilities "https://github.com/fantasyele/llava-kd/tree/main/llavakd/utils"
    click ExtDep "https://github.com/fantasyele/llava-kd/tree/main/bitsandbytes"

    %% Styles
    classDef input fill:#efe,stroke:#333,stroke-width:2px;
    classDef dataPreproc fill:#f9f,stroke:#333,stroke-width:2px;
    classDef model fill:#bbf,stroke:#333,stroke-width:2px;
    classDef connector fill:#bfb,stroke:#333,stroke-width:2px;
    classDef train fill:#ff9,stroke:#333,stroke-width:2px;
    classDef eval fill:#fd8,stroke:#333,stroke-width:2px;
    classDef config fill:#d9f,stroke:#333,stroke-width:2px;
    classDef util fill:#cff,stroke:#333,stroke-width:2px;
    classDef extdep fill:#fdd,stroke:#333,stroke-width:2px;
```
