How fantasyele/llava-kd works
The project is a research framework for distilling multimodal large language models (MLLM) into smaller models. In other words, it is designed to transfer the knowledge from a larger, more computationally intensive model (l‑MLLM) into a smaller, resource-friendly model (s‑MLLM) while preserving as much capability as possible. The goal is to support tasks that involve both visual and textual inputs by aligning visual-textual representations and then refining the model with fine-tuning.
Main Components
• Data Processing & Preprocessing Module:
– Modules under llavakd/data handle dataset preparation, image preprocessing, text preprocessing, and template formatting. They define how input images and text queries are turned into model-ready data.
• Model Components:
– The “llavakd/model” directory is subdivided into several key parts:
▪ LLM Modules: The “llavakd/model/llm” folder includes implementations of various language models (e.g., Qwen, tinyllama, phi, etc.).
▪ Vision Tower Modules: Under “llavakd/model/vision_tower,” you have different visual feature extractors (e.g., clip, dinov2, openclip).
▪ Connector Modules: The “llavakd/model/connector” folder includes various connectors (e.g., linear, mlp, qformer) which are essential for bridging the representations between vision and language components.
▪ Configuration & Loading: Files like configuration_tinyllava.py and load_model.py configure and initialize the models.
• Distillation and Training Pipeline:
– Although the training code is planned (listed in the TODO), the repository includes evaluation scripts and utilities for distillation (e.g., methods for Multimodal Distillation (MDist) and Relation Distillation (RDist) in the evaluation and utils modules).
• Evaluation and Benchmarking:
– The “llavakd/eval” directory and “scripts/eval” folder contain multiple evaluation scripts for different benchmarks (e.g., for textVQA, science QA, etc.).
– There are also conversion scripts in the “scripts” directory for adapting data to various benchmark formats.
• Supporting Utilities:
– The “llavakd/utils” folder offers helper functions, logging, argument parsing, and methods for data and evaluation utilities.
• External Dependencies:
– Dependencies such as bitsandbytes, flash-attention, TorchVision (and PyTorch) are used as indicated in the README, supporting efficient computing and model inference.
Relationships and Interactions
• Flow of Data:
– The system accepts inputs (typically an image and associated text query) which are preprocessed by components in the “data” modules.
– Preprocessed data is fed into the Visual Tower and LLM components via connector modules that align the visual and text representations.
– Distillation methods compare the outputs of the large (teacher) and small (student) models thereby guiding the training (or fine-tuning) process.
• Inference & Evaluation:
– The quick_inference.py entry point represents the inference pipeline where a pre-trained distilled model is loaded and used to answer queries based on input images.
– Multiple评 scripts and evaluation modules maintain consistency in benchmarking across different evaluation datasets.
• External Tools Integration:
Architectural Patterns and Design Principles
• Modularity: The clear separation into modules for data management, model architecture (LLM, vision tower, connectors), evaluation, and utilities.
• Separation of Concerns: Distinct directories take care of preprocessing, model configuration, evaluation, and conversion scripts highlights purposeful decoupling.
• Extensibility: The design allows for adding new evaluation benchmarks, modifying distillation techniques, or swapping underlying model architectures without disrupting the entire pipeline.
Technologies, Frameworks, and Libraries
• Programming Framework: PyTorch (with TorchVision) for model implementation.
• External Libraries: Bitsandbytes and Flash-Attention to accelerate and compress model computations.
• Evaluation: Shell scripts in the “scripts/eval” folder and Python-based evaluation in the “llavakd/eval” directory.