# magic-research/pllava architecture

> Architecture diagram of the GitHub repository magic-research/pllava, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/magic-research/pllava
- Repository: https://github.com/magic-research/pllava

## Overview

##### Main Project Type and Purpose
- This is a research prototype that adapts an image-language model to process videos for dense captioning tasks.
- Its primary functionality involves training (using distributed strategies via Hugging Face’s Accelerate) and evaluation (including demo scripts and gallery visualization) of the extended model.
- The README also indicates integration with Hugging Face models, benchmark evaluation, and how to reproduce experiments via training and evaluation scripts.

##### File Structure to Highlight Key Components
- Top-Level Directories:
- dataset: Contains modules for handling and processing datasets (e.g., base_dataset, specialized video utilities) which serve as the entry point for video and image data.
- models: Contains the PLLaVA implementation (modeling, configuration, processing modules) which defines the network architecture and any conversion scripts.
- tasks: Subdivided into “train” and “eval” (as well as “eval” demos for mvbench, recaption, vcgbench, videoqabench). These orchestrate training and evaluation procedures.
- scripts: Provides shell scripts and configuration files for running distributed training (with different acceleration configurations) and evaluation demos.
- utils: Provides supporting modules (configuration, logging, distributed training modules, optimizers, etc.) that are used across components.
- python_scripts: Contains a helper for Hugging Face model integration.
- example and docs: Useful for demo media content and documentation (e.g., PDFs, images, sample videos).
- Configuration files (YAML for deepspeed, accelerate configuration) and multiple requirements files indicate the use of popular libraries such as PyTorch, Hugging Face Transformers, and Accelerate.

##### Core Components and Relationships
a. Data Ingestion and Preprocessing
- Component: Dataset Manager (represented by modules in dataset/ and parts of tasks/train/instruction_data.py)
- Role: Loads video and image data, applies pre-processing with utilities (video_utils.py, utils modules).
b. Model Architecture and Configuration
- Component: PLLaVA Model (located in models/pllava/)
- Role: Contains the configuration (configuration_pllava.py), the core model definition (modeling_pllava.py), weight conversion routines, and processing methods. It interfaces with Hugging Face’s transformers.
c. Training Pipeline
- Component: Training Tasks (tasks/train and associated scripts like train_pllava.sh)
- Role: Handles training configurations (config_pllava_nframe.py) using distributed training strategies (configuration and utils modules) and accelerate configurations (scripts/accel_config_*.yaml). Training outputs (lora/projector weights) are stored for later use.
d. Evaluation/Inference Pipeline
- Component: Evaluation Tasks (tasks/eval/ folders including demos and benchmarking scripts)
- Role: Evaluates models on multiple benchmarks (VideoQA, MVBench, recaption, etc.), generates result galleries and provides demo interfaces (e.g., tasks/eval/demo/pllava_demo.py).
e. External Integration and Environment Setup
- Component: Hugging Face Integration (python_scripts/hf.py, requirements.txt)
- Role: Downloads and integrates pretrained models as well as ensuring the right library and accelerator configurations are in place.
f. Supporting Infrastructure
- Component: Utility/Configuration Layer (utils, scripts for config and logging)
- Role: Provides shared services like logging, distributed computing utilities, and configuration handling.

##### Architectural Patterns/Design Principles
- Modular design: Each component (data, model, training, evaluation) is separated into different directories.
- Clear separation of concerns: The system isolates model logic from training scripts and dataset management.
- Extensibility: Thanks to a clear directory separation (e.g., different evaluations for VideoQA, MVBench, etc.), adding new tasks or benchmarks should be straightforward.
- Distributed Training: Supporting multiple configurations from multi-GPU to multinode via YAML configs and using Accelerate.
- Integration with Hugging Face: Ensures compatibility with popular transformers and easy model sharing via the Hugging Face ecosystem.

##### Detailed Steps to Convey
- The “Training Pipeline” (tasks/train) utilizes the “PLLaVA Model” (models/pllava) and outputs trained weights/models.
- In parallel, a “Hugging Face Integration” module (python_scripts/hf.py) links externally downloaded model assets into the model directory.
- “Utilities” (utils/) across all modules to denote shared services like logging, distributed setup, configuration updates.

## Diagram (Mermaid)

```mermaid
flowchart TD
    E("Environment Setup"):::env
    DM("Dataset Manager"):::data
    TP("Training Pipeline"):::pipeline
    PM("PLLaVA Model"):::model
    EI("Evaluation/Inference Pipeline"):::eval
    HF("Hugging Face Integration"):::external
    UC("Utility & Config Module"):::util

    E -->|"provides-configs"| DM
    DM -->|"supplies-preprocessed-data"| TP
    TP -->|"trains-using"| PM
    HF -->|"integrates-pretrained-assets"| PM
    PM -->|"outputs-trained-weights"| EI
    UC ---|"shared-services"| DM
    UC ---|"shared-services"| TP
    UC ---|"shared-services"| PM
    UC ---|"shared-services"| EI
    UC ---|"shared-services"| HF

    classDef env fill:#ffeb3b,stroke:#333,stroke-width:2px;
    classDef data fill:#90caf9,stroke:#333,stroke-width:2px;
    classDef pipeline fill:#a5d6a7,stroke:#333,stroke-width:2px;
    classDef model fill:#ffab91,stroke:#333,stroke-width:2px;
    classDef eval fill:#ce93d8,stroke:#333,stroke-width:2px;
    classDef external fill:#bcaaa4,stroke:#333,stroke-width:2px;
    classDef util fill:#f48fb1,stroke:#333,stroke-width:2px;

    click DM "https://github.com/magic-research/pllava/tree/main/dataset/"
    click DM "https://github.com/magic-research/pllava/blob/main/dataset/base_dataset.py"
    click DM "https://github.com/magic-research/pllava/blob/main/dataset/video_utils.py"

    click PM "https://github.com/magic-research/pllava/tree/main/models/pllava/"
    click PM "https://github.com/magic-research/pllava/blob/main/models/pllava/configuration_pllava.py"
    click PM "https://github.com/magic-research/pllava/blob/main/models/pllava/modeling_pllava.py"
    click PM "https://github.com/magic-research/pllava/blob/main/models/pllava/processing_pllava.py"
    click PM "https://github.com/magic-research/pllava/blob/main/models/pllava/convert_pllava_weights_to_hf.py"

    click TP "https://github.com/magic-research/pllava/tree/main/tasks/train/"
    click TP "https://github.com/magic-research/pllava/blob/main/tasks/train/train_pllava_nframe_accel.py"
    click TP "https://github.com/magic-research/pllava/blob/main/tasks/train/instruction_data.py"
    click TP "https://github.com/magic-research/pllava/blob/main/tasks/train/config_pllava_nframe.py"
    click TP "https://github.com/magic-research/pllava/blob/main/tasks/train/config_pllava_nframe_yiprompt.py"
    click TP "https://github.com/magic-research/pllava/blob/main/scripts/train_pllava.sh"

    click EI "https://github.com/magic-research/pllava/tree/main/tasks/eval/"
    click EI "https://github.com/magic-research/pllava/blob/main/tasks/eval/demo/pllava_demo.py"
    click EI "https://github.com/magic-research/pllava/blob/main/tasks/eval/eval_utils.py"
    click EI "https://github.com/magic-research/pllava/blob/main/tasks/eval/model_utils.py"
    click EI "https://github.com/magic-research/pllava/blob/main/scripts/eval.sh"
    click EI "https://github.com/magic-research/pllava/blob/main/scripts/eval_yiprompt.sh"
    click EI "https://github.com/magic-research/pllava/blob/main/scripts/gallery.sh"

    click HF "https://github.com/magic-research/pllava/blob/main/python_scripts/hf.py"
    click HF "https://github.com/magic-research/pllava/blob/main/requirements.txt"

    click UC "https://github.com/magic-research/pllava/tree/main/utils/"
    click UC "https://github.com/magic-research/pllava/blob/main/utils/config.py"
    click UC "https://github.com/magic-research/pllava/blob/main/utils/distributed.py"
    click UC "https://github.com/magic-research/pllava/blob/main/utils/logger.py"
    click UC "https://github.com/magic-research/pllava/blob/main/scripts/accel_config_deepspeed_zero2.yaml"
```
