How khurramhashmi/llava-v1.6-mistral-7b-finetune-orpo-rlaif-v works
Based on our analysis of the file tree and README, this project is a specialized fork aimed at fine-tuning the LLaVA 1.6 – Mistral-7B model using ORPO (Optimized Rank Preference Optimization) with data derived from RLAIF-V (and optionally HH-RLHF), employing parameter-efficient methods such as LoRA and QLoRA. Essentially, this is a multimodal fine-tuning pipeline built around a foundation from the LLaVA repository. The system supports everything from data preprocessing, end-to-end training, evaluation, and interactive deployment via Gradio.
Main Components & Their Roles
• Data Preparation & Dataset Management:
• Scripts such as create_splits.py and download_hh-rlhf_dataset.py indicate that dataset splitting and preparation are important.
• Training Pipeline:
• The training code resides in llava/train (train.py, train_orpo.py, train_orpo_RLAIF_HH-RLHF.py), along with specific fine-tuning scripts and supporting modules (flash attention patches, etc.).
• LoRA adapter integration and ORPO adjustments in this layer.
• Model Architecture & Modules:
• The llava/model directory details how base and delta (adapter) parts are built. Here you specify the language model components (e.g., llava_llama.py, llava_mistral.py, llava_mpt.py) and multimodal encoder/projector modules.
• It’s beneficial to denote these as “Base Model”, “LoRA Adapter”, and “Merged Model” components.
• Evaluation:
• The llava/eval folder contains multiple scripts for evaluation specific to text, vision-QA, and summarization.
• Separate evaluation workflows (e.g., eval_lora_orpo_rlaif.py, model_vqa*.py) along with output tables and visual feedback (webpage, dialogs, charts).
• Serving & Deployment:
• The llava/serve directory includes server components, CLI, web server (Gradio-based interface) and worker registration.
• That once fine-tuning completes, there’s a step to merge the LoRA adapter with base model weights (via peft.merge_adapters) before deploying a Gradio app (gradio_app.py).
• DevOps and Environment Setup:
• The .devcontainer and .github directories indicate that there are configurations for a containerized development environment and project collaboration (issue templates, etc.).
• CI/CD integration to automate tests and deployments.
Relationships and Interactions
• Data Flow:
• Dataset preparation scripts feed data into the training pipelines.
• Training outputs generate checkpoints and LoRA adapters that are then evaluated by the evaluation scripts.
• Model Flow:
• The base model is loaded, then fine-tuned using scripts in llava/train, with intermediary LoRA components maintained during the training phase.
• After training, a merge process creates the final, deployable model that feeds into the serving layer (Gradio demo).
• Service Communication:
• That the evaluation scripts may report performance to external services (e.g., wandb).
Architectural Patterns & Key Design Principles
• Modularity:
• Clearly separate modules for data preparation, training, evaluation, and serving. This separation makes the model reusable and simplifies debugging.
• Parameter-Efficient Fine-Tuning:
• Use of LoRA and QLoRA techniques, showing how the fine-tuning mechanism is modularized away from the base model (as seen in the llava/model and train directory).
• Scalability & Extensibility:
• The use of Docker containers (.devcontainer folder) ensures that the environment is easily reproducible.
• CI/CD configuration and modular test/workflow scripts in GitHub (.github folder) indicate a modern development workflow.
• Avoid over-complication by grouping scripts that handle similar functions into a single component (e.g., merging multiple evaluation scripts into an “Evaluation Suite” block).
Happy diagramming!