How fuxiaoliu/lrv-instruction works
Project Type & Purpose
• Type: Research-grade, open-source, multi-modal instruction-tuning toolkit built atop two base model frameworks (MiniGPT-4 & mPLUG-Owl).
• Purpose: Provide scripts, configs, data pipelines, and evaluation tools to fine-tune large vision-and-language models (VLMs) on a robust visual instruction dataset, then evaluate hallucination using GPT-4-assisted GAVIE.
File Structure Analysis
Top‐Level Components:
• /MiniGPT-4: Core codebase for version-1 (MiniGPT-4) fine-tuning & inference (configs, models, runners, datasets, processors).
• /model_weight, /train_configs, train.py: Training orchestration for stage-wise fine-tuning.
• /data & /prompts: Data‐generation scripts and prompt templates used to build the visual instruction dataset.
• /Evaluation: GAVIE evaluation pipeline (evaluation_set.json, evaluate.py, prompt).
• /utils: Helper scripts for serving & inference with mPLUG-Owl (model_worker.py, inference.py).
• /examples & /sample_images: Illustration assets for demoing.
• environment.yml, LICENSE, README.md: Environment and documentation.
README Insights
• Two pipeline variants:
– V1: Fine-tune MiniGPT-4 + Vicuna-7B via LoRA.
– V2: Fine-tune mPLUG-Owl + LoRA.
• Dataset: 320k+ visual instructions (positive & negative), based on Visual Genome.
• Evaluation: GPT-4 “smart teacher” uses dense captions for hallucination scoring.
• External Dependencies: PyTorch, Transformers, PEFT, Gradio, Hydra-style config.
Main Components & Interactions
A. Data Layer
– Instruction Dataset: Generated by data/data_generation.py using prompts/*.txt.
– Filter & alignment metadata in MiniGPT-4/cc_sbu_align.
B. Preprocessing & Dataset Builders
– miniGPT4/minigpt4/datasets/builders: base_dataset_builder.py, image_text_pair_builder.py.
– Data utils to load, filter, augment (randaugment).
C. Model Training & Fine-Tuning
– Two paths (choose via config):
1. Stage 1 & 2 MiniGPT-4 training (train.py + train_configs).
2. PLUG-Owl LoRA fine-tuning (modify serve/model_worker.py in utils).
– Key modules: mini_gpt4/models, tasks/image_text_pretrain.py, runners/runner_base.py.
– Config orchestration: miniGPT4/configs/{models,datasets,default}.
D. Inference & Serving
– MiniGPT-4: inference.py + conversation module.
– mPLUG-Owl: utils/model_worker.py + utils/inference.py + serve/web_server.
– Gradio demos (demo.py or serve.web_server).
E. Evaluation (GAVIE)
– Evaluation/evaluate.py calls GPT-4 API.
– Prompts in prompts/GAVIE.txt define teacher instructions.
– Output: hallucination & relevancy scores per image/instruction.
F. External Services
– GPT-4 API for dataset generation & for GAVIE scoring.
– Hugging Face checkpoints (Vicuna, mPLUG-Owl).
Architectural Patterns & Principles
• Modular structure: Separate data builders, model definitions, tasks, and runners.
• Config‐driven: All paths (datasets, models, weights) set via YAML.
• Plugin/LoRA pattern: PEFT hooks inserted into base models.
• Evaluation as a service: decoupled GAVIE module treating GPT-4 as an external scoring engine.