# fuxiaoliu/lrv-instruction architecture

> Architecture diagram of the GitHub repository fuxiaoliu/lrv-instruction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fuxiaoliu/lrv-instruction
- Repository: https://github.com/fuxiaoliu/lrv-instruction

## Overview

##### Project Type & Purpose
- Type: Research-grade, open-source, multi-modal instruction-tuning toolkit built atop two base model frameworks (MiniGPT-4 & mPLUG-Owl).
- Purpose: Provide scripts, configs, data pipelines, and evaluation tools to fine-tune large vision-and-language models (VLMs) on a robust visual instruction dataset, then evaluate hallucination using GPT-4-assisted GAVIE.

##### File Structure Analysis
Top‐Level Components:
- /MiniGPT-4: Core codebase for version-1 (MiniGPT-4) fine-tuning & inference (configs, models, runners, datasets, processors).
- /model_weight, /train_configs, train.py: Training orchestration for stage-wise fine-tuning.
- /data & /prompts: Data‐generation scripts and prompt templates used to build the visual instruction dataset.
- /Evaluation: GAVIE evaluation pipeline (evaluation_set.json, evaluate.py, prompt).
- /utils: Helper scripts for serving & inference with mPLUG-Owl (model_worker.py, inference.py).
- /examples & /sample_images: Illustration assets for demoing.
- environment.yml, LICENSE, README.md: Environment and documentation.

##### README Insights
- Two pipeline variants:
– V1: Fine-tune MiniGPT-4 + Vicuna-7B via LoRA.
– V2: Fine-tune mPLUG-Owl + LoRA.
- Dataset: 320k+ visual instructions (positive & negative), based on Visual Genome.
- Evaluation: GPT-4 “smart teacher” uses dense captions for hallucination scoring.
- External Dependencies: PyTorch, Transformers, PEFT, Gradio, Hydra-style config.

##### Main Components & Interactions
A. Data Layer
– Instruction Dataset: Generated by data/data_generation.py using prompts/*.txt.
– Filter & alignment metadata in MiniGPT-4/cc_sbu_align.
B. Preprocessing & Dataset Builders
– miniGPT4/minigpt4/datasets/builders: base_dataset_builder.py, image_text_pair_builder.py.
– Data utils to load, filter, augment (randaugment).
C. Model Training & Fine-Tuning
– Two paths (choose via config):
1. Stage 1 & 2 MiniGPT-4 training (train.py + train_configs).
2. PLUG-Owl LoRA fine-tuning (modify serve/model_worker.py in utils).
– Key modules: mini_gpt4/models, tasks/image_text_pretrain.py, runners/runner_base.py.
– Config orchestration: miniGPT4/configs/{models,datasets,default}.
D. Inference & Serving
– MiniGPT-4: inference.py + conversation module.
– mPLUG-Owl: utils/model_worker.py + utils/inference.py + serve/web_server.
– Gradio demos (demo.py or serve.web_server).
E. Evaluation (GAVIE)
– Evaluation/evaluate.py calls GPT-4 API.
– Prompts in prompts/GAVIE.txt define teacher instructions.
– Output: hallucination & relevancy scores per image/instruction.
F. External Services
– GPT-4 API for dataset generation & for GAVIE scoring.
– Hugging Face checkpoints (Vicuna, mPLUG-Owl).

##### Architectural Patterns & Principles
- Modular structure: Separate data builders, model definitions, tasks, and runners.
- Config‐driven: All paths (datasets, models, weights) set via YAML.
- Plugin/LoRA pattern: PEFT hooks inserted into base models.
- Evaluation as a service: decoupled GAVIE module treating GPT-4 as an external scoring engine.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DV["Visual Feature Dataset"]:::data
        ID["Instruction Dataset"]:::data
        DG["Data Generation"]:::data
        PP["Positive Prompts"]:::data
        NP1["Negative1 Prompts"]:::data
        NP2["Negative2 Prompts"]:::data
    end

    PP -->|"uses"| DG
    NP1 -->|"uses"| DG
    NP2 -->|"uses"| DG
    DG -->|"generate"| ID
    DV -->|"source"| ID

    %% Preprocessing Layer
    subgraph "Preprocessing & Builders"
        direction TB
        BDB["BaseDatasetBuilder"]:::pre
        ITPB["ImageTextPairBuilder"]:::pre
        DU["Data Utils"]:::pre
        subgraph "Processors"
            direction TB
            BP["BaseProcessor"]:::pre
            RA["RandAugment"]:::pre
            BLIP["BLIP Processors"]:::pre
        end
    end

    ID -->|"load/transform"| BDB
    ID -->|"load/transform"| ITPB
    BDB -->|"util"| DU
    ITPB -->|"util"| DU
    DU -->|"preprocess"| BP
    DU -->|"augment"| RA
    DU -->|"process"| BLIP

    %% Configuration Layer
    CFG["YAML Configs"]:::config

    CFG -->|configures| BDB
    CFG -->|configures| ITPB
    CFG -->|configures| TrainingV1
    CFG -->|configures| TrainingV2
    CFG -->|configures| Inference
    CFG -->|configures| Evaluation

    %% Training Layer
    subgraph "Training"
        direction TB
        subgraph "V1: MiniGPT-4 Training"
            direction TB
            Runner["Runner (runner_base.py)"]:::train
            TrainScript["train.py"]:::train
            subgraph "Model Definition"
                direction TB
                MG4["MiniGPT-4 Model"]:::train
                BM["Base Model"]:::train
                B2["BLIP2 Module"]:::train
                EVA["EVA Vision"]:::train
                Qf["Q-Former"]:::train
            end
        end
        subgraph "V2: mPLUG-Owl + LoRA"
            direction TB
            MW["Model Worker"]:::train
            IU["Inference Utils"]:::train
        end
    end

    BP -->|feeds| Runner
    DU -->|feeds| Runner
    Runner -->|fine-tune| MG4
    Runner -->|fine-tune| BM
    Runner -->|fine-tune| B2
    Runner -->|fine-tune| EVA
    Runner -->|fine-tune| Qf
    MG4 -->|trained model| Inference
    MW -->|LoRA fine-tune| IU
    IU -->|trained model| Inference

    %% Inference Layer
    subgraph "Inference"
        direction TB
        IF_MG4["MiniGPT-4 Inference"]:::infer
        Demo["Demo (demo.py)"]:::infer
        Conv["Conversation Module"]:::infer
        MW["Model Worker"]:::infer
        INF_UTIL["Inference Utils"]:::infer
    end

    MG4 -->|serve| IF_MG4
    IF_MG4 -->|demo| Demo
    IF_MG4 -->|chat| Conv
    IU -->|serve| MW
    MW -->|infer| INF_UTIL

    %% Evaluation Layer
    subgraph "Evaluation (GAVIE)"
        direction TB
        EvalScript["evaluate.py"]:::eval
        EvalSet["evaluation_set.json"]:::eval
        GPrompt["GAVIE Prompts"]:::eval
        Scores["Hallucination & Relevancy Scores"]:::eval
    end

    Demo -->|outputs| EvalScript
    IF_MG4 -->|outputs| EvalScript
    INF_UTIL -->|outputs| EvalScript
    EvalScript -->|uses| EvalSet
    EvalScript -->|uses| GPrompt
    EvalScript -->|evaluate| GPT4API
    GPT4API -->|returns| Scores

    %% External Services
    subgraph "External Services"
        direction TB
        GPT4API["GPT-4 API"]:::external
        HF["Hugging Face Model Hub"]:::external
    end

    HF -->|provides| BM
    HF -->|provides| B2
    HF -->|provides| EVA
    HF -->|provides| Qf

    %% Click Events
    click DG "https://github.com/fuxiaoliu/lrv-instruction/blob/main/data/data_generation.py"
    click PP "https://github.com/fuxiaoliu/lrv-instruction/blob/main/prompts/positive_generation_prompt.txt"
    click NP1 "https://github.com/fuxiaoliu/lrv-instruction/blob/main/prompts/negative1_generation_prompt.txt"
    click NP2 "https://github.com/fuxiaoliu/lrv-instruction/blob/main/prompts/negative2_generation_prompt.txt"
    click GPrompt "https://github.com/fuxiaoliu/lrv-instruction/blob/main/prompts/GAVIE.txt"
    click BDB "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/datasets/builders/base_dataset_builder.py"
    click ITPB "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/datasets/builders/image_text_pair_builder.py"
    click DU "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/datasets/data_utils.py"
    click BP "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/processors/base_processor.py"
    click RA "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/processors/randaugment.py"
    click BLIP "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/processors/blip_processors.py"
    click CFG "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/configs/default.yaml"
    click Runner "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/runners/runner_base.py"
    click TrainScript "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/train.py"
    click MG4 "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/models/mini_gpt4.py"
    click BM "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/models/base_model.py"
    click B2 "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/models/blip2.py"
    click EVA "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/models/eva_vit.py"
    click Qf "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/models/Qformer.py"
    click MW "https://github.com/fuxiaoliu/lrv-instruction/blob/main/utils/model_worker.py"
    click INF_UTIL "https://github.com/fuxiaoliu/lrv-instruction/blob/main/utils/inference.py"
    click IF_MG4 "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/inference.py"
    click Demo "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/demo.py"
    click Conv "https://github.com/fuxiaoliu/lrv-instruction/blob/main/MiniGPT-4/minigpt4/conversation/conversation.py"
    click EvalScript "https://github.com/fuxiaoliu/lrv-instruction/blob/main/Evaluation/evaluate.py"
    click EvalSet "https://github.com/fuxiaoliu/lrv-instruction/blob/main/Evaluation/evaluation_set.json"

    %% Styles
    classDef data fill:#e5ffe5,stroke:#1b5e20;
    classDef pre fill:#e0f7fa,stroke:#006064;
    classDef config fill:#fff3e0,stroke:#e65100;
    classDef train fill:#ffe0b2,stroke:#ef6c00;
    classDef infer fill:#f3e5f5,stroke:#6a1b9a;
    classDef eval fill:#ffebee,stroke:#b71c1c;
    classDef external fill:#eceff1,stroke:#455a64;
```
