# jingyaogong/minimind-v architecture

> Architecture diagram of the GitHub repository jingyaogong/minimind-v, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jingyaogong/minimind-v
- Repository: https://github.com/jingyaogong/minimind-v

## Overview

##### Project type and purpose
- This is neither a traditional webapp nor microservices suite, but a research-oriented, full ML pipeline for training and serving a compact Vision-Language Model (VLM).
- Key goal: from raw image + instruction data ⇒ pretrain & finetune a VLM (MiniMind-V) ⇒ export weights ⇒ evaluate & demo via CLI or Web UI.

##### File structure
- dataset/
- lm_dataset.py: Data loader for JSONL “conversations” + image references
- eval_images, eval_multi_images: sample visuals for evaluation
- model/
- model_minimind.py: base LLM architecture
- model_vlm.py: VLM subclass adding Visual Encoder + projection layers
- vision_model/: CLIP Vit encoder submodule (external git clone)
- tokenizer.json, tokenizer_config.json
- trainer/
- train_pretrain_vlm.py: pretraining pipeline
- train_sft_vlm.py: supervised finetune pipeline
- scripts/
- convert_vlm.py: weight conversion utilities
- web_demo_vlm.py: lightweight FastAPI/Flask demo for Chat+Image
- Top level
- eval_vlm.py: CLI evaluation harness
- requirements.txt, README files

##### Extract architecture from README
- Two-stage training: Pretrain (image description) ⇒ SFT (instruction following)
- Core model = Huggingface CLIP Vit (Visual Encoder) + tiny GPT-style LLM (MiniMind) + linear projection
- Inference modes: CLI (eval_vlm.py) or simple Web UI (web_demo_vlm.py)

##### Main components & interactions
a. Data Ingestion & Preprocessing
- Raw JSONL + images → lm_dataset.py → PyTorch DataLoader
b. Training Pipelines
- Pretrain: train_pretrain_vlm.py → loads Visual Encoder (frozen) + projection (+ last LLM layer) → outputs pretrain weights (*.pth)
- SFT: train_sft_vlm.py → loads pretrain weights → finetunes full projection + LLM → outputs sft weights (*.pth)
- Orchestrated via torchrun/DDP or deepspeed
c. Model Definition
- Visual Encoder (CLIP Vit)
- Projection layers (768→LLM embedding dim)
- MiniMind LLM (GPT-style Transformer)
- Tokenizer (text tokens + “@@@…@@@” image placeholders)
d. Weight Conversion & Export
- convert_vlm.py: convert PyTorch .pth → Huggingface style if needed
e. Inference & Serving
- eval_vlm.py (CLI)
- web_demo_vlm.py (Web UI)
- Both load:
– Visual Encoder & projection + LLM
– Tokenizer
– Weights from /out or Huggingface repo
f. External Services
- Huggingface/ModelScope: initial CLIP model & base LLM weights + dataset storage

##### Architectural patterns & design principles
- Modular separation:
– dataset (I/O)
– model (definition)
– trainer (compute pipelines)
– scripts (conversion & serving)
- Pipeline pattern: sequential stages (Pretrain → SFT → Eval/Serve)
- Dependency injection: external CLIP & base LLM loaded at runtime
- Scalability: DDP/DeepSpeed for multi-GPU training

##### Technologies & frameworks
- Python 3.10+, PyTorch
- Huggingface Transformers tokenizer module
- CLIP Vit from openai/clip-vit-base-patch16
- torchrun / deepspeed for distributed training
- Optional: wandb for logging
- Web UI: Flask/FastAPI (in web_demo_vlm.py)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        RawData["Raw JSONL & Images"]:::data
        DataLoader["lm_dataset.py\n(Data Loader)"]:::data
        EvalImages["eval_images"]:::data
        EvalMultiImages["eval_multi_images"]:::data
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        VisualEncoder["CLIP ViT (frozen)"]:::model
        Projection["Projection Layer\n(768→LLM dim)"]:::model
        LLM["MiniMind GPT LLM"]:::model
        VLMDef["model_vlm.py\n(VLM Definition)"]:::model
        Tokenizer["tokenizer.json &\ntokenizer_config.json"]:::model
    end

    %% Training Pipelines
    subgraph "Training Pipelines"
        direction TB
        Pretrain["train_pretrain_vlm.py\n(Pretrain)"]:::train
        SFT["train_sft_vlm.py\n(SFT)"]:::train
    end

    %% Weight Storage
    Weights["/out weights (pth)"]:::storage

    %% Inference & Serving
    subgraph "Inference & Serving"
        direction TB
        EvalCLI["eval_vlm.py\n(CLI)"]:::inference
        WebDemo["web_demo_vlm.py\n(Web UI)"]:::inference
        Convert["convert_vlm.py\n(Weight Conversion)"]:::inference
    end

    %% External Services
    subgraph "External Services"
        direction TB
        Huggingface["Huggingface/ModelScope\n(CLIP & LLM weights)"]:::external
    end

    %% Connections: Data to Training
    RawData --> DataLoader
    DataLoader --> Pretrain
    DataLoader --> SFT

    %% Model to Training
    VisualEncoder --> Pretrain
    Projection --> Pretrain
    LLM --> Pretrain
    Tokenizer --> Pretrain

    VisualEncoder --> SFT
    Projection --> SFT
    LLM --> SFT
    Tokenizer --> SFT

    %% Training to Storage
    Pretrain --> Weights
    SFT --> Weights

    %% Inference & Serving flows
    Weights --> EvalCLI
    Weights --> WebDemo
    VisualEncoder --> EvalCLI
    Projection --> EvalCLI
    LLM --> EvalCLI
    Tokenizer --> EvalCLI

    VisualEncoder --> WebDemo
    Projection --> WebDemo
    LLM --> WebDemo
    Tokenizer --> WebDemo

    %% Conversion utility
    Weights --> Convert

    %% External to Model
    Huggingface --> VisualEncoder
    Huggingface --> LLM

    %% Demo data
    EvalImages --> EvalCLI
    EvalMultiImages --> EvalCLI

    %% Click Events
    click DataLoader "https://github.com/jingyaogong/minimind-v/blob/master/dataset/lm_dataset.py"
    click EvalImages "https://github.com/jingyaogong/minimind-v/tree/master/dataset/eval_images"
    click EvalMultiImages "https://github.com/jingyaogong/minimind-v/tree/master/dataset/eval_multi_images"
    click Pretrain "https://github.com/jingyaogong/minimind-v/blob/master/trainer/train_pretrain_vlm.py"
    click SFT "https://github.com/jingyaogong/minimind-v/blob/master/trainer/train_sft_vlm.py"
    click LLM "https://github.com/jingyaogong/minimind-v/blob/master/model/model_minimind.py"
    click VLMDef "https://github.com/jingyaogong/minimind-v/blob/master/model/model_vlm.py"
    click VisualEncoder "https://github.com/jingyaogong/minimind-v/tree/master/model/vision_model/"
    click Tokenizer "https://github.com/jingyaogong/minimind-v/blob/master/model/tokenizer.json"
    click Tokenizer "https://github.com/jingyaogong/minimind-v/blob/master/model/tokenizer_config.json"
    click EvalCLI "https://github.com/jingyaogong/minimind-v/blob/master/eval_vlm.py"
    click WebDemo "https://github.com/jingyaogong/minimind-v/blob/master/scripts/web_demo_vlm.py"
    click Convert "https://github.com/jingyaogong/minimind-v/blob/master/scripts/convert_vlm.py"

    %% Styles
    classDef data fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef model fill:#d4edda,stroke:#333,stroke-width:1px
    classDef train fill:#ffe5b4,stroke:#333,stroke-width:1px
    classDef inference fill:#e5ccff,stroke:#333,stroke-width:1px
    classDef external fill:#e2e3e5,stroke:#333,stroke-width:1px
    classDef storage fill:#f7f7f7,stroke:#333,stroke-width:1px
```
