How jingyaogong/minimind-v works
Project type and purpose
• This is neither a traditional webapp nor microservices suite, but a research-oriented, full ML pipeline for training and serving a compact Vision-Language Model (VLM).
• Key goal: from raw image + instruction data ⇒ pretrain & finetune a VLM (MiniMind-V) ⇒ export weights ⇒ evaluate & demo via CLI or Web UI.
File structure
• dataset/
• lm_dataset.py: Data loader for JSONL “conversations” + image references
• eval_images, eval_multi_images: sample visuals for evaluation
• model/
• model_minimind.py: base LLM architecture
• model_vlm.py: VLM subclass adding Visual Encoder + projection layers
• vision_model/: CLIP Vit encoder submodule (external git clone)
• tokenizer.json, tokenizer_config.json
• trainer/
• train_pretrain_vlm.py: pretraining pipeline
• train_sft_vlm.py: supervised finetune pipeline
• scripts/
• convert_vlm.py: weight conversion utilities
• web_demo_vlm.py: lightweight FastAPI/Flask demo for Chat+Image
• Top level
• eval_vlm.py: CLI evaluation harness
• requirements.txt, README files
Extract architecture from README
• Two-stage training: Pretrain (image description) ⇒ SFT (instruction following)
• Core model = Huggingface CLIP Vit (Visual Encoder) + tiny GPT-style LLM (MiniMind) + linear projection
• Inference modes: CLI (eval_vlm.py) or simple Web UI (web_demo_vlm.py)
Main components & interactions
a. Data Ingestion & Preprocessing
• Raw JSONL + images → lm_dataset.py → PyTorch DataLoader
b. Training Pipelines
• Pretrain: train_pretrain_vlm.py → loads Visual Encoder (frozen) + projection (+ last LLM layer) → outputs pretrain weights (*.pth)
• SFT: train_sft_vlm.py → loads pretrain weights → finetunes full projection + LLM → outputs sft weights (*.pth)
• Orchestrated via torchrun/DDP or deepspeed
c. Model Definition
• Visual Encoder (CLIP Vit)
• Projection layers (768→LLM embedding dim)
• MiniMind LLM (GPT-style Transformer)
• Tokenizer (text tokens + “@@@…@@@” image placeholders)
d. Weight Conversion & Export
• convert_vlm.py: convert PyTorch .pth → Huggingface style if needed
e. Inference & Serving
• eval_vlm.py (CLI)
• web_demo_vlm.py (Web UI)
• Both load:
– Visual Encoder & projection + LLM
– Tokenizer
– Weights from /out or Huggingface repo
f. External Services
• Huggingface/ModelScope: initial CLIP model & base LLM weights + dataset storage
Architectural patterns & design principles
• Modular separation:
– dataset (I/O)
– model (definition)
– trainer (compute pipelines)
– scripts (conversion & serving)
• Pipeline pattern: sequential stages (Pretrain → SFT → Eval/Serve)
• Dependency injection: external CLIP & base LLM loaded at runtime
• Scalability: DDP/DeepSpeed for multi-GPU training
Technologies & frameworks
• Python 3.10+, PyTorch
• Huggingface Transformers tokenizer module
• CLIP Vit from openai/clip-vit-base-patch16
• torchrun / deepspeed for distributed training
• Optional: wandb for logging
• Web UI: Flask/FastAPI (in web_demo_vlm.py)