How 2u1/qwen-vl-series-finetune works
This repository is a Qwen vision-language fine-tuning toolkit. Its main workflow takes prepared conversation/image/video data through task-specific datasets and a training entry point, loads a compatible model, then trains with task-specific trainers and saves adapted weights. It also supports classification, DPO/GRPO preference optimization, LoRA merging, and Gradio inference. The README documents these capabilities; edges among implementation components are limited to relationships shown by the SOURCE INDEX or excerpts, so unsampled internals are not expanded.
Main entry points: load_model.py Qwen model loader · train_grpo.py GRPO entry point · train_cls.py Classification entry point · train_sft.py SFT entry point
Components 26
Data preparation
- SFT dataset
src/dataset/sft_dataset.py - DPO dataset
src/dataset/dpo_dataset.py - GRPO dataset
src/dataset/grpo_dataset.py - Classification dataset
src/dataset/cls_dataset.py - Multimodal data utilities
src/dataset/data_utils.py
Models and adaptation
- Qwen model loader
src/model/load_model.py - Classification models
src/model/modeling_cls.py - Mixed-modality forwards
src/train/monkey_patch_forward.py - Vision patch
src/train/monkey_patch_vision.py
Training workflows
- Training parameters
src/params.py - SFT entry point
src/train/train_sft.py - DPO entry point
src/train/train_dpo.py - GRPO entry point
src/train/train_grpo.py - Classification entry point
src/train/train_cls.py - SFT trainer
src/trainer/sft_trainer.py - DPO trainer
src/trainer/dpo_trainer.py - GRPO trainer
src/trainer/grpo_trainer.py - Classification trainer
src/trainer/cls_trainer.py - Classification losses
src/loss/loss_factory.py - Reward functions
src/train/reward_funcs.py - Training utilities
src/train/train_utils.py
Inference and artifacts
- Fine-tuned weights
- LoRA weight merger
src/merge_lora_weights.py - Gradio inference app
src/serve/app.py - Model loading utilities
src/utils.py
Other components
- Fine-tuning user
Connections 42
- Fine-tuning user → SFT entry point: runs SFT
README.md - Fine-tuning user → DPO entry point: runs DPO
README.md - Fine-tuning user → GRPO entry point: runs GRPO
README.md - Fine-tuning user → Classification entry point: runs classification
README.md - SFT entry point → Training parameters: parses arguments
src/train/train_sft.py - DPO entry point → Training parameters: parses arguments
src/train/train_dpo.py - GRPO entry point → Training parameters: parses arguments
src/train/train_grpo.py - Classification entry point → Training parameters: parses arguments
src/train/train_cls.py - SFT entry point → SFT dataset: builds data
src/train/train_sft.py - DPO entry point → DPO dataset: builds data
src/train/train_dpo.py - GRPO entry point → GRPO dataset: builds data
src/train/train_grpo.py - Classification entry point → Classification dataset: builds data
src/train/train_cls.py - SFT dataset → Multimodal data utilities: uses multimodal helpers
src/dataset/sft_dataset.py - DPO dataset → Multimodal data utilities: uses multimodal helpers
src/dataset/dpo_dataset.py - GRPO dataset → Multimodal data utilities: uses multimodal helpers
src/dataset/grpo_dataset.py - Classification dataset → Multimodal data utilities: uses multimodal helpers
src/dataset/cls_dataset.py - SFT entry point → Qwen model loader: loads model
src/train/train_sft.py - DPO entry point → Qwen model loader: loads model
src/train/train_dpo.py - GRPO entry point → Qwen model loader: loads model
src/train/train_grpo.py - Classification entry point → Qwen model loader: loads model
src/train/train_cls.py - Qwen model loader → Classification models: selects classifier
src/model/load_model.py - Qwen model loader → Mixed-modality forwards: applies forward patches
src/model/load_model.py - Qwen model loader → Vision patch: applies vision patch
src/model/load_model.py - SFT entry point → SFT trainer: creates trainer
src/train/train_sft.py - DPO entry point → DPO trainer: creates trainer
src/train/train_dpo.py - GRPO entry point → GRPO trainer: creates trainer
src/train/train_grpo.py - Classification entry point → Classification trainer: creates trainer
src/train/train_cls.py - Classification entry point → Classification losses: configures loss
src/train/train_cls.py - GRPO entry point → Reward functions: loads rewards
src/train/train_grpo.py - SFT trainer → Training utilities: uses save helpers
src/trainer/sft_trainer.py - DPO trainer → Training utilities: uses save helpers
src/trainer/dpo_trainer.py - GRPO trainer → Training utilities: uses save helpers
src/trainer/grpo_trainer.py - Classification trainer → Training utilities: uses save helpers
src/trainer/cls_trainer.py - SFT entry point → Fine-tuned weights: saves checkpoints
README.md - DPO entry point → Fine-tuned weights: saves checkpoints
README.md - GRPO entry point → Fine-tuned weights: saves checkpoints
README.md - Classification entry point → Fine-tuned weights: saves checkpoints
README.md - Fine-tuning user → LoRA weight merger: merges adapters
README.md - LoRA weight merger → Fine-tuned weights: writes merged weights
src/merge_lora_weights.py - Fine-tuning user → Gradio inference app: uses web UI
README.md - Gradio inference app → Model loading utilities: loads model
src/serve/app.py - Model loading utilities → Qwen model loader: loads Qwen backbone
src/utils.py