# 2u1/qwen-vl-series-finetune architecture

> Architecture diagram of the GitHub repository 2u1/qwen-vl-series-finetune, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/2u1/qwen-vl-series-finetune
- Repository: https://github.com/2u1/qwen-vl-series-finetune

## Overview

This repository is a Qwen vision-language fine-tuning toolkit. Its main workflow takes prepared conversation/image/video data through task-specific datasets and a training entry point, loads a compatible model, then trains with task-specific trainers and saves adapted weights. It also supports classification, DPO/GRPO preference optimization, LoRA merging, and Gradio inference. The README documents these capabilities; edges among implementation components are limited to relationships shown by the SOURCE INDEX or excerpts, so unsampled internals are not expanded.

Main entry points: [`src/model/load_model.py`](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/model/load_model.py) (Qwen model loader), [`src/train/train_grpo.py`](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/train_grpo.py) (GRPO entry point), [`src/train/train_cls.py`](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/train_cls.py) (Classification entry point), [`src/train/train_sft.py`](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/train_sft.py) (SFT entry point)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_sft_data["SFT dataset<br/>[sft_dataset.py]"]
  node_dpo_data["DPO dataset<br/>[dpo_dataset.py]"]
  node_grpo_data["GRPO dataset<br/>[grpo_dataset.py]"]
  node_cls_data["Classification dataset<br/>[cls_dataset.py]"]
  node_data_utils["Multimodal data utilities<br/>[data_utils.py]"]
end

subgraph group_models["Models and adaptation"]
  node_model_loader["Qwen model loader<br/>[load_model.py]"]
  node_cls_model["Classification models<br/>[modeling_cls.py]"]
  node_forward_patch["Mixed-modality forwards"]
  node_vision_patch["Vision patch"]
end

subgraph group_training["Training workflows"]
  node_params["Training parameters<br/>[params.py]"]
  node_sft_entry["SFT entry point<br/>[train_sft.py]"]
  node_dpo_entry["DPO entry point<br/>[train_dpo.py]"]
  node_grpo_entry["GRPO entry point<br/>[train_grpo.py]"]
  node_cls_entry["Classification entry point<br/>[train_cls.py]"]
  node_sft_trainer["SFT trainer<br/>[sft_trainer.py]"]
  node_dpo_trainer["DPO trainer<br/>[dpo_trainer.py]"]
  node_grpo_trainer["GRPO trainer<br/>[grpo_trainer.py]"]
  node_cls_trainer["Classification trainer<br/>[cls_trainer.py]"]
  node_loss_factory["Classification losses<br/>[loss_factory.py]"]
  node_reward_funcs["Reward functions<br/>[reward_funcs.py]"]
  node_train_utils["Training utilities<br/>[train_utils.py]"]
end

subgraph group_runtime["Inference and artifacts"]
  node_saved_weights["Fine-tuned weights"]
  node_merge["LoRA weight merger"]
  node_inference["Gradio inference app<br/>[app.py]"]
  node_model_utils["Model loading utilities<br/>[utils.py]"]
end

node_engineer(("Fine-tuning user"))

node_engineer -->|"runs SFT"| node_sft_entry
node_engineer -->|"runs DPO"| node_dpo_entry
node_engineer -->|"runs GRPO"| node_grpo_entry
node_engineer -->|"runs classification"| node_cls_entry
node_sft_entry -->|"parses arguments"| node_params
node_dpo_entry -->|"parses arguments"| node_params
node_grpo_entry -->|"parses arguments"| node_params
node_cls_entry -->|"parses arguments"| node_params
node_sft_entry -->|"builds data"| node_sft_data
node_dpo_entry -->|"builds data"| node_dpo_data
node_grpo_entry -->|"builds data"| node_grpo_data
node_cls_entry -->|"builds data"| node_cls_data
node_sft_data -->|"uses multimodal helpers"| node_data_utils
node_dpo_data -->|"uses multimodal helpers"| node_data_utils
node_grpo_data -->|"uses multimodal helpers"| node_data_utils
node_cls_data -->|"uses multimodal helpers"| node_data_utils
node_sft_entry -->|"loads model"| node_model_loader
node_dpo_entry -->|"loads model"| node_model_loader
node_grpo_entry -->|"loads model"| node_model_loader
node_cls_entry -->|"loads model"| node_model_loader
node_model_loader -->|"selects classifier"| node_cls_model
node_model_loader -->|"applies forward patches"| node_forward_patch
node_model_loader -->|"applies vision patch"| node_vision_patch
node_sft_entry -->|"creates trainer"| node_sft_trainer
node_dpo_entry -->|"creates trainer"| node_dpo_trainer
node_grpo_entry -->|"creates trainer"| node_grpo_trainer
node_cls_entry -->|"creates trainer"| node_cls_trainer
node_cls_entry -->|"configures loss"| node_loss_factory
node_grpo_entry -->|"loads rewards"| node_reward_funcs
node_sft_trainer -->|"uses save helpers"| node_train_utils
node_dpo_trainer -->|"uses save helpers"| node_train_utils
node_grpo_trainer -->|"uses save helpers"| node_train_utils
node_cls_trainer -->|"uses save helpers"| node_train_utils
node_sft_entry -->|"saves checkpoints"| node_saved_weights
node_dpo_entry -->|"saves checkpoints"| node_saved_weights
node_grpo_entry -->|"saves checkpoints"| node_saved_weights
node_cls_entry -->|"saves checkpoints"| node_saved_weights
node_engineer -->|"merges adapters"| node_merge
node_merge -->|"writes merged weights"| node_saved_weights
node_engineer -->|"uses web UI"| node_inference
node_inference -->|"loads model"| node_model_utils
node_model_utils -->|"loads Qwen backbone"| node_model_loader

click node_sft_data "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/dataset/sft_dataset.py"
click node_dpo_data "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/dataset/dpo_dataset.py"
click node_grpo_data "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/dataset/grpo_dataset.py"
click node_cls_data "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/dataset/cls_dataset.py"
click node_data_utils "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/dataset/data_utils.py"
click node_params "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/params.py"
click node_model_loader "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/model/load_model.py"
click node_cls_model "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/model/modeling_cls.py"
click node_forward_patch "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/monkey_patch_forward.py"
click node_vision_patch "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/monkey_patch_vision.py"
click node_sft_entry "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/train_sft.py"
click node_dpo_entry "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/train_dpo.py"
click node_grpo_entry "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/train_grpo.py"
click node_cls_entry "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/train_cls.py"
click node_sft_trainer "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/trainer/sft_trainer.py"
click node_dpo_trainer "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/trainer/dpo_trainer.py"
click node_grpo_trainer "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/trainer/grpo_trainer.py"
click node_cls_trainer "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/trainer/cls_trainer.py"
click node_loss_factory "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/loss/loss_factory.py"
click node_reward_funcs "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/reward_funcs.py"
click node_train_utils "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/train_utils.py"
click node_merge "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/merge_lora_weights.py"
click node_inference "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/serve/app.py"
click node_model_utils "https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sft_data,node_dpo_data,node_grpo_data,node_cls_data,node_data_utils,node_engineer toneBlue
class node_model_loader,node_cls_model,node_forward_patch,node_vision_patch toneAmber
class node_params,node_sft_entry,node_dpo_entry,node_grpo_entry,node_cls_entry,node_sft_trainer,node_dpo_trainer,node_grpo_trainer,node_cls_trainer,node_loss_factory,node_reward_funcs,node_train_utils toneMint
class node_saved_weights,node_merge,node_inference,node_model_utils toneRose
```

## Components

### Data preparation

- [SFT dataset](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/dataset/sft_dataset.py): `src/dataset/sft_dataset.py`
- [DPO dataset](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/dataset/dpo_dataset.py): `src/dataset/dpo_dataset.py`
- [GRPO dataset](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/dataset/grpo_dataset.py): `src/dataset/grpo_dataset.py`
- [Classification dataset](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/dataset/cls_dataset.py): `src/dataset/cls_dataset.py`
- [Multimodal data utilities](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/dataset/data_utils.py): `src/dataset/data_utils.py`

### Models and adaptation

- [Qwen model loader](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/model/load_model.py): `src/model/load_model.py`
- [Classification models](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/model/modeling_cls.py): `src/model/modeling_cls.py`
- [Mixed-modality forwards](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/monkey_patch_forward.py): `src/train/monkey_patch_forward.py`
- [Vision patch](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/monkey_patch_vision.py): `src/train/monkey_patch_vision.py`

### Training workflows

- [Training parameters](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/params.py): `src/params.py`
- [SFT entry point](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/train_sft.py): `src/train/train_sft.py`
- [DPO entry point](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/train_dpo.py): `src/train/train_dpo.py`
- [GRPO entry point](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/train_grpo.py): `src/train/train_grpo.py`
- [Classification entry point](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/train_cls.py): `src/train/train_cls.py`
- [SFT trainer](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/trainer/sft_trainer.py): `src/trainer/sft_trainer.py`
- [DPO trainer](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/trainer/dpo_trainer.py): `src/trainer/dpo_trainer.py`
- [GRPO trainer](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/trainer/grpo_trainer.py): `src/trainer/grpo_trainer.py`
- [Classification trainer](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/trainer/cls_trainer.py): `src/trainer/cls_trainer.py`
- [Classification losses](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/loss/loss_factory.py): `src/loss/loss_factory.py`
- [Reward functions](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/reward_funcs.py): `src/train/reward_funcs.py`
- [Training utilities](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/train/train_utils.py): `src/train/train_utils.py`

### Inference and artifacts

- **Fine-tuned weights**
- [LoRA weight merger](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/merge_lora_weights.py): `src/merge_lora_weights.py`
- [Gradio inference app](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/serve/app.py): `src/serve/app.py`
- [Model loading utilities](https://github.com/2u1/qwen-vl-series-finetune/blob/master/src/utils.py): `src/utils.py`

### Other components

- **Fine-tuning user**

## Connections

- Fine-tuning user → SFT entry point: runs SFT (evidence: `README.md`)
- Fine-tuning user → DPO entry point: runs DPO (evidence: `README.md`)
- Fine-tuning user → GRPO entry point: runs GRPO (evidence: `README.md`)
- Fine-tuning user → Classification entry point: runs classification (evidence: `README.md`)
- SFT entry point → Training parameters: parses arguments (evidence: `src/train/train_sft.py`)
- DPO entry point → Training parameters: parses arguments (evidence: `src/train/train_dpo.py`)
- GRPO entry point → Training parameters: parses arguments (evidence: `src/train/train_grpo.py`)
- Classification entry point → Training parameters: parses arguments (evidence: `src/train/train_cls.py`)
- SFT entry point → SFT dataset: builds data (evidence: `src/train/train_sft.py`)
- DPO entry point → DPO dataset: builds data (evidence: `src/train/train_dpo.py`)
- GRPO entry point → GRPO dataset: builds data (evidence: `src/train/train_grpo.py`)
- Classification entry point → Classification dataset: builds data (evidence: `src/train/train_cls.py`)
- SFT dataset → Multimodal data utilities: uses multimodal helpers (evidence: `src/dataset/sft_dataset.py`)
- DPO dataset → Multimodal data utilities: uses multimodal helpers (evidence: `src/dataset/dpo_dataset.py`)
- GRPO dataset → Multimodal data utilities: uses multimodal helpers (evidence: `src/dataset/grpo_dataset.py`)
- Classification dataset → Multimodal data utilities: uses multimodal helpers (evidence: `src/dataset/cls_dataset.py`)
- SFT entry point → Qwen model loader: loads model (evidence: `src/train/train_sft.py`)
- DPO entry point → Qwen model loader: loads model (evidence: `src/train/train_dpo.py`)
- GRPO entry point → Qwen model loader: loads model (evidence: `src/train/train_grpo.py`)
- Classification entry point → Qwen model loader: loads model (evidence: `src/train/train_cls.py`)
- Qwen model loader → Classification models: selects classifier (evidence: `src/model/load_model.py`)
- Qwen model loader → Mixed-modality forwards: applies forward patches (evidence: `src/model/load_model.py`)
- Qwen model loader → Vision patch: applies vision patch (evidence: `src/model/load_model.py`)
- SFT entry point → SFT trainer: creates trainer (evidence: `src/train/train_sft.py`)
- DPO entry point → DPO trainer: creates trainer (evidence: `src/train/train_dpo.py`)
- GRPO entry point → GRPO trainer: creates trainer (evidence: `src/train/train_grpo.py`)
- Classification entry point → Classification trainer: creates trainer (evidence: `src/train/train_cls.py`)
- Classification entry point → Classification losses: configures loss (evidence: `src/train/train_cls.py`)
- GRPO entry point → Reward functions: loads rewards (evidence: `src/train/train_grpo.py`)
- SFT trainer → Training utilities: uses save helpers (evidence: `src/trainer/sft_trainer.py`)
- DPO trainer → Training utilities: uses save helpers (evidence: `src/trainer/dpo_trainer.py`)
- GRPO trainer → Training utilities: uses save helpers (evidence: `src/trainer/grpo_trainer.py`)
- Classification trainer → Training utilities: uses save helpers (evidence: `src/trainer/cls_trainer.py`)
- SFT entry point → Fine-tuned weights: saves checkpoints (evidence: `README.md`)
- DPO entry point → Fine-tuned weights: saves checkpoints (evidence: `README.md`)
- GRPO entry point → Fine-tuned weights: saves checkpoints (evidence: `README.md`)
- Classification entry point → Fine-tuned weights: saves checkpoints (evidence: `README.md`)
- Fine-tuning user → LoRA weight merger: merges adapters (evidence: `README.md`)
- LoRA weight merger → Fine-tuned weights: writes merged weights (evidence: `src/merge_lora_weights.py`)
- Fine-tuning user → Gradio inference app: uses web UI (evidence: `README.md`)
- Gradio inference app → Model loading utilities: loads model (evidence: `src/serve/app.py`)
- Model loading utilities → Qwen model loader: loads Qwen backbone (evidence: `src/utils.py`)
