How wangbo-7423/finetune works
This repository documents and implements GSM8K fine-tuning and evaluation for Qwen3.5-4B. The main workflow prepares datasets, trains a LoRA SFT adapter, applies rule-reward GRPO, merges both adapters back into a complete model, and evaluates results; chat and visualization support inspection. Preference-pair data is prepared for a reward-model stage, but the README says GRPO uses rule rewards and no reward-model trainer is present. Edges reflect shown code or README statements; the source excerpts do not establish additional runtime wiring.
Main entry points: train_grpo.py Rule-reward GRPO · merge_grpo.py Adapter merge · prepare_gsm8k.py Dataset preparation · strain_sft.py SFT training
Components 16
GSM8K data
- Source GSM8K
datasets/gsm8k - Dataset preparation
prepare_gsm8k.py - Prepared splits
data/gsm8k - Preference pairs
data/gsm8k/rm_train.jsonl
Model training
- SFT training
qwen-finetune/strain_sft.py - Rule-reward GRPO
qwen-finetune/train_grpo.py
Inference and evaluation
- GSM8K evaluation
qwen-finetune/eval_gsm8k.py - Evaluation results
qwen-finetune/outputs - Interactive chat
qwen-finetune/chat.py - Result visualization
qwen-finetune/visualize.py
Model artifacts
- Qwen base model
- SFT adapter
qwen-finetune/outputs/sft - GRPO adapter
qwen-finetune/outputs/grpo - Adapter merge
qwen-finetune/merge_grpo.py - Merged model
qwen-finetune/outputs/grpo_merged
Other components
- Engineer
Connections 25
- Engineer → Dataset preparation: runs preparation
README.md - Dataset preparation → Source GSM8K: reads parquet
prepare_gsm8k.py - Dataset preparation → Prepared splits: writes splits
prepare_gsm8k.py - Dataset preparation → Preference pairs: builds pairs
prepare_gsm8k.py - Engineer → SFT training: starts training
README.md - SFT training → Qwen base model: loads base
qwen-finetune/strain_sft.py - SFT training → Prepared splits: loads SFT splits
qwen-finetune/strain_sft.py - SFT training → SFT adapter: saves adapter
qwen-finetune/strain_sft.py - Engineer → Rule-reward GRPO: starts training
README.md - Rule-reward GRPO → Qwen base model: loads base
qwen-finetune/train_grpo.py - Rule-reward GRPO → SFT adapter: merges SFT
qwen-finetune/train_grpo.py - Rule-reward GRPO → Prepared splits: loads RL prompts
qwen-finetune/train_grpo.py - Rule-reward GRPO → GRPO adapter: saves adapter
qwen-finetune/train_grpo.py - Engineer → Adapter merge: runs merge
README.md - Adapter merge → Qwen base model: loads base
qwen-finetune/merge_grpo.py - Adapter merge → SFT adapter: applies adapter
qwen-finetune/merge_grpo.py - Adapter merge → GRPO adapter: applies adapter
qwen-finetune/merge_grpo.py - Adapter merge → Merged model: saves model
qwen-finetune/merge_grpo.py - Engineer → GSM8K evaluation: runs evaluation
README.md - GSM8K evaluation → Prepared splits: reads test set
qwen-finetune/eval_gsm8k.py - GSM8K evaluation → Merged model: loads model
README.md - GSM8K evaluation → Evaluation results: writes predictions
qwen-finetune/eval_gsm8k.py - Engineer → Interactive chat: starts chat
README.md - Interactive chat → SFT adapter: loads SFT model
qwen-finetune/chat.py - Result visualization → Evaluation results: reads results
qwen-finetune/visualize.py