How dbiswas55/vlm-qlora-finetuning works
This repository is a compact QLoRA fine-tuning and inference setup for chart question answering with vision-language models. Its documented workflow is smoke test, train, evaluate, then predict; the graph focuses on the training, evaluation, comparison, and prediction runtime. Training formats ChartQA examples, prepares masked image/text batches, loads a quantized model, and saves a LoRA adapter. Evaluation scores adapter predictions; comparison reports base-versus-adapter results. The README documents distributed launch and other capabilities, but their implementation details are not sampled here.
Main entry points: train.py Training run · compare.py Before-after comparison · model.py Model loader · evaluate.py Adapter evaluation
Components 14
Fine-tuning
- Training run
src/train.py - SFT trainer
src/train.py - LoRA adapter
src/train.py
Shared model and data
- Example formatting
src/utils/data.py - Batch collator
src/utils/data.py - Model loader
src/utils/model.py
Evaluation and comparison
- Adapter evaluation
src/evaluate.py - Relaxed accuracy
src/evaluate.py - Before-after comparison
src/compare.py - Comparison artifacts
src/compare.py
Prediction
- Single-image prediction
src/predict.py
Other components
- Researcher
- ChartQA dataset
- Model hub
Connections 23
- Researcher → Training run: runs training
README.md - Training run → ChartQA dataset: loads examples
src/train.py - Training run → Example formatting: formats examples
src/train.py - Training run → Batch collator: builds batches
src/train.py - Training run → Model loader: loads model
src/train.py - Model loader → Model hub: loads weights
src/utils/model.py - Batch collator → Model loader: gets image token
src/utils/data.py - Training run → SFT trainer: configures trainer
src/train.py - SFT trainer → LoRA adapter: saves adapter
src/train.py - Researcher → Adapter evaluation: runs evaluation
README.md - Adapter evaluation → ChartQA dataset: loads test split
src/evaluate.py - Adapter evaluation → Model loader: loads model
src/evaluate.py - Adapter evaluation → LoRA adapter: attaches adapter
src/evaluate.py - Adapter evaluation → Relaxed accuracy: scores answers
src/evaluate.py - Researcher → Before-after comparison: runs comparison
README.md - Before-after comparison → ChartQA dataset: evaluates split
src/compare.py - Before-after comparison → Model loader: loads base model
src/compare.py - Before-after comparison → LoRA adapter: attaches adapter
src/compare.py - Before-after comparison → Relaxed accuracy: reuses scoring
src/compare.py - Before-after comparison → Comparison artifacts: writes report
src/compare.py - Researcher → Single-image prediction: runs prediction
README.md - Single-image prediction → Model loader: loads model
src/predict.py - Single-image prediction → LoRA adapter: optionally attaches
src/predict.py