# dbiswas55/vlm-qlora-finetuning architecture

> Architecture diagram of the GitHub repository dbiswas55/vlm-qlora-finetuning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dbiswas55/vlm-qlora-finetuning
- Repository: https://github.com/dbiswas55/vlm-qlora-finetuning

## Overview

This repository is a compact QLoRA fine-tuning and inference setup for chart question answering with vision-language models. Its documented workflow is smoke test, train, evaluate, then predict; the graph focuses on the training, evaluation, comparison, and prediction runtime. Training formats ChartQA examples, prepares masked image/text batches, loads a quantized model, and saves a LoRA adapter. Evaluation scores adapter predictions; comparison reports base-versus-adapter results. The README documents distributed launch and other capabilities, but their implementation details are not sampled here.

Main entry points: [`src/train.py`](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/train.py) (Training run), [`src/compare.py`](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/compare.py) (Before-after comparison), [`src/utils/model.py`](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/utils/model.py) (Model loader), [`src/evaluate.py`](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/evaluate.py) (Adapter evaluation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_training["Fine-tuning"]
  node_train["Training run<br/>[train.py]"]
  node_trainer["SFT trainer<br/>[train.py]"]
  node_adapter["LoRA adapter<br/>[train.py]"]
end

subgraph group_shared["Shared model and data"]
  node_format["Example formatting<br/>[data.py]"]
  node_collator["Batch collator<br/>[data.py]"]
  node_loader["Model loader<br/>[model.py]"]
end

subgraph group_assessment["Evaluation and comparison"]
  node_evaluate["Adapter evaluation<br/>[evaluate.py]"]
  node_metric["Relaxed accuracy<br/>[evaluate.py]"]
  node_compare["Before-after comparison<br/>[compare.py]"]
  node_report["Comparison artifacts<br/>[compare.py]"]
end

subgraph group_inference["Prediction"]
  node_predict["Single-image prediction<br/>[predict.py]"]
end

node_engineer(("Researcher"))
node_chartqa[("ChartQA dataset")]
node_modelhub["Model hub"]

node_engineer -->|"runs training"| node_train
node_train -->|"loads examples"| node_chartqa
node_train -->|"formats examples"| node_format
node_train -->|"builds batches"| node_collator
node_train -->|"loads model"| node_loader
node_loader -->|"loads weights"| node_modelhub
node_collator -->|"gets image token"| node_loader
node_train -->|"configures trainer"| node_trainer
node_trainer -->|"saves adapter"| node_adapter
node_engineer -->|"runs evaluation"| node_evaluate
node_evaluate -->|"loads test split"| node_chartqa
node_evaluate -->|"loads model"| node_loader
node_evaluate -->|"attaches adapter"| node_adapter
node_evaluate -->|"scores answers"| node_metric
node_engineer -->|"runs comparison"| node_compare
node_compare -->|"evaluates split"| node_chartqa
node_compare -->|"loads base model"| node_loader
node_compare -->|"attaches adapter"| node_adapter
node_compare -->|"reuses scoring"| node_metric
node_compare -->|"writes report"| node_report
node_engineer -->|"runs prediction"| node_predict
node_predict -->|"loads model"| node_loader
node_predict -.->|"optionally attaches"| node_adapter

click node_train "https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/train.py"
click node_format "https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/utils/data.py"
click node_collator "https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/utils/data.py"
click node_loader "https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/utils/model.py"
click node_trainer "https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/train.py"
click node_adapter "https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/train.py"
click node_evaluate "https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/evaluate.py"
click node_metric "https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/evaluate.py"
click node_compare "https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/compare.py"
click node_report "https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/compare.py"
click node_predict "https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/predict.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_train,node_trainer,node_adapter toneBlue
class node_format,node_collator,node_loader,node_chartqa toneAmber
class node_evaluate,node_metric,node_compare,node_report toneMint
class node_predict toneRose
class node_engineer,node_modelhub toneIndigo
```

## Components

### Fine-tuning

- [Training run](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/train.py): `src/train.py`
- [SFT trainer](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/train.py): `src/train.py`
- [LoRA adapter](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/train.py): `src/train.py`

### Shared model and data

- [Example formatting](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/utils/data.py): `src/utils/data.py`
- [Batch collator](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/utils/data.py): `src/utils/data.py`
- [Model loader](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/utils/model.py): `src/utils/model.py`

### Evaluation and comparison

- [Adapter evaluation](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/evaluate.py): `src/evaluate.py`
- [Relaxed accuracy](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/evaluate.py): `src/evaluate.py`
- [Before-after comparison](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/compare.py): `src/compare.py`
- [Comparison artifacts](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/compare.py): `src/compare.py`

### Prediction

- [Single-image prediction](https://github.com/dbiswas55/vlm-qlora-finetuning/blob/master/src/predict.py): `src/predict.py`

### Other components

- **Researcher**
- **ChartQA dataset**
- **Model hub**

## Connections

- Researcher → Training run: runs training (evidence: `README.md`)
- Training run → ChartQA dataset: loads examples (evidence: `src/train.py`)
- Training run → Example formatting: formats examples (evidence: `src/train.py`)
- Training run → Batch collator: builds batches (evidence: `src/train.py`)
- Training run → Model loader: loads model (evidence: `src/train.py`)
- Model loader → Model hub: loads weights (evidence: `src/utils/model.py`)
- Batch collator → Model loader: gets image token (evidence: `src/utils/data.py`)
- Training run → SFT trainer: configures trainer (evidence: `src/train.py`)
- SFT trainer → LoRA adapter: saves adapter (evidence: `src/train.py`)
- Researcher → Adapter evaluation: runs evaluation (evidence: `README.md`)
- Adapter evaluation → ChartQA dataset: loads test split (evidence: `src/evaluate.py`)
- Adapter evaluation → Model loader: loads model (evidence: `src/evaluate.py`)
- Adapter evaluation → LoRA adapter: attaches adapter (evidence: `src/evaluate.py`)
- Adapter evaluation → Relaxed accuracy: scores answers (evidence: `src/evaluate.py`)
- Researcher → Before-after comparison: runs comparison (evidence: `README.md`)
- Before-after comparison → ChartQA dataset: evaluates split (evidence: `src/compare.py`)
- Before-after comparison → Model loader: loads base model (evidence: `src/compare.py`)
- Before-after comparison → LoRA adapter: attaches adapter (evidence: `src/compare.py`)
- Before-after comparison → Relaxed accuracy: reuses scoring (evidence: `src/compare.py`)
- Before-after comparison → Comparison artifacts: writes report (evidence: `src/compare.py`)
- Researcher → Single-image prediction: runs prediction (evidence: `README.md`)
- Single-image prediction → Model loader: loads model (evidence: `src/predict.py`)
- Single-image prediction → LoRA adapter: optionally attaches (evidence: `src/predict.py`)