# iadtya/sarvamai-vlm-finetuning architecture

> Architecture diagram of the GitHub repository iadtya/sarvamai-vlm-finetuning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/iadtya/sarvamai-vlm-finetuning
- Repository: https://github.com/iadtya/sarvamai-vlm-finetuning

## Overview

Based on an analysis of the file tree and README, this project is a machine learning fine-tuning effort for a vision-language model (VLM) that leverages LoRA (Low-Rank Adaptation) adjustments – hence “LoRA” appears prominently in the model artifacts. The aim is to drastically improve evaluation metrics such as WER, CER, and sequence accuracy. The project integrates dataset processing, fine-tuning scripts, evaluation metrics, and a final model artifact (lora_model_V2) that users subsequently deploy via Google Colab and Google Drive.

##### Main Components
- Data Ingestion & Preprocessing:
- Components: Dataset folder (with processed_dataset_V6.json, train, test); split_dataset.py script.
- Role: Splitting and processing raw image/text data into training and test sets.
- Model Training & Fine-Tuning:
- Components: Fine_tune.ipynb and Fine_tune_V2.ipynb notebooks (with HF.ipynb for Hugging Face integration).
- Role: Running experiments, executing the fine-tuning process, and interfacing with external frameworks (Hugging Face).
- Evaluation & Metrics Calculation:
- Components: metrics.py, ground_truth.py, baseline_metrics_v2.txt, Fine_tuned_metrics.txt, test_ground_truth.json.
- Role: Generating evaluation metrics and validating performance improvements.
- Model Artifact and Deployment Setup:
- Components: lora_model_V2 directory (holding adapter_config.json, adapter_model weights, tokenizer/tokenizer_config files, and chat_template.json), with its own README.md which acts as a model card.
- Role: Containing the fine-tuned model files that can be pushed to a Google Drive for inference in Colab.
- External & Integration Aspects:
- External Services: Hugging Face Hub (both for model and dataset uploads – links provided in the README).
- Deployment Environment: Google Colab and Google Drive usage as indicated in the README.

##### Relationships and Data Flow
- Use split_dataset.py to partition data into train and test groups.
- The processed data (processed_dataset_V6.json) is fed into the fine-tuning notebooks (Fine_tune.ipynb and Fine_tune_V2.ipynb), where model training happens.
- Ground truth scripts (ground_truth.py) and other evaluation scripts (metrics.py) are used to generate performance metrics (baseline_metrics_v2.txt, Fine_tuned_metrics.txt) which facilitate comparison between the baseline and fine-tuned models.
- The output from fine-tuning is stored in the lora_model_V2 folder (incorporating adapter models, tokenizers, and necessary configuration files).
- Instructions specify an integration step where users push the lora_model_V2 folder to Google Drive and run inference through a Colab notebook (link provided).

##### Architectural Patterns and Principles
- Separation of Concerns: The project clearly distinguishes between data preprocessing, model training, evaluation, and deployment.
- Modularity: Distinct scripts/notebooks for data splitting, training, evaluation, and model packaging.
- External Integration: Use of external resources (Hugging Face) and deployment on cloud environments (Google Colab/Drive).

##### Relevant Technologies and Frameworks
- Primary Languages/Tools: Python (scripts and notebooks), Jupyter Notebook.
- Machine Learning Techniques: Fine-tuning with LoRA adjustments, evaluation via metrics.
- Third-Party Integrations: Hugging Face for sourcing models/datasets and Google Colab for deployment/testing.
- Data Management: JSON files for configuration and processed data.

##### Details
- Every component is clearly labeled with its function.
- Incorporate any dependencies (e.g., the environment is set up via requirements.txt).
- Files (e.g., processed_dataset, fine-tuning notebooks, evaluation scripts, and model artifacts) because these are the heart of the system.
- Directional flow and interaction between modules to provide a comprehensive view of this fine-tuning system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        D1["Dataset (raw data)"]
        D2["Data Splitting (split_dataset.py)"]
    end

    %% Model Training & Fine-Tuning
    subgraph "Model Training & Fine-Tuning"
        T1["Fine_tune.ipynb"]
        T2["Fine_tune_V2.ipynb"]
        T3["HF.ipynb"]
        TF["Hugging Face Hub"]
    end

    %% Evaluation & Metrics Calculation
    subgraph "Evaluation & Metrics Calculation"
        E1["metrics.py"]
        E2["ground_truth.py"]
        E3["baseline_metrics_v2.txt"]
        E4["Fine_tuned_metrics.txt"]
        E5["test_ground_truth.json"]
    end

    %% Model Artifact and Deployment Setup
    subgraph "Model Artifact & Deployment Setup"
        subgraph "Model Artifact (lora_model_V2)"
            A1["adapter_config.json"]
            A2["adapter_model.safetensors"]
            A3["chat_template.json"]
            A4["preprocessor_config.json"]
            A5["special_tokens_map.json"]
            A6["tokenizer.json"]
            A7["tokenizer_config.json"]
        end
        DPLY["Deployment (Google Colab/Drive)"]
    end

    %% Data Flow
    D1 -->|"splits"| D2
    D2 -->|"feeds"| T1
    D2 -->|"feeds"| T2
    D2 -->|"feeds"| T3
    T3 -->|"integrates"| TF

    D2 -->|"provides_GT"| E2
    T1 -->|"evaluates"| E1
    E1 -->|"reports"| E3
    E1 -->|"reports"| E4
    E2 -->|"compares"| E5

    T1 -->|"outputs_model"| A1
    A1 -->|"deployed_to"| DPLY

    %% Click Events
    click D1 "https://github.com/iadtya/sarvamai-vlm-finetuning/tree/main/dataset"
    click D2 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/split_dataset.py"

    click T1 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/Fine_tune.ipynb"
    click T2 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/Fine_tune_V2.ipynb"
    click T3 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/HF.ipynb"

    click E1 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/metrics.py"
    click E2 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/ground_truth.py"
    click E3 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/baseline_metrics_v2.txt"
    click E4 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/Fine_tuned_metrics.txt"
    click E5 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/test_ground_truth.json"

    click A1 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/lora_model_V2/adapter_config.json"
    click A2 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/lora_model_V2/adapter_model.safetensors"
    click A3 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/lora_model_V2/chat_template.json"
    click A4 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/lora_model_V2/preprocessor_config.json"
    click A5 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/lora_model_V2/special_tokens_map.json"
    click A6 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/lora_model_V2/tokenizer.json"
    click A7 "https://github.com/iadtya/sarvamai-vlm-finetuning/blob/main/lora_model_V2/tokenizer_config.json"

    %% Styles
    classDef data fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef training fill:#A9DFBF,stroke:#145A32,stroke-width:2px;
    classDef evaluation fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef artifact fill:#E8DAEF,stroke:#7D3C98,stroke-width:2px;
    classDef external fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef deployment fill:#D6EAF8,stroke:#2874A6,stroke-width:2px;

    class D1,D2 data;
    class T1,T2,T3 training;
    class TF external;
    class E1,E2,E3,E4,E5 evaluation;
    class A1,A2,A3,A4,A5,A6,A7 artifact;
    class DPLY deployment;
```
