How iadtya/sarvamai-vlm-finetuning works
Based on an analysis of the file tree and README, this project is a machine learning fine-tuning effort for a vision-language model (VLM) that leverages LoRA (Low-Rank Adaptation) adjustments – hence “LoRA” appears prominently in the model artifacts. The aim is to drastically improve evaluation metrics such as WER, CER, and sequence accuracy. The project integrates dataset processing, fine-tuning scripts, evaluation metrics, and a final model artifact (lora_model_V2) that users subsequently deploy via Google Colab and Google Drive.
Main Components
• Data Ingestion & Preprocessing:
• Components: Dataset folder (with processed_dataset_V6.json, train, test); split_dataset.py script.
• Role: Splitting and processing raw image/text data into training and test sets.
• Model Training & Fine-Tuning:
• Components: Fine_tune.ipynb and Fine_tune_V2.ipynb notebooks (with HF.ipynb for Hugging Face integration).
• Role: Running experiments, executing the fine-tuning process, and interfacing with external frameworks (Hugging Face).
• Evaluation & Metrics Calculation:
• Components: metrics.py, ground_truth.py, baseline_metrics_v2.txt, Fine_tuned_metrics.txt, test_ground_truth.json.
• Role: Generating evaluation metrics and validating performance improvements.
• Model Artifact and Deployment Setup:
• Components: lora_model_V2 directory (holding adapter_config.json, adapter_model weights, tokenizer/tokenizer_config files, and chat_template.json), with its own README.md which acts as a model card.
• Role: Containing the fine-tuned model files that can be pushed to a Google Drive for inference in Colab.
• External & Integration Aspects:
• External Services: Hugging Face Hub (both for model and dataset uploads – links provided in the README).
• Deployment Environment: Google Colab and Google Drive usage as indicated in the README.
Relationships and Data Flow
• Use split_dataset.py to partition data into train and test groups.
• The processed data (processed_dataset_V6.json) is fed into the fine-tuning notebooks (Fine_tune.ipynb and Fine_tune_V2.ipynb), where model training happens.
• Ground truth scripts (ground_truth.py) and other evaluation scripts (metrics.py) are used to generate performance metrics (baseline_metrics_v2.txt, Fine_tuned_metrics.txt) which facilitate comparison between the baseline and fine-tuned models.
• The output from fine-tuning is stored in the lora_model_V2 folder (incorporating adapter models, tokenizers, and necessary configuration files).
• Instructions specify an integration step where users push the lora_model_V2 folder to Google Drive and run inference through a Colab notebook (link provided).
Architectural Patterns and Principles
• Separation of Concerns: The project clearly distinguishes between data preprocessing, model training, evaluation, and deployment.
• Modularity: Distinct scripts/notebooks for data splitting, training, evaluation, and model packaging.
• External Integration: Use of external resources (Hugging Face) and deployment on cloud environments (Google Colab/Drive).
Relevant Technologies and Frameworks
• Primary Languages/Tools: Python (scripts and notebooks), Jupyter Notebook.
• Machine Learning Techniques: Fine-tuning with LoRA adjustments, evaluation via metrics.
• Third-Party Integrations: Hugging Face for sourcing models/datasets and Google Colab for deployment/testing.
• Data Management: JSON files for configuration and processed data.
Details
• Every component is clearly labeled with its function.
• Incorporate any dependencies (e.g., the environment is set up via requirements.txt).
• Files (e.g., processed_dataset, fine-tuning notebooks, evaluation scripts, and model artifacts) because these are the heart of the system.
• Directional flow and interaction between modules to provide a comprehensive view of this fine-tuning system.