How sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms works
This repository builds a from-scratch vision-language model and provides a separate caption-data preparation/loading path. In inference, a user supplies an image and prompt; the VLM encodes image patches, compresses them into query tokens, projects them into language-model space, and generates text. Dataset utilities download caption rows to Parquet and load images and captions as model-ready tensors. The README describes future training stages, but no training implementation or wiring is present. Connections reflect the README and shown source; no end-to-end dataset-to-training link is evidenced.
Main entry points: lm_to_vlm.py VLM orchestration · __init__.py Caption loader · filter_dataset.py Dataset downloader · q_former.py Q-Former
Components 16
Caption data
- Caption Parquet
- Dataset downloader
vlm_train/utils/filter_dataset.py - Caption loader
vlm_train/datasets/__init__.py - Image transforms
vlm_train/datasets/__init__.py
Vision-language model
- VLM orchestration
vlm_train/networks/lm_to_vlm.py - ViT encoder
vlm_train/networks/vit_encoder.py - Q-Former
vlm_train/networks/q_former.py - Cross-attention blocks
vlm_train/networks/q_former.py - Visual adapter
vlm_train/networks/lm_to_vlm.py - DistilBERT backbone
Language generation
- Causal language model
vlm_train/networks/lm_to_vlm.py - Text tokenizer
vlm_train/networks/lm_to_vlm.py
Other components
- User
- Dataset hub
- Image URLs
- Pretrained model hub
Connections 17
- User → Dataset downloader: starts download
README.md - Dataset downloader → Dataset hub: loads rows
vlm_train/utils/filter_dataset.py - Dataset downloader → Caption Parquet: writes rows
vlm_train/utils/filter_dataset.py - Caption loader → Caption Parquet: reads rows
vlm_train/datasets/__init__.py - Caption loader → Image URLs: downloads fallback
vlm_train/datasets/__init__.py - Caption loader → Image transforms: preprocesses images
vlm_train/datasets/__init__.py - User → VLM orchestration: provides image and prompt
README.md - VLM orchestration → ViT encoder: encodes image
vlm_train/networks/lm_to_vlm.py - ViT encoder → Pretrained model hub: loads pretrained ViT
vlm_train/networks/vit_encoder.py - VLM orchestration → Q-Former: compresses patches
vlm_train/networks/lm_to_vlm.py - Q-Former → DistilBERT backbone: runs transformer layers
vlm_train/networks/q_former.py - Q-Former → Cross-attention blocks: attends to image features
vlm_train/networks/q_former.py - VLM orchestration → Visual adapter: projects query tokens
vlm_train/networks/lm_to_vlm.py - VLM orchestration → Causal language model: generates from embeddings
vlm_train/networks/lm_to_vlm.py - Causal language model → Pretrained model hub: loads pretrained model
vlm_train/networks/lm_to_vlm.py - VLM orchestration → Text tokenizer: tokenizes prompt and decodes output
vlm_train/networks/lm_to_vlm.py - Text tokenizer → Pretrained model hub: loads pretrained tokenizer
vlm_train/networks/lm_to_vlm.py