# sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms architecture

> Architecture diagram of the GitHub repository sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms
- Repository: https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms

## Overview

This repository builds a from-scratch vision-language model and provides a separate caption-data preparation/loading path. In inference, a user supplies an image and prompt; the VLM encodes image patches, compresses them into query tokens, projects them into language-model space, and generates text. Dataset utilities download caption rows to Parquet and load images and captions as model-ready tensors. The README describes future training stages, but no training implementation or wiring is present. Connections reflect the README and shown source; no end-to-end dataset-to-training link is evidenced.

Main entry points: [`vlm_train/networks/lm_to_vlm.py`](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/lm_to_vlm.py) (VLM orchestration), [`vlm_train/datasets/__init__.py`](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/datasets/__init__.py) (Caption loader), [`vlm_train/utils/filter_dataset.py`](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/utils/filter_dataset.py) (Dataset downloader), [`vlm_train/networks/q_former.py`](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/q_former.py) (Q-Former)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Caption data"]
  node_parquet[("Caption Parquet")]
  node_filter["Dataset downloader<br/>[filter_dataset.py]"]
  node_caption_dataset["Caption loader<br/>[__init__.py]"]
  node_image_transform["Image transforms<br/>[__init__.py]"]
end

subgraph group_model["Vision-language model"]
  node_vlm["VLM orchestration<br/>[lm_to_vlm.py]"]
  node_vit["ViT encoder<br/>[vit_encoder.py]"]
  node_qformer["Q-Former<br/>[q_former.py]"]
  node_cross_attention["Cross-attention blocks<br/>[q_former.py]"]
  node_adapter["Visual adapter<br/>[lm_to_vlm.py]"]
  node_distilbert["DistilBERT backbone"]
end

subgraph group_language["Language generation"]
  node_llm["Causal language model<br/>[lm_to_vlm.py]"]
  node_tokenizer["Text tokenizer<br/>[lm_to_vlm.py]"]
end

node_user(("User"))
node_dataset_hub{{"Dataset hub"}}
node_image_web{{"Image URLs"}}
node_model_hub{{"Pretrained model hub"}}

node_user -->|"starts download"| node_filter
node_filter -->|"loads rows"| node_dataset_hub
node_filter -->|"writes rows"| node_parquet
node_caption_dataset -->|"reads rows"| node_parquet
node_caption_dataset -->|"downloads fallback"| node_image_web
node_caption_dataset -->|"preprocesses images"| node_image_transform
node_user -->|"provides image and prompt"| node_vlm
node_vlm -->|"encodes image"| node_vit
node_vit -->|"loads pretrained ViT"| node_model_hub
node_vlm -->|"compresses patches"| node_qformer
node_qformer -->|"runs transformer layers"| node_distilbert
node_qformer -->|"attends to image features"| node_cross_attention
node_vlm -->|"projects query tokens"| node_adapter
node_vlm -->|"generates from embeddings"| node_llm
node_llm -->|"loads pretrained model"| node_model_hub
node_vlm -->|"tokenizes prompt and decodes output"| node_tokenizer
node_tokenizer -->|"loads pretrained tokenizer"| node_model_hub

click node_filter "https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/utils/filter_dataset.py"
click node_caption_dataset "https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/datasets/__init__.py"
click node_image_transform "https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/datasets/__init__.py"
click node_vlm "https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/lm_to_vlm.py"
click node_vit "https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/vit_encoder.py"
click node_qformer "https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/q_former.py"
click node_cross_attention "https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/q_former.py"
click node_adapter "https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/lm_to_vlm.py"
click node_llm "https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/lm_to_vlm.py"
click node_tokenizer "https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/lm_to_vlm.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_parquet,node_filter,node_caption_dataset,node_image_transform,node_user toneBlue
class node_vlm,node_vit,node_qformer,node_cross_attention,node_adapter,node_distilbert toneAmber
class node_llm,node_tokenizer toneMint
class node_dataset_hub,node_image_web,node_model_hub toneIndigo
```

## Components

### Caption data

- **Caption Parquet**
- [Dataset downloader](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/utils/filter_dataset.py): `vlm_train/utils/filter_dataset.py`
- [Caption loader](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/datasets/__init__.py): `vlm_train/datasets/__init__.py`
- [Image transforms](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/datasets/__init__.py): `vlm_train/datasets/__init__.py`

### Vision-language model

- [VLM orchestration](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/lm_to_vlm.py): `vlm_train/networks/lm_to_vlm.py`
- [ViT encoder](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/vit_encoder.py): `vlm_train/networks/vit_encoder.py`
- [Q-Former](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/q_former.py): `vlm_train/networks/q_former.py`
- [Cross-attention blocks](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/q_former.py): `vlm_train/networks/q_former.py`
- [Visual adapter](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/lm_to_vlm.py): `vlm_train/networks/lm_to_vlm.py`
- **DistilBERT backbone**

### Language generation

- [Causal language model](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/lm_to_vlm.py): `vlm_train/networks/lm_to_vlm.py`
- [Text tokenizer](https://github.com/sujithsai-kalakonda/train_vlms_from_scratch_using_text_llms/blob/main/vlm_train/networks/lm_to_vlm.py): `vlm_train/networks/lm_to_vlm.py`

### Other components

- **User**
- **Dataset hub**
- **Image URLs**
- **Pretrained model hub**

## Connections

- User → Dataset downloader: starts download (evidence: `README.md`)
- Dataset downloader → Dataset hub: loads rows (evidence: `vlm_train/utils/filter_dataset.py`)
- Dataset downloader → Caption Parquet: writes rows (evidence: `vlm_train/utils/filter_dataset.py`)
- Caption loader → Caption Parquet: reads rows (evidence: `vlm_train/datasets/__init__.py`)
- Caption loader → Image URLs: downloads fallback (evidence: `vlm_train/datasets/__init__.py`)
- Caption loader → Image transforms: preprocesses images (evidence: `vlm_train/datasets/__init__.py`)
- User → VLM orchestration: provides image and prompt (evidence: `README.md`)
- VLM orchestration → ViT encoder: encodes image (evidence: `vlm_train/networks/lm_to_vlm.py`)
- ViT encoder → Pretrained model hub: loads pretrained ViT (evidence: `vlm_train/networks/vit_encoder.py`)
- VLM orchestration → Q-Former: compresses patches (evidence: `vlm_train/networks/lm_to_vlm.py`)
- Q-Former → DistilBERT backbone: runs transformer layers (evidence: `vlm_train/networks/q_former.py`)
- Q-Former → Cross-attention blocks: attends to image features (evidence: `vlm_train/networks/q_former.py`)
- VLM orchestration → Visual adapter: projects query tokens (evidence: `vlm_train/networks/lm_to_vlm.py`)
- VLM orchestration → Causal language model: generates from embeddings (evidence: `vlm_train/networks/lm_to_vlm.py`)
- Causal language model → Pretrained model hub: loads pretrained model (evidence: `vlm_train/networks/lm_to_vlm.py`)
- VLM orchestration → Text tokenizer: tokenizes prompt and decodes output (evidence: `vlm_train/networks/lm_to_vlm.py`)
- Text tokenizer → Pretrained model hub: loads pretrained tokenizer (evidence: `vlm_train/networks/lm_to_vlm.py`)
