# shivendrra/smalllanguagemodel architecture

> Architecture diagram of the GitHub repository shivendrra/smalllanguagemodel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shivendrra/smalllanguagemodel
- Repository: https://github.com/shivendrra/smalllanguagemodel

## Overview

SmallLanguageModel is a from-scratch language-model project spanning corpus collection, text preparation, tokenization, model training, and generation. Its collection workflows gather encyclopedia pages or YouTube captions; processing utilities prepare files, while the model directory provides tokenizer, training/runtime code, and BERT, GPT, and Seq2Seq implementations. The graph preserves those documented model families and the JavaScript collection tools. Relationships are limited to calls and module references visible in excerpts, the source index, or README; unsampled processing and model internals are not inferred.

Main entry points: [`Models/tokenizer.py`](https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/tokenizer.py) (Tokenizer), [`Data Collection/britannica/main.py`](https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/britannica/main.py) (Britannica Scraper), [`Data Collection/youtube_transcripts/main.py`](https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/youtube_transcripts/main.py) (Transcript Collector), [`Models/run.py`](https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/run.py) (Training and Generation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Data Collection"]
  node_britannica_queries["Britannica Queries<br/>[queries.py]"]
  node_britannica_scraper["Britannica Scraper<br/>[main.py]"]
  node_britannica_urls["Britannica URL Fetcher<br/>[URLFetcher.py]"]
  node_wiki_scraper["Wikipedia Scraper<br/>[main.py]"]
  node_wiki_url_scraper["Wikipedia URL Scraper<br/>[fetch_urls.py]"]
  node_youtube_snippets["Channel ID Samples<br/>[snippets.py]"]
  node_youtube_collector["Transcript Collector<br/>[main.py]"]
  node_js_collection["JavaScript Scrapers"]
end

subgraph group_preparation["Data Preparation"]
  node_file_processing["File Processing Utilities"]
  node_training_data["Text Corpora<br/>[training_data.txt]"]
  node_captions_data["Caption Data<br/>[captions.txt]"]
end

subgraph group_modeling["Modeling and Inference"]
  node_tokenizer["Tokenizer<br/>[tokenizer.py]"]
  node_gpt["GPT Decoder<br/>[decoder.py]"]
  node_bert_encoder["BERT Encoder<br/>[encoder.py]"]
  node_seq2seq["Seq2Seq Model<br/>[sequence.py]"]
  node_training_runtime["Training and Generation<br/>[run.py]"]
  node_lora["LoRA Fine-Tuning<br/>[lora.py]"]
  node_qlora["QLoRA Fine-Tuning<br/>[qlora.py]"]
  node_model_output["Generated Text"]
end

node_user(("Project User"))
node_youtube_api{{"YouTube APIs"}}
node_web_sources{{"Web Sources"}}

node_user -->|"selects topics"| node_britannica_queries
node_user -->|"supplies queries"| node_wiki_scraper
node_user -->|"selects channels"| node_youtube_snippets
node_britannica_queries -->|"provides queries"| node_britannica_scraper
node_britannica_scraper -->|"inherits and fetches URLs"| node_britannica_urls
node_britannica_urls -->|"requests search pages"| node_web_sources
node_britannica_scraper -->|"requests articles"| node_web_sources
node_wiki_scraper -->|"requests pages"| node_web_sources
node_wiki_url_scraper -->|"requests URLs"| node_web_sources
node_youtube_snippets -->|"supplies channel IDs"| node_youtube_collector
node_youtube_collector -->|"fetches videos and captions"| node_youtube_api
node_youtube_collector -->|"appends transcripts"| node_captions_data
node_js_collection -.->|"scrapes web"| node_web_sources
node_user -->|"prepares files"| node_file_processing
node_training_data -.->|"provides text"| node_tokenizer
node_captions_data -.->|"provides text"| node_tokenizer
node_tokenizer -->|"tokenizes input"| node_training_runtime
node_training_runtime -->|"trains and generates"| node_gpt
node_training_runtime -->|"produces text"| node_model_output
node_user -->|"chooses model family"| node_bert_encoder
node_user -->|"chooses model family"| node_seq2seq
node_lora -->|"uses decoder"| node_gpt
node_lora -->|"uses tokenizer"| node_tokenizer
node_qlora -->|"uses decoder"| node_gpt
node_qlora -->|"uses tokenizer"| node_tokenizer

click node_britannica_queries "https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/britannica/queries.py"
click node_britannica_scraper "https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/britannica/main.py"
click node_britannica_urls "https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/britannica/URLFetcher.py"
click node_wiki_scraper "https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/wikipedia/main.py"
click node_wiki_url_scraper "https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/wikipedia/fetch_urls.py"
click node_youtube_snippets "https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/youtube_transcripts/snippets.py"
click node_youtube_collector "https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/youtube_transcripts/main.py"
click node_js_collection "https://github.com/shivendrra/smalllanguagemodel/tree/main/Data%20Collection/javascript"
click node_file_processing "https://github.com/shivendrra/smalllanguagemodel/tree/main/Data%20Processing"
click node_training_data "https://github.com/shivendrra/smalllanguagemodel/blob/main/Data/training_data.txt"
click node_captions_data "https://github.com/shivendrra/smalllanguagemodel/blob/main/Data/captions.txt"
click node_tokenizer "https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/tokenizer.py"
click node_gpt "https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/decoder.py"
click node_bert_encoder "https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/encoder.py"
click node_seq2seq "https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/sequence.py"
click node_training_runtime "https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/run.py"
click node_lora "https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/lora.py"
click node_qlora "https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/qlora.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_britannica_queries,node_britannica_scraper,node_britannica_urls,node_wiki_scraper,node_wiki_url_scraper,node_youtube_snippets,node_youtube_collector,node_js_collection,node_user toneBlue
class node_file_processing,node_training_data,node_captions_data toneAmber
class node_tokenizer,node_gpt,node_bert_encoder,node_seq2seq,node_training_runtime,node_lora,node_qlora,node_model_output,node_youtube_api toneMint
class node_web_sources toneIndigo
```

## Components

### Data Collection

- [Britannica Queries](https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/britannica/queries.py): `Data Collection/britannica/queries.py`
- [Britannica Scraper](https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/britannica/main.py): `Data Collection/britannica/main.py`
- [Britannica URL Fetcher](https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/britannica/URLFetcher.py): `Data Collection/britannica/URLFetcher.py`
- [Wikipedia Scraper](https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/wikipedia/main.py): `Data Collection/wikipedia/main.py`
- [Wikipedia URL Scraper](https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/wikipedia/fetch_urls.py): `Data Collection/wikipedia/fetch_urls.py`
- [Channel ID Samples](https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/youtube_transcripts/snippets.py): `Data Collection/youtube_transcripts/snippets.py`
- [Transcript Collector](https://github.com/shivendrra/smalllanguagemodel/blob/main/Data%20Collection/youtube_transcripts/main.py): `Data Collection/youtube_transcripts/main.py`
- [JavaScript Scrapers](https://github.com/shivendrra/smalllanguagemodel/tree/main/Data%20Collection/javascript): `Data Collection/javascript`

### Data Preparation

- [File Processing Utilities](https://github.com/shivendrra/smalllanguagemodel/tree/main/Data%20Processing): `Data Processing`
- [Text Corpora](https://github.com/shivendrra/smalllanguagemodel/blob/main/Data/training_data.txt): `Data/training_data.txt`
- [Caption Data](https://github.com/shivendrra/smalllanguagemodel/blob/main/Data/captions.txt): `Data/captions.txt`

### Modeling and Inference

- [Tokenizer](https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/tokenizer.py): `Models/tokenizer.py`
- [GPT Decoder](https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/decoder.py): `Models/decoder.py`
- [BERT Encoder](https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/encoder.py): `Models/encoder.py`
- [Seq2Seq Model](https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/sequence.py): `Models/sequence.py`
- [Training and Generation](https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/run.py): `Models/run.py`
- [LoRA Fine-Tuning](https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/lora.py): `Models/lora.py`
- [QLoRA Fine-Tuning](https://github.com/shivendrra/smalllanguagemodel/blob/main/Models/qlora.py): `Models/qlora.py`
- **Generated Text**

### Other components

- **Project User**
- **YouTube APIs**
- **Web Sources**

## Connections

- Project User → Britannica Queries: selects topics
- Project User → Wikipedia Scraper: supplies queries (evidence: `Data Collection/wikipedia/main.py`)
- Project User → Channel ID Samples: selects channels
- Britannica Queries → Britannica Scraper: provides queries
- Britannica Scraper → Britannica URL Fetcher: inherits and fetches URLs (evidence: `Data Collection/britannica/main.py`)
- Britannica URL Fetcher → Web Sources: requests search pages (evidence: `Data Collection/britannica/URLFetcher.py`)
- Britannica Scraper → Web Sources: requests articles (evidence: `Data Collection/britannica/main.py`)
- Wikipedia Scraper → Web Sources: requests pages (evidence: `Data Collection/wikipedia/main.py`)
- Wikipedia URL Scraper → Web Sources: requests URLs (evidence: `Data Collection/wikipedia/fetch_urls.py`)
- Channel ID Samples → Transcript Collector: supplies channel IDs
- Transcript Collector → YouTube APIs: fetches videos and captions (evidence: `Data Collection/youtube_transcripts/main.py`)
- Transcript Collector → Caption Data: appends transcripts (evidence: `Data Collection/youtube_transcripts/main.py`)
- JavaScript Scrapers → Web Sources: scrapes web
- Project User → File Processing Utilities: prepares files (evidence: `README.md`)
- Text Corpora → Tokenizer: provides text
- Caption Data → Tokenizer: provides text
- Tokenizer → Training and Generation: tokenizes input (evidence: `Models/run.py`)
- Training and Generation → GPT Decoder: trains and generates (evidence: `Models/run.py`)
- Training and Generation → Generated Text: produces text (evidence: `Models/run.py`)
- Project User → BERT Encoder: chooses model family (evidence: `README.md`)
- Project User → Seq2Seq Model: chooses model family (evidence: `README.md`)
- LoRA Fine-Tuning → GPT Decoder: uses decoder (evidence: `Models/lora.py`)
- LoRA Fine-Tuning → Tokenizer: uses tokenizer (evidence: `Models/lora.py`)
- QLoRA Fine-Tuning → GPT Decoder: uses decoder (evidence: `Models/qlora.py`)
- QLoRA Fine-Tuning → Tokenizer: uses tokenizer (evidence: `Models/qlora.py`)