How shivendrra/smalllanguagemodel works
SmallLanguageModel is a from-scratch language-model project spanning corpus collection, text preparation, tokenization, model training, and generation. Its collection workflows gather encyclopedia pages or YouTube captions; processing utilities prepare files, while the model directory provides tokenizer, training/runtime code, and BERT, GPT, and Seq2Seq implementations. The graph preserves those documented model families and the JavaScript collection tools. Relationships are limited to calls and module references visible in excerpts, the source index, or README; unsampled processing and model internals are not inferred.
Main entry points: tokenizer.py Tokenizer · britannica/main.py Britannica Scraper · youtube_transcripts/main.py Transcript Collector · run.py Training and Generation
Components 22
Data Collection
- Britannica Queries
Data Collection/britannica/queries.py - Britannica Scraper
Data Collection/britannica/main.py - Britannica URL Fetcher
Data Collection/britannica/URLFetcher.py - Wikipedia Scraper
Data Collection/wikipedia/main.py - Wikipedia URL Scraper
Data Collection/wikipedia/fetch_urls.py - Channel ID Samples
Data Collection/youtube_transcripts/snippets.py - Transcript Collector
Data Collection/youtube_transcripts/main.py - JavaScript Scrapers
Data Collection/javascript
Data Preparation
- File Processing Utilities
Data Processing - Text Corpora
Data/training_data.txt - Caption Data
Data/captions.txt
Modeling and Inference
- Tokenizer
Models/tokenizer.py - GPT Decoder
Models/decoder.py - BERT Encoder
Models/encoder.py - Seq2Seq Model
Models/sequence.py - Training and Generation
Models/run.py - LoRA Fine-Tuning
Models/lora.py - QLoRA Fine-Tuning
Models/qlora.py - Generated Text
Other components
- Project User
- YouTube APIs
- Web Sources
Connections 25
- Project User → Britannica Queries: selects topics
- Project User → Wikipedia Scraper: supplies queries
Data Collection/wikipedia/main.py - Project User → Channel ID Samples: selects channels
- Britannica Queries → Britannica Scraper: provides queries
- Britannica Scraper → Britannica URL Fetcher: inherits and fetches URLs
Data Collection/britannica/main.py - Britannica URL Fetcher → Web Sources: requests search pages
Data Collection/britannica/URLFetcher.py - Britannica Scraper → Web Sources: requests articles
Data Collection/britannica/main.py - Wikipedia Scraper → Web Sources: requests pages
Data Collection/wikipedia/main.py - Wikipedia URL Scraper → Web Sources: requests URLs
Data Collection/wikipedia/fetch_urls.py - Channel ID Samples → Transcript Collector: supplies channel IDs
- Transcript Collector → YouTube APIs: fetches videos and captions
Data Collection/youtube_transcripts/main.py - Transcript Collector → Caption Data: appends transcripts
Data Collection/youtube_transcripts/main.py - JavaScript Scrapers → Web Sources: scrapes web
- Project User → File Processing Utilities: prepares files
README.md - Text Corpora → Tokenizer: provides text
- Caption Data → Tokenizer: provides text
- Tokenizer → Training and Generation: tokenizes input
Models/run.py - Training and Generation → GPT Decoder: trains and generates
Models/run.py - Training and Generation → Generated Text: produces text
Models/run.py - Project User → BERT Encoder: chooses model family
README.md - Project User → Seq2Seq Model: chooses model family
README.md - LoRA Fine-Tuning → GPT Decoder: uses decoder
Models/lora.py - LoRA Fine-Tuning → Tokenizer: uses tokenizer
Models/lora.py - QLoRA Fine-Tuning → GPT Decoder: uses decoder
Models/qlora.py - QLoRA Fine-Tuning → Tokenizer: uses tokenizer
Models/qlora.py