How bhooyas/storynet works
Project Type and Purpose
• This is a machine-learning pipeline for a decoder-only transformer specialized in story generation.
• It includes data preprocessing, tokenizer training, model pretraining, fine-tuning, and inference steps.
Main Architectural Components
a. Data Ingestion & Preparation
• process_data.py reads raw text (story.txt and other story corpora)
• Builds/updates the tokenizer model (tokenizer.yttm)
b. Model Configuration
• config.py holds hyperparameters (e.g., layer counts, embedding sizes, learning rates)
c. Model Definition & Utilities
• layers.py: transformer block definitions
• model.py: assembles layers into a decoder-only transformer
• utils.py: helper routines (e.g., data loaders, checkpoint I/O)
d. Training Pipeline
• pre_train.py: trains from scratch on a large corpus → produces checkpoint (e.g., transformer.safetensors)
• train.py: fine-tunes the pretrained model on target data
e. Inference Pipeline
• inference.py: loads a checkpoint, tokenizes input, generates story continuations, detokenizes output
f. Artifacts & Storage
• tokenizer.yttm: subword tokenization model
• transformer.safetensors & other checkpoints
Component Interactions and Data Flow
• Raw story data → process_data
↳ trains/updates tokenizer → tokenizer.yttm
↳ outputs tokenized training sets
• Tokenized data + config → pre_train → pretrained checkpoint
• Pretrained checkpoint + tokenized dataset → train → fine-tuned checkpoint
• Fine-tuned checkpoint + inference input → inference → generated text
Architectural Patterns & Design Principles
• Pipeline pattern: sequential stages for data prep, training, inference
• Modularization: each Python script addresses a single responsibility
• Config-driven: hyperparameters isolated in config.py for easy experimentation
• Checkpointing: model weights saved and re-used across stages
Technologies & Frameworks
• Python 3.x
• Byte-Pair Encoding via youtokentome (tokenizer.yttm)
• PyTorch (implied by .safetensors format, typical for transformer weights)
• Standard Python libraries (numpy, logging, etc.) as per requirements.txt