How arian-urdu/tinieststoryteller works
Main Components
• Data Layer:
• The “data” folder contains training and validation data (e.g., TinyStories-valid.txt).
• Additional data aspects may involve fetching the full dataset from external sources (as indicated in the README).
• Preprocessing Module:
• Located in the “preprocessing” directory, it houses the scripts that clean and filter raw text (e.g., remove_forbidden_letters.py, filter_data_with_ollama.py).
• Responsible for preparing textual data before training.
• Core Model Training:
• train.py initiates the training process using settings defined in config.py.
• transformer_model.py defines the Transformer architecture – the central component of the model building.
• Text Generation:
• generate.py is used to generate text using a trained model, loading checkpoints from the output folder.
• Evaluation and Benchmarking:
• The “evaluation” and “benchmarks” directories contain various scripts and datasets for both automatic and manual evaluation (e.g., rouge_and_bleu_score.py, evaluation of creativity, manual evaluation logic).
• This component ensures model performance can be quantitatively and qualitatively assessed.
• Curriculum Learning:
• The “curriculum_learning” folder includes scripts and files for organizing training curriculum (e.g., create_curriculum.py, layer_freezing.py).
• This indicates experimentation with progressive learning strategies.
• Tokenization:
• The “tokenizers” folder provides various tokenization methods (BPE, unigram, wordpiece) along with their configuration (JSON) files.
• Important for converting raw text into inputs suitable for the Transformer.
• Reporting:
• The “report” directory holds a LaTeX-based report (with auxiliary files, configuration for fonts, etc.), indicating that model results and findings are documented.
• Configuration & Miscellaneous:
• config.py defines crucial training hyperparameters (batch_size, block_size, etc.) and logging (e.g., wandb).
• Other scripts like test.py and demo/demo.py show examples/demos of functionality.
Relationships and Interactions
• Data flows through the Preprocessing module (cleaning, filtering) before entering the training pipeline.
• The training component uses preprocessed data and configuration parameters to instantiate and train the transformer model.
• The trained model can then be evaluated (using evaluation/benchmarks scripts) and used for text generation.
• Tokenizers are an integral part of both preprocessing and training as they convert text data to numerical tokens.
• Curriculum Learning strategies influence the training process (e.g., via layer freezing or curriculum configuration files).
• Reporting and demos capture outputs and evaluation metrics for documentation and presentation.
Architectural Patterns and Design Principles
• Separation of Concerns: Each component is isolated in its directory (data, preprocessing, training, evaluation) for clarity and maintainability.
• Modularity: Independent modules (tokenizers, curriculum learning, evaluation scripts) can be updated or replaced without affecting the core training logic.
• Pipeline Architecture: The project follows a linear data flow starting from data acquisition to preprocessing, training, evaluation, and finally text generation.
• Configurability: Using config.py provides quick adjustments to hyperparameters and experimental settings.
Technologies, Frameworks, and Libraries
• Python is the central language, as illustrated by the multitude of Python scripts.
• PyTorch is implied given the usage of torch.load for checkpoint loading within generate.py.
• Wandb might be used for logging based on the config instructions.
• LaTeX is utilized in the reporting module for creating professional academic reports.
• Data Source → Preprocessing → Training (including tokenization as a part or a submodule) → Model Checkpoint → Text Generation.
• Parallel to training, include Evaluation and Benchmarking as interactive modules that feed back into training adjustments.
• Lastly, add Reporting as an output channel that formalizes evaluation findings.
• Detail any notable configuration points (e.g., configurable hyperparameters and curriculum design choices) to ensure clarity in system dependencies.