# arian-urdu/tinieststoryteller architecture

> Architecture diagram of the GitHub repository arian-urdu/tinieststoryteller, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arian-urdu/tinieststoryteller
- Repository: https://github.com/arian-urdu/tinieststoryteller

## Overview

##### Main Components

- Data Layer:
- The “data” folder contains training and validation data (e.g., TinyStories-valid.txt).
- Additional data aspects may involve fetching the full dataset from external sources (as indicated in the README).

- Preprocessing Module:
- Located in the “preprocessing” directory, it houses the scripts that clean and filter raw text (e.g., remove_forbidden_letters.py, filter_data_with_ollama.py).
- Responsible for preparing textual data before training.

- Core Model Training:
- train.py initiates the training process using settings defined in config.py.
- transformer_model.py defines the Transformer architecture – the central component of the model building.

- Text Generation:
- generate.py is used to generate text using a trained model, loading checkpoints from the output folder.

- Evaluation and Benchmarking:
- The “evaluation” and “benchmarks” directories contain various scripts and datasets for both automatic and manual evaluation (e.g., rouge_and_bleu_score.py, evaluation of creativity, manual evaluation logic).
- This component ensures model performance can be quantitatively and qualitatively assessed.

- Curriculum Learning:
- The “curriculum_learning” folder includes scripts and files for organizing training curriculum (e.g., create_curriculum.py, layer_freezing.py).
- This indicates experimentation with progressive learning strategies.

- Tokenization:
- The “tokenizers” folder provides various tokenization methods (BPE, unigram, wordpiece) along with their configuration (JSON) files.
- Important for converting raw text into inputs suitable for the Transformer.

- Reporting:
- The “report” directory holds a LaTeX-based report (with auxiliary files, configuration for fonts, etc.), indicating that model results and findings are documented.

- Configuration & Miscellaneous:
- config.py defines crucial training hyperparameters (batch_size, block_size, etc.) and logging (e.g., wandb).
- Other scripts like test.py and demo/demo.py show examples/demos of functionality.

##### Relationships and Interactions

- Data flows through the Preprocessing module (cleaning, filtering) before entering the training pipeline.
- The training component uses preprocessed data and configuration parameters to instantiate and train the transformer model.
- The trained model can then be evaluated (using evaluation/benchmarks scripts) and used for text generation.
- Tokenizers are an integral part of both preprocessing and training as they convert text data to numerical tokens.
- Curriculum Learning strategies influence the training process (e.g., via layer freezing or curriculum configuration files).
- Reporting and demos capture outputs and evaluation metrics for documentation and presentation.

##### Architectural Patterns and Design Principles

- Separation of Concerns: Each component is isolated in its directory (data, preprocessing, training, evaluation) for clarity and maintainability.
- Modularity: Independent modules (tokenizers, curriculum learning, evaluation scripts) can be updated or replaced without affecting the core training logic.
- Pipeline Architecture: The project follows a linear data flow starting from data acquisition to preprocessing, training, evaluation, and finally text generation.
- Configurability: Using config.py provides quick adjustments to hyperparameters and experimental settings.

##### Technologies, Frameworks, and Libraries

- Python is the central language, as illustrated by the multitude of Python scripts.
- PyTorch is implied given the usage of torch.load for checkpoint loading within generate.py.
- Wandb might be used for logging based on the config instructions.
- LaTeX is utilized in the reporting module for creating professional academic reports.

- Data Source → Preprocessing → Training (including tokenization as a part or a submodule) → Model Checkpoint → Text Generation.
- Parallel to training, include Evaluation and Benchmarking as interactive modules that feed back into training adjustments.
- Lastly, add Reporting as an output channel that formalizes evaluation findings.

- Detail any notable configuration points (e.g., configurable hyperparameters and curriculum design choices) to ensure clarity in system dependencies.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Data Layer"]:::data
    B["Preprocessing Module"]:::preprocess
    C["Tokenization Module"]:::tokenizer

    subgraph "Core Model Training"
        D["train.py"]:::training
        E["transformer_model.py"]:::training
        F["config.py"]:::training
    end

    G["Curriculum Learning"]:::curriculum
    H["Text Generation"]:::textgen
    I1["Evaluation Scripts"]:::evaluation
    I2["Benchmark Scripts"]:::evaluation
    J["Reporting"]:::report

    A -->|"prepares"| B
    B -->|"cleans"| C
    C -->|"tokenizes"| D
    G -->|"influences"| D
    D -->|"model_checkpoint"| H
    D -->|"model_checkpoint"| I1
    D -->|"model_checkpoint"| I2
    I1 -->|"feedback"| D
    I1 -->|"outputs"| J
    I2 -->|"outputs"| J

    click A "https://github.com/arian-urdu/tinieststoryteller/blob/main/data/TinyStories-valid.txt"
    click B "https://github.com/arian-urdu/tinieststoryteller/tree/main/preprocessing/"
    click C "https://github.com/arian-urdu/tinieststoryteller/tree/main/tokenizers/"
    click D "https://github.com/arian-urdu/tinieststoryteller/blob/main/train.py"
    click E "https://github.com/arian-urdu/tinieststoryteller/blob/main/transformer_model.py"
    click F "https://github.com/arian-urdu/tinieststoryteller/blob/main/config.py"
    click G "https://github.com/arian-urdu/tinieststoryteller/tree/main/curriculum_learning/"
    click H "https://github.com/arian-urdu/tinieststoryteller/blob/main/generate.py"
    click I1 "https://github.com/arian-urdu/tinieststoryteller/tree/main/evaluation/"
    click I2 "https://github.com/arian-urdu/tinieststoryteller/tree/main/benchmarks/"
    click J "https://github.com/arian-urdu/tinieststoryteller/tree/main/report/"

    classDef data fill:#ffcccc,stroke:#990000,stroke-width:2px;
    classDef preprocess fill:#ccffcc,stroke:#006600,stroke-width:2px;
    classDef tokenizer fill:#ccffff,stroke:#008080,stroke-width:2px;
    classDef training fill:#ffffcc,stroke:#cccc00,stroke-width:2px;
    classDef curriculum fill:#ffccff,stroke:#cc0066,stroke-width:2px;
    classDef textgen fill:#e6e6fa,stroke:#9370db,stroke-width:2px;
    classDef evaluation fill:#d3d3d3,stroke:#696969,stroke-width:2px;
    classDef report fill:#ffe4b5,stroke:#cd853f,stroke-width:2px;
```
