# bhooyas/storynet architecture

> Architecture diagram of the GitHub repository bhooyas/storynet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bhooyas/storynet
- Repository: https://github.com/bhooyas/storynet

## Overview

##### Project Type and Purpose
- This is a machine-learning pipeline for a decoder-only transformer specialized in story generation.
- It includes data preprocessing, tokenizer training, model pretraining, fine-tuning, and inference steps.

##### Main Architectural Components
a. Data Ingestion & Preparation
- process_data.py reads raw text (`story.txt` and other story corpora)
- Builds/updates the tokenizer model (`tokenizer.yttm`)
b. Model Configuration
- config.py holds hyperparameters (e.g., layer counts, embedding sizes, learning rates)
c. Model Definition & Utilities
- layers.py: transformer block definitions
- model.py: assembles layers into a decoder-only transformer
- utils.py: helper routines (e.g., data loaders, checkpoint I/O)
d. Training Pipeline
- pre_train.py: trains from scratch on a large corpus → produces checkpoint (e.g., `transformer.safetensors`)
- train.py: fine-tunes the pretrained model on target data
e. Inference Pipeline
- inference.py: loads a checkpoint, tokenizes input, generates story continuations, detokenizes output
f. Artifacts & Storage
- tokenizer.yttm: subword tokenization model
- transformer.safetensors & other checkpoints

##### Component Interactions and Data Flow
- Raw story data → process_data
↳ trains/updates tokenizer → tokenizer.yttm
↳ outputs tokenized training sets
- Tokenized data + config → pre_train → pretrained checkpoint
- Pretrained checkpoint + tokenized dataset → train → fine-tuned checkpoint
- Fine-tuned checkpoint + inference input → inference → generated text

##### Architectural Patterns & Design Principles
- Pipeline pattern: sequential stages for data prep, training, inference
- Modularization: each Python script addresses a single responsibility
- Config-driven: hyperparameters isolated in config.py for easy experimentation
- Checkpointing: model weights saved and re-used across stages

##### Technologies & Frameworks
- Python 3.x
- Byte-Pair Encoding via `youtokentome` (tokenizer.yttm)
- PyTorch (implied by `.safetensors` format, typical for transformer weights)
- Standard Python libraries (numpy, logging, etc.) as per requirements.txt

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preparation Stage
    subgraph "Data Preparation"
        raw_data["Raw Story Corpus\n(story.txt)"]:::storage
        process_data["process_data.py"]:::process
        tokenized_data["Tokenized Dataset"]:::storage
        tokenizer_model["Tokenizer Model\n(tokenizer.yttm)"]:::storage
    end

    %% Configuration
    config["config.py"]:::config

    %% Model Definition & Utilities
    subgraph "Model Definition"
        layers["layers.py"]:::code
        model["model.py"]:::code
        utils["utils.py"]:::code
    end

    %% Training Stage
    subgraph "Training"
        pre_train["pre_train.py"]:::process
        pretrained_ckpt["Pretrained Checkpoint\n(transformer.safetensors)"]:::storage
        train["train.py"]:::process
        fine_tuned_ckpt["Fine-Tuned Checkpoint"]:::storage
    end

    %% Inference Stage
    subgraph "Inference"
        inference["inference.py"]:::process
        generated_text["Generated Story Output"]:::output
    end

    %% Data Flows
    raw_data -->|"raw text"| process_data
    process_data -->|"tokenized data"| tokenized_data
    process_data -->|"updates tokenizer"| tokenizer_model

    tokenized_data -->|"tokenized data"| pre_train
    config -->|config params| pre_train
    pre_train -->|"checkpoint"| pretrained_ckpt

    pretrained_ckpt -->|"pretrained checkpoint"| train
    tokenized_data -->|"tokenized data"| train
    config -->|config params| train
    train -->|"fine-tuned checkpoint"| fine_tuned_ckpt

    fine_tuned_ckpt -->|"checkpoint"| inference
    tokenizer_model -->|"tokenizer model"| inference
    config -->|config params| inference
    inference -->|"generated text"| generated_text

    %% Click Events
    click process_data "https://github.com/bhooyas/storynet/blob/main/process_data.py"
    click raw_data "https://github.com/bhooyas/storynet/blob/main/story.txt"
    click tokenizer_model "https://github.com/bhooyas/storynet/blob/main/tokenizer.yttm"
    click config "https://github.com/bhooyas/storynet/blob/main/config.py"
    click layers "https://github.com/bhooyas/storynet/blob/main/layers.py"
    click model "https://github.com/bhooyas/storynet/blob/main/model.py"
    click utils "https://github.com/bhooyas/storynet/blob/main/utils.py"
    click pre_train "https://github.com/bhooyas/storynet/blob/main/pre_train.py"
    click train "https://github.com/bhooyas/storynet/blob/main/train.py"
    click pretrained_ckpt "https://github.com/bhooyas/storynet/blob/main/transformer.safetensors"
    click inference "https://github.com/bhooyas/storynet/blob/main/inference.py"

    %% Styles
    classDef process fill:#bbdefb,stroke:#1e88e5,stroke-width:2px
    classDef storage fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
    classDef config fill:#ffe0b2,stroke:#e65100,stroke-width:2px
    classDef code fill:#e1bee7,stroke:#6a1b9a,stroke-width:2px
    classDef output fill:#ffccbc,stroke:#bf360c,stroke-width:2px
```
