# kr4t0n/tiny-language-model architecture

> Architecture diagram of the GitHub repository kr4t0n/tiny-language-model, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kr4t0n/tiny-language-model
- Repository: https://github.com/kr4t0n/tiny-language-model

## Overview

##### Main Components

A. Training Pipeline Components:
- Data Processing Layer (data_utils.py)
- Model Architecture (modeling_tlm.py)
- Training Infrastructure (train.py, modal_train.py)

B. Project Variants:
- Basic Implementation (tlm folder)
- Professional Implementation (tlm-pro folder)

**┌─────────────────────────────────────┐**
- TLM Project               │
- ┬─────────────────────┘
- ┌───────────┴────────────┐
- ▼                        ▼
**┌─────────┐            ┌──────────┐**
- TLM   │            │ TLM-Pro  │
- ┘            └──────────┘

**┌─────────────────────────────────────────────────┐**
- Data Layer                      │
- (data_utils.py)                     │
- ┬─────────────────────────┘
- ▼
**┌─────────────────────────────────────────────────┐**
- Model Architecture                  │
- (modeling_tlm.py)                    │
- ┬─────────────────────────┘
- ▼
**┌─────────────────────────────────────────────────┐**
- Training Pipeline                   │
- (train.py, modal_train.py)              │
- ┘

##### Important Elements to Highlight

a. Data Flow:
- How data moves from raw text files (train.txt, valid.txt) through the processing pipeline
- Transformation steps in data_utils.py

b. Model Architecture:
- Llama-style architecture components in modeling_tlm.py
- Connection between model definition and training process

c. Training Infrastructure:
- Relationship between train.py and modal_train.py
- Validation/evaluation feedback loops

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Storage Layer
    subgraph DataStorage["Data Storage Layer"]
        TrainData[("Training Data")]:::data
        ValidData[("Validation Data")]:::data
        ModelCheckpoints[("Model Checkpoints")]:::data
    end

    %% Data Processing Layer
    subgraph DataProcessing["Data Processing Layer"]
        DataUtils["Data Utils"]:::processing
        Tokenization["Tokenization"]:::processing
        Preprocessing["Text Preprocessing"]:::processing
        DataQuality["Data Quality Checks"]:::validation
    end

    %% Model Architecture Layer
    subgraph ModelArch["Model Architecture Layer"]
        EmbeddingLayer["Embedding Layer"]:::model
        subgraph AttentionBlock["Self-Attention Blocks"]
            MultiHead["Multi-Head Attention"]:::model
            AttentionMask["Attention Masks"]:::model
        end
        FFN["Feed-Forward Networks"]:::model
        LayerNorm["Layer Normalization"]:::model
        OutputLayer["Output Layer"]:::model
    end

    %% Training Infrastructure Layer
    subgraph TrainingInfra["Training Infrastructure"]
        TrainingLoop["Training Loop"]:::training
        ModalTrain["Distributed Training"]:::training
        ValidationProcess["Validation Process"]:::validation
        Metrics["Performance Metrics"]:::validation
    end

    %% Environment Layer
    subgraph Environments["Execution Environments"]
        DevEnv["Interactive Environment"]:::env
        ProdEnv["Production Environment"]:::env
    end

    %% Configuration Layer
    subgraph Config["Configuration Management"]
        Requirements["Dependencies"]:::config
        Hyperparameters["Model Hyperparameters"]:::config
    end

    %% Data Flow Connections
    TrainData --> DataUtils
    ValidData --> DataUtils
    DataUtils --> Tokenization
    Tokenization --> Preprocessing
    Preprocessing --> DataQuality
    DataQuality --> EmbeddingLayer

    EmbeddingLayer --> MultiHead
    MultiHead --> AttentionMask
    AttentionMask --> FFN
    FFN --> LayerNorm
    LayerNorm --> OutputLayer

    OutputLayer --> TrainingLoop
    TrainingLoop --> ModelCheckpoints
    TrainingLoop --> ValidationProcess
    ValidationProcess --> Metrics
    Metrics --> TrainingLoop

    ModalTrain --> TrainingLoop
    Requirements --> ProdEnv
    Hyperparameters --> TrainingLoop

    %% Environment Connections
    DevEnv --> DataUtils
    ProdEnv --> DataUtils

    %% Click Events
    click DataUtils "https://github.com/kr4t0n/tiny-language-model/blob/main/tlm-pro/tlm_pro/data_utils.py"
    click ModelArch "https://github.com/kr4t0n/tiny-language-model/blob/main/tlm-pro/tlm_pro/modeling_tlm.py"
    click TrainingLoop "https://github.com/kr4t0n/tiny-language-model/blob/main/tlm-pro/tlm_pro/train.py"
    click ModalTrain "https://github.com/kr4t0n/tiny-language-model/blob/main/tlm-pro/modal_train.py"
    click DevEnv "https://github.com/kr4t0n/tiny-language-model/blob/main/tlm/tlm.ipynb"
    click TrainData "https://github.com/kr4t0n/tiny-language-model/blob/main/tlm/train.txt"
    click ValidData "https://github.com/kr4t0n/tiny-language-model/blob/main/tlm/valid.txt"
    click ProdEnv "https://github.com/kr4t0n/tiny-language-model/tree/main/tlm-pro/tlm_pro/"
    click Requirements "https://github.com/kr4t0n/tiny-language-model/blob/main/tlm-pro/requirements.txt"

    %% Styling
    classDef data fill:#gray,stroke:#333,stroke-width:2px
    classDef processing fill:#3498db,stroke:#2980b9,color:white
    classDef model fill:#2ecc71,stroke:#27ae60,color:white
    classDef training fill:#e67e22,stroke:#d35400,color:white
    classDef validation fill:#9b59b6,stroke:#8e44ad,color:white
    classDef env fill:#34495e,stroke:#2c3e50,color:white
    classDef config fill:#95a5a6,stroke:#7f8c8d,color:white

    %% Legend
    subgraph Legend["Legend"]
        L1["Data Storage"]:::data
        L2["Processing Component"]:::processing
        L3["Model Component"]:::model
        L4["Training Component"]:::training
        L5["Validation Component"]:::validation
        L6["Environment"]:::env
        L7["Configuration"]:::config
    end
```
