# bhargavmupparisetty/bi-gram-transformer architecture

> Architecture diagram of the GitHub repository bhargavmupparisetty/bi-gram-transformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bhargavmupparisetty/bi-gram-transformer
- Repository: https://github.com/bhargavmupparisetty/bi-gram-transformer

## Overview

##### Project type and purpose
- This is a machine‐learning research/demo project: a Bigram Transformer text‐generation model trained on Shakespeare’s works, implemented in Python using PyTorch. Its purpose is to showcase bigram tokenization and a simplified transformer architecture for language modeling and generation.

##### File structure
- Top level:
- README.md – project overview, features, dependencies, usage
- Transformer.ipynb – Jupyter notebook containing all code for data loading, preprocessing, model definition, training loop, and text generation
- No separate “src” or “modules” directories, so everything is self‐contained in the notebook.
- Key configuration is implicit in code cells (hyperparameters, model dimensions, training settings).

##### README for additional insights
- Core components described: bigram tokenization, transformer architecture, text generation
- Toolchain: Python, PyTorch, NumPy/Pandas, Shakespeare dataset
- Future enhancements hint at possible architectural extensions (n‐gram tokenization, positional encodings, larger models)

##### Main components
a. Data source
– Shakespeare text files (raw `.txt` dataset)
b. Preprocessing pipeline
– Text loading → bigram tokenizer → token‐index mapping → numerical dataset
c. Data loader
– PyTorch Dataset & DataLoader for batching
d. Model architecture
– Embedding layer
– Self‐attention blocks (query/key/value, softmax attention)
– Feed‐forward layers
– Output linear + softmax to predict next token
e. Training loop
– Forward pass → loss computation → backward pass → optimizer step → checkpointing/logging
f. Inference/generation module
– Seed prompt → autoregressive sampling loop → text detokenization
g. Supporting libraries
– PyTorch, NumPy, Pandas

##### Relationships and interactions
- Data flows from raw text into tokenizer, then into the Dataset object.
- DataLoader feeds batches to the model.
- Model emits predictions → loss → gradients → optimizer updates parameters.
- Checkpoints/logs record model state and metrics.
- For generation, the trained model takes a seed, iteratively samples bigram tokens, and converts them back to text.

##### Architectural patterns and design principles
- Pipeline separation: distinct stages for preprocessing, training, and inference
- Modular ML design: clear interfaces between data handling, model definition, and training logic
- Single‐notebook approach: best for research prototyping but could evolve into a folder structure with `data/`, `models/`, `train.py`, `generate.py` modules for production

##### Elements
- Technology callouts near each component (e.g., “PyTorch nn.Module,” “NumPy/Pandas”)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Documentation
    Overview["Project Overview & Dependencies"]:::doc
    CoreComp["Transformer Notebook (all core components)"]:::doc

    %% Data Components
    RawText["Shakespeare Raw Text Files"]:::data
    Tokenizer["Bigram Tokenizer"]:::pre
    TokenIndex["Token-Index Mapping"]:::pre
    Dataset["Numerical Dataset (PyTorch Dataset)"]:::pre
    DataLoader["DataLoader (Batching)"]:::pre

    %% Model Internals
    Embedding["Embedding Layer"]:::model
    SelfAttn["Self-Attention Block"]:::model
    FFN["Feed-Forward Layer"]:::model
    Output["Output Linear + Softmax"]:::model
    TrainedModel["Trained Model"]:::model

    %% Training Pipeline
    Loss["Loss Computation"]:::train
    Optimizer["Optimizer Step"]:::train
    Checkpoint["Checkpointing/Logging"]:::train

    %% Inference Pipeline
    Seed["Seed Prompt"]:::train
    GenLoop["Autoregressive Generation Loop"]:::train
    Detoken["Text Detokenization"]:::train
    Generated["Generated Text"]:::train

    %% Supporting Libraries
    PyTorch["PyTorch"]:::lib
    NumPy["NumPy / Pandas"]:::lib

    %% Data Flow
    RawText --> Tokenizer --> TokenIndex --> Dataset --> DataLoader --> Embedding
    Embedding --> SelfAttn --> FFN --> Output --> Loss --> Optimizer --> Checkpoint --> TrainedModel
    TrainedModel --> Seed --> GenLoop --> Detoken --> Generated

    %% Library Dependencies
    PyTorch -.-> DataLoader
    PyTorch -.-> Embedding
    NumPy -.-> Tokenizer
    NumPy -.-> TokenIndex

    %% Documentation Click Events
    click Overview "https://github.com/bhargavmupparisetty/bi-gram-transformer/blob/main/README.md"
    click CoreComp "https://github.com/bhargavmupparisetty/bi-gram-transformer/blob/main/Transformer.ipynb"

    %% Library Click Events
    click PyTorch "https://github.com/bhargavmupparisetty/bi-gram-transformer/blob/main/README.md"
    click NumPy "https://github.com/bhargavmupparisetty/bi-gram-transformer/blob/main/README.md"

    %% Styles
    classDef data fill:#f0f8ff,stroke:#000,stroke-width:1px;
    classDef pre fill:#add8e6,stroke:#000,stroke-width:1px;
    classDef model fill:#98fb98,stroke:#000,stroke-width:1px;
    classDef train fill:#ffdab9,stroke:#000,stroke-width:1px;
    classDef lib fill:#ffe4e1,stroke:#000,stroke-width:1px;
    classDef doc fill:#fafad2,stroke:#000,stroke-width:1px;
```
