# meetvasoya10/transformer-blocks architecture

> Architecture diagram of the GitHub repository meetvasoya10/transformer-blocks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/meetvasoya10/transformer-blocks
- Repository: https://github.com/meetvasoya10/transformer-blocks

## Overview

##### Project Type and Purpose
- This is an educational, open-source PyTorch library providing modular Transformer building blocks (token embedding, positional encoding, multi-head attention, encoder/decoder layers).
- It’s not a full training or serving system but a toolkit to assemble and experiment with Transformer architectures for pre-training and research.

##### File-Structure Analysis
- Top‐level modules:
- token-embeddings/token_embedding.py
- positional-encoding/positional_encoding.py
- multi-head-attention/multi_head_attention.py
- encoder-layer/encoder_layer.py
- decoder-layer/decoder_layer.py
- Each directory defines one core Transformer block.
- requirements.txt indicates a PyTorch dependency; README.md describes how to assemble these into a full encoder/decoder model.

##### Key README Insights
- Core classes: TokenEmbedding, PositionalEncoding, MultiHeadAttention, FullEncoder, FullDecoder.
- Demonstrates how to compose blocks into a Transformer (Encoder–Decoder) for MLM and sequence reconstruction.
- Modularity: each block is independent, well-documented, and meant to be stacked.

##### Main Components
a. Data Inputs
– Source tokens (src)
– Target tokens (tgt)
b. Preprocessing Blocks
– TokenEmbedding
– PositionalEncoding
c. Attention & Feed-Forward Blocks
– MultiHeadAttention (self-attention in encoder; masked self-attention in decoder; cross-attention in decoder)
– Feed-forward sublayer (inside encoder_layer.py and decoder_layer.py)
– LayerNorm & residual connections (inside each layer)
d. Higher-Level Assemblies
– EncoderLayer (one block)
– FullEncoder (stack of N EncoderLayer)
– DecoderLayer (one block)
– FullDecoder (stack of N DecoderLayer)
e. Final Output Projection
– Linear + softmax (implied in FullDecoder or user-provided head)

##### Relationships and Data Flow
1. src → TokenEmbedding → PositionalEncoding → repeated [
- Self-MultiHeadAttention → Add & Norm
- Feed-Forward → Add & Norm
] (FullEncoder) → encoder_output
2. tgt → TokenEmbedding → PositionalEncoding → repeated [
- Masked Self-MultiHeadAttention → Add & Norm
- Cross-MultiHeadAttention(encoder_output) → Add & Norm
- Feed-Forward → Add & Norm
] (FullDecoder) → decoder_output → projection → logits

##### Architectural Patterns and Design Principles
- Layered architecture: blocks composed by stacking identical layers.
- Modularity: each block encapsulates one core function (attention, FFN, embedding).
- Residual + normalization pattern throughout.

##### Technology Stack
- Python 3.x, PyTorch
- Standard data-flow via torch.nn.Module
- No training loops or data-loaders included

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Inputs
    src["src Tokens"]:::io
    tgt["tgt Tokens"]:::io

    %% Embedding & Encoding
    src --> TE
    TE["TokenEmbedding"]:::embeddings
    click TE "https://github.com/meetvasoya10/transformer-blocks/blob/main/token-embeddings/token_embedding.py"
    TE --> PE
    PE["PositionalEncoding"]:::embeddings
    click PE "https://github.com/meetvasoya10/transformer-blocks/blob/main/positional-encoding/positional_encoding.py"
    PE --> FE

    subgraph "FullEncoder" 
        direction TB
        FE["FullEncoder (N× EncoderLayer)"]:::assembly
        click FE "https://github.com/meetvasoya10/transformer-blocks/blob/main/encoder-layer/encoder_layer.py"
        subgraph "EncoderLayer" 
            direction TB
            MHA1["MultiHeadAttention"]:::attention
            click MHA1 "https://github.com/meetvasoya10/transformer-blocks/blob/main/multi-head-attention/multi_head_attention.py"
            AN1["Add & Norm"]:::user
            FF1["Feed-Forward"]:::ff
            AN2["Add & Norm"]:::user
            MHA1 --> AN1 --> FF1 --> AN2
        end
        FE --> AN2
    end

    AN2 --> EO["encoder_output"]:::io

    %% Decoding
    tgt --> TE2
    TE2["TokenEmbedding"]:::embeddings
    click TE2 "https://github.com/meetvasoya10/transformer-blocks/blob/main/token-embeddings/token_embedding.py"
    TE2 --> PE2
    PE2["PositionalEncoding"]:::embeddings
    click PE2 "https://github.com/meetvasoya10/transformer-blocks/blob/main/positional-encoding/positional_encoding.py"
    PE2 --> FD

    subgraph "FullDecoder"
        direction TB
        FD["FullDecoder (N× DecoderLayer)"]:::assembly
        click FD "https://github.com/meetvasoya10/transformer-blocks/blob/main/decoder-layer/decoder_layer.py"
        subgraph "DecoderLayer"
            direction TB
            MHA2["Masked Self-Attention"]:::attention
            AN3["Add & Norm"]:::user
            MHA3["Cross-Attention"]:::attention
            AN4["Add & Norm"]:::user
            FF2["Feed-Forward"]:::ff
            AN5["Add & Norm"]:::user
            EO --> MHA3
            PE2 --> MHA2
            MHA2 --> AN3 --> MHA3 --> AN4 --> FF2 --> AN5
        end
        FD --> AN5
    end

    AN5 --> DP["Decoder Output"]:::io
    DP --> PROJ

    %% Projection
    PROJ["Linear + Softmax"]:::projection

    %% Legend
    subgraph Legend
        direction TB
        E["Embeddings"]:::embeddings
        A["Attention"]:::attention
        F["Feed-Forward"]:::ff
        L["Layer Container"]:::layer
        S["Stack Container"]:::assembly
        P["Projection"]:::projection
    end

    classDef io fill:#FFFFFF,stroke:#333,stroke-width:1px;
    classDef embeddings fill:#ADD8E6,stroke:#333,stroke-width:1px;
    classDef attention fill:#98FB98,stroke:#333,stroke-width:1px;
    classDef ff fill:#FFFACD,stroke:#333,stroke-width:1px;
    classDef layer fill:#D8BFD8,stroke:#333,stroke-width:1px;
    classDef assembly fill:#FFA500,stroke:#333,stroke-width:1px;
    classDef projection fill:#D3D3D3,stroke:#333,stroke-width:1px;
    classDef user stroke-dasharray: 5 5;
```
