# tranquoctrinh/transformer architecture

> Architecture diagram of the GitHub repository tranquoctrinh/transformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tranquoctrinh/transformer
- Repository: https://github.com/tranquoctrinh/transformer

## Overview

##### Project Type and Purpose
- This is an open-source PyTorch implementation of the "Transformer – Attention is All You Need" model, tailored for English→Vietnamese machine translation.
- It’s not a web or full-stack app but an ML research/training/inference pipeline composed of data handling, model definition, training, evaluation, and inference modules.

##### File‐Structure Analysis
- Top-level scripts:
- datasets.py – data loading and preprocessing
- models.py – Transformer model components (PositionalEncoder, SelfAttention, MultiHeadAttention, EncoderLayer, DecoderLayer, Encoder, Decoder, Transformer)
- train.py – training loop, checkpointing, logging
- evaluate.py – BLEU‐score evaluation on validation set
- translate.py – inference (load model + tokenizer; beam‐search decode)
- utils.py – hyperparameters, config loader, tokenization utilities
- logs/ – runtime logs and artifacts (loss curves, predictions)
- .gitignore, LICENSE, README.md

##### Key Architectural Components
a. Data Ingestion & Preprocessing (datasets.py)
– Reads bilingual corpus (data_en_vi/)
– Tokenizes sentences, builds vocabularies, creates DataLoader with source/target tensors and masks
b. Model Definition (models.py)
– Positional Encoding
– Scaled Dot-Product Self-Attention
– Multi-Head Attention
– Encoder Layer (self-attention + feed-forward + residual + norm)
– Decoder Layer (self-attention + encoder-decoder attention + feed-forward + residual + norm)
– Encoder Stack (N layers) + embedding + positional encoding
– Decoder Stack (N layers) + embedding + positional encoding
– Final linear projection to target vocabulary
c. Training Pipeline (train.py)
– Loads config from utils
– Instantiates Transformer model, optimizer, loss criterion
– Iterates epochs: forward pass → compute cross-entropy loss → backward → optimizer step → early stopping → log metrics to logs/
d. Evaluation Module (evaluate.py)
– Loads best checkpoint + tokenizers
– Runs validation set → decodes predictions → computes BLEU-2/3/4 via NLTK → logs results
e. Inference Service (translate.py)
– Loads pretrained model + tokenizers
– Defines translate() with beam search over decoder
– Exposes simple script/API to translate input strings

##### Detail Level
- Don’t just show “Model” as a monolith; expand to Encoder vs. Decoder, multi-head attention vs. feed-forward
- Masks (source_mask, target_mask) flowing into attention sub-modules

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        RawData["Data Files\n(data_en_vi/)"]:::storage
        DataLoader["Data Loader\n(datasets.py)"]:::data
    end

    %% Preprocessing & Tokenization
    subgraph "Preprocessing & Tokenization"
        Tokenizer["Tokenizer\n(utils.py)"]:::preprocess
        TokenizedBatches["Tokenized Batches\nwith Masks"]:::preprocess
    end

    %% Model Layer
    subgraph "Model Layer"
        subgraph "Encoder Stack" 
            PE["Positional Encoding"]:::model
            SA["Self-Attention"]:::model
            MHA_E["Multi-Head Attention"]:::model
            FFN_E["Feed-Forward"]:::model
        end
        subgraph "Decoder Stack"
            PE2["Positional Encoding"]:::model
            SA2["Self-Attention"]:::model
            EncDecAtt["Encoder-Decoder Attention"]:::model
            MHA_D["Multi-Head Attention"]:::model
            FFN_D["Feed-Forward"]:::model
        end
        LinearProj["Final Linear Projection\n(models.py)"]:::model
    end

    %% Training / Evaluation / Inference Control Flow
    subgraph "Workflows"
        TrainLoop["Training Loop\n(train.py)"]:::train
        EvalModule["BLEU Evaluator\n(evaluate.py)"]:::train
        InferModule["Beam Search Inference\n(translate.py)"]:::inference
    end

    %% Logging & Artifacts
    subgraph "Logging & Artifacts"
        Logs["Logs & Checkpoints\n(logs/)"]:::storage
    end

    %% External Dependencies
    subgraph "External Libraries"
        PyTorch["PyTorch"]:::external
        NLTK["NLTK (BLEU)"]:::external
        gdown["gdown (Data Download)"]:::external
    end

    %% Data Flow
    RawData -->|reads| DataLoader
    DataLoader -->|outputs| TokenizedBatches
    TokenizedBatches -->|input| PE
    TokenizedBatches -->|input| PE2

    %% Model internals flow
    PE --> SA
    SA --> MHA_E
    MHA_E --> FFN_E
    FFN_E -->|Encoder Output| LinearProj

    PE2 --> SA2
    SA2 --> EncDecAtt
    EncDecAtt --> MHA_D
    MHA_D --> FFN_D
    FFN_D -->|Decoder Output| LinearProj

    LinearProj -->|logits| TrainLoop

    %% Training Flow
    TrainLoop -->|saves| Logs
    TrainLoop -->|depends on| PyTorch
    TrainLoop -->|uses| Tokenizer
    TrainLoop -->|loads config| Tokenizer

    %% Evaluation
    Logs -->|loads best| EvalModule
    EvalModule -->|uses| Tokenizer
    EvalModule -->|calls| NLTK
    EvalModule -->|logs| Logs

    %% Inference
    Logs -->|loads pretrained| InferModule
    InferModule -->|uses| Tokenizer
    InferModule -->|outputs| RawOutput["Translated Text"]:::inference

    %% Dependencies
    TrainLoop -.->|imports| models.py
    EvalModule -.->|imports| models.py
    InferModule -.->|imports| models.py

    %% Click Events
    click DataLoader "https://github.com/tranquoctrinh/transformer/blob/main/datasets.py"
    click Tokenizer "https://github.com/tranquoctrinh/transformer/blob/main/utils.py"
    click PE "https://github.com/tranquoctrinh/transformer/blob/main/models.py"
    click SA "https://github.com/tranquoctrinh/transformer/blob/main/models.py"
    click MHA_E "https://github.com/tranquoctrinh/transformer/blob/main/models.py"
    click FFN_E "https://github.com/tranquoctrinh/transformer/blob/main/models.py"
    click PE2 "https://github.com/tranquoctrinh/transformer/blob/main/models.py"
    click SA2 "https://github.com/tranquoctrinh/transformer/blob/main/models.py"
    click EncDecAtt "https://github.com/tranquoctrinh/transformer/blob/main/models.py"
    click MHA_D "https://github.com/tranquoctrinh/transformer/blob/main/models.py"
    click FFN_D "https://github.com/tranquoctrinh/transformer/blob/main/models.py"
    click LinearProj "https://github.com/tranquoctrinh/transformer/blob/main/models.py"
    click TrainLoop "https://github.com/tranquoctrinh/transformer/blob/main/train.py"
    click EvalModule "https://github.com/tranquoctrinh/transformer/blob/main/evaluate.py"
    click InferModule "https://github.com/tranquoctrinh/transformer/blob/main/translate.py"
    click Logs "https://github.com/tranquoctrinh/transformer/tree/main/logs/"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#000
    classDef preprocess fill:#87CEFA,stroke:#000
    classDef model fill:#90EE90,stroke:#000
    classDef train fill:#FFA07A,stroke:#000
    classDef inference fill:#D8BFD8,stroke:#000
    classDef storage fill:#F0E68C,stroke:#000
    classDef external fill:#E0E0E0,stroke:#000
```
