How tranquoctrinh/transformer works
Project Type and Purpose
• This is an open-source PyTorch implementation of the "Transformer – Attention is All You Need" model, tailored for English→Vietnamese machine translation.
• It’s not a web or full-stack app but an ML research/training/inference pipeline composed of data handling, model definition, training, evaluation, and inference modules.
File‐Structure Analysis
• Top-level scripts:
• datasets.py – data loading and preprocessing
• models.py – Transformer model components (PositionalEncoder, SelfAttention, MultiHeadAttention, EncoderLayer, DecoderLayer, Encoder, Decoder, Transformer)
• train.py – training loop, checkpointing, logging
• evaluate.py – BLEU‐score evaluation on validation set
• translate.py – inference (load model + tokenizer; beam‐search decode)
• utils.py – hyperparameters, config loader, tokenization utilities
• logs/ – runtime logs and artifacts (loss curves, predictions)
• .gitignore, LICENSE, README.md
Key Architectural Components
a. Data Ingestion & Preprocessing (datasets.py)
– Reads bilingual corpus (data_en_vi/)
– Tokenizes sentences, builds vocabularies, creates DataLoader with source/target tensors and masks
b. Model Definition (models.py)
– Positional Encoding
– Scaled Dot-Product Self-Attention
– Multi-Head Attention
– Encoder Layer (self-attention + feed-forward + residual + norm)
– Decoder Layer (self-attention + encoder-decoder attention + feed-forward + residual + norm)
– Encoder Stack (N layers) + embedding + positional encoding
– Decoder Stack (N layers) + embedding + positional encoding
– Final linear projection to target vocabulary
c. Training Pipeline (train.py)
– Loads config from utils
– Instantiates Transformer model, optimizer, loss criterion
– Iterates epochs: forward pass → compute cross-entropy loss → backward → optimizer step → early stopping → log metrics to logs/
d. Evaluation Module (evaluate.py)
– Loads best checkpoint + tokenizers
– Runs validation set → decodes predictions → computes BLEU-2/3/4 via NLTK → logs results
e. Inference Service (translate.py)
– Loads pretrained model + tokenizers
– Defines translate() with beam search over decoder
– Exposes simple script/API to translate input strings
Detail Level
• Don’t just show “Model” as a monolith; expand to Encoder vs. Decoder, multi-head attention vs. feed-forward
• Masks (source_mask, target_mask) flowing into attention sub-modules