How jadore801120/attention-is-all-you-need-pytorch works
Purpose and entry points
This repository is a PyTorch sequence-to-sequence Transformer implementation for machine translation. The executable pipeline is:
• Dataset preparation: preprocess.py
• Optional BPE vocabulary/code learning and application: learn_bpe.py, apply_bpe.py
• Model training/checkpoint creation: train.py
• Beam-search translation from a trained checkpoint: translate.py
• Reproducible Multi30k German→English workflow: train_multi30k_de_en.sh
Data preparation and tokenization
preprocess.py builds serialized dataset/vocabulary artifacts from multilingual translation data using torchtext and spaCy. It supports a standard tokenized-data path and a separate BPE-oriented path, which are explicitly not fully unified.
• preprocess.py
• learn_bpe.py
• apply_bpe.py
The resulting pickle is the main contract between preprocessing, training, and translation; BPE mode can additionally use train/validation text paths.
Transformer model assembly
The Transformer is assembled in transformer/Models.py, defining encoder/decoder stacks, embeddings, positional encodings, output projection, and masking-aware forward flow. Model-level constants, including special token IDs, are centralized separately.
• transformer/Models.py
• transformer/Constants.py
• transformer/Layers.py
Source token IDs flow through encoder embeddings and self-attention layers; shifted target IDs flow through the decoder, which attends both to prior target positions and encoder outputs. Decoder representations are projected to target-vocabulary logits.
Attention and layer primitives
Attention mechanics are factored below the model assembly layer. Modules.py contains scaled dot-product attention; SubLayers.py composes multi-head attention and position-wise feed-forward blocks; Layers.py packages these into encoder and decoder layers.
• transformer/Modules.py
• transformer/SubLayers.py
• transformer/Layers.py
This separation creates the core architectural boundary: reusable tensor-level attention primitives versus sequence-model orchestration in Models.py.
Training and optimization
train.py loads prepared data, constructs the Transformer, executes batched training/validation, and persists checkpoints under a requested output directory. It exposes architectural choices such as embedding/projection weight sharing and label smoothing.
• train.py
• transformer/Models.py
• transformer/Optim.py
transformer/Optim.py implements the Transformer learning-rate schedule with warmup, wrapping the PyTorch optimizer. Training depends on PyTorch, torchtext, spaCy tokenization/models, and the selected translation corpus.
Inference and decoding
translate.py restores a trained checkpoint and source/target vocabulary metadata, then invokes the translation component to generate output text. Decoding behavior is isolated from training/model definition.
• translate.py
• transformer/Translator.py
• transformer/Models.py
Translator.py is the inference boundary: it drives autoregressive decoding/beam search over model logits until terminal tokens or length limits. The README identifies BPE translation completion as unfinished, so the standard serialized-data path is the supported inference flow.
Operational boundaries
The repository is a command-line ML workflow, not a service. Persistent artifacts cross stages: preprocessed pickle/vocabulary data, optional BPE codes/text files, model checkpoints, logs, and translation output. The only provided orchestration wrapper is:
• train_multi30k_de_en.sh
• requirements.txt
Main entry points: train.py Training CLI · Models.py Transformer model · preprocess.py Preprocess CLI · Translator.py Beam-search decoder
Components 19
Data preparation
Builds the serialized translation-data contract.
- Translation corpus
- Preprocess CLI · Supports standard tokenized and separate BPE-oriented paths.
preprocess.py - Learn BPE
learn_bpe.py - Apply BPE
apply_bpe.py - Prepared data + vocab · Primary contract for training and supported translation.
- BPE codes + text · BPE inference completion is unfinished.
Transformer core
Attention primitives and sequence-model assembly.
- Model constants
transformer/Constants.py - Attention primitive
transformer/Modules.py - Attention + FFN blocks
transformer/SubLayers.py - Encoder/decoder layers
transformer/Layers.py - Transformer model · Embeddings, positional encoding, masks, stacks, and vocabulary projection.
transformer/Models.py
Training and inference
CLI execution, checkpoints, and decoding.
- Warmup optimizer
transformer/Optim.py - Training CLI · Runs batched training and validation.
train.py - Model checkpoints
- Beam-search decoder · Stops on terminal tokens or length limits.
transformer/Translator.py - Translation CLI
translate.py - Translated text
- Multi30k DE→EN workflow
train_multi30k_de_en.sh
Other components
- ML dependencies · PyTorch, torchtext, and spaCy dependencies.
requirements.txt
Connections 21
- Translation corpus → Preprocess CLI: raw parallel text
- Translation corpus → Learn BPE: training text
- Learn BPE → Apply BPE: BPE codes
- Apply BPE → BPE codes + text: segmented text
- Preprocess CLI → Prepared data + vocab: serialize vocab + splits
- Model constants → Transformer model: special token IDs
- Attention primitive → Attention + FFN blocks: attention operation
- Attention + FFN blocks → Encoder/decoder layers: multi-head attention + FFN
- Encoder/decoder layers → Transformer model: encoder/decoder stacks
- Prepared data + vocab → Training CLI: load splits + vocab
- Transformer model → Training CLI: construct and optimize
- Warmup optimizer → Training CLI: warmup LR schedule
- Training CLI → Model checkpoints: persist model state
- Model checkpoints → Translation CLI: restore model state
- Prepared data + vocab → Translation CLI: source/target vocab
- Translation CLI → Beam-search decoder: invoke decoding
- Transformer model → Beam-search decoder: model logits
- Beam-search decoder → Translated text: generated text
- Multi30k DE→EN workflow → Training CLI: reproducible training
- ML dependencies → Preprocess CLI: torchtext + spaCy
- ML dependencies → Training CLI: PyTorch runtime