# jadore801120/attention-is-all-you-need-pytorch architecture

> Architecture diagram of the GitHub repository jadore801120/attention-is-all-you-need-pytorch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jadore801120/attention-is-all-you-need-pytorch
- Repository: https://github.com/jadore801120/attention-is-all-you-need-pytorch

## Overview

#### Purpose and entry points
This repository is a PyTorch sequence-to-sequence Transformer implementation for machine translation. The executable pipeline is:
- Dataset preparation: `preprocess.py`
- Optional BPE vocabulary/code learning and application: `learn_bpe.py`, `apply_bpe.py`
- Model training/checkpoint creation: `train.py`
- Beam-search translation from a trained checkpoint: `translate.py`
- Reproducible Multi30k German→English workflow: `train_multi30k_de_en.sh`

#### Data preparation and tokenization
`preprocess.py` builds serialized dataset/vocabulary artifacts from multilingual translation data using torchtext and spaCy. It supports a standard tokenized-data path and a separate BPE-oriented path, which are explicitly not fully unified.
- `preprocess.py`
- `learn_bpe.py`
- `apply_bpe.py`

The resulting pickle is the main contract between preprocessing, training, and translation; BPE mode can additionally use train/validation text paths.

#### Transformer model assembly
The Transformer is assembled in `transformer/Models.py`, defining encoder/decoder stacks, embeddings, positional encodings, output projection, and masking-aware forward flow. Model-level constants, including special token IDs, are centralized separately.
- `transformer/Models.py`
- `transformer/Constants.py`
- `transformer/Layers.py`

Source token IDs flow through encoder embeddings and self-attention layers; shifted target IDs flow through the decoder, which attends both to prior target positions and encoder outputs. Decoder representations are projected to target-vocabulary logits.

#### Attention and layer primitives
Attention mechanics are factored below the model assembly layer. `Modules.py` contains scaled dot-product attention; `SubLayers.py` composes multi-head attention and position-wise feed-forward blocks; `Layers.py` packages these into encoder and decoder layers.
- `transformer/Modules.py`
- `transformer/SubLayers.py`
- `transformer/Layers.py`

This separation creates the core architectural boundary: reusable tensor-level attention primitives versus sequence-model orchestration in `Models.py`.

#### Training and optimization
`train.py` loads prepared data, constructs the Transformer, executes batched training/validation, and persists checkpoints under a requested output directory. It exposes architectural choices such as embedding/projection weight sharing and label smoothing.
- `train.py`
- `transformer/Models.py`
- `transformer/Optim.py`

`transformer/Optim.py` implements the Transformer learning-rate schedule with warmup, wrapping the PyTorch optimizer. Training depends on PyTorch, torchtext, spaCy tokenization/models, and the selected translation corpus.

#### Inference and decoding
`translate.py` restores a trained checkpoint and source/target vocabulary metadata, then invokes the translation component to generate output text. Decoding behavior is isolated from training/model definition.
- `translate.py`
- `transformer/Translator.py`
- `transformer/Models.py`

`Translator.py` is the inference boundary: it drives autoregressive decoding/beam search over model logits until terminal tokens or length limits. The README identifies BPE translation completion as unfinished, so the standard serialized-data path is the supported inference flow.

#### Operational boundaries
The repository is a command-line ML workflow, not a service. Persistent artifacts cross stages: preprocessed pickle/vocabulary data, optional BPE codes/text files, model checkpoints, logs, and translation output. The only provided orchestration wrapper is:
- `train_multi30k_de_en.sh`
- `requirements.txt`

Main entry points: [`train.py`](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/train.py) (Training CLI), [`transformer/Models.py`](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Models.py) (Transformer model), [`preprocess.py`](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/preprocess.py) (Preprocess CLI), [`transformer/Translator.py`](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Translator.py) (Beam-search decoder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_corpus[("Translation corpus<br/>external data")]
  node_preprocess["Preprocess CLI<br/>dataset builder<br/>[preprocess.py]"]
  node_learn_bpe["Learn BPE<br/>BPE learner<br/>[learn_bpe.py]"]
  node_apply_bpe["Apply BPE<br/>BPE tokenizer<br/>[apply_bpe.py]"]
  node_prepared_data["Prepared data + vocab<br/>pickle artifact"]
  node_bpe_artifacts["BPE codes + text<br/>BPE artifacts"]
end

subgraph group_model["Transformer core"]
  node_constants["Model constants<br/>token IDs<br/>[Constants.py]"]
  node_attention["Attention primitive<br/>scaled dot-product<br/>[Modules.py]"]
  node_sublayers["Attention + FFN blocks<br/>neural sublayers<br/>[SubLayers.py]"]
  node_layers["Encoder/decoder layers<br/>stack layers<br/>[Layers.py]"]
  node_transformer{{"Transformer model<br/>seq2seq assembly<br/>[Models.py]"}}
end

subgraph group_runtime["Training and inference"]
  node_optimizer["Warmup optimizer<br/>LR scheduler<br/>[Optim.py]"]
  node_train["Training CLI<br/>trainer<br/>[train.py]"]
  node_checkpoints["Model checkpoints<br/>persistent artifact"]
  node_translator["Beam-search decoder<br/>autoregressive inference<br/>[Translator.py]"]
  node_translate["Translation CLI<br/>inference entry point<br/>[translate.py]"]
  node_translation_output["Translated text<br/>output artifact"]
  node_workflow["Multi30k DE→EN workflow<br/>shell wrapper"]
end

node_dependencies["ML dependencies<br/>runtime requirements<br/>[requirements.txt]"]

node_corpus -->|"raw parallel text"| node_preprocess
node_corpus -->|"training text"| node_learn_bpe
node_learn_bpe -->|"BPE codes"| node_apply_bpe
node_apply_bpe -->|"segmented text"| node_bpe_artifacts
node_preprocess -->|"serialize vocab + splits"| node_prepared_data
node_constants -->|"special token IDs"| node_transformer
node_attention -->|"attention operation"| node_sublayers
node_sublayers -->|"multi-head attention + FFN"| node_layers
node_layers -->|"encoder/decoder stacks"| node_transformer
node_prepared_data -->|"load splits + vocab"| node_train
node_transformer -->|"construct and optimize"| node_train
node_optimizer -->|"warmup LR schedule"| node_train
node_train -->|"persist model state"| node_checkpoints
node_checkpoints -->|"restore model state"| node_translate
node_prepared_data -->|"source/target vocab"| node_translate
node_translate -->|"invoke decoding"| node_translator
node_transformer -->|"model logits"| node_translator
node_translator -->|"generated text"| node_translation_output
node_workflow -->|"reproducible training"| node_train
node_dependencies -.->|"torchtext + spaCy"| node_preprocess
node_dependencies -.->|"PyTorch runtime"| node_train

click node_preprocess "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/preprocess.py"
click node_learn_bpe "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/learn_bpe.py"
click node_apply_bpe "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/apply_bpe.py"
click node_constants "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Constants.py"
click node_attention "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Modules.py"
click node_sublayers "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/SubLayers.py"
click node_layers "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Layers.py"
click node_transformer "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Models.py"
click node_optimizer "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Optim.py"
click node_train "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/train.py"
click node_translator "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Translator.py"
click node_translate "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/translate.py"
click node_workflow "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/train_multi30k_de_en.sh"
click node_dependencies "https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/requirements.txt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_corpus,node_preprocess,node_learn_bpe,node_apply_bpe,node_prepared_data,node_bpe_artifacts toneBlue
class node_constants,node_attention,node_sublayers,node_layers,node_transformer toneAmber
class node_optimizer,node_train,node_checkpoints,node_translator,node_translate,node_translation_output,node_workflow toneMint
class node_dependencies toneNeutral
```

## Components

### Data preparation

Builds the serialized translation-data contract.

- **Translation corpus**: external data
- [Preprocess CLI](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/preprocess.py): dataset builder · Supports standard tokenized and separate BPE-oriented paths. · `preprocess.py`
- [Learn BPE](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/learn_bpe.py): BPE learner · `learn_bpe.py`
- [Apply BPE](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/apply_bpe.py): BPE tokenizer · `apply_bpe.py`
- **Prepared data + vocab**: pickle artifact · Primary contract for training and supported translation.
- **BPE codes + text**: BPE artifacts · BPE inference completion is unfinished.

### Transformer core

Attention primitives and sequence-model assembly.

- [Model constants](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Constants.py): token IDs · `transformer/Constants.py`
- [Attention primitive](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Modules.py): scaled dot-product · `transformer/Modules.py`
- [Attention + FFN blocks](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/SubLayers.py): neural sublayers · `transformer/SubLayers.py`
- [Encoder/decoder layers](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Layers.py): stack layers · `transformer/Layers.py`
- [Transformer model](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Models.py): seq2seq assembly · Embeddings, positional encoding, masks, stacks, and vocabulary projection. · `transformer/Models.py`

### Training and inference

CLI execution, checkpoints, and decoding.

- [Warmup optimizer](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Optim.py): LR scheduler · `transformer/Optim.py`
- [Training CLI](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/train.py): trainer · Runs batched training and validation. · `train.py`
- **Model checkpoints**: persistent artifact
- [Beam-search decoder](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/transformer/Translator.py): autoregressive inference · Stops on terminal tokens or length limits. · `transformer/Translator.py`
- [Translation CLI](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/translate.py): inference entry point · `translate.py`
- **Translated text**: output artifact
- [Multi30k DE→EN workflow](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/train_multi30k_de_en.sh): shell wrapper · `train_multi30k_de_en.sh`

### Other components

- [ML dependencies](https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/requirements.txt): runtime requirements · PyTorch, torchtext, and spaCy dependencies. · `requirements.txt`

## Connections

- Translation corpus → Preprocess CLI: raw parallel text
- Translation corpus → Learn BPE: training text
- Learn BPE → Apply BPE: BPE codes
- Apply BPE → BPE codes + text: segmented text
- Preprocess CLI → Prepared data + vocab: serialize vocab + splits
- Model constants → Transformer model: special token IDs
- Attention primitive → Attention + FFN blocks: attention operation
- Attention + FFN blocks → Encoder/decoder layers: multi-head attention + FFN
- Encoder/decoder layers → Transformer model: encoder/decoder stacks
- Prepared data + vocab → Training CLI: load splits + vocab
- Transformer model → Training CLI: construct and optimize
- Warmup optimizer → Training CLI: warmup LR schedule
- Training CLI → Model checkpoints: persist model state
- Model checkpoints → Translation CLI: restore model state
- Prepared data + vocab → Translation CLI: source/target vocab
- Translation CLI → Beam-search decoder: invoke decoding
- Transformer model → Beam-search decoder: model logits
- Beam-search decoder → Translated text: generated text
- Multi30k DE→EN workflow → Training CLI: reproducible training
- ML dependencies → Preprocess CLI: torchtext + spaCy
- ML dependencies → Training CLI: PyTorch runtime
