How tensorflow/nmt works
Project Type & Purpose
• This is an open-source tutorial/toolkit for building Neural Machine Translation (NMT) systems using TensorFlow’s seq2seq API.
• It provides:
• A vanilla seq2seq model (encoder–decoder)
• Attention-based variants (Luong/Bahdanau)
• Google’s GNMT architecture
• A complete training, evaluation, and inference pipeline
• Utility scripts for data download, BLEU/ROUGE scoring, and standard hyperparameter presets
High-Level File Structure & Architectural Patterns
• Top-level:
• train.py – entry point for building and running the training graph
• inference.py – entry point for batch inference
• nmt/nmt.py – unified CLI that dispatches to training or inference based on flags
• nmt/scripts/ – shell + Python scripts (download data, compute BLEU/ROUGE)
• nmt/standard_hparams/ – JSON presets for common configurations (IWSLT, WMT)
• nmt/testdata/ – small sample data for unit tests
• Core package (nmt/):
• model.py – vanilla seq2seq definition (embeddings, multi-layer RNNs, loss, optimizer)
• attention_model.py – wraps decoder with tf.contrib.seq2seq.AttentionWrapper
• gnmt_model.py – GNMTAttentionMultiCell for multi-GPU, parallelizable decoding
• model_helper.py – shared helpers for graph construction and checkpoint management
• inference.py / nmt.py – build inference graph (greedy & beam-search)
• utils/ – data pipeline (iterator_utils.py), vocab lookup (vocab_utils.py), evaluation (evaluation_utils.py), misc
Key Insights from README
• Emphasizes three separate TensorFlow graphs: Training, Evaluation, Inference
• Uses tf.data Datasets + iterators for efficient batching, bucketing, padding
• Describes core NMT components: Embedding → Encoder RNN → (Attention) → Decoder RNN → Projection → Loss → Optimizer
• Details on greedy decoding vs. beam search and multi-GPU placement
A. Entry Layer (CLI)
• “nmt/nmt.py” reads flags → chooses TRAIN or INFER path → invokes appropriate graph builder in train.py or inference.py
B. Data Ingestion & Pre-processing
• Scripts: download_iwslt15.sh, wmt16_en_de.sh → raw parallel text files
• tf.data Pipeline (utils/iterator_utils.py):
• TextLineDataset → tokenization → vocabulary lookup (vocab_utils) → zip source/target → padded_batch → iterator
C. Graph Builders
• Training Graph (train.py + model_helper):
• Builds Embedding layers; Encoder RNN (optionally bidirectional); AttentionWrapper (if flag set) or vanilla decoder
• Projection layer; Loss (cross-entropy with padding mask); Gradient clipping; Optimizer (SGD/Adam)
• Summaries for TensorBoard; Saver for checkpoints
• Evaluation Graph (train.py):
• Shares variable scopes; same forward pass; additional ops for BLEU/perplexity
• Inference Graph (inference.py):
• Builds encoder and decoder with GreedyEmbeddingHelper or BeamSearchDecoder; no backprop ops
D. Model Variants
• Vanilla seq2seq: nmt/model.py
• Attention NMT: nmt/attention_model.py + tf.contrib.seq2seq.LuongAttention or BahdanauAttention
• GNMT: nmt/gnmt_model.py (GNMTAttentionMultiCell for multi-GPU)
E. Utilities & Scripts
• evaluation_utils.py: BLEU, ROUGE computation
• scripts/bleu.py, scripts/rouge.py: wrapper CLI scripts
• standard_hparams: JSON files tie hyperparameters to pre-trained checkpoints
F. External Services
• TensorBoard (logs from training graph)
• Filesystem (checkpoints, vocab, data, inference outputs)
Architectural Patterns & Design Principles
• Modular separation: CLI ↔ Data Pipeline ↔ Graph Builders ↔ Model Variants ↔ Utils
• Clear separation of graphs (training/eval/inference) via separate scopes & sessions
• Use of tf.data iterators for pipeline flexibility & performance
• Pluggable attention mechanisms and multi-GPU parallelism
• Configuration-driven (flags & JSON hparams) for reproducibility