How tensorflow/tensor2tensor works
Project type and purpose
• Tensor2Tensor is a modular deep-learning framework and toolkit:
• Offers CLI entrypoints (t2t-datagen, t2t-trainer, t2t-decoder, etc.)
• Provides reusable components—Problems (data), Models, Layers, Hyperparams, Utils
• Supports training, evaluation, inference, model export, and web-based “insights” visualization
• Integrates with TensorFlow, Cloud ML Engine, TPUs, distributed setups
File structure
• Top-level files: setup.py, README.md, configuration (travis, pylintrc, floyd.yml)
• Directories and roles:
• tensor2tensor/bin → CLI scripts (entrypoints)
• tensor2tensor/data_generators → Problem definitions & TFRecord data pipeline
• tensor2tensor/models → Model implementations (Transformer, ResNet, research)
• tensor2tensor/layers → Reusable NN layers (attention, Transformer components)
• tensor2tensor/utils → Registry, HParams, trainer_lib, checkpoint hooks, metrics, TFRecord readers
• tensor2tensor/serving → SavedModel export & inference API
• tensor2tensor/insights → Query processor, web server, Polymer front-end UI
• Other: notebooks, docs, oss_scripts, test_data
Extract architecture hints from the README
• Core concepts: Problems ↔ Models ↔ Hyperparam sets, all registered via a central registry
• Binaries: t2t-datagen (data prep), t2t-trainer (train/eval/infer), t2t-decoder/bleu/exporter
• Deployment targets: local GPUs, multi-GPU with parameter servers, Cloud ML Engine, Cloud TPU
• Serving & Insights components for production inference and monitoring
Main components and their interactions
a. CLI layer
• t2t-datagen → invokes Data Generator registry → writes TFRecord files to DataDir
• t2t-trainer → uses registry to instantiate Problem, Model, HParams → builds TF graph → trains/evaluates via TensorFlow → writes checkpoints + summaries
• t2t-decoder / t2t-exporter → load checkpoints + graph → produce translations or SavedModel
b. Core library
• Registry service (problems, models, hparams)
• DataGenerators (produce TFExample → TFRecord)
• Layers & Modalities (bottom/top transformations)
• Models (tensor-to-tensor graphs)
• Utils (trainer_lib for session management, beam search, metrics, checkpoint compat)
c. Storage & compute
• Data storage: TFRecord files on disk or cloud storage
• Checkpoints: model.ckpt in OutputDir
• Logs/summaries: TensorBoard–readable event files
• Compute backends: local GPU(s), parameter servers, Cloud TPU pods, Cloud ML Engine
d. Serving & Insights
• Serving: export.py → SavedModel → query.py serves inference requests
• Insights: query_processor loads metrics & checkpoints → insights/server (Python) → Polymer-based front-end (HTML/JS)
Architectural patterns & design principles
• Plugin/registry pattern: loose coupling of Problems, Models, HParams via central registry
• Layered architecture: clear separation of CLI, core library, data layer, compute layer, serving layer, and visualization/UI
• MVC (or MV*) in Insights: server/controller, query_processor/model, Polymer frontend/view
• Modular & extensible: add new Problems, Models, hyperparam sets without core code changes
Technologies and external services
• Python, TensorFlow (and tf.data, tf.estimator or low-level APIs)
• TFRecord + proto buffers
• REST or RPC (for serving/query)
• JavaScript + Polymer for UI
• Cloud ML Engine, Cloud TPU, parameter-server / multi-worker distributed TensorFlow