How opennmt/ctranslate2 works
Main components
• User Interfaces
– CLI tool (cli/translator.cc, cli/CMakeLists.txt)
– Python API (python/ctranslate2 package + Python bindings in python/cpp)
• Core C++ Runtime
– Core engine (src/*.cc + include/ctranslate2)
– Model loader & converter modules (include/ctranslate2/models + python/ctranslate2/converters)
– Translation/Generation pipeline (translator.cc, generation.cc, decoding.cc)
• Backends
– CPU backend (src/cpu, device_dispatch, primitives)
– CUDA backend (src/cuda, cublas_stub, nccl_stub, primitives.cu)
– Automatic dispatch module (device_dispatch.h, devices.cc) selects CPU/GPU and instruction set
• External Dependencies & Libraries
– BLAS & math libraries (oneDNN, MKL, OpenBLAS, Ruy, Apple Accelerate)
– Third‑party headers and utilities (third_party/*, nlohmann/json)
– Python build system (setuptools in python/setup.py)
• Models & Data
– Optimized model format on disk (converted via converters)
– Vocabulary & specification files (include/ctranslate2/vocabulary*)
• Tools & Examples
– Converters (python/ctranslate2/converters) for Fairseq, Marian, Transformers, etc.
– Examples (examples/llama*, examples/wngt2020)
– Benchmarking suite (tools/benchmark)
• Documentation & Tests
– Sphinx docs (docs/)
– Unit tests (tests/, python/tests/)
– CI workflows (.github/workflows/ci.yml)
Interactions and data flow
– User issues a command via CLI or Python API
→ Python API calls into C++ runtime via pybind11 bindings (python/cpp/module.cc)
→ CLI directly invokes the C++ Translator class
– Translator/Generator loads an optimized model (ModelReader → ModelFactory → weight tensors)
– Device dispatch chooses CPU or GPU path
– The execution pipeline applies layers (encoder, decoder, attention, etc.)
→ Calls into ops library (src/ops + include/ctranslate2/ops)
→ Ops are dispatched to CPU kernels (src/ops/*_cpu.cc) or GPU kernels (src/ops/*_gpu.cu)
– BLAS/math calls forwarded to selected backend library (oneDNN, MKL, etc.)
– Outputs (token IDs, probabilities) are returned to Python or printed by CLI
Architectural patterns
– Layered architecture: User interface → API/CLI → Core engine → Backends → Hardware
– Plugin‑style converters: abstract “converter” interface with multiple implementations
– Device Dispatch pattern: single binary supports multiple ISAs/backends selected at runtime
– Pipeline pattern: translate/generate pipeline with clear stages (load model → encode → decode → post‑process)
– Quantization & compression modules separate from core ops