# opennmt/ctranslate2 architecture

> Architecture diagram of the GitHub repository opennmt/ctranslate2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opennmt/ctranslate2
- Repository: https://github.com/opennmt/ctranslate2

## Overview

##### Main components
- User Interfaces
– CLI tool (cli/translator.cc, cli/CMakeLists.txt)
– Python API (python/ctranslate2 package + Python bindings in python/cpp)
- Core C++ Runtime
– Core engine (src/*.cc + include/ctranslate2)
– Model loader & converter modules (include/ctranslate2/models + python/ctranslate2/converters)
– Translation/Generation pipeline (translator.cc, generation.cc, decoding.cc)
- Backends
– CPU backend (src/cpu, device_dispatch, primitives)
– CUDA backend (src/cuda, cublas_stub, nccl_stub, primitives.cu)
– Automatic dispatch module (device_dispatch.h, devices.cc) selects CPU/GPU and instruction set
- External Dependencies & Libraries
– BLAS & math libraries (oneDNN, MKL, OpenBLAS, Ruy, Apple Accelerate)
– Third‑party headers and utilities (third_party/*, nlohmann/json)
– Python build system (setuptools in python/setup.py)
- Models & Data
– Optimized model format on disk (converted via converters)
– Vocabulary & specification files (include/ctranslate2/vocabulary*)
- Tools & Examples
– Converters (python/ctranslate2/converters) for Fairseq, Marian, Transformers, etc.
– Examples (examples/llama*, examples/wngt2020)
– Benchmarking suite (tools/benchmark)
- Documentation & Tests
– Sphinx docs (docs/)
– Unit tests (tests/, python/tests/)
– CI workflows (.github/workflows/ci.yml)

##### Interactions and data flow
– User issues a command via CLI or Python API
→ Python API calls into C++ runtime via pybind11 bindings (python/cpp/module.cc)
→ CLI directly invokes the C++ Translator class
– Translator/Generator loads an optimized model (ModelReader → ModelFactory → weight tensors)
– Device dispatch chooses CPU or GPU path
– The execution pipeline applies layers (encoder, decoder, attention, etc.)
→ Calls into ops library (src/ops + include/ctranslate2/ops)
→ Ops are dispatched to CPU kernels (src/ops/*_cpu.cc) or GPU kernels (src/ops/*_gpu.cu)
– BLAS/math calls forwarded to selected backend library (oneDNN, MKL, etc.)
– Outputs (token IDs, probabilities) are returned to Python or printed by CLI

##### Architectural patterns
– Layered architecture: User interface → API/CLI → Core engine → Backends → Hardware
– Plugin‑style converters: abstract “converter” interface with multiple implementations
– Device Dispatch pattern: single binary supports multiple ISAs/backends selected at runtime
– Pipeline pattern: translate/generate pipeline with clear stages (load model → encode → decode → post‑process)
– Quantization & compression modules separate from core ops

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interfaces
    subgraph "User Interfaces"
        CLITool["CLI Tool"]:::userFacing
        PythonAPI["Python API"]:::userFacing
        PythonBindings["pybind11 Bindings"]:::userFacing
    end

    %% Core C++ Runtime
    subgraph "Core C++ Runtime"
        CoreEngine["Core Engine"]:::core
        ModelLoader["Model Loader & Factory"]:::core
        TranslationPipeline["Translation/Generation Pipeline"]:::core
        DeviceDispatch["Automatic Device Dispatch"]:::core
        OpsLibrary["Operations Library"]:::core
        ThreadPool["Thread Pool & Async Execution"]:::core
    end

    %% Backends
    subgraph "Backends"
        CPUBackend["CPU Backend"]:::backend
        GPUBackend["CUDA GPU Backend"]:::backend
    end

    %% External Dependencies & Data
    subgraph "External Dependencies & Data"
        ExtDeps["Third-Party Libraries"]:::external
        PythonBuild["Python Build System"]:::external
        Vocab["Vocabulary & Spec Files"]:::external
        CI["CI Workflows"]:::external
        ModelFiles["Model Files on Disk"]:::data
    end

    %% Tools & Examples
    subgraph "Tools, Converters, Examples, Tests, Docs"
        Converters["Converters (Plugins)"]:::tools
        Examples["Examples"]:::tools
        Benchmark["Benchmarking Suite"]:::tools
        Tests["Unit Tests"]:::tools
        Docs["Documentation"]:::tools
    end

    %% Interactions
    CLITool -->|"invoke Translator"| CoreEngine
    PythonAPI -->|"API call"| PythonBindings
    PythonBindings -->|"bind to"| CoreEngine

    CoreEngine -->|"load_model()"| ModelLoader
    ModelLoader -->|"reads"| ModelFiles

    CoreEngine -->|"translate()/generate()"| TranslationPipeline
    TranslationPipeline -->|"use threads"| ThreadPool

    TranslationPipeline -->|"dispatch ops"| DeviceDispatch
    DeviceDispatch -->|"CPU path"| CPUBackend
    DeviceDispatch -->|"GPU path"| GPUBackend

    CPUBackend -->|"calls ops"| OpsLibrary
    GPUBackend -->|"calls ops"| OpsLibrary

    OpsLibrary -->|"BLAS/math calls"| ExtDeps
    GPUBackend -->|"GPU drivers"| ExtDeps

    Converters -.->|"dashed plugin"| ModelLoader
    Examples -.->|"use CLI/Python"| CLITool
    Examples -.->|"use API"| PythonAPI
    Benchmark -.->|"benchmark CLI/API"| CLITool
    Tests -.->|"test Core/API"| CoreEngine
    Tests -.->|"test Python"| PythonAPI
    Docs -.->|"docs for users"| CLITool

    %% Click Events
    click CLITool "https://github.com/opennmt/ctranslate2/blob/master/cli/translator.cc"
    click PythonAPI "https://github.com/opennmt/ctranslate2/blob/master/python/ctranslate2/__init__.py"
    click PythonBindings "https://github.com/opennmt/ctranslate2/blob/master/python/cpp/module.cc"
    click CoreEngine "https://github.com/opennmt/ctranslate2/blob/master/src/translator.cc"
    click ModelLoader "https://github.com/opennmt/ctranslate2/blob/master/src/models/model_reader.cc"
    click TranslationPipeline "https://github.com/opennmt/ctranslate2/blob/master/src/encoder.cc"
    click DeviceDispatch "https://github.com/opennmt/ctranslate2/blob/master/src/device_dispatch.h"
    click CPUBackend "https://github.com/opennmt/ctranslate2/blob/master/src/cpu/backend.cc"
    click GPUBackend "https://github.com/opennmt/ctranslate2/blob/master/src/cuda/primitives.cu"
    click OpsLibrary "https://github.com/opennmt/ctranslate2/blob/master/include/ctranslate2/ops/ops.h"
    click ThreadPool "https://github.com/opennmt/ctranslate2/blob/master/src/thread_pool.cc"
    click Converters "https://github.com/opennmt/ctranslate2/blob/master/python/ctranslate2/converters/converter.py"
    click Examples "https://github.com/opennmt/ctranslate2/tree/master/examples/llama2"
    click Benchmark "https://github.com/opennmt/ctranslate2/blob/master/tools/benchmark/benchmark.py"
    click Tests "https://github.com/opennmt/ctranslate2/tree/master/tests/"
    click Docs "https://github.com/opennmt/ctranslate2/tree/master/docs/"
    click ExtDeps "https://github.com/opennmt/ctranslate2/tree/master/third_party/"
    click PythonBuild "https://github.com/opennmt/ctranslate2/blob/master/python/setup.py"
    click Vocab "https://github.com/opennmt/ctranslate2/blob/master/include/ctranslate2/vocabulary.h"
    click CI "https://github.com/opennmt/ctranslate2/blob/master/.github/workflows/ci.yml"

    %% Styles
    classDef userFacing fill:#B3DFFF,stroke:#333,stroke-width:1px;
    classDef core fill:#FFFFFF,stroke:#333,stroke-width:1px;
    classDef backend fill:#D4F7C5,stroke:#333,stroke-width:1px;
    classDef external fill:#EEEEEE,stroke:#333,stroke-width:1px;
    classDef tools fill:#FFD8A8,stroke:#333,stroke-width:1px;
    classDef data fill:#FFEEAA,stroke:#333,stroke-width:1px;
```
