# triton-inference-server/server architecture

> Architecture diagram of the GitHub repository triton-inference-server/server, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/triton-inference-server/server
- Repository: https://github.com/triton-inference-server/server

## Overview

##### Project Type and Purpose
- Triton Inference Server is a modular, full-stack inference‐serving system. It comprises:
- A high-performance C++ core (src/) implementing HTTP/gRPC servers, scheduler, batcher, model lifecycle, shared memory.
- Multiple execution backends (TensorRT, TensorFlow, ONNX, Python, custom) loaded at runtime.
- Language bindings and clients: Python wheel, FastAPI‐based frontend (python/openai), C API, Java API, and SDK tools.
- Deployment and orchestration artifacts: Dockerfiles, Docker Compose, Helm charts (deploy/…), Kubernetes configurations.
- Extensive QA suite and examples (qa/, docs/examples).

##### File‐Structure Analysis
- Top‐level: Dockerfile.* (containers), CMakeLists.txt (build), compose.py, build.py (build orchestration), deploy/ (Helm charts for AWS, GCP, on‐prem), docker/ (entrypoints), docs/ (user & developer guides), python/ (OpenAI front end), src/ (server core, gRPC/HTTP handlers, Python bindings), qa/ (thousands of L0 integration/unit tests), tools/ (scripts).
- Architectural patterns:
- Modular plugin/backends architecture (pluggable shared‐object backends).
- Layered networking: HTTP/gRPC handlers → scheduler & batcher → model backends → model instances.
- Microservices-friendly container images and Helm charts.
- CMake-based build for cross-platform compilation.

##### Key README Insights
- Core features: concurrent model execution, dynamic & sequence batching, ensemble pipelines, BLS, extensible backend API, multiple protocols (HTTP/REST, gRPC, in-process).
- Deployment: recommends Docker and Helm (deploy/*) for cloud or edge.
- Clients: Python & C++ client libraries with examples.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        direction TB
        Python_Client["Python Client SDK"]:::client
        Cpp_SDK["C++ SDK"]:::client
        Java_SDK["Java SDK"]:::client
        Custom_Apps["Custom Applications"]:::client
    end

    %% Server Transport Layer
    subgraph "Transport Layer"
        direction TB
        HTTP_Server["HTTP Server\n(src/http_server.cc)"]:::server
        gRPC_Server["gRPC Server\n(src/grpc/grpc_server.cc)"]:::server
        FastAPI_Router["FastAPI Router\n(routers/)"]:::server
        FastAPI_Main["FastAPI Main\n(main.py)"]:::server
    end

    %% Core Engine
    subgraph "Core Engine"
        direction TB
        Scheduler_Batcher["Scheduler & Batcher\n(src/common.cc)"]:::server
        Model_Repo_Manager["Model Repository Manager\n(src/command_line_parser.cc)"]:::server
        Shared_Memory_Manager["Shared Memory Manager\n(src/shared_memory_manager.cc)"]:::server
        Metrics_Logging["Metrics & Logging"]:::storage
    end

    %% Backends Plugins
    subgraph "Backend Plugins"
        direction TB
        TensorRT["TensorRT Backend"]:::server
        TensorFlow["TensorFlow Backend"]:::server
        ONNX["ONNX Backend"]:::server
        Python_Backend["Python Backend"]:::server
        Custom_Backend["Custom Backend"]:::server
    end

    %% Model Repository Storage
    Model_Repository["Model Repository\n(filesystem, S3, Azure)"]:::storage

    %% Language Bindings & In-Process API
    subgraph "In-Process APIs"
        direction TB
        Python_Bindings["Python Wheel\n(setup.py)"]:::server
        C_API_Bindings["C API Bindings\n(tritonfrontend/_c/)"]:::server
        Py_API_Wrappers["Python API Wrappers\n(tritonfrontend/_api/)"]:::server
    end

    %% Infrastructure & Deployment
    subgraph "Deployment & Infra"
        direction TB
        Docker_Images["Docker Images\n(Dockerfile.*)"]:::infra
        Entrypoints["Entrypoints\n(docker/entrypoint.d/)"]:::infra
        Compose["Docker Compose\n(compose.py)"]:::infra
        Kubernetes["Kubernetes & Helm\n(deploy/)"]:::infra
        Cloud_Storage["Cloud Storage Services"]:::infra
        CI_CD["CI/CD Pipelines"]:::infra
    end

    %% QA Suite
    QA_Suite["QA Suite & Tests\n(qa/)"]:::qa

    %% Documentation & Examples
    Docs_Examples["Documentation & Examples\n(docs/examples/)"]:::qa

    %% Connections
    Python_Client -->|HTTP/gRPC Request| HTTP_Server
    Cpp_SDK -->|HTTP/gRPC Request| HTTP_Server
    Java_SDK -->|HTTP/gRPC Request| gRPC_Server
    Custom_Apps -->|HTTP/gRPC Request| HTTP_Server
    HTTP_Server -->|forwards| Scheduler_Batcher
    gRPC_Server -->|forwards| Scheduler_Batcher
    FastAPI_Router -->|forwards| Scheduler_Batcher
    FastAPI_Main -->|routes| FastAPI_Router
    Scheduler_Batcher -->|requests| Model_Repo_Manager
    Model_Repo_Manager -->|loads models| Model_Repository
    Scheduler_Batcher -->|uses| Shared_Memory_Manager
    Scheduler_Batcher -->|emits| Metrics_Logging
    Scheduler_Batcher -->|dispatches| TensorRT
    Scheduler_Batcher -->|dispatches| TensorFlow
    Scheduler_Batcher -->|dispatches| ONNX
    Scheduler_Batcher -->|dispatches| Python_Backend
    Scheduler_Batcher -->|dispatches| Custom_Backend
    TensorRT -->|reads models| Model_Repository
    TensorFlow -->|reads models| Model_Repository
    ONNX -->|reads models| Model_Repository
    Python_Backend -->|reads models| Model_Repository
    Custom_Backend -->|reads models| Model_Repository
    Python_Bindings -->|in-process call| Scheduler_Batcher
    C_API_Bindings -->|in-process call| Scheduler_Batcher
    Py_API_Wrappers -->|wraps| Python_Bindings
    Docker_Images -->|package| HTTP_Server
    Entrypoints -->|initialize| Docker_Images
    Compose -->|orchestrates| Docker_Images
    Kubernetes -->|deploys| Docker_Images
    Kubernetes -->|uses| Cloud_Storage
    QA_Suite -->|tests| HTTP_Server
    QA_Suite -->|tests| gRPC_Server
    QA_Suite -->|tests| FastAPI_Router
    Docs_Examples -->|illustrate| Model_Repo_Manager

    %% Click Events
    click HTTP_Server "https://github.com/triton-inference-server/server/blob/main/src/http_server.cc"
    click gRPC_Server "https://github.com/triton-inference-server/server/blob/main/src/grpc/grpc_server.cc"
    click Shared_Memory_Manager "https://github.com/triton-inference-server/server/blob/main/src/shared_memory_manager.cc"
    click Scheduler_Batcher "https://github.com/triton-inference-server/server/blob/main/src/common.cc"
    click Model_Repo_Manager "https://github.com/triton-inference-server/server/blob/main/src/command_line_parser.cc"
    click FastAPI_Router "https://github.com/triton-inference-server/server/tree/main/python/openai/openai_frontend/frontend/fastapi/routers/"
    click FastAPI_Main "https://github.com/triton-inference-server/server/blob/main/python/openai/openai_frontend/main.py"
    click Python_Bindings "https://github.com/triton-inference-server/server/blob/main/src/python/setup.py"
    click Python_C_API "https://github.com/triton-inference-server/server/tree/main/src/python/tritonfrontend/_c/"
    click Py_API_Wrappers "https://github.com/triton-inference-server/server/tree/main/src/python/tritonfrontend/_api/"
    click Python_Client "https://github.com/triton-inference-server/server/tree/main/python/openai/"
    click Cpp_SDK "https://github.com/triton-inference-server/server/tree/main/qa/L0_sdk/"
    click Entrypoints "https://github.com/triton-inference-server/server/tree/main/docker/entrypoint.d/"
    click Compose "https://github.com/triton-inference-server/server/blob/main/compose.py"
    click Kubernetes "https://github.com/triton-inference-server/server/tree/main/deploy/"
    click QA_Suite "https://github.com/triton-inference-server/server/tree/main/qa/"
    click Docs_Examples "https://github.com/triton-inference-server/server/tree/main/docs/examples/"

    %% Styles
    classDef client fill:#ffcc99,stroke:#333,stroke-width:1px
    classDef server fill:#99ccff,stroke:#333,stroke-width:1px
    classDef storage fill:#ccffcc,stroke:#333,stroke-width:1px
    classDef infra fill:#cc99ff,stroke:#333,stroke-width:1px
    classDef qa fill:#ff9999,stroke:#333,stroke-width:1px
```
