# tenstorrent/tt-inference-server architecture

> Architecture diagram of the GitHub repository tenstorrent/tt-inference-server, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tenstorrent/tt-inference-server
- Repository: https://github.com/tenstorrent/tt-inference-server

## Overview

tt-inference-server deploys and tests model-serving workloads on Tenstorrent hardware. The user-facing path is an inference request entering an API, being handled by a model service and worker runtime, then returning a response; the repository also supports workflow-driven deployment, performance benchmarking, and accuracy evaluation. The graph covers the sampled Python server and C++ LLM path, plus documented workflow and evaluation capabilities. Some subsystems are represented by their existing directories because their implementation was not sampled; dashed links mark documented capabilities whose internal wiring is not shown.

Main entry points: [`tt-media-server/cpp_server/src/api/llm_controller.cpp`](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/src/api/llm_controller.cpp) (C++ chat API), [`tt-media-server/main.py`](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/main.py) (Python API server), [`tt-media-server/cpp_server/src/services/llm_service.cpp`](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/src/services/llm_service.cpp) (LLM service), [`tt-media-server/utils/job_manager.py`](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/utils/job_manager.py) (Job management)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Inference serving"]
  node_python_app["Python API server<br/>[main.py]"]
  node_openai_api["OpenAI API routes"]
  node_cpp_llm_api["C++ chat API<br/>[llm_controller.cpp]"]
  node_response_writers["Response writers"]
  node_stream_formatter["Stream formatter"]
  node_embed_api["Embedding API"]
  node_llm_response["LLM response model<br/>[llm_response.hpp]"]
end

subgraph group_runtime["Model runtime"]
  node_model_config["Model configuration<br/>[settings.py]"]
  node_vllm_config["vLLM settings<br/>[vllm_settings.py]"]
  node_llm_service["LLM service<br/>[llm_service.cpp]"]
  node_llm_pipeline["LLM pipeline<br/>[llm_pipeline.hpp]"]
  node_request_pipeline["Worker request pipeline"]
  node_worker_runtime["Device workers<br/>[device_worker.py]"]
  node_service_container["Service container"]
  node_prefill_gateway["Prefill gateway<br/>[main.cpp]"]
  node_settings_cpp["C++ server settings<br/>[settings.hpp]"]
  node_base_service["Python service base<br/>[base_service.py]"]
end

subgraph group_ops["Jobs and operations"]
  node_job_manager["Job management<br/>[job_manager.py]"]
  node_job_database[("Job persistence<br/>[job_database.py]")]
  node_workflows["Workflow automation"]
  node_py_logger["Python logging<br/>[logger.py]"]
  node_cpp_logger["C++ logging<br/>[logger.hpp]"]
end

subgraph group_validation["Benchmarking and evaluation"]
  node_benchmark["Performance benchmarks"]
  node_evaluation["Accuracy evaluations"]
end

node_client(("Inference client"))

node_client -->|"sends request"| node_python_app
node_python_app -->|"mounts routes"| node_openai_api
node_python_app -->|"shuts down"| node_job_manager
node_python_app -.->|"supports automation"| node_workflows
node_client -->|"posts completions"| node_cpp_llm_api
node_cpp_llm_api -->|"dispatches requests"| node_llm_pipeline
node_cpp_llm_api -->|"resolves service"| node_service_container
node_cpp_llm_api -->|"writes responses"| node_response_writers
node_cpp_llm_api -->|"formats stream"| node_stream_formatter
node_cpp_llm_api -->|"checks readiness"| node_llm_service
node_embed_api -->|"enqueues request"| node_request_pipeline
node_embed_api -->|"uses services"| node_service_container
node_llm_pipeline -->|"uses response type"| node_llm_response
node_request_pipeline -.->|"dispatches work"| node_worker_runtime
node_service_container -->|"uses config types"| node_settings_cpp
node_model_config -->|"loads settings"| node_vllm_config
node_job_manager -->|"persists jobs"| node_job_database
node_job_manager -->|"logs operations"| node_py_logger
node_llm_service -.->|"uses worker pipeline"| node_request_pipeline
node_prefill_gateway -.->|"routes prefill"| node_client
node_workflows -.->|"runs benchmarks"| node_benchmark
node_workflows -.->|"runs evaluations"| node_evaluation
node_benchmark -.->|"tests serving"| node_base_service
node_llm_service -.->|"logs activity"| node_cpp_logger

click node_python_app "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/main.py"
click node_openai_api "https://github.com/tenstorrent/tt-inference-server/tree/main/tt-media-server/open_ai_api"
click node_cpp_llm_api "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/src/api/llm_controller.cpp"
click node_response_writers "https://github.com/tenstorrent/tt-inference-server/tree/main/tt-media-server/cpp_server/include/api/response_writer"
click node_stream_formatter "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/api/stream_event_formatter.hpp"
click node_embed_api "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/src/api/embedding_controller.cpp"
click node_model_config "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/config/settings.py"
click node_vllm_config "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/config/vllm_settings.py"
click node_llm_service "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/src/services/llm_service.cpp"
click node_llm_pipeline "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/services/llm_pipeline.hpp"
click node_request_pipeline "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/services/request_pipeline.hpp"
click node_worker_runtime "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/device_workers/device_worker.py"
click node_service_container "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/services/service_container.hpp"
click node_prefill_gateway "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/prefill_gateway/src/main.cpp"
click node_job_manager "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/utils/job_manager.py"
click node_job_database "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/utils/job_database.py"
click node_workflows "https://github.com/tenstorrent/tt-inference-server/tree/main/workflows"
click node_benchmark "https://github.com/tenstorrent/tt-inference-server/tree/main/test_module/benchmark_tests"
click node_evaluation "https://github.com/tenstorrent/tt-inference-server/tree/main/test_module/eval_tests"
click node_settings_cpp "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/config/settings.hpp"
click node_llm_response "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/domain/llm/llm_response.hpp"
click node_base_service "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/model_services/base_service.py"
click node_py_logger "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/utils/logger.py"
click node_cpp_logger "https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/utils/logger.hpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_python_app,node_openai_api,node_cpp_llm_api,node_response_writers,node_stream_formatter,node_embed_api,node_llm_response,node_client toneBlue
class node_model_config,node_vllm_config,node_llm_service,node_llm_pipeline,node_request_pipeline,node_worker_runtime,node_service_container,node_prefill_gateway,node_settings_cpp,node_base_service toneAmber
class node_job_manager,node_job_database,node_workflows,node_py_logger,node_cpp_logger toneMint
class node_benchmark,node_evaluation toneRose
```

## Components

### Inference serving

- [Python API server](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/main.py): `tt-media-server/main.py`
- [OpenAI API routes](https://github.com/tenstorrent/tt-inference-server/tree/main/tt-media-server/open_ai_api): `tt-media-server/open_ai_api`
- [C++ chat API](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/src/api/llm_controller.cpp): `tt-media-server/cpp_server/src/api/llm_controller.cpp`
- [Response writers](https://github.com/tenstorrent/tt-inference-server/tree/main/tt-media-server/cpp_server/include/api/response_writer): `tt-media-server/cpp_server/include/api/response_writer`
- [Stream formatter](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/api/stream_event_formatter.hpp): `tt-media-server/cpp_server/include/api/stream_event_formatter.hpp`
- [Embedding API](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/src/api/embedding_controller.cpp): `tt-media-server/cpp_server/src/api/embedding_controller.cpp`
- [LLM response model](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/domain/llm/llm_response.hpp): `tt-media-server/cpp_server/include/domain/llm/llm_response.hpp`

### Model runtime

- [Model configuration](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/config/settings.py): `tt-media-server/config/settings.py`
- [vLLM settings](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/config/vllm_settings.py): `tt-media-server/config/vllm_settings.py`
- [LLM service](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/src/services/llm_service.cpp): `tt-media-server/cpp_server/src/services/llm_service.cpp`
- [LLM pipeline](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/services/llm_pipeline.hpp): `tt-media-server/cpp_server/include/services/llm_pipeline.hpp`
- [Worker request pipeline](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/services/request_pipeline.hpp): `tt-media-server/cpp_server/include/services/request_pipeline.hpp`
- [Device workers](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/device_workers/device_worker.py): `tt-media-server/device_workers/device_worker.py`
- [Service container](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/services/service_container.hpp): `tt-media-server/cpp_server/include/services/service_container.hpp`
- [Prefill gateway](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/prefill_gateway/src/main.cpp): `tt-media-server/prefill_gateway/src/main.cpp`
- [C++ server settings](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/config/settings.hpp): `tt-media-server/cpp_server/include/config/settings.hpp`
- [Python service base](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/model_services/base_service.py): `tt-media-server/model_services/base_service.py`

### Jobs and operations

- [Job management](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/utils/job_manager.py): `tt-media-server/utils/job_manager.py`
- [Job persistence](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/utils/job_database.py): `tt-media-server/utils/job_database.py`
- [Workflow automation](https://github.com/tenstorrent/tt-inference-server/tree/main/workflows): `workflows`
- [Python logging](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/utils/logger.py): `tt-media-server/utils/logger.py`
- [C++ logging](https://github.com/tenstorrent/tt-inference-server/blob/main/tt-media-server/cpp_server/include/utils/logger.hpp): `tt-media-server/cpp_server/include/utils/logger.hpp`

### Benchmarking and evaluation

- [Performance benchmarks](https://github.com/tenstorrent/tt-inference-server/tree/main/test_module/benchmark_tests): `test_module/benchmark_tests`
- [Accuracy evaluations](https://github.com/tenstorrent/tt-inference-server/tree/main/test_module/eval_tests): `test_module/eval_tests`

### Other components

- **Inference client**

## Connections

- Inference client → Python API server: sends request (evidence: `tt-media-server/main.py`)
- Python API server → OpenAI API routes: mounts routes (evidence: `tt-media-server/main.py`)
- Python API server → Job management: shuts down (evidence: `tt-media-server/main.py`)
- Python API server → Workflow automation: supports automation (evidence: `README.md`)
- Inference client → C++ chat API: posts completions
- C++ chat API → LLM pipeline: dispatches requests (evidence: `tt-media-server/cpp_server/src/api/llm_controller.cpp`)
- C++ chat API → Service container: resolves service (evidence: `tt-media-server/cpp_server/src/api/llm_controller.cpp`)
- C++ chat API → Response writers: writes responses (evidence: `tt-media-server/cpp_server/src/api/llm_controller.cpp`)
- C++ chat API → Stream formatter: formats stream (evidence: `tt-media-server/cpp_server/src/api/llm_controller.cpp`)
- C++ chat API → LLM service: checks readiness (evidence: `tt-media-server/cpp_server/src/api/llm_controller.cpp`)
- Embedding API → Worker request pipeline: enqueues request (evidence: `tt-media-server/cpp_server/src/api/embedding_controller.cpp`)
- Embedding API → Service container: uses services (evidence: `tt-media-server/cpp_server/src/api/embedding_controller.cpp`)
- LLM pipeline → LLM response model: uses response type (evidence: `tt-media-server/cpp_server/include/services/llm_pipeline.hpp`)
- Worker request pipeline → Device workers: dispatches work
- Service container → C++ server settings: uses config types (evidence: `tt-media-server/cpp_server/include/services/service_container.hpp`)
- Model configuration → vLLM settings: loads settings (evidence: `tt-media-server/config/settings.py`)
- Job management → Job persistence: persists jobs (evidence: `tt-media-server/utils/job_manager.py`)
- Job management → Python logging: logs operations (evidence: `tt-media-server/utils/job_manager.py`)
- LLM service → Worker request pipeline: uses worker pipeline
- Prefill gateway → Inference client: routes prefill
- Workflow automation → Performance benchmarks: runs benchmarks (evidence: `README.md`)
- Workflow automation → Accuracy evaluations: runs evaluations (evidence: `README.md`)
- Performance benchmarks → Python service base: tests serving
- LLM service → C++ logging: logs activity
