How tenstorrent/tt-inference-server works
tt-inference-server deploys and tests model-serving workloads on Tenstorrent hardware. The user-facing path is an inference request entering an API, being handled by a model service and worker runtime, then returning a response; the repository also supports workflow-driven deployment, performance benchmarking, and accuracy evaluation. The graph covers the sampled Python server and C++ LLM path, plus documented workflow and evaluation capabilities. Some subsystems are represented by their existing directories because their implementation was not sampled; dashed links mark documented capabilities whose internal wiring is not shown.
Main entry points: llm_controller.cpp C++ chat API · main.py Python API server · llm_service.cpp LLM service · job_manager.py Job management
Components 25
Inference serving
- Python API server
tt-media-server/main.py - OpenAI API routes
tt-media-server/open_ai_api - C++ chat API
tt-media-server/cpp_server/src/api/llm_controller.cpp - Response writers
tt-media-server/cpp_server/include/api/response_writer - Stream formatter
tt-media-server/cpp_server/include/api/stream_event_formatter.hpp - Embedding API
tt-media-server/cpp_server/src/api/embedding_controller.cpp - LLM response model
tt-media-server/cpp_server/include/domain/llm/llm_response.hpp
Model runtime
- Model configuration
tt-media-server/config/settings.py - vLLM settings
tt-media-server/config/vllm_settings.py - LLM service
tt-media-server/cpp_server/src/services/llm_service.cpp - LLM pipeline
tt-media-server/cpp_server/include/services/llm_pipeline.hpp - Worker request pipeline
tt-media-server/cpp_server/include/services/request_pipeline.hpp - Device workers
tt-media-server/device_workers/device_worker.py - Service container
tt-media-server/cpp_server/include/services/service_container.hpp - Prefill gateway
tt-media-server/prefill_gateway/src/main.cpp - C++ server settings
tt-media-server/cpp_server/include/config/settings.hpp - Python service base
tt-media-server/model_services/base_service.py
Jobs and operations
- Job management
tt-media-server/utils/job_manager.py - Job persistence
tt-media-server/utils/job_database.py - Workflow automation
workflows - Python logging
tt-media-server/utils/logger.py - C++ logging
tt-media-server/cpp_server/include/utils/logger.hpp
Benchmarking and evaluation
- Performance benchmarks
test_module/benchmark_tests - Accuracy evaluations
test_module/eval_tests
Other components
- Inference client
Connections 24
- Inference client → Python API server: sends request
tt-media-server/main.py - Python API server → OpenAI API routes: mounts routes
tt-media-server/main.py - Python API server → Job management: shuts down
tt-media-server/main.py - Python API server → Workflow automation: supports automation
README.md - Inference client → C++ chat API: posts completions
- C++ chat API → LLM pipeline: dispatches requests
tt-media-server/cpp_server/src/api/llm_controller.cpp - C++ chat API → Service container: resolves service
tt-media-server/cpp_server/src/api/llm_controller.cpp - C++ chat API → Response writers: writes responses
tt-media-server/cpp_server/src/api/llm_controller.cpp - C++ chat API → Stream formatter: formats stream
tt-media-server/cpp_server/src/api/llm_controller.cpp - C++ chat API → LLM service: checks readiness
tt-media-server/cpp_server/src/api/llm_controller.cpp - Embedding API → Worker request pipeline: enqueues request
tt-media-server/cpp_server/src/api/embedding_controller.cpp - Embedding API → Service container: uses services
tt-media-server/cpp_server/src/api/embedding_controller.cpp - LLM pipeline → LLM response model: uses response type
tt-media-server/cpp_server/include/services/llm_pipeline.hpp - Worker request pipeline → Device workers: dispatches work
- Service container → C++ server settings: uses config types
tt-media-server/cpp_server/include/services/service_container.hpp - Model configuration → vLLM settings: loads settings
tt-media-server/config/settings.py - Job management → Job persistence: persists jobs
tt-media-server/utils/job_manager.py - Job management → Python logging: logs operations
tt-media-server/utils/job_manager.py - LLM service → Worker request pipeline: uses worker pipeline
- Prefill gateway → Inference client: routes prefill
- Workflow automation → Performance benchmarks: runs benchmarks
README.md - Workflow automation → Accuracy evaluations: runs evaluations
README.md - Performance benchmarks → Python service base: tests serving
- LLM service → C++ logging: logs activity