How nvidia/tensorrt-llm works
TensorRT-LLM is an inference framework for large language and visual-generation models, combining a Python runtime with specialized GPU kernels. The main request path is user input through model execution to generated output; cache management supports efficient serving, while visual-generation pipelines provide a distinct generation path. The README establishes these product capabilities, but the sampled source excerpts are partial and do not expose all runtime wiring, so several subsystem links are intentionally omitted.
Main entry points: pipeline.py Visual Pipeline · model_engine.py Python Model Engine · executor.h C++ Executor · batch_manager Batch Manager
Components 14
LLM Runtime
- Python Model Engine
tensorrt_llm/_torch/pyexecutor/model_engine.py - C++ Executor
cpp/include/tensorrt_llm/executor/executor.h - Batch Manager
cpp/tensorrt_llm/batch_manager - Generated Output
Cache Management
- KV Cache Manager
cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp - Cache Storage
cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.cpp - Python Cache Storage
tensorrt_llm/runtime/kv_cache_manager_v2/_storage_manager.py
GPU Kernels
- Attention Kernels
cpp/kernels/fmha_v2/src/fused_multihead_attention.cpp - Decode Kernels
cpp/kernels/xqa/mha.h
Visual Generation
- Visual Pipeline
tensorrt_llm/_torch/visual_gen/pipeline.py - Cosmos Pipeline
tensorrt_llm/_torch/visual_gen/models/cosmos3/pipeline_cosmos3.py - LTX-2 Pipeline
tensorrt_llm/_torch/visual_gen/models/ltx2/pipeline_ltx2.py - Layered Image Pipeline
tensorrt_llm/_torch/visual_gen/models/qwen_image_layered/pipeline_qwen_image_layered.py
Other components
- Inference User
Connections 13
- Inference User → Python Model Engine: submits request
- Python Model Engine → C++ Executor: runs inference
- C++ Executor → Batch Manager: dispatches work
- C++ Executor → Generated Output: returns output
- Batch Manager → KV Cache Manager: manages cache
- KV Cache Manager → Cache Storage: uses storage
- Python Model Engine → Attention Kernels: executes kernels
- Python Model Engine → Decode Kernels: executes kernels
- Inference User → Visual Pipeline: requests generation
- Visual Pipeline → Generated Output: produces output
- Visual Pipeline → Cosmos Pipeline: supports model pipeline
- Visual Pipeline → LTX-2 Pipeline: supports model pipeline
- Visual Pipeline → Layered Image Pipeline: supports model pipeline