# nvidia/tensorrt-llm architecture

> Architecture diagram of the GitHub repository nvidia/tensorrt-llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/tensorrt-llm
- Repository: https://github.com/nvidia/tensorrt-llm

## Overview

TensorRT-LLM is an inference framework for large language and visual-generation models, combining a Python runtime with specialized GPU kernels. The main request path is user input through model execution to generated output; cache management supports efficient serving, while visual-generation pipelines provide a distinct generation path. The README establishes these product capabilities, but the sampled source excerpts are partial and do not expose all runtime wiring, so several subsystem links are intentionally omitted.

Main entry points: [`tensorrt_llm/_torch/visual_gen/pipeline.py`](https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/_torch/visual_gen/pipeline.py) (Visual Pipeline), [`tensorrt_llm/_torch/pyexecutor/model_engine.py`](https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/_torch/pyexecutor/model_engine.py) (Python Model Engine), [`cpp/include/tensorrt_llm/executor/executor.h`](https://github.com/nvidia/tensorrt-llm/blob/main/cpp/include/tensorrt_llm/executor/executor.h) (C++ Executor), [`cpp/tensorrt_llm/batch_manager`](https://github.com/nvidia/tensorrt-llm/tree/main/cpp/tensorrt_llm/batch_manager) (Batch Manager)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["LLM Runtime"]
  node_engine["Python Model Engine<br/>[model_engine.py]"]
  node_executor["C++ Executor<br/>[executor.h]"]
  node_batch["Batch Manager"]
  node_result["Generated Output"]
end

subgraph group_cache["Cache Management"]
  node_kvmanager["KV Cache Manager<br/>[kvCacheManager.cpp]"]
  node_storage[("Cache Storage<br/>[storageManager.cpp]")]
  node_py_storage["Python Cache Storage"]
end

subgraph group_kernels["GPU Kernels"]
  node_attention["Attention Kernels"]
  node_decode["Decode Kernels<br/>[mha.h]"]
end

subgraph group_visual["Visual Generation"]
  node_visual_pipeline["Visual Pipeline<br/>[pipeline.py]"]
  node_cosmos["Cosmos Pipeline"]
  node_ltx["LTX-2 Pipeline<br/>[pipeline_ltx2.py]"]
  node_layered["Layered Image Pipeline"]
end

node_user(("Inference User"))

node_user -->|"submits request"| node_engine
node_engine -->|"runs inference"| node_executor
node_executor -->|"dispatches work"| node_batch
node_executor -->|"returns output"| node_result
node_batch -->|"manages cache"| node_kvmanager
node_kvmanager -->|"uses storage"| node_storage
node_engine -->|"executes kernels"| node_attention
node_engine -->|"executes kernels"| node_decode
node_user -->|"requests generation"| node_visual_pipeline
node_visual_pipeline -->|"produces output"| node_result
node_visual_pipeline -->|"supports model pipeline"| node_cosmos
node_visual_pipeline -->|"supports model pipeline"| node_ltx
node_visual_pipeline -->|"supports model pipeline"| node_layered

click node_engine "https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/_torch/pyexecutor/model_engine.py"
click node_executor "https://github.com/nvidia/tensorrt-llm/blob/main/cpp/include/tensorrt_llm/executor/executor.h"
click node_batch "https://github.com/nvidia/tensorrt-llm/tree/main/cpp/tensorrt_llm/batch_manager"
click node_kvmanager "https://github.com/nvidia/tensorrt-llm/blob/main/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp"
click node_storage "https://github.com/nvidia/tensorrt-llm/blob/main/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.cpp"
click node_py_storage "https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/runtime/kv_cache_manager_v2/_storage_manager.py"
click node_attention "https://github.com/nvidia/tensorrt-llm/blob/main/cpp/kernels/fmha_v2/src/fused_multihead_attention.cpp"
click node_decode "https://github.com/nvidia/tensorrt-llm/blob/main/cpp/kernels/xqa/mha.h"
click node_visual_pipeline "https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/_torch/visual_gen/pipeline.py"
click node_cosmos "https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/_torch/visual_gen/models/cosmos3/pipeline_cosmos3.py"
click node_ltx "https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/_torch/visual_gen/models/ltx2/pipeline_ltx2.py"
click node_layered "https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/_torch/visual_gen/models/qwen_image_layered/pipeline_qwen_image_layered.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_engine,node_executor,node_batch,node_result,node_user toneBlue
class node_kvmanager,node_storage,node_py_storage toneAmber
class node_attention,node_decode toneMint
class node_visual_pipeline,node_cosmos,node_ltx,node_layered toneRose
```

## Components

### LLM Runtime

- [Python Model Engine](https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/_torch/pyexecutor/model_engine.py): `tensorrt_llm/_torch/pyexecutor/model_engine.py`
- [C++ Executor](https://github.com/nvidia/tensorrt-llm/blob/main/cpp/include/tensorrt_llm/executor/executor.h): `cpp/include/tensorrt_llm/executor/executor.h`
- [Batch Manager](https://github.com/nvidia/tensorrt-llm/tree/main/cpp/tensorrt_llm/batch_manager): `cpp/tensorrt_llm/batch_manager`
- **Generated Output**

### Cache Management

- [KV Cache Manager](https://github.com/nvidia/tensorrt-llm/blob/main/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp): `cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp`
- [Cache Storage](https://github.com/nvidia/tensorrt-llm/blob/main/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.cpp): `cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.cpp`
- [Python Cache Storage](https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/runtime/kv_cache_manager_v2/_storage_manager.py): `tensorrt_llm/runtime/kv_cache_manager_v2/_storage_manager.py`

### GPU Kernels

- [Attention Kernels](https://github.com/nvidia/tensorrt-llm/blob/main/cpp/kernels/fmha_v2/src/fused_multihead_attention.cpp): `cpp/kernels/fmha_v2/src/fused_multihead_attention.cpp`
- [Decode Kernels](https://github.com/nvidia/tensorrt-llm/blob/main/cpp/kernels/xqa/mha.h): `cpp/kernels/xqa/mha.h`

### Visual Generation

- [Visual Pipeline](https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/_torch/visual_gen/pipeline.py): `tensorrt_llm/_torch/visual_gen/pipeline.py`
- [Cosmos Pipeline](https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/_torch/visual_gen/models/cosmos3/pipeline_cosmos3.py): `tensorrt_llm/_torch/visual_gen/models/cosmos3/pipeline_cosmos3.py`
- [LTX-2 Pipeline](https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/_torch/visual_gen/models/ltx2/pipeline_ltx2.py): `tensorrt_llm/_torch/visual_gen/models/ltx2/pipeline_ltx2.py`
- [Layered Image Pipeline](https://github.com/nvidia/tensorrt-llm/blob/main/tensorrt_llm/_torch/visual_gen/models/qwen_image_layered/pipeline_qwen_image_layered.py): `tensorrt_llm/_torch/visual_gen/models/qwen_image_layered/pipeline_qwen_image_layered.py`

### Other components

- **Inference User**

## Connections

- Inference User → Python Model Engine: submits request
- Python Model Engine → C++ Executor: runs inference
- C++ Executor → Batch Manager: dispatches work
- C++ Executor → Generated Output: returns output
- Batch Manager → KV Cache Manager: manages cache
- KV Cache Manager → Cache Storage: uses storage
- Python Model Engine → Attention Kernels: executes kernels
- Python Model Engine → Decode Kernels: executes kernels
- Inference User → Visual Pipeline: requests generation
- Visual Pipeline → Generated Output: produces output
- Visual Pipeline → Cosmos Pipeline: supports model pipeline
- Visual Pipeline → LTX-2 Pipeline: supports model pipeline
- Visual Pipeline → Layered Image Pipeline: supports model pipeline