# flashrt-project/flashrt architecture

> Architecture diagram of the GitHub repository flashrt-project/flashrt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/flashrt-project/flashrt
- Repository: https://github.com/flashrt-project/flashrt

## Overview

FlashRT is a real-time inference engine for latency-sensitive, small-batch AI workloads. A typical VLA request enters a model-specific frontend/pipeline, runs optimized model stages on a hardware backend, and produces an action; the README also documents language, vision-language, image-generation, audio, and video workloads. The repository combines Python model pipelines, C++ runtime components, and CUDA/CUTLASS, AMD, and NPU implementations. Some documented product branches are represented at subsystem-directory level because their source was not sampled; edges for those branches rely on README statements rather than inspected call sites.

Main entry points: [`flash_rt/models`](https://github.com/flashrt-project/flashrt/tree/main/flash_rt/models) (VLA pipelines), [`flash_rt/models/pi05/pipeline_rtx.py`](https://github.com/flashrt-project/flashrt/blob/main/flash_rt/models/pi05/pipeline_rtx.py) (Pi0.5 RTX), [`flash_wm/src/bagel_fp8_engine.py`](https://github.com/flashrt-project/flashrt/blob/main/flash_wm/src/bagel_fp8_engine.py) (BAGEL generation), [`serving/qwen36_agent/qwen36_engine.py`](https://github.com/flashrt-project/flashrt/blob/main/serving/qwen36_agent/qwen36_engine.py) (Qwen serving)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipelines["Model pipelines"]
  node_vla["VLA pipelines"]
  node_pi05rtx["Pi0.5 RTX<br/>[pipeline_rtx.py]"]
  node_pi05amd["Pi0.5 AMD<br/>[pipeline.py]"]
  node_grootnpu["GROOT NPU<br/>[pipeline.py]"]
  node_othermodels["Other workloads"]
  node_bagel["BAGEL generation"]
  node_qwenfront["Qwen frontends<br/>[qwen36_rtx.py]"]
end

subgraph group_runtime["Runtime and state"]
  node_buffers["Stable buffers"]
  node_graphs["Captured graphs"]
  node_fusion["Temporal fusion"]
end

subgraph group_hardware["Hardware kernels"]
  node_cuda["CUDA kernels"]
  node_cutlass["CUTLASS pipelines"]
  node_amd["AMD kernels"]
  node_npu["NPU kernels"]
end

subgraph group_integrations["Serving integrations"]
  node_qwenserve["Qwen serving<br/>[qwen36_engine.py]"]
  node_audio["Audio serving<br/>[server.py]"]
  node_llmserve["LLM serving<br/>[main.rs]"]
end

subgraph group_cpp["C++ runtime"]
  node_cppmodel["Pi0.5 model runtime<br/>[model_runtime.cpp]"]
  node_modalities["Input modalities"]
  node_providers["Llama.cpp provider<br/>[provider_open.cpp]"]
end

node_caller(("Application caller"))
node_result(("Actions or generated output"))

node_caller -->|"requests policy"| node_vla
node_vla -.->|"selects pipeline"| node_pi05rtx
node_vla -.->|"selects backend"| node_pi05amd
node_vla -.->|"selects backend"| node_grootnpu
node_caller -->|"requests generation"| node_bagel
node_caller -.->|"sends prompt"| node_qwenserve
node_caller -->|"requests speech"| node_audio
node_caller -->|"sends prompt"| node_llmserve
node_pi05rtx -->|"uses scratch"| node_buffers
node_pi05rtx -->|"captures execution"| node_graphs
node_pi05rtx -.->|"dispatches kernels"| node_cuda
node_pi05amd -->|"dispatches kernels"| node_amd
node_grootnpu -->|"uses operators"| node_npu
node_cuda -.->|"uses primitives"| node_cutlass
node_othermodels -->|"supports fusion"| node_fusion
node_bagel -->|"uses kernels"| node_cuda
node_qwenserve -->|"selects frontend"| node_qwenfront
node_caller -.->|"opens model"| node_cppmodel
node_cppmodel -.->|"consumes inputs"| node_modalities
node_providers -.->|"adapts runtime"| node_cppmodel
node_vla -->|"returns actions"| node_result
node_bagel -->|"generates images"| node_result
node_qwenserve -.->|"streams tokens"| node_result
node_audio -->|"returns audio"| node_result
node_llmserve -->|"streams tokens"| node_result

click node_vla "https://github.com/flashrt-project/flashrt/tree/main/flash_rt/models"
click node_pi05rtx "https://github.com/flashrt-project/flashrt/blob/main/flash_rt/models/pi05/pipeline_rtx.py"
click node_pi05amd "https://github.com/flashrt-project/flashrt/blob/main/flash_rt/amd/models/pi05/pipeline.py"
click node_grootnpu "https://github.com/flashrt-project/flashrt/blob/main/flash_rt/npu/models/groot_n17/pipeline.py"
click node_othermodels "https://github.com/flashrt-project/flashrt/tree/main/flash_rt/models"
click node_bagel "https://github.com/flashrt-project/flashrt/blob/main/flash_wm/src/bagel_fp8_engine.py"
click node_qwenfront "https://github.com/flashrt-project/flashrt/blob/main/flash_rt/frontends/torch/qwen36_rtx.py"
click node_buffers "https://github.com/flashrt-project/flashrt/tree/main/flash_rt/core"
click node_graphs "https://github.com/flashrt-project/flashrt/tree/main/flash_rt/core"
click node_fusion "https://github.com/flashrt-project/flashrt/blob/main/flash_rt/runtime/rtc_temporal_fusion.py"
click node_cuda "https://github.com/flashrt-project/flashrt/tree/main/csrc"
click node_cutlass "https://github.com/flashrt-project/flashrt/tree/main/csrc/attention/flash_attn_2_src/cutlass/include"
click node_amd "https://github.com/flashrt-project/flashrt/tree/main/csrc/amd"
click node_npu "https://github.com/flashrt-project/flashrt/tree/main/flash_rt/npu"
click node_cppmodel "https://github.com/flashrt-project/flashrt/blob/main/cpp/models/pi05/src/model_runtime.cpp"
click node_modalities "https://github.com/flashrt-project/flashrt/tree/main/cpp/modalities/src"
click node_providers "https://github.com/flashrt-project/flashrt/blob/main/cpp/providers/llama_cpp/src/provider_open.cpp"
click node_qwenserve "https://github.com/flashrt-project/flashrt/blob/main/serving/qwen36_agent/qwen36_engine.py"
click node_audio "https://github.com/flashrt-project/flashrt/blob/main/serving/omnivoice_agent/server.py"
click node_llmserve "https://github.com/flashrt-project/flashrt/blob/main/serving/llm_agent/src/main.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_vla,node_pi05rtx,node_pi05amd,node_grootnpu,node_othermodels,node_bagel,node_qwenfront toneBlue
class node_buffers,node_graphs,node_fusion toneAmber
class node_cuda,node_cutlass,node_amd,node_npu toneMint
class node_qwenserve,node_audio,node_llmserve toneRose
class node_cppmodel,node_modalities,node_providers,node_caller,node_result toneIndigo
```

## Components

### Model pipelines

- [VLA pipelines](https://github.com/flashrt-project/flashrt/tree/main/flash_rt/models): `flash_rt/models`
- [Pi0.5 RTX](https://github.com/flashrt-project/flashrt/blob/main/flash_rt/models/pi05/pipeline_rtx.py): `flash_rt/models/pi05/pipeline_rtx.py`
- [Pi0.5 AMD](https://github.com/flashrt-project/flashrt/blob/main/flash_rt/amd/models/pi05/pipeline.py): `flash_rt/amd/models/pi05/pipeline.py`
- [GROOT NPU](https://github.com/flashrt-project/flashrt/blob/main/flash_rt/npu/models/groot_n17/pipeline.py): `flash_rt/npu/models/groot_n17/pipeline.py`
- [Other workloads](https://github.com/flashrt-project/flashrt/tree/main/flash_rt/models): `flash_rt/models`
- [BAGEL generation](https://github.com/flashrt-project/flashrt/blob/main/flash_wm/src/bagel_fp8_engine.py): `flash_wm/src/bagel_fp8_engine.py`
- [Qwen frontends](https://github.com/flashrt-project/flashrt/blob/main/flash_rt/frontends/torch/qwen36_rtx.py): `flash_rt/frontends/torch/qwen36_rtx.py`

### Runtime and state

- [Stable buffers](https://github.com/flashrt-project/flashrt/tree/main/flash_rt/core): `flash_rt/core`
- [Captured graphs](https://github.com/flashrt-project/flashrt/tree/main/flash_rt/core): `flash_rt/core`
- [Temporal fusion](https://github.com/flashrt-project/flashrt/blob/main/flash_rt/runtime/rtc_temporal_fusion.py): `flash_rt/runtime/rtc_temporal_fusion.py`

### Hardware kernels

- [CUDA kernels](https://github.com/flashrt-project/flashrt/tree/main/csrc): `csrc`
- [CUTLASS pipelines](https://github.com/flashrt-project/flashrt/tree/main/csrc/attention/flash_attn_2_src/cutlass/include): `csrc/attention/flash_attn_2_src/cutlass/include`
- [AMD kernels](https://github.com/flashrt-project/flashrt/tree/main/csrc/amd): `csrc/amd`
- [NPU kernels](https://github.com/flashrt-project/flashrt/tree/main/flash_rt/npu): `flash_rt/npu`

### Serving integrations

- [Qwen serving](https://github.com/flashrt-project/flashrt/blob/main/serving/qwen36_agent/qwen36_engine.py): `serving/qwen36_agent/qwen36_engine.py`
- [Audio serving](https://github.com/flashrt-project/flashrt/blob/main/serving/omnivoice_agent/server.py): `serving/omnivoice_agent/server.py`
- [LLM serving](https://github.com/flashrt-project/flashrt/blob/main/serving/llm_agent/src/main.rs): `serving/llm_agent/src/main.rs`

### C++ runtime

- [Pi0.5 model runtime](https://github.com/flashrt-project/flashrt/blob/main/cpp/models/pi05/src/model_runtime.cpp): `cpp/models/pi05/src/model_runtime.cpp`
- [Input modalities](https://github.com/flashrt-project/flashrt/tree/main/cpp/modalities/src): `cpp/modalities/src`
- [Llama.cpp provider](https://github.com/flashrt-project/flashrt/blob/main/cpp/providers/llama_cpp/src/provider_open.cpp): `cpp/providers/llama_cpp/src/provider_open.cpp`

### Other components

- **Application caller**
- **Actions or generated output**

## Connections

- Application caller → VLA pipelines: requests policy (evidence: `README.md`)
- VLA pipelines → Pi0.5 RTX: selects pipeline (evidence: `flash_rt/models/pi05/pipeline_rtx.py`)
- VLA pipelines → Pi0.5 AMD: selects backend
- VLA pipelines → GROOT NPU: selects backend
- Application caller → BAGEL generation: requests generation (evidence: `README.md`)
- Application caller → Qwen serving: sends prompt
- Application caller → Audio serving: requests speech (evidence: `README.md`)
- Application caller → LLM serving: sends prompt (evidence: `README.md`)
- Pi0.5 RTX → Stable buffers: uses scratch (evidence: `flash_rt/models/pi05/pipeline_rtx.py`)
- Pi0.5 RTX → Captured graphs: captures execution (evidence: `flash_rt/models/pi05/pipeline_rtx.py`)
- Pi0.5 RTX → CUDA kernels: dispatches kernels
- Pi0.5 AMD → AMD kernels: dispatches kernels (evidence: `flash_rt/amd/models/pi05/pipeline.py`)
- GROOT NPU → NPU kernels: uses operators (evidence: `flash_rt/npu/models/groot_n17/pipeline.py`)
- CUDA kernels → CUTLASS pipelines: uses primitives (evidence: `csrc/gemm/mega/cutlass/pipeline/sm100_pipeline.hpp`)
- Other workloads → Temporal fusion: supports fusion (evidence: `README.md`)
- BAGEL generation → CUDA kernels: uses kernels (evidence: `flash_wm/src/bagel_fp8_engine.py`)
- Qwen serving → Qwen frontends: selects frontend (evidence: `serving/qwen36_agent/qwen36_engine.py`)
- Application caller → Pi0.5 model runtime: opens model
- Pi0.5 model runtime → Input modalities: consumes inputs
- Llama.cpp provider → Pi0.5 model runtime: adapts runtime
- VLA pipelines → Actions or generated output: returns actions (evidence: `README.md`)
- BAGEL generation → Actions or generated output: generates images (evidence: `README.md`)
- Qwen serving → Actions or generated output: streams tokens
- Audio serving → Actions or generated output: returns audio (evidence: `README.md`)
- LLM serving → Actions or generated output: streams tokens (evidence: `README.md`)
