How flashrt-project/flashrt works
FlashRT is a real-time inference engine for latency-sensitive, small-batch AI workloads. A typical VLA request enters a model-specific frontend/pipeline, runs optimized model stages on a hardware backend, and produces an action; the README also documents language, vision-language, image-generation, audio, and video workloads. The repository combines Python model pipelines, C++ runtime components, and CUDA/CUTLASS, AMD, and NPU implementations. Some documented product branches are represented at subsystem-directory level because their source was not sampled; edges for those branches rely on README statements rather than inspected call sites.
Main entry points: models VLA pipelines · pipeline_rtx.py Pi0.5 RTX · bagel_fp8_engine.py BAGEL generation · qwen36_engine.py Qwen serving
Components 22
Model pipelines
- VLA pipelines
flash_rt/models - Pi0.5 RTX
flash_rt/models/pi05/pipeline_rtx.py - Pi0.5 AMD
flash_rt/amd/models/pi05/pipeline.py - GROOT NPU
flash_rt/npu/models/groot_n17/pipeline.py - Other workloads
flash_rt/models - BAGEL generation
flash_wm/src/bagel_fp8_engine.py - Qwen frontends
flash_rt/frontends/torch/qwen36_rtx.py
Runtime and state
- Stable buffers
flash_rt/core - Captured graphs
flash_rt/core - Temporal fusion
flash_rt/runtime/rtc_temporal_fusion.py
Hardware kernels
- CUDA kernels
csrc - CUTLASS pipelines
csrc/attention/flash_attn_2_src/cutlass/include - AMD kernels
csrc/amd - NPU kernels
flash_rt/npu
Serving integrations
- Qwen serving
serving/qwen36_agent/qwen36_engine.py - Audio serving
serving/omnivoice_agent/server.py - LLM serving
serving/llm_agent/src/main.rs
C++ runtime
- Pi0.5 model runtime
cpp/models/pi05/src/model_runtime.cpp - Input modalities
cpp/modalities/src - Llama.cpp provider
cpp/providers/llama_cpp/src/provider_open.cpp
Other components
- Application caller
- Actions or generated output
Connections 25
- Application caller → VLA pipelines: requests policy
README.md - VLA pipelines → Pi0.5 RTX: selects pipeline
flash_rt/models/pi05/pipeline_rtx.py - VLA pipelines → Pi0.5 AMD: selects backend
- VLA pipelines → GROOT NPU: selects backend
- Application caller → BAGEL generation: requests generation
README.md - Application caller → Qwen serving: sends prompt
- Application caller → Audio serving: requests speech
README.md - Application caller → LLM serving: sends prompt
README.md - Pi0.5 RTX → Stable buffers: uses scratch
flash_rt/models/pi05/pipeline_rtx.py - Pi0.5 RTX → Captured graphs: captures execution
flash_rt/models/pi05/pipeline_rtx.py - Pi0.5 RTX → CUDA kernels: dispatches kernels
- Pi0.5 AMD → AMD kernels: dispatches kernels
flash_rt/amd/models/pi05/pipeline.py - GROOT NPU → NPU kernels: uses operators
flash_rt/npu/models/groot_n17/pipeline.py - CUDA kernels → CUTLASS pipelines: uses primitives
csrc/gemm/mega/cutlass/pipeline/sm100_pipeline.hpp - Other workloads → Temporal fusion: supports fusion
README.md - BAGEL generation → CUDA kernels: uses kernels
flash_wm/src/bagel_fp8_engine.py - Qwen serving → Qwen frontends: selects frontend
serving/qwen36_agent/qwen36_engine.py - Application caller → Pi0.5 model runtime: opens model
- Pi0.5 model runtime → Input modalities: consumes inputs
- Llama.cpp provider → Pi0.5 model runtime: adapts runtime
- VLA pipelines → Actions or generated output: returns actions
README.md - BAGEL generation → Actions or generated output: generates images
README.md - Qwen serving → Actions or generated output: streams tokens
- Audio serving → Actions or generated output: returns audio
README.md - LLM serving → Actions or generated output: streams tokens
README.md