# flashinfer-ai/flashinfer architecture

> Architecture diagram of the GitHub repository flashinfer-ai/flashinfer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/flashinfer-ai/flashinfer
- Repository: https://github.com/flashinfer-ai/flashinfer
- Video tour (about a minute): https://gitdiagram.com/flashinfer-ai/flashinfer/video

## Overview

This project – FlashInfer – is a high-performance kernel library aimed at LLM serving and inference. Its purpose is to provide efficient, customizable GPU kernels (CUDA, Tensor Cores) for various attention operations (such as FlashAttention, SparseAttention, and more) that power LLM deployments. The library has multiple interfaces (Python via PyTorch, C++ header-only, and TVM bindings) and uses a suite of build tools and kernel generators.

##### Main Components
- User/Application Layer: Illustrate how end users (or an LLM inference server) interact with FlashInfer. List the different APIs:
- PyTorch API (from the “flashinfer” Python package)
- C++ API (using header files from “include/flashinfer”)
- TVM Bindings (in “tvm_binding”)
- Kernel Generation & Build System:
- AOT Build Utilities: Represented by the “aot_build_utils” directory, these modules generate and configure kernel code at build time.
- CMake & other build scripts (e.g., “CMakeLists.txt”, “Jenkinsfile”, Docker and devcontainer configurations) that compile the C++/CUDA kernels.
- Core Compute/Kernel Library:
- C++/CUDA Kernels: Group items from the “csrc” directory (which contains CUDA kernels, templated and Jinja configurations) and the “flashinfer” folder implementing CPU/PyTorch integration.
- Custom and JIT Kernel components: Represent the use of Jinja templates (e.g., “*.jinja” files) and the use of CUDAGraph, torch.compile compatibility.
- Supporting and Auxiliary Modules:
- 3rdparty libraries: This includes dependencies like CUTLASS, googletest, spdlog, etc., which provide lower-level routines and testing frameworks.
- Benchmarking & Profiling: Components in “benchmarks”, “profiler” and “tests” showcasing performance measurement and validation.
- Documentation & Examples: “docs” and “README.md” for guides, tutorials and API references.

##### Relationships and Interactions
- User/Application Layer calls the Python API (FlashInfer’s “flashinfer” package) which then invokes the underlying GPU kernels provided by the C++/CUDA implementation.
- The Python API communicates with custom C++ modules and links to the compiled CUDA kernels (located in “csrc”, “include/flashinfer”, and “flashinfer/triton”).
- AOT Build Utilities assist in kernel code generation. Their outputs feed into the compilation process managed by CMake and integrated in Docker/CI pipelines.
- The TVM binding layer (in “tvm_binding”) interfaces with the kernel library allowing use in TVM-based workflows.
- Benchmark and test components interact with almost every part of the system to ensure correctness and performance.
- 3rdparty components act as foundational layers that support both kernel operations (such as CUTLASS for efficient GEMM operations) and robust software practices (googletest for unit testing).

##### Architectural Patterns and Design Principles
- Modular Separation: Distinct separation between frontend (Python and C++ API interfaces) and backend (GPU kernel implementations and generation).
- Extensibility via JIT and Kernel Generation: The use of templates and on-the-fly kernel customization permits easy extensions and tuning for different LLM serving scenarios.
- Build & Deployment Abstraction: Containerization support (Docker, devcontainer) along with CI/CD integrations (Jenkins, GitHub Actions) is key in how the software is built, tested, and released.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User/Application Layer
    User["User/Application"]:::frontend

    %% API Interfaces
    subgraph "API Interfaces"
        PythonAPI["Python API (flashinfer)"]:::frontend
        CppAPI["C++ API (include/flashinfer)"]:::frontend
        TVMBindings["TVM Bindings (tvm_binding)"]:::frontend
    end

    %% Kernel Generation & Build System
    subgraph "Kernel Generation & Build System"
        AOTBuildUtils["AOT Build Utilities (aot_build_utils)"]:::build
        BuildCI["Build & CI/CD Tools"]:::build
    end

    %% Core GPU Kernel Library
    subgraph "GPU Kernel Library"
        CoreKernel["Core GPU Kernel Library (csrc)"]:::backend
    end

    %% Supporting and Auxiliary Modules
    subgraph "Auxiliary Modules"
        ThirdParty["3rdparty Dependencies (3rdparty)"]:::auxiliary
        Benchmarking["Benchmarking & Profiling (benchmarks/profiler/tests)"]:::auxiliary
        Documentation["Documentation & Examples (docs/README.md)"]:::auxiliary
    end

    %% Connections between User and API Interfaces
    User -->|"calls"| PythonAPI
    User -->|"calls"| CppAPI
    User -->|"calls"| TVMBindings

    %% Connections from API Interfaces to Core GPU Kernel Library
    PythonAPI -->|"invokes"| CoreKernel
    CppAPI -->|"integrates"| CoreKernel
    TVMBindings -->|"interfaces"| CoreKernel

    %% Connections from Kernel Generation & Build System to Core GPU Kernel Library
    AOTBuildUtils -->|"generates"| CoreKernel
    BuildCI -->|"compiles"| CoreKernel

    %% Connections from Core GPU Kernel Library to Auxiliary Modules
    CoreKernel -->|"utilizes"| ThirdParty
    CoreKernel -->|"validated by"| Benchmarking

    %% Documentation References
    PythonAPI -->|"references"| Documentation
    CppAPI -->|"references"| Documentation
    TVMBindings -->|"references"| Documentation

    %% Click Events
    click PythonAPI "https://github.com/flashinfer-ai/flashinfer/tree/main/flashinfer"
    click CppAPI "https://github.com/flashinfer-ai/flashinfer/tree/main/include/flashinfer"
    click TVMBindings "https://github.com/flashinfer-ai/flashinfer/tree/main/tvm_binding"
    click AOTBuildUtils "https://github.com/flashinfer-ai/flashinfer/tree/main/aot_build_utils"
    click BuildCI "https://github.com/flashinfer-ai/flashinfer/blob/main/CMakeLists.txt"
    click CoreKernel "https://github.com/flashinfer-ai/flashinfer/tree/main/csrc"
    click ThirdParty "https://github.com/flashinfer-ai/flashinfer/tree/main/3rdparty"
    click Benchmarking "https://github.com/flashinfer-ai/flashinfer/tree/main/benchmarks"
    click Documentation "https://github.com/flashinfer-ai/flashinfer/tree/main/docs"

    %% Styles
    classDef frontend fill:#8EC0E4,stroke:#333,stroke-width:2px;
    classDef backend fill:#F4A261,stroke:#333,stroke-width:2px;
    classDef build fill:#A8DADC,stroke:#333,stroke-width:2px;
    classDef auxiliary fill:#E9C46A,stroke:#333,stroke-width:2px;
```
