# microsoft/onnxruntime-genai architecture

> Architecture diagram of the GitHub repository microsoft/onnxruntime-genai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/onnxruntime-genai
- Repository: https://github.com/microsoft/onnxruntime-genai

## Overview

##### Project type and purpose
- This is an open-source SDK / runtime library (ONNX Runtime GenAI) that provides a generative AI inference engine for ONNX models.
- It implements the full GenAI loop—pre/post processing, tokenization, search & sampling, KV cache management—exposed via multiple language bindings (Python, C#, Java, Objective-C, C/C++) and accelerated by various execution providers (CUDA, DirectML, QNN, OpenVINO, WebGPU, CPU).

##### File structure
- Top-level “src” folder contains:
- core modules: models/, search.cpp/h, sampling and logits processors, config.h/cpp
- execution-provider folders: cuda/, dml/, qnn/, openvino/, webgpu/, cpu/
- language bindings: python/, csharp/, java/, objectivec/
- C API entrypoints in ort_genai_c.cpp/h
- “examples” demonstrates usage in each language and client application.
- “test” verifies core functionality, language APIs, platform packages.
- “cmake”, “CMakeLists.txt” drive the build and packaging.
- “.github” and “.pipelines” define CI/CD workflows and publishing steps (NuGet, PyPI, Cocoapods, AAR).

##### README for additional insights
- Purpose: “Run generative AI models with ONNX Runtime on-device”
- Lists supported model architectures, languages, platforms, acceleration backends
- Describes API layers (Model, Tokenizer, Generator, GeneratorParams) and sample usage in Python

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Engine
    subgraph "Core GenAI Engine" 
        direction TB
        CoreModels["Models & Pipelines"]:::core
        CoreConfig["config.cpp / config.h"]:::core
        CoreSearch["search.cpp / search.h"]:::core
        CoreGenerators["generators.cpp / generators.h"]:::core
        CoreSampling["constrained_logits_processor.cpp / .h"]:::core
    end

    %% C API
    CAPI["C API Layer\n(ort_genai_c.cpp / ort_genai_c.h)"]:::core

    %% Execution Providers
    subgraph "Execution Provider Plugins"
        direction TB
        CUDA["CUDA EP Plugin"]:::ep
        DML["DirectML EP Plugin"]:::ep
        QNN["QNN EP Plugin"]:::ep
        OpenVINO["OpenVINO EP Plugin"]:::ep
        WebGPU["WebGPU EP Plugin"]:::ep
        CPU["CPU EP Plugin"]:::ep
    end

    %% Language Bindings
    subgraph "Language Bindings"
        direction TB
        Python["Python Wrapper"]:::binding
        CSharp["C# Managed Package"]:::binding
        Java["Java JNI / AAR"]:::binding
        ObjC["Objective-C Pod"]:::binding
        NativeCPP["Native C/C++ API"]:::binding
    end

    %% Packaging & CI
    subgraph "Packaging & CI/CD"
        direction TB
        CMakeBuild["CMake Build\n(CMakeLists.txt / cmake/)"]:::ci
        NuGet["NuGet Publishing"]:::ci
        PyPI["PyPI Publishing"]:::ci
        CocoaPods["CocoaPods Publishing"]:::ci
        AAR["Android AAR Publishing"]:::ci
    end

    %% Examples & Tests
    subgraph "Examples & Tests"
        direction TB
        Examples["Examples (Python, C#, Java, ObjC, C)"]:::example
        Tests["Tests (core, API, sampling)"]:::example
    end

    %% Relationships
    Python -->|"uses C API (SWIG)"| CAPI
    CSharp -->|"PInvoke / C API"| CAPI
    Java -->|"JNI / C API"| CAPI
    ObjC -->|"ObjC wrapper / C API"| CAPI
    NativeCPP -->|"Direct call"| CAPI

    CAPI -->|"routes calls"| CoreModels
    CAPI -->|"routes calls"| CoreConfig
    CAPI -->|"routes calls"| CoreSearch
    CAPI -->|"routes calls"| CoreGenerators
    CAPI -->|"routes calls"| CoreSampling

    CoreModels -->|"dispatch compute"| CUDA
    CoreModels -->|"dispatch compute"| DML
    CoreModels -->|"dispatch compute"| QNN
    CoreModels -->|"dispatch compute"| OpenVINO
    CoreModels -->|"dispatch compute"| WebGPU
    CoreModels -->|"dispatch compute"| CPU

    CUDA -->|"GPU drivers"| Hardware["Hardware Backends"]:::external
    DML -->|"Windows DirectML"| Hardware
    QNN -->|"Qualcomm QNN runtime"| Hardware
    OpenVINO -->|"Intel OpenVINO"| Hardware
    WebGPU -->|"WebGPU runtime"| Hardware
    CPU -->|"CPU execution"| Hardware

    CMakeBuild --> NuGet
    CMakeBuild --> PyPI
    CMakeBuild --> CocoaPods
    CMakeBuild --> AAR

    NuGet --> Examples
    PyPI --> Examples
    CocoaPods --> Examples
    AAR --> Examples

    NuGet --> Tests
    PyPI --> Tests
    CocoaPods --> Tests
    AAR --> Tests

    Examples -->|consume| Python
    Examples -->|consume| CSharp
    Examples -->|consume| Java
    Examples -->|consume| ObjC
    Examples -->|consume| NativeCPP

    Tests -->|verify| CAPI
    Tests -->|verify| Python
    Tests -->|verify| CSharp
    Tests -->|verify| Java
    Tests -->|verify| ObjC

    %% Click Events
    click CoreModels "https://github.com/microsoft/onnxruntime-genai/tree/main/src/models/"
    click CoreConfig "https://github.com/microsoft/onnxruntime-genai/blob/main/src/config.cpp"
    click CoreSearch "https://github.com/microsoft/onnxruntime-genai/blob/main/src/search.cpp"
    click CoreGenerators "https://github.com/microsoft/onnxruntime-genai/blob/main/src/generators.cpp"
    click CoreSampling "https://github.com/microsoft/onnxruntime-genai/blob/main/src/constrained_logits_processor.cpp"
    click CUDA "https://github.com/microsoft/onnxruntime-genai/tree/main/src/cuda/"
    click DML "https://github.com/microsoft/onnxruntime-genai/tree/main/src/dml/"
    click QNN "https://github.com/microsoft/onnxruntime-genai/tree/main/src/qnn/"
    click OpenVINO "https://github.com/microsoft/onnxruntime-genai/tree/main/src/openvino/"
    click WebGPU "https://github.com/microsoft/onnxruntime-genai/tree/main/src/webgpu/"
    click CPU "https://github.com/microsoft/onnxruntime-genai/tree/main/src/cpu/"
    click CAPI "https://github.com/microsoft/onnxruntime-genai/blob/main/src/ort_genai_c.cpp"
    click Python "https://github.com/microsoft/onnxruntime-genai/tree/main/src/python/"
    click CSharp "https://github.com/microsoft/onnxruntime-genai/blob/main/src/csharp/Microsoft.ML.OnnxRuntimeGenAI.csproj"
    click Java "https://github.com/microsoft/onnxruntime-genai/tree/main/src/java/"
    click ObjC "https://github.com/microsoft/onnxruntime-genai/tree/main/src/objectivec/"
    click CMakeBuild "https://github.com/microsoft/onnxruntime-genai/blob/main/CMakeLists.txt"

    %% Styles
    classDef core fill:#D0E1FF,stroke:#0366D6,color:#0366D6
    classDef ep fill:#D0FFD8,stroke:#028A0F,color:#028A0F
    classDef binding fill:#FFEFD0,stroke:#D87F04,color:#D87F04
    classDef ci fill:#F3D0FF,stroke:#A205A2,color:#A205A2
    classDef example fill:#F0F0F0,stroke:#888888,color:#888888
    classDef external fill:#FFFFFF,stroke:#AAAAAA,color:#AAAAAA
```
