How microsoft/onnxruntime-genai works
Project type and purpose
• This is an open-source SDK / runtime library (ONNX Runtime GenAI) that provides a generative AI inference engine for ONNX models.
• It implements the full GenAI loop—pre/post processing, tokenization, search & sampling, KV cache management—exposed via multiple language bindings (Python, C#, Java, Objective-C, C/C++) and accelerated by various execution providers (CUDA, DirectML, QNN, OpenVINO, WebGPU, CPU).
File structure
• Top-level “src” folder contains:
• core modules: models/, search.cpp/h, sampling and logits processors, config.h/cpp
• execution-provider folders: cuda/, dml/, qnn/, openvino/, webgpu/, cpu/
• language bindings: python/, csharp/, java/, objectivec/
• C API entrypoints in ort_genai_c.cpp/h
• “examples” demonstrates usage in each language and client application.
• “test” verifies core functionality, language APIs, platform packages.
• “cmake”, “CMakeLists.txt” drive the build and packaging.
• “.github” and “.pipelines” define CI/CD workflows and publishing steps (NuGet, PyPI, Cocoapods, AAR).
README for additional insights
• Purpose: “Run generative AI models with ONNX Runtime on-device”
• Lists supported model architectures, languages, platforms, acceleration backends
• Describes API layers (Model, Tokenizer, Generator, GeneratorParams) and sample usage in Python