# michalpitr/inference_engine architecture

> Architecture diagram of the GitHub repository michalpitr/inference_engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/michalpitr/inference_engine
- Repository: https://github.com/michalpitr/inference_engine

## Overview

1. This repository is a small ONNX-based inference platform split into two major runtimes: a C++ inference engine that executes models, and a Go HTTP server that wraps the engine for network access.

2. The core subsystem lives under `src/` and is the actual inference engine. It appears to handle model loading, graph construction, tensor management, operator execution, and device-specific execution through pluggable “execution providers.”

3. Model ingestion starts from ONNX assets in `models/` and protobuf definitions in `src/onnx-ml.proto` plus generated header `src/onnx-ml.pb.h`. Helper code in `onnx_helper.*`, `model_config.*`, `graph.*`, `node.*`, and `attribute.*` likely translates ONNX data into an internal computation graph.

4. Runtime execution is centered around `inference_session.*`, which is the boundary between model metadata/loading and actual inference. It likely owns the loaded graph, allocators, provider selection, and the sequence that prepares inputs, runs operators, and produces outputs.

5. Computation is organized around operator implementations in `operators.*` and low-level math kernels in `gemm_cpu.*` and `kernels.cu`. The presence of both CPU and CUDA kernels suggests the engine can execute some paths on the CPU and some on the GPU.

6. Device abstraction is explicit. `execution_provider.h`, `cpu_provider.*`, `cuda_provider.*`, and `cuda_provider_unoptimized.*` indicate a provider pattern where the session dispatches work to a backend. This is a key architectural boundary: graph/operator logic is separated from hardware-specific execution.

7. Memory management is its own subsystem. Files such as `memory_allocator.h`, `cpu_allocator.h`, `pinned_cpu_allocator.h`, `cuda_allocator.h`, and `cuda_memory_pool.h` indicate custom allocators for host, pinned host, and GPU memory, which is important for performance and for data transfer boundaries between CPU and CUDA.

8. Tensor handling is isolated in `tensor.*`, with input loading in `input_loader.*`. Together these likely define the data container API, shape/dtype handling, and the conversion from raw example inputs into engine-ready tensors.

9. The `inputs/` directory contains MNIST-style `.ubyte` sample data, and the repo’s `utils/` scripts (`mnist.py`, `matrix.py`, `infer.py`, `infer_server.py`) are supporting tools for generating inputs, benchmarking, and exercising the engine. These are not core runtime components, but they form the test and demo data flow.

10. The `server/` directory is a separate Go service that exposes the C++ engine over HTTP. Its `main.go` is the server entrypoint, while `inference_wrapper.cpp/.h` bridge Go to the C++ backend, making the server a thin API layer over the native inference runtime.

11. The `server/` boundary is important architecturally: request handling and transport live in Go, but actual model execution stays in C++. This keeps the inference engine reusable outside HTTP and isolates network concerns from performance-sensitive native code.

12. `model_repository/mnist.yaml` suggests the server can load model metadata/configuration from a repository-style manifest, similar to model-serving systems. That makes model selection/configuration a separate concern from the compiled engine binary itself.

13. The repo also includes `src/test/` with unit and benchmark targets (`tensor_test`, `operators_test`, `node_test`, `gemm_test`, CUDA-specific tests, and `gemm_bench`). This indicates the engine is developed as a systems project with explicit validation and performance measurement around core primitives.

14. Build and environment tooling is split between CMake for the native code, Go modules for the server, and a Docker/devcontainer setup for reproducible development. The architecture therefore spans native C++, Go service code, and Python utilities, with the C++ engine as the central execution core.

Main entry points: [`src/inference_session.h`](https://github.com/michalpitr/inference_engine/blob/main/src/inference_session.h) (Session), [`src/operators.h`](https://github.com/michalpitr/inference_engine/blob/main/src/operators.h) (Operators), [`server/main.go`](https://github.com/michalpitr/inference_engine/blob/main/server/main.go) (Server API), [`src/execution_provider.h`](https://github.com/michalpitr/inference_engine/blob/main/src/execution_provider.h) (Providers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_native["Native engine"]
  node_node_main_cpp["Engine CLI<br/>C++ entrypoint<br/>[main.cpp]"]
  node_node_session["Session<br/>runtime core"]
  node_node_modeling["Model parse<br/>onnx ingestion<br/>[onnx_helper.h]"]
  node_node_graph{{"Graph IR<br/>graph model<br/>[graph.h]"}}
  node_node_node_attr["Node attrs<br/>graph metadata<br/>[node.h]"]
  node_node_tensor["Tensor API<br/>data model<br/>[tensor.h]"]
  node_node_io["Input loader<br/>tensor ingest<br/>[input_loader.h]"]
  node_node_ops{{"Operators<br/>ops registry<br/>[operators.h]"}}
  node_node_gemm_cpu["CPU math<br/>kernel math<br/>[gemm_cpu.h]"]
  node_node_cuda["CUDA kernels<br/>gpu kernels<br/>[kernels.h]"]
  node_node_ep{{"Providers<br/>backend API"}}
  node_node_cpu_ep["CPU provider<br/>backend cpu<br/>[cpu_provider.h]"]
  node_node_cuda_ep["CUDA provider<br/>backend gpu<br/>[cuda_provider.h]"]
  node_node_alloc[("Allocators<br/>memory mgmt<br/>[memory_allocator.h]")]
  node_node_tests["Core tests<br/>validation<br/>[CMakeLists.txt]"]
end

subgraph group_group_server["HTTP server"]
  node_node_go_main["Server API<br/>Go entrypoint<br/>[main.go]"]
  node_node_bridge["Native bridge<br/>cgo bridge"]
end

subgraph group_group_assets["Models and data"]
  node_node_manifest["Model repo<br/>server config<br/>[mnist.yaml]"]
  node_node_models["ONNX models<br/>model assets<br/>[mnist_ffn.onnx]"]
  node_node_inputs["Sample inputs<br/>data assets<br/>[image_0.ubyte]"]
end

subgraph group_group_tools["Tooling"]
  node_node_scripts["Python tools<br/>support scripts<br/>[infer.py]"]
  node_node_build["Build setup<br/>build system<br/>[CMakeLists.txt]"]
end

node_node_main_cpp -->|"starts"| node_node_session
node_node_session -->|"loads"| node_node_modeling
node_node_modeling -->|"builds"| node_node_graph
node_node_graph -->|"annotates"| node_node_node_attr
node_node_session -->|"uses"| node_node_tensor
node_node_io -->|"fills"| node_node_tensor
node_node_session -->|"executes"| node_node_ops
node_node_ops -->|"calls"| node_node_gemm_cpu
node_node_ops -->|"calls"| node_node_cuda
node_node_session -->|"dispatches via"| node_node_ep
node_node_ep -->|"implements"| node_node_cpu_ep
node_node_ep -->|"implements"| node_node_cuda_ep
node_node_cpu_ep -->|"allocates"| node_node_alloc
node_node_cuda_ep -->|"allocates"| node_node_alloc
node_node_cuda_ep -->|"launches"| node_node_cuda
node_node_tests -->|"verifies"| node_node_tensor
node_node_tests -->|"verifies"| node_node_ops
node_node_go_main -->|"invokes"| node_node_bridge
node_node_bridge -->|"calls"| node_node_session
node_node_manifest -->|"configures"| node_node_go_main
node_node_models -->|"consumed by"| node_node_modeling
node_node_inputs -->|"feeds"| node_node_io
node_node_scripts -->|"generates"| node_node_inputs
node_node_scripts -->|"drives"| node_node_go_main
node_node_build -->|"builds"| node_node_main_cpp
node_node_build -->|"builds"| node_node_go_main

click node_node_main_cpp "https://github.com/michalpitr/inference_engine/blob/main/src/main.cpp"
click node_node_session "https://github.com/michalpitr/inference_engine/blob/main/src/inference_session.h"
click node_node_modeling "https://github.com/michalpitr/inference_engine/blob/main/src/onnx_helper.h"
click node_node_graph "https://github.com/michalpitr/inference_engine/blob/main/src/graph.h"
click node_node_node_attr "https://github.com/michalpitr/inference_engine/blob/main/src/node.h"
click node_node_tensor "https://github.com/michalpitr/inference_engine/blob/main/src/tensor.h"
click node_node_io "https://github.com/michalpitr/inference_engine/blob/main/src/input_loader.h"
click node_node_ops "https://github.com/michalpitr/inference_engine/blob/main/src/operators.h"
click node_node_gemm_cpu "https://github.com/michalpitr/inference_engine/blob/main/src/gemm_cpu.h"
click node_node_cuda "https://github.com/michalpitr/inference_engine/blob/main/src/kernels.h"
click node_node_ep "https://github.com/michalpitr/inference_engine/blob/main/src/execution_provider.h"
click node_node_cpu_ep "https://github.com/michalpitr/inference_engine/blob/main/src/cpu_provider.h"
click node_node_cuda_ep "https://github.com/michalpitr/inference_engine/blob/main/src/cuda_provider.h"
click node_node_alloc "https://github.com/michalpitr/inference_engine/blob/main/src/memory_allocator.h"
click node_node_tests "https://github.com/michalpitr/inference_engine/blob/main/src/test/CMakeLists.txt"
click node_node_go_main "https://github.com/michalpitr/inference_engine/blob/main/server/main.go"
click node_node_bridge "https://github.com/michalpitr/inference_engine/blob/main/server/inference_wrapper.h"
click node_node_manifest "https://github.com/michalpitr/inference_engine/blob/main/model_repository/mnist.yaml"
click node_node_models "https://github.com/michalpitr/inference_engine/blob/main/models/mnist_ffn.onnx"
click node_node_inputs "https://github.com/michalpitr/inference_engine/blob/main/inputs/image_0.ubyte"
click node_node_scripts "https://github.com/michalpitr/inference_engine/blob/main/utils/infer.py"
click node_node_build "https://github.com/michalpitr/inference_engine/blob/main/CMakeLists.txt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_main_cpp,node_node_session,node_node_modeling,node_node_graph,node_node_node_attr,node_node_tensor,node_node_io,node_node_ops,node_node_gemm_cpu,node_node_cuda,node_node_ep,node_node_cpu_ep,node_node_cuda_ep,node_node_alloc,node_node_tests toneBlue
class node_node_go_main,node_node_bridge toneAmber
class node_node_manifest,node_node_models,node_node_inputs toneMint
class node_node_scripts,node_node_build toneRose
```

## Components

### Native engine

C++ inference runtime, execution providers, tensors, allocators, and tests.

- [Engine CLI](https://github.com/michalpitr/inference_engine/blob/main/src/main.cpp): C++ entrypoint · Native executable entry that wires the inference engine for local runs. · `src/main.cpp`
- [Session](https://github.com/michalpitr/inference_engine/blob/main/src/inference_session.h): runtime core · Owns model loading, provider selection, execution flow, and inference lifecycle. · `src/inference_session.h`
- [Model parse](https://github.com/michalpitr/inference_engine/blob/main/src/onnx_helper.h): onnx ingestion · Parses ONNX metadata and builds the internal graph representation. · `src/onnx_helper.h`
- [Graph IR](https://github.com/michalpitr/inference_engine/blob/main/src/graph.h): graph model · Internal computation graph and node structure used during execution. · `src/graph.h`
- [Node attrs](https://github.com/michalpitr/inference_engine/blob/main/src/node.h): graph metadata · Node and attribute definitions that enrich the graph with operator parameters. · `src/node.h`
- [Tensor API](https://github.com/michalpitr/inference_engine/blob/main/src/tensor.h): data model · Tensor shapes, dtypes, and runtime data containers. · `src/tensor.h`
- [Input loader](https://github.com/michalpitr/inference_engine/blob/main/src/input_loader.h): tensor ingest · Loads external example data into engine-ready tensors. · `src/input_loader.h`
- [Operators](https://github.com/michalpitr/inference_engine/blob/main/src/operators.h): ops registry · Operator implementations that map graph nodes to executable work. · `src/operators.h`
- [CPU math](https://github.com/michalpitr/inference_engine/blob/main/src/gemm_cpu.h): kernel math · CPU-side matrix multiplication and low-level numeric kernels. · `src/gemm_cpu.h`
- [CUDA kernels](https://github.com/michalpitr/inference_engine/blob/main/src/kernels.h): gpu kernels · GPU kernels and CUDA execution paths for accelerated operators. · `src/kernels.h`
- [Providers](https://github.com/michalpitr/inference_engine/blob/main/src/execution_provider.h): backend API · Execution-provider interface used to dispatch work to CPU or CUDA backends. · `src/execution_provider.h`
- [CPU provider](https://github.com/michalpitr/inference_engine/blob/main/src/cpu_provider.h): backend cpu · Host execution provider and CPU-specific runtime integration. · `src/cpu_provider.h`
- [CUDA provider](https://github.com/michalpitr/inference_engine/blob/main/src/cuda_provider.h): backend gpu · GPU execution provider and optimized CUDA dispatch implementation. · `src/cuda_provider.h`
- [Allocators](https://github.com/michalpitr/inference_engine/blob/main/src/memory_allocator.h): memory mgmt · Memory allocation interfaces for host, pinned host, and GPU memory. · `src/memory_allocator.h`
- [Core tests](https://github.com/michalpitr/inference_engine/blob/main/src/test/CMakeLists.txt): validation · Unit and benchmark coverage for tensors, nodes, operators, and GEMM paths. · `src/test/CMakeLists.txt`

### HTTP server

Go wrapper that exposes the native engine over the network.

- [Server API](https://github.com/michalpitr/inference_engine/blob/main/server/main.go): Go entrypoint · HTTP service entrypoint that exposes inference requests to clients. · `server/main.go`
- [Native bridge](https://github.com/michalpitr/inference_engine/blob/main/server/inference_wrapper.h): cgo bridge · Go/C++ interop layer that forwards server requests into the native engine. · `server/inference_wrapper.h`

### Models and data

ONNX models, manifests, and sample inputs used by the engine and demos.

- [Model repo](https://github.com/michalpitr/inference_engine/blob/main/model_repository/mnist.yaml): server config · Model-serving manifest that points the server at a deployable model configuration. · `model_repository/mnist.yaml`
- [ONNX models](https://github.com/michalpitr/inference_engine/blob/main/models/mnist_ffn.onnx): model assets · Packaged ONNX model files consumed by the native inference engine. · `models/mnist_ffn.onnx`
- [Sample inputs](https://github.com/michalpitr/inference_engine/blob/main/inputs/image_0.ubyte): data assets · MNIST-style example input data used for demos and local inference runs. · `inputs/image_0.ubyte`

### Tooling

Support scripts and build/development setup.

- [Python tools](https://github.com/michalpitr/inference_engine/blob/main/utils/infer.py): support scripts · Utility scripts for generating inputs, benchmarking, and driving inference. · `utils/infer.py`
- [Build setup](https://github.com/michalpitr/inference_engine/blob/main/CMakeLists.txt): build system · Native and containerized build/development configuration. · `CMakeLists.txt`

## Connections

- Engine CLI → Session: starts · Native executable initializes the runtime session.
- Session → Model parse: loads · Session ingests ONNX model metadata and constructs runtime state.
- Model parse → Graph IR: builds · ONNX structures are translated into the internal graph IR.
- Graph IR → Node attrs: annotates · Graph nodes carry operator and attribute metadata.
- Session → Tensor API: uses · Inference session manages inputs, outputs, and intermediate tensor state.
- Input loader → Tensor API: fills · External input bytes are converted into tensors.
- Session → Operators: executes · Session dispatches graph work to operator implementations.
- Operators → CPU math: calls · Operators use CPU kernels for numeric work when scheduled on host.
- Operators → CUDA kernels: calls · Operators use CUDA kernels for GPU-accelerated work.
- Session → Providers: dispatches via · Execution provider abstraction selects the backend for a node or subgraph.
- Providers → CPU provider: implements · CPU provider realizes the execution-provider interface.
- Providers → CUDA provider: implements · CUDA provider realizes the execution-provider interface.
- CPU provider → Allocators: allocates · CPU execution relies on host and pinned-host memory management.
- CUDA provider → Allocators: allocates · GPU execution relies on CUDA allocations and pools.
- CUDA provider → CUDA kernels: launches · CUDA provider schedules GPU kernels for execution.
- Core tests → Tensor API: verifies · Tests cover tensor behavior and memory semantics.
- Core tests → Operators: verifies · Tests validate operator correctness and GEMM paths.
- Server API → Native bridge: invokes · HTTP service forwards requests through the native bridge.
- Native bridge → Session: calls · Go wrapper delegates inference to the C++ session.
- Model repo → Server API: configures · Model repository manifest tells the server what to serve.
- ONNX models → Model parse: consumed by · Packaged ONNX models are parsed by the native model loader.
- Sample inputs → Input loader: feeds · Sample input files are ingested by the input loader.
- Python tools → Sample inputs: generates · Support scripts create or transform example input data.
- Python tools → Server API: drives · Utility scripts exercise the server for demos and benchmarking.
- Build setup → Engine CLI: builds · Build tooling compiles the native engine.
- Build setup → Server API: builds · Build tooling also supports the Go server.
