How michalpitr/inference_engine works
1. This repository is a small ONNX-based inference platform split into two major runtimes: a C++ inference engine that executes models, and a Go HTTP server that wraps the engine for network access.
2. The core subsystem lives under src/ and is the actual inference engine. It appears to handle model loading, graph construction, tensor management, operator execution, and device-specific execution through pluggable “execution providers.”
3. Model ingestion starts from ONNX assets in models/ and protobuf definitions in src/onnx-ml.proto plus generated header src/onnx-ml.pb.h. Helper code in onnx_helper.*, model_config.*, graph.*, node.*, and attribute.* likely translates ONNX data into an internal computation graph.
4. Runtime execution is centered around inference_session.*, which is the boundary between model metadata/loading and actual inference. It likely owns the loaded graph, allocators, provider selection, and the sequence that prepares inputs, runs operators, and produces outputs.
5. Computation is organized around operator implementations in operators.* and low-level math kernels in gemm_cpu.* and kernels.cu. The presence of both CPU and CUDA kernels suggests the engine can execute some paths on the CPU and some on the GPU.
6. Device abstraction is explicit. execution_provider.h, cpu_provider.*, cuda_provider.*, and cuda_provider_unoptimized.* indicate a provider pattern where the session dispatches work to a backend. This is a key architectural boundary: graph/operator logic is separated from hardware-specific execution.
7. Memory management is its own subsystem. Files such as memory_allocator.h, cpu_allocator.h, pinned_cpu_allocator.h, cuda_allocator.h, and cuda_memory_pool.h indicate custom allocators for host, pinned host, and GPU memory, which is important for performance and for data transfer boundaries between CPU and CUDA.
8. Tensor handling is isolated in tensor.*, with input loading in input_loader.*. Together these likely define the data container API, shape/dtype handling, and the conversion from raw example inputs into engine-ready tensors.
9. The inputs/ directory contains MNIST-style .ubyte sample data, and the repo’s utils/ scripts (mnist.py, matrix.py, infer.py, infer_server.py) are supporting tools for generating inputs, benchmarking, and exercising the engine. These are not core runtime components, but they form the test and demo data flow.
10. The server/ directory is a separate Go service that exposes the C++ engine over HTTP. Its main.go is the server entrypoint, while inference_wrapper.cpp/.h bridge Go to the C++ backend, making the server a thin API layer over the native inference runtime.
11. The server/ boundary is important architecturally: request handling and transport live in Go, but actual model execution stays in C++. This keeps the inference engine reusable outside HTTP and isolates network concerns from performance-sensitive native code.
12. model_repository/mnist.yaml suggests the server can load model metadata/configuration from a repository-style manifest, similar to model-serving systems. That makes model selection/configuration a separate concern from the compiled engine binary itself.
13. The repo also includes src/test/ with unit and benchmark targets (tensor_test, operators_test, node_test, gemm_test, CUDA-specific tests, and gemm_bench). This indicates the engine is developed as a systems project with explicit validation and performance measurement around core primitives.
14. Build and environment tooling is split between CMake for the native code, Go modules for the server, and a Docker/devcontainer setup for reproducible development. The architecture therefore spans native C++, Go service code, and Python utilities, with the C++ engine as the central execution core.
Main entry points: inference_session.h Session · operators.h Operators · main.go Server API · execution_provider.h Providers
Components 22
Native engine
C++ inference runtime, execution providers, tensors, allocators, and tests.
- Engine CLI · Native executable entry that wires the inference engine for local runs.
src/main.cpp - Session · Owns model loading, provider selection, execution flow, and inference lifecycle.
src/inference_session.h - Model parse · Parses ONNX metadata and builds the internal graph representation.
src/onnx_helper.h - Graph IR · Internal computation graph and node structure used during execution.
src/graph.h - Node attrs · Node and attribute definitions that enrich the graph with operator parameters.
src/node.h - Tensor API · Tensor shapes, dtypes, and runtime data containers.
src/tensor.h - Input loader · Loads external example data into engine-ready tensors.
src/input_loader.h - Operators · Operator implementations that map graph nodes to executable work.
src/operators.h - CPU math · CPU-side matrix multiplication and low-level numeric kernels.
src/gemm_cpu.h - CUDA kernels · GPU kernels and CUDA execution paths for accelerated operators.
src/kernels.h - Providers · Execution-provider interface used to dispatch work to CPU or CUDA backends.
src/execution_provider.h - CPU provider · Host execution provider and CPU-specific runtime integration.
src/cpu_provider.h - CUDA provider · GPU execution provider and optimized CUDA dispatch implementation.
src/cuda_provider.h - Allocators · Memory allocation interfaces for host, pinned host, and GPU memory.
src/memory_allocator.h - Core tests · Unit and benchmark coverage for tensors, nodes, operators, and GEMM paths.
src/test/CMakeLists.txt
HTTP server
Go wrapper that exposes the native engine over the network.
- Server API · HTTP service entrypoint that exposes inference requests to clients.
server/main.go - Native bridge · Go/C++ interop layer that forwards server requests into the native engine.
server/inference_wrapper.h
Models and data
ONNX models, manifests, and sample inputs used by the engine and demos.
- Model repo · Model-serving manifest that points the server at a deployable model configuration.
model_repository/mnist.yaml - ONNX models · Packaged ONNX model files consumed by the native inference engine.
models/mnist_ffn.onnx - Sample inputs · MNIST-style example input data used for demos and local inference runs.
inputs/image_0.ubyte
Tooling
Support scripts and build/development setup.
- Python tools · Utility scripts for generating inputs, benchmarking, and driving inference.
utils/infer.py - Build setup · Native and containerized build/development configuration.
CMakeLists.txt
Connections 26
- Engine CLI → Session: starts · Native executable initializes the runtime session.
- Session → Model parse: loads · Session ingests ONNX model metadata and constructs runtime state.
- Model parse → Graph IR: builds · ONNX structures are translated into the internal graph IR.
- Graph IR → Node attrs: annotates · Graph nodes carry operator and attribute metadata.
- Session → Tensor API: uses · Inference session manages inputs, outputs, and intermediate tensor state.
- Input loader → Tensor API: fills · External input bytes are converted into tensors.
- Session → Operators: executes · Session dispatches graph work to operator implementations.
- Operators → CPU math: calls · Operators use CPU kernels for numeric work when scheduled on host.
- Operators → CUDA kernels: calls · Operators use CUDA kernels for GPU-accelerated work.
- Session → Providers: dispatches via · Execution provider abstraction selects the backend for a node or subgraph.
- Providers → CPU provider: implements · CPU provider realizes the execution-provider interface.
- Providers → CUDA provider: implements · CUDA provider realizes the execution-provider interface.
- CPU provider → Allocators: allocates · CPU execution relies on host and pinned-host memory management.
- CUDA provider → Allocators: allocates · GPU execution relies on CUDA allocations and pools.
- CUDA provider → CUDA kernels: launches · CUDA provider schedules GPU kernels for execution.
- Core tests → Tensor API: verifies · Tests cover tensor behavior and memory semantics.
- Core tests → Operators: verifies · Tests validate operator correctness and GEMM paths.
- Server API → Native bridge: invokes · HTTP service forwards requests through the native bridge.
- Native bridge → Session: calls · Go wrapper delegates inference to the C++ session.
- Model repo → Server API: configures · Model repository manifest tells the server what to serve.
- ONNX models → Model parse: consumed by · Packaged ONNX models are parsed by the native model loader.
- Sample inputs → Input loader: feeds · Sample input files are ingested by the input loader.
- Python tools → Sample inputs: generates · Support scripts create or transform example input data.
- Python tools → Server API: drives · Utility scripts exercise the server for demos and benchmarking.
- Build setup → Engine CLI: builds · Build tooling compiles the native engine.
- Build setup → Server API: builds · Build tooling also supports the Go server.