# holyapelsin/cpu-ai-inference-benchmark architecture

> Architecture diagram of the GitHub repository holyapelsin/cpu-ai-inference-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/holyapelsin/cpu-ai-inference-benchmark
- Repository: https://github.com/holyapelsin/cpu-ai-inference-benchmark

## Overview

This WIP project compares CPU MobileNetV2 inference in PyTorch and ONNX Runtime, with ONNX export and dynamic INT8 quantization. The user-facing workflow is to export a model, optionally quantize it, run benchmarks, and record latency and throughput metrics. Benchmark scripts also support direct PyTorch inference. The source excerpts establish these calls and artifact paths; the README describes the comparison and quantization, but reports no results yet.

Main entry points: [`src/bench_onnx.py`](https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/bench_onnx.py) (ONNX benchmark), [`src/bench_pytorch.py`](https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/bench_pytorch.py) (PyTorch benchmark), [`src/export_onnx.py`](https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/export_onnx.py) (ONNX export), [`src/quant_dynamic.py`](https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/quant_dynamic.py) (Dynamic INT8)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_model_prep["Model preparation"]
  node_export["ONNX export<br/>[export_onnx.py]"]
  node_quantize["Dynamic INT8<br/>[quant_dynamic.py]"]
  node_onnx_model["ONNX models"]
end

subgraph group_benchmarks["CPU benchmarks"]
  node_pytorch_bench["PyTorch benchmark<br/>[bench_pytorch.py]"]
  node_onnx_bench["ONNX benchmark<br/>[bench_onnx.py]"]
end

subgraph group_shared["Shared utilities"]
  node_utils["Input and metrics helpers<br/>[utils.py]"]
end

node_user(("CLI user"))
node_csv["Metrics CSV"]

node_user -->|"runs"| node_export
node_user -->|"runs"| node_quantize
node_user -->|"runs"| node_pytorch_bench
node_user -->|"runs"| node_onnx_bench
node_export -->|"writes FP32"| node_onnx_model
node_quantize -->|"reads and writes INT8"| node_onnx_model
node_onnx_bench -->|"loads model"| node_onnx_model
node_pytorch_bench -->|"uses helpers"| node_utils
node_onnx_bench -->|"uses helpers"| node_utils
node_pytorch_bench -->|"appends metrics"| node_csv
node_onnx_bench -->|"appends metrics"| node_csv

click node_export "https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/export_onnx.py"
click node_quantize "https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/quant_dynamic.py"
click node_pytorch_bench "https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/bench_pytorch.py"
click node_onnx_bench "https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/bench_onnx.py"
click node_utils "https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_export,node_quantize,node_onnx_model,node_user toneBlue
class node_pytorch_bench,node_onnx_bench toneAmber
class node_utils toneMint
class node_csv toneIndigo
```

## Components

### Model preparation

- [ONNX export](https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/export_onnx.py): `src/export_onnx.py`
- [Dynamic INT8](https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/quant_dynamic.py): `src/quant_dynamic.py`
- **ONNX models**

### CPU benchmarks

- [PyTorch benchmark](https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/bench_pytorch.py): `src/bench_pytorch.py`
- [ONNX benchmark](https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/bench_onnx.py): `src/bench_onnx.py`

### Shared utilities

- [Input and metrics helpers](https://github.com/holyapelsin/cpu-ai-inference-benchmark/blob/main/src/utils.py): `src/utils.py`

### Other components

- **CLI user**
- **Metrics CSV**

## Connections

- CLI user → ONNX export: runs
- CLI user → Dynamic INT8: runs
- CLI user → PyTorch benchmark: runs
- CLI user → ONNX benchmark: runs
- ONNX export → ONNX models: writes FP32 (evidence: `src/export_onnx.py`)
- Dynamic INT8 → ONNX models: reads and writes INT8 (evidence: `src/quant_dynamic.py`)
- ONNX benchmark → ONNX models: loads model (evidence: `src/bench_onnx.py`)
- PyTorch benchmark → Input and metrics helpers: uses helpers (evidence: `src/bench_pytorch.py`)
- ONNX benchmark → Input and metrics helpers: uses helpers (evidence: `src/bench_onnx.py`)
- PyTorch benchmark → Metrics CSV: appends metrics (evidence: `src/bench_pytorch.py`)
- ONNX benchmark → Metrics CSV: appends metrics (evidence: `src/bench_onnx.py`)
