How holyapelsin/cpu-ai-inference-benchmark works
This WIP project compares CPU MobileNetV2 inference in PyTorch and ONNX Runtime, with ONNX export and dynamic INT8 quantization. The user-facing workflow is to export a model, optionally quantize it, run benchmarks, and record latency and throughput metrics. Benchmark scripts also support direct PyTorch inference. The source excerpts establish these calls and artifact paths; the README describes the comparison and quantization, but reports no results yet.
Main entry points: bench_onnx.py ONNX benchmark · bench_pytorch.py PyTorch benchmark · export_onnx.py ONNX export · quant_dynamic.py Dynamic INT8
Components 8
Model preparation
- ONNX export
src/export_onnx.py - Dynamic INT8
src/quant_dynamic.py - ONNX models
CPU benchmarks
- PyTorch benchmark
src/bench_pytorch.py - ONNX benchmark
src/bench_onnx.py
Shared utilities
- Input and metrics helpers
src/utils.py
Other components
- CLI user
- Metrics CSV
Connections 11
- CLI user → ONNX export: runs
- CLI user → Dynamic INT8: runs
- CLI user → PyTorch benchmark: runs
- CLI user → ONNX benchmark: runs
- ONNX export → ONNX models: writes FP32
src/export_onnx.py - Dynamic INT8 → ONNX models: reads and writes INT8
src/quant_dynamic.py - ONNX benchmark → ONNX models: loads model
src/bench_onnx.py - PyTorch benchmark → Input and metrics helpers: uses helpers
src/bench_pytorch.py - ONNX benchmark → Input and metrics helpers: uses helpers
src/bench_onnx.py - PyTorch benchmark → Metrics CSV: appends metrics
src/bench_pytorch.py - ONNX benchmark → Metrics CSV: appends metrics
src/bench_onnx.py