How yawpremuh/ml-inference-optimization works
This repository is an ML inference serving and optimization project: users submit images to a FastAPI endpoint, which preprocesses each image, runs pretrained ResNet18 inference, and returns top predictions. A parallel evaluation workflow compares PyTorch and ONNX Runtime across batch sizes and hardware, measuring latency, throughput, and memory; Locust exercises the API under concurrent load. The benchmark and load-test implementations are represented at their tree paths, but their source was not sampled, so those connections follow README descriptions rather than inspected call details.
Main entry points: models ONNX Model · model.py ResNet18 PyTorch · main.py FastAPI Routes · benchmark.py Runtime Benchmarks
Components 20
Image Serving
- FastAPI Routes
app/main.py - JSON Response
app/main.py
Inference Models
- Image Preprocessing
app/model.py - ResNet18 PyTorch
app/model.py - Top Predictions
app/model.py - CLI Inference
predict_data.py
Runtime Evaluation
- ONNX Export
scripts/export_to_onnx.py - ONNX Model
models - ONNX Validation
scripts/verify_onnx.py - Batch Validation
scripts/verify_batch.py - ONNX Shape Inspection
scripts/onnx_shape.py - Runtime Benchmarks
benchmarks/benchmark.py - Hardware Benchmarks
benchmarks/benchmark_hardware.py - Benchmark Results
benchmarks/results
Load Testing
- Locust Requests
load_tests/locustfile.py - Load Results
load_tests/results - Load Summary
load_tests/summarize_load.py - Load Plots
load_tests/plot_load.py
Other components
- Image Client
- Concurrent Users
Connections 21
- Image Client → FastAPI Routes: uploads image
- FastAPI Routes → Image Preprocessing: calls inference
- Image Preprocessing → ResNet18 PyTorch: runs model
- ResNet18 PyTorch → Top Predictions: produces scores
- FastAPI Routes → JSON Response: returns predictions
- CLI Inference → ResNet18 PyTorch: runs inference
- ONNX Export → ONNX Model: exports model
- ONNX Validation → ONNX Model: loads model
- Batch Validation → ONNX Model: loads model
- ONNX Shape Inspection → ONNX Model: inspects model
- Runtime Benchmarks → ResNet18 PyTorch: benchmarks runtime
- Runtime Benchmarks → ONNX Model: benchmarks runtime
- Hardware Benchmarks → ResNet18 PyTorch: tests hardware
- Hardware Benchmarks → ONNX Model: tests hardware
- Runtime Benchmarks → Benchmark Results: records metrics
- Hardware Benchmarks → Benchmark Results: records metrics
- Concurrent Users → Locust Requests: starts requests
- Locust Requests → FastAPI Routes: posts images
- Load Summary → Load Results: reads statistics
- Load Summary → Load Results: writes summary
- Load Plots → Load Results: reads summary