# yawpremuh/ml-inference-optimization architecture

> Architecture diagram of the GitHub repository yawpremuh/ml-inference-optimization, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yawpremuh/ml-inference-optimization
- Repository: https://github.com/yawpremuh/ml-inference-optimization

## Overview

This repository is an ML inference serving and optimization project: users submit images to a FastAPI endpoint, which preprocesses each image, runs pretrained ResNet18 inference, and returns top predictions. A parallel evaluation workflow compares PyTorch and ONNX Runtime across batch sizes and hardware, measuring latency, throughput, and memory; Locust exercises the API under concurrent load. The benchmark and load-test implementations are represented at their tree paths, but their source was not sampled, so those connections follow README descriptions rather than inspected call details.

Main entry points: [`models`](https://github.com/yawpremuh/ml-inference-optimization/tree/main/models) (ONNX Model), [`app/model.py`](https://github.com/yawpremuh/ml-inference-optimization/blob/main/app/model.py) (ResNet18 PyTorch), [`app/main.py`](https://github.com/yawpremuh/ml-inference-optimization/blob/main/app/main.py) (FastAPI Routes), [`benchmarks/benchmark.py`](https://github.com/yawpremuh/ml-inference-optimization/blob/main/benchmarks/benchmark.py) (Runtime Benchmarks)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Image Serving"]
  node_api["FastAPI Routes<br/>[main.py]"]
  node_json_response["JSON Response<br/>[main.py]"]
end

subgraph group_inference["Inference Models"]
  node_preprocess["Image Preprocessing<br/>[model.py]"]
  node_pytorch_model["ResNet18 PyTorch<br/>[model.py]"]
  node_predictions["Top Predictions<br/>[model.py]"]
  node_cli_inference["CLI Inference<br/>[predict_data.py]"]
end

subgraph group_evaluation["Runtime Evaluation"]
  node_onnx_export["ONNX Export<br/>[export_to_onnx.py]"]
  node_onnx_artifact["ONNX Model"]
  node_onnx_validation["ONNX Validation<br/>[verify_onnx.py]"]
  node_batch_validation["Batch Validation<br/>[verify_batch.py]"]
  node_onnx_shape["ONNX Shape Inspection<br/>[onnx_shape.py]"]
  node_benchmark["Runtime Benchmarks<br/>[benchmark.py]"]
  node_hardware_benchmark["Hardware Benchmarks"]
  node_benchmark_results["Benchmark Results"]
end

subgraph group_load["Load Testing"]
  node_locust["Locust Requests<br/>[locustfile.py]"]
  node_load_results["Load Results"]
  node_load_summary["Load Summary<br/>[summarize_load.py]"]
  node_load_plots["Load Plots<br/>[plot_load.py]"]
end

node_image_user(("Image Client"))
node_load_user(("Concurrent Users"))

node_image_user -->|"uploads image"| node_api
node_api -->|"calls inference"| node_preprocess
node_preprocess -->|"runs model"| node_pytorch_model
node_pytorch_model -->|"produces scores"| node_predictions
node_api -->|"returns predictions"| node_json_response
node_cli_inference -->|"runs inference"| node_pytorch_model
node_onnx_export -->|"exports model"| node_onnx_artifact
node_onnx_validation -->|"loads model"| node_onnx_artifact
node_batch_validation -->|"loads model"| node_onnx_artifact
node_onnx_shape -->|"inspects model"| node_onnx_artifact
node_benchmark -->|"benchmarks runtime"| node_pytorch_model
node_benchmark -->|"benchmarks runtime"| node_onnx_artifact
node_hardware_benchmark -->|"tests hardware"| node_pytorch_model
node_hardware_benchmark -->|"tests hardware"| node_onnx_artifact
node_benchmark -->|"records metrics"| node_benchmark_results
node_hardware_benchmark -->|"records metrics"| node_benchmark_results
node_load_user -->|"starts requests"| node_locust
node_locust -->|"posts images"| node_api
node_load_summary -->|"reads statistics"| node_load_results
node_load_summary -->|"writes summary"| node_load_results
node_load_plots -->|"reads summary"| node_load_results

click node_api "https://github.com/yawpremuh/ml-inference-optimization/blob/main/app/main.py"
click node_preprocess "https://github.com/yawpremuh/ml-inference-optimization/blob/main/app/model.py"
click node_pytorch_model "https://github.com/yawpremuh/ml-inference-optimization/blob/main/app/model.py"
click node_predictions "https://github.com/yawpremuh/ml-inference-optimization/blob/main/app/model.py"
click node_json_response "https://github.com/yawpremuh/ml-inference-optimization/blob/main/app/main.py"
click node_cli_inference "https://github.com/yawpremuh/ml-inference-optimization/blob/main/predict_data.py"
click node_onnx_export "https://github.com/yawpremuh/ml-inference-optimization/blob/main/scripts/export_to_onnx.py"
click node_onnx_artifact "https://github.com/yawpremuh/ml-inference-optimization/tree/main/models"
click node_onnx_validation "https://github.com/yawpremuh/ml-inference-optimization/blob/main/scripts/verify_onnx.py"
click node_batch_validation "https://github.com/yawpremuh/ml-inference-optimization/blob/main/scripts/verify_batch.py"
click node_onnx_shape "https://github.com/yawpremuh/ml-inference-optimization/blob/main/scripts/onnx_shape.py"
click node_benchmark "https://github.com/yawpremuh/ml-inference-optimization/blob/main/benchmarks/benchmark.py"
click node_hardware_benchmark "https://github.com/yawpremuh/ml-inference-optimization/blob/main/benchmarks/benchmark_hardware.py"
click node_benchmark_results "https://github.com/yawpremuh/ml-inference-optimization/tree/main/benchmarks/results"
click node_locust "https://github.com/yawpremuh/ml-inference-optimization/blob/main/load_tests/locustfile.py"
click node_load_results "https://github.com/yawpremuh/ml-inference-optimization/tree/main/load_tests/results"
click node_load_summary "https://github.com/yawpremuh/ml-inference-optimization/blob/main/load_tests/summarize_load.py"
click node_load_plots "https://github.com/yawpremuh/ml-inference-optimization/blob/main/load_tests/plot_load.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_json_response,node_image_user,node_load_user toneBlue
class node_preprocess,node_pytorch_model,node_predictions,node_cli_inference toneAmber
class node_onnx_export,node_onnx_artifact,node_onnx_validation,node_batch_validation,node_onnx_shape,node_benchmark,node_hardware_benchmark,node_benchmark_results toneMint
class node_locust,node_load_results,node_load_summary,node_load_plots toneRose
```

## Components

### Image Serving

- [FastAPI Routes](https://github.com/yawpremuh/ml-inference-optimization/blob/main/app/main.py): `app/main.py`
- [JSON Response](https://github.com/yawpremuh/ml-inference-optimization/blob/main/app/main.py): `app/main.py`

### Inference Models

- [Image Preprocessing](https://github.com/yawpremuh/ml-inference-optimization/blob/main/app/model.py): `app/model.py`
- [ResNet18 PyTorch](https://github.com/yawpremuh/ml-inference-optimization/blob/main/app/model.py): `app/model.py`
- [Top Predictions](https://github.com/yawpremuh/ml-inference-optimization/blob/main/app/model.py): `app/model.py`
- [CLI Inference](https://github.com/yawpremuh/ml-inference-optimization/blob/main/predict_data.py): `predict_data.py`

### Runtime Evaluation

- [ONNX Export](https://github.com/yawpremuh/ml-inference-optimization/blob/main/scripts/export_to_onnx.py): `scripts/export_to_onnx.py`
- [ONNX Model](https://github.com/yawpremuh/ml-inference-optimization/tree/main/models): `models`
- [ONNX Validation](https://github.com/yawpremuh/ml-inference-optimization/blob/main/scripts/verify_onnx.py): `scripts/verify_onnx.py`
- [Batch Validation](https://github.com/yawpremuh/ml-inference-optimization/blob/main/scripts/verify_batch.py): `scripts/verify_batch.py`
- [ONNX Shape Inspection](https://github.com/yawpremuh/ml-inference-optimization/blob/main/scripts/onnx_shape.py): `scripts/onnx_shape.py`
- [Runtime Benchmarks](https://github.com/yawpremuh/ml-inference-optimization/blob/main/benchmarks/benchmark.py): `benchmarks/benchmark.py`
- [Hardware Benchmarks](https://github.com/yawpremuh/ml-inference-optimization/blob/main/benchmarks/benchmark_hardware.py): `benchmarks/benchmark_hardware.py`
- [Benchmark Results](https://github.com/yawpremuh/ml-inference-optimization/tree/main/benchmarks/results): `benchmarks/results`

### Load Testing

- [Locust Requests](https://github.com/yawpremuh/ml-inference-optimization/blob/main/load_tests/locustfile.py): `load_tests/locustfile.py`
- [Load Results](https://github.com/yawpremuh/ml-inference-optimization/tree/main/load_tests/results): `load_tests/results`
- [Load Summary](https://github.com/yawpremuh/ml-inference-optimization/blob/main/load_tests/summarize_load.py): `load_tests/summarize_load.py`
- [Load Plots](https://github.com/yawpremuh/ml-inference-optimization/blob/main/load_tests/plot_load.py): `load_tests/plot_load.py`

### Other components

- **Image Client**
- **Concurrent Users**

## Connections

- Image Client → FastAPI Routes: uploads image
- FastAPI Routes → Image Preprocessing: calls inference
- Image Preprocessing → ResNet18 PyTorch: runs model
- ResNet18 PyTorch → Top Predictions: produces scores
- FastAPI Routes → JSON Response: returns predictions
- CLI Inference → ResNet18 PyTorch: runs inference
- ONNX Export → ONNX Model: exports model
- ONNX Validation → ONNX Model: loads model
- Batch Validation → ONNX Model: loads model
- ONNX Shape Inspection → ONNX Model: inspects model
- Runtime Benchmarks → ResNet18 PyTorch: benchmarks runtime
- Runtime Benchmarks → ONNX Model: benchmarks runtime
- Hardware Benchmarks → ResNet18 PyTorch: tests hardware
- Hardware Benchmarks → ONNX Model: tests hardware
- Runtime Benchmarks → Benchmark Results: records metrics
- Hardware Benchmarks → Benchmark Results: records metrics
- Concurrent Users → Locust Requests: starts requests
- Locust Requests → FastAPI Routes: posts images
- Load Summary → Load Results: reads statistics
- Load Summary → Load Results: writes summary
- Load Plots → Load Results: reads summary