coderoj/inference-benchmarker

How coderoj/inference-benchmarker works

Inference Benchmarker is a CLI tool for load-testing OpenAI-compatible text-generation servers. The main workflow configures a run, selects a profile and workload, generates and sends requests under a chosen load pattern, aggregates latency/throughput results, and writes JSON for later visualization. The README also documents Kubernetes and Slurm deployment. Control-flow details come from sampled Rust excerpts; deployment and some library wiring are represented only where the README or tree supports them.

Main entry points: lib.rs Run orchestration · scheduler.rs Load scheduler · executors.rs Load executors · requests.rs Streaming API client

Components 21

Run Setup

CLI configuration src/main.rs
Run orchestration src/lib.rs
Workload profiles src/profiles.rs

Workload Generation

Dataset and prompts src/requests.rs
Tokenizer src/lib.rs
Request generation src/requests.rs

Benchmark Execution

Benchmark modes src/benchmark.rs
Load scheduler src/scheduler.rs
Load executors src/executors.rs
Streaming API client src/requests.rs

Results and Review

Metric aggregation src/results.rs
JSON results writer src/writers.rs
JSON result files
Interactive console src/app.rs
Results conversion extra/dashboard/parse_results.py
Results dashboard extra/dashboard/app.py

Deployment

Kubernetes chart extra/k8s/inference-benchmarker
Slurm example extra/slurm/benchmark.py

Other components

Benchmark operator
Hugging Face dataset
Inference server

Connections 25

Benchmark operator → CLI configuration: configures
CLI configuration → Run orchestration: starts run
Run orchestration → Workload profiles: applies profile
Run orchestration → Tokenizer: loads tokenizer
Run orchestration → Dataset and prompts: loads workload
Hugging Face dataset → Dataset and prompts: provides dataset
Dataset and prompts → Request generation: supplies prompts
Tokenizer → Request generation: tokenizes prompts
Run orchestration → Benchmark modes: configures benchmark
Benchmark modes → Load scheduler: schedules runs
Load scheduler → Load executors: selects load
Load executors → Request generation: requests workload
Load executors → Streaming API client: dispatches requests
Streaming API client → Inference server: streams API requests
Streaming API client → Load scheduler: returns responses
Load scheduler → Metric aggregation: records responses
Run orchestration → JSON results writer: writes report
Metric aggregation → JSON results writer: provides metrics
JSON results writer → JSON result files: serializes JSON
Run orchestration → Interactive console: updates console
Benchmark operator → Results dashboard: views results
JSON result files → Results conversion: loads JSON
Results conversion → Results dashboard: provides tabular data
Benchmark operator → Kubernetes chart: deploys benchmarker
Benchmark operator → Slurm example: runs example

More Rust diagrams: code-payments/code-vm · jkitchin/pounce · aeroxy/drift · input-output-hk/partner-chains · jnietzsche/wemod-pro-unlocker

Drawn by GitDiagram on September 28, 2026 from the repository's file tree, README and source files. Read as Markdown ·