# coderoj/inference-benchmarker architecture

> Architecture diagram of the GitHub repository coderoj/inference-benchmarker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/coderoj/inference-benchmarker
- Repository: https://github.com/coderoj/inference-benchmarker

## Overview

Inference Benchmarker is a CLI tool for load-testing OpenAI-compatible text-generation servers. The main workflow configures a run, selects a profile and workload, generates and sends requests under a chosen load pattern, aggregates latency/throughput results, and writes JSON for later visualization. The README also documents Kubernetes and Slurm deployment. Control-flow details come from sampled Rust excerpts; deployment and some library wiring are represented only where the README or tree supports them.

Main entry points: [`src/lib.rs`](https://github.com/coderoj/inference-benchmarker/blob/main/src/lib.rs) (Run orchestration), [`src/scheduler.rs`](https://github.com/coderoj/inference-benchmarker/blob/main/src/scheduler.rs) (Load scheduler), [`src/executors.rs`](https://github.com/coderoj/inference-benchmarker/blob/main/src/executors.rs) (Load executors), [`src/requests.rs`](https://github.com/coderoj/inference-benchmarker/blob/main/src/requests.rs) (Streaming API client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_setup["Run Setup"]
  node_cli["CLI configuration<br/>[main.rs]"]
  node_run["Run orchestration<br/>[lib.rs]"]
  node_profiles["Workload profiles<br/>[profiles.rs]"]
end

subgraph group_workload["Workload Generation"]
  node_dataset["Dataset and prompts<br/>[requests.rs]"]
  node_tokenizer["Tokenizer<br/>[lib.rs]"]
  node_requestgen["Request generation<br/>[requests.rs]"]
end

subgraph group_execution["Benchmark Execution"]
  node_benchmark["Benchmark modes<br/>[benchmark.rs]"]
  node_scheduler["Load scheduler<br/>[scheduler.rs]"]
  node_executors["Load executors<br/>[executors.rs]"]
  node_backend["Streaming API client<br/>[requests.rs]"]
end

subgraph group_results["Results and Review"]
  node_results["Metric aggregation<br/>[results.rs]"]
  node_writer["JSON results writer<br/>[writers.rs]"]
  node_resultfiles["JSON result files"]
  node_console["Interactive console<br/>[app.rs]"]
  node_dashboardparse["Results conversion<br/>[parse_results.py]"]
  node_dashboard["Results dashboard<br/>[app.py]"]
end

subgraph group_deployment["Deployment"]
  node_helm["Kubernetes chart"]
  node_slurm["Slurm example<br/>[benchmark.py]"]
end

node_operator(("Benchmark operator"))
node_hfdata["Hugging Face dataset"]
node_server["Inference server"]

node_operator -->|"configures"| node_cli
node_cli -->|"starts run"| node_run
node_run -->|"applies profile"| node_profiles
node_run -->|"loads tokenizer"| node_tokenizer
node_run -->|"loads workload"| node_dataset
node_hfdata -->|"provides dataset"| node_dataset
node_dataset -->|"supplies prompts"| node_requestgen
node_tokenizer -->|"tokenizes prompts"| node_requestgen
node_run -->|"configures benchmark"| node_benchmark
node_benchmark -->|"schedules runs"| node_scheduler
node_scheduler -->|"selects load"| node_executors
node_executors -->|"requests workload"| node_requestgen
node_executors -->|"dispatches requests"| node_backend
node_backend -->|"streams API requests"| node_server
node_backend -->|"returns responses"| node_scheduler
node_scheduler -->|"records responses"| node_results
node_run -->|"writes report"| node_writer
node_results -->|"provides metrics"| node_writer
node_writer -->|"serializes JSON"| node_resultfiles
node_run -.->|"updates console"| node_console
node_operator -->|"views results"| node_dashboard
node_resultfiles -->|"loads JSON"| node_dashboardparse
node_dashboardparse -->|"provides tabular data"| node_dashboard
node_operator -->|"deploys benchmarker"| node_helm
node_operator -->|"runs example"| node_slurm

click node_cli "https://github.com/coderoj/inference-benchmarker/blob/main/src/main.rs"
click node_run "https://github.com/coderoj/inference-benchmarker/blob/main/src/lib.rs"
click node_profiles "https://github.com/coderoj/inference-benchmarker/blob/main/src/profiles.rs"
click node_dataset "https://github.com/coderoj/inference-benchmarker/blob/main/src/requests.rs"
click node_tokenizer "https://github.com/coderoj/inference-benchmarker/blob/main/src/lib.rs"
click node_requestgen "https://github.com/coderoj/inference-benchmarker/blob/main/src/requests.rs"
click node_benchmark "https://github.com/coderoj/inference-benchmarker/blob/main/src/benchmark.rs"
click node_scheduler "https://github.com/coderoj/inference-benchmarker/blob/main/src/scheduler.rs"
click node_executors "https://github.com/coderoj/inference-benchmarker/blob/main/src/executors.rs"
click node_backend "https://github.com/coderoj/inference-benchmarker/blob/main/src/requests.rs"
click node_results "https://github.com/coderoj/inference-benchmarker/blob/main/src/results.rs"
click node_writer "https://github.com/coderoj/inference-benchmarker/blob/main/src/writers.rs"
click node_console "https://github.com/coderoj/inference-benchmarker/blob/main/src/app.rs"
click node_dashboardparse "https://github.com/coderoj/inference-benchmarker/blob/main/extra/dashboard/parse_results.py"
click node_dashboard "https://github.com/coderoj/inference-benchmarker/blob/main/extra/dashboard/app.py"
click node_helm "https://github.com/coderoj/inference-benchmarker/tree/main/extra/k8s/inference-benchmarker"
click node_slurm "https://github.com/coderoj/inference-benchmarker/blob/main/extra/slurm/benchmark.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_run,node_profiles toneBlue
class node_dataset,node_tokenizer,node_requestgen toneAmber
class node_benchmark,node_scheduler,node_executors,node_backend,node_server toneMint
class node_results,node_writer,node_resultfiles,node_console,node_dashboardparse,node_dashboard toneRose
class node_helm,node_slurm,node_operator,node_hfdata toneIndigo
```

## Components

### Run Setup

- [CLI configuration](https://github.com/coderoj/inference-benchmarker/blob/main/src/main.rs): `src/main.rs`
- [Run orchestration](https://github.com/coderoj/inference-benchmarker/blob/main/src/lib.rs): `src/lib.rs`
- [Workload profiles](https://github.com/coderoj/inference-benchmarker/blob/main/src/profiles.rs): `src/profiles.rs`

### Workload Generation

- [Dataset and prompts](https://github.com/coderoj/inference-benchmarker/blob/main/src/requests.rs): `src/requests.rs`
- [Tokenizer](https://github.com/coderoj/inference-benchmarker/blob/main/src/lib.rs): `src/lib.rs`
- [Request generation](https://github.com/coderoj/inference-benchmarker/blob/main/src/requests.rs): `src/requests.rs`

### Benchmark Execution

- [Benchmark modes](https://github.com/coderoj/inference-benchmarker/blob/main/src/benchmark.rs): `src/benchmark.rs`
- [Load scheduler](https://github.com/coderoj/inference-benchmarker/blob/main/src/scheduler.rs): `src/scheduler.rs`
- [Load executors](https://github.com/coderoj/inference-benchmarker/blob/main/src/executors.rs): `src/executors.rs`
- [Streaming API client](https://github.com/coderoj/inference-benchmarker/blob/main/src/requests.rs): `src/requests.rs`

### Results and Review

- [Metric aggregation](https://github.com/coderoj/inference-benchmarker/blob/main/src/results.rs): `src/results.rs`
- [JSON results writer](https://github.com/coderoj/inference-benchmarker/blob/main/src/writers.rs): `src/writers.rs`
- **JSON result files**
- [Interactive console](https://github.com/coderoj/inference-benchmarker/blob/main/src/app.rs): `src/app.rs`
- [Results conversion](https://github.com/coderoj/inference-benchmarker/blob/main/extra/dashboard/parse_results.py): `extra/dashboard/parse_results.py`
- [Results dashboard](https://github.com/coderoj/inference-benchmarker/blob/main/extra/dashboard/app.py): `extra/dashboard/app.py`

### Deployment

- [Kubernetes chart](https://github.com/coderoj/inference-benchmarker/tree/main/extra/k8s/inference-benchmarker): `extra/k8s/inference-benchmarker`
- [Slurm example](https://github.com/coderoj/inference-benchmarker/blob/main/extra/slurm/benchmark.py): `extra/slurm/benchmark.py`

### Other components

- **Benchmark operator**
- **Hugging Face dataset**
- **Inference server**

## Connections

- Benchmark operator → CLI configuration: configures
- CLI configuration → Run orchestration: starts run
- Run orchestration → Workload profiles: applies profile
- Run orchestration → Tokenizer: loads tokenizer
- Run orchestration → Dataset and prompts: loads workload
- Hugging Face dataset → Dataset and prompts: provides dataset
- Dataset and prompts → Request generation: supplies prompts
- Tokenizer → Request generation: tokenizes prompts
- Run orchestration → Benchmark modes: configures benchmark
- Benchmark modes → Load scheduler: schedules runs
- Load scheduler → Load executors: selects load
- Load executors → Request generation: requests workload
- Load executors → Streaming API client: dispatches requests
- Streaming API client → Inference server: streams API requests
- Streaming API client → Load scheduler: returns responses
- Load scheduler → Metric aggregation: records responses
- Run orchestration → JSON results writer: writes report
- Metric aggregation → JSON results writer: provides metrics
- JSON results writer → JSON result files: serializes JSON
- Run orchestration → Interactive console: updates console
- Benchmark operator → Results dashboard: views results
- JSON result files → Results conversion: loads JSON
- Results conversion → Results dashboard: provides tabular data
- Benchmark operator → Kubernetes chart: deploys benchmarker
- Benchmark operator → Slurm example: runs example