# pranavvm26/llmperf architecture

> Architecture diagram of the GitHub repository pranavvm26/llmperf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pranavvm26/llmperf
- Repository: https://github.com/pranavvm26/llmperf

## Overview

1. **What this repo is**
   This is a **local benchmarking CLI**, not a server. The main executable (`sm-benchmarker`) drives load tests against external LLM endpoints and collects latency/throughput metrics. The core runtime is **Python + Ray**, with AWS-focused adapters plus a few non-AWS backends.

2. **Top-level flow**
   The benchmark flow is:
   **CLI args → prompt/token setup → optional warmup → power-of-2 concurrency sweep → Ray-dispatched requests → per-request metrics → aggregation → terminal tables + JSON + PDF report**.
   There is no durable service state; outputs are written only to the chosen results directory.

3. **Orchestration layer**
   `token_benchmark_ray.py` is the central entry point for running benchmarks. It likely owns argument handling, concurrency stepping, and test lifecycle. `requests_launcher.py` appears to coordinate issuing batches of benchmark requests at each concurrency level, while `ray_llm_client.py` provides the worker-side client abstraction used by Ray tasks.

4. **Execution model**
   Ray is the primary parallelism boundary. Each concurrent request is executed in a Ray worker, which isolates request execution, timing, and backend-specific client behavior. This makes the benchmark distributed across local Ray workers but still controlled by a single driver process.

5. **Backend adapter boundary**
   The `src/llmperf/ray_clients/` package contains the provider-specific adapters:
   - `litellm_client.py` for LiteLLM-based access
   - `sagemaker_direct_client.py` for direct boto3 SageMaker calls, including inference components and multimodal payloads
   - `sagemaker_client.py` for the legacy SageMaker path
   - `openai_chat_completions_client.py` for OpenAI-compatible APIs
   - `vertexai_client.py` for Vertex AI
   These adapters are the main integration boundary between the benchmark engine and external model providers.

6. **Prompt generation and tokenization**
   `common.py`, `tokenizer_factory.py`, and `sonnet.txt` support request synthesis and token counting. The repo’s design requires a model-specific tokenizer via `--tokenizer`, so token accounting is intentionally decoupled from the endpoint itself. For vision mode, requests use a bundled image plus instructions rather than sampled text prompts.

7. **Metrics and aggregation**
   `common_metrics.py` and `models.py` likely define the per-request and aggregate metric objects: TTFT, ITL, E2E latency, output throughput, token counts, and error summaries. These are collected per request and then rolled up per concurrency step and across the full sweep.

8. **Reporting and UX**
   `rich_output.py` handles formatted terminal tables/panels, while `report.py` generates the PDF benchmark report. The README’s output structure implies the repo produces both machine-readable JSON summaries and a human-readable PDF, with the terminal view showing percentile distributions and a final concurrency comparison table.

9. **AWS and multimodal specifics**
   The main architectural differentiator is the **`sagemaker_direct`** path. It bypasses LiteLLM when the endpoint needs **Inference Components** or **vision/multimodal** payloads, and it uses boto3 directly. That creates a clean boundary between generic benchmark orchestration and AWS-specific payload/serialization concerns.

10. **Supporting artifacts**
    `sample-output/` contains an example generated report, and `analyze-token-benchmark-results.ipynb` is an offline analysis notebook for post-processing benchmark output. `llm_correctness.py` appears to be a separate utility or experiment script, not part of the benchmark execution path.

11. **External dependencies that shape the design**
    The architecture depends on **Ray** for concurrency, **AWS SageMaker/Boto3** for direct invocation, **LiteLLM** for gateway-style access, and optional **OpenAI/Vertex AI** compatibility. These are not interchangeable implementation details; they define the transport and payload shape at the backend boundary.

12. **Diagram guidance**
    A good architecture diagram would show:
    - a single CLI/driver node,
    - a Ray worker pool beneath it,
    - a backend adapter layer separating request execution from providers,
    - a metrics/aggregation layer,
    - and output sinks for terminal, JSON files, and PDF reports.
    The important conceptual split is between **benchmark orchestration** and **provider-specific invocation logic**.

Main entry points: [`src/llmperf/token_benchmark_ray.py`](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/token_benchmark_ray.py) (Benchmark CLI), [`src/llmperf/ray_llm_client.py`](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/ray_llm_client.py) (Ray Client), [`src/llmperf/requests_launcher.py`](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/requests_launcher.py) (Request Launcher), [`src/llmperf/common_metrics.py`](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/common_metrics.py) (Metrics Core)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_orchestration["Orchestration"]
  node_node_cli_driver["Benchmark CLI<br/>entrypoint"]
  node_node_request_launcher["Request Launcher<br/>orchestrator"]
end

subgraph group_group_runtime["Ray Runtime"]
  node_node_ray_client["Ray Client<br/>worker API<br/>[ray_llm_client.py]"]
end

subgraph group_group_backends["Backend Adapters"]
  node_node_litellm["LiteLLM Client<br/>adapter<br/>[litellm_client.py]"]
  node_node_sagemaker_direct["SageMaker Direct<br/>adapter"]
  node_node_sagemaker_legacy["SageMaker Client<br/>adapter"]
  node_node_openai["OpenAI Compat<br/>adapter"]
  node_node_vertexai["Vertex AI<br/>adapter<br/>[vertexai_client.py]"]
end

subgraph group_group_metrics["Metrics & Reporting"]
  node_node_metrics["Metrics Core<br/>[common_metrics.py]"]
  node_node_models["Data Models<br/>schema<br/>[models.py]"]
  node_node_rich["Terminal Output<br/>presentation<br/>[rich_output.py]"]
  node_node_report["PDF Report<br/>reporting<br/>[report.py]"]
end

subgraph group_group_support["Shared Support"]
  node_node_common["Prompt Setup<br/>support<br/>[common.py]"]
  node_node_tokenizer["Tokenizer Factory<br/>support"]
  node_node_sonnet["Sonnet Prompt<br/>asset<br/>[sonnet.txt]"]
  node_node_utils["Utilities<br/>support<br/>[utils.py]"]
end

node_node_cli_driver -->|"drives"| node_node_request_launcher
node_node_cli_driver -->|"builds prompts"| node_node_common
node_node_cli_driver -->|"counts tokens"| node_node_tokenizer
node_node_cli_driver -->|"prints status"| node_node_rich
node_node_request_launcher -->|"dispatches"| node_node_ray_client
node_node_request_launcher -->|"collects"| node_node_metrics
node_node_request_launcher -->|"materializes"| node_node_models
node_node_ray_client -->|"uses"| node_node_litellm
node_node_ray_client -->|"uses"| node_node_sagemaker_direct
node_node_ray_client -->|"uses"| node_node_sagemaker_legacy
node_node_ray_client -->|"uses"| node_node_openai
node_node_ray_client -->|"uses"| node_node_vertexai
node_node_common -->|"loads"| node_node_sonnet
node_node_common -->|"depends on"| node_node_tokenizer
node_node_metrics -->|"fills"| node_node_models
node_node_metrics -->|"summarizes to"| node_node_rich
node_node_metrics -->|"exports to"| node_node_report
node_node_models -->|"feeds"| node_node_report
node_node_cli_driver -->|"writes"| node_node_report
node_node_cli_driver -->|"uses"| node_node_utils

click node_node_cli_driver "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/token_benchmark_ray.py"
click node_node_request_launcher "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/requests_launcher.py"
click node_node_ray_client "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/ray_llm_client.py"
click node_node_litellm "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/ray_clients/litellm_client.py"
click node_node_sagemaker_direct "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/ray_clients/sagemaker_direct_client.py"
click node_node_sagemaker_legacy "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/ray_clients/sagemaker_client.py"
click node_node_openai "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/ray_clients/openai_chat_completions_client.py"
click node_node_vertexai "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/ray_clients/vertexai_client.py"
click node_node_common "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/common.py"
click node_node_tokenizer "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/tokenizer_factory.py"
click node_node_sonnet "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/sonnet.txt"
click node_node_metrics "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/common_metrics.py"
click node_node_models "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/models.py"
click node_node_rich "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/rich_output.py"
click node_node_report "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/report.py"
click node_node_utils "https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_cli_driver,node_node_request_launcher toneBlue
class node_node_ray_client toneAmber
class node_node_litellm,node_node_sagemaker_direct,node_node_sagemaker_legacy,node_node_openai,node_node_vertexai toneMint
class node_node_metrics,node_node_models,node_node_rich,node_node_report toneRose
class node_node_common,node_node_tokenizer,node_node_sonnet,node_node_utils toneIndigo
```

## Components

### Orchestration

CLI driver and benchmark lifecycle control.

- [Benchmark CLI](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/token_benchmark_ray.py): entrypoint · Main benchmark executable that parses arguments and drives the full run lifecycle. · `src/llmperf/token_benchmark_ray.py`
- [Request Launcher](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/requests_launcher.py): orchestrator · Coordinates request batches across concurrency steps and manages warmup/sweep execution. · `src/llmperf/requests_launcher.py`

### Ray Runtime

Worker-side execution and request dispatch.

- [Ray Client](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/ray_llm_client.py): worker API · Worker-facing client abstraction used by Ray tasks to execute benchmark requests. · `src/llmperf/ray_llm_client.py`

### Backend Adapters

Provider-specific invocation layers.

- [LiteLLM Client](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/ray_clients/litellm_client.py): adapter · Adapter for LiteLLM-based access to model endpoints. · `src/llmperf/ray_clients/litellm_client.py`
- [SageMaker Direct](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/ray_clients/sagemaker_direct_client.py): adapter · Direct boto3 SageMaker path for inference components and multimodal payloads. · `src/llmperf/ray_clients/sagemaker_direct_client.py`
- [SageMaker Client](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/ray_clients/sagemaker_client.py): adapter · Legacy SageMaker invocation path. · `src/llmperf/ray_clients/sagemaker_client.py`
- [OpenAI Compat](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/ray_clients/openai_chat_completions_client.py): adapter · Client for OpenAI-compatible chat completions APIs. · `src/llmperf/ray_clients/openai_chat_completions_client.py`
- [Vertex AI](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/ray_clients/vertexai_client.py): adapter · Client for Vertex AI endpoints. · `src/llmperf/ray_clients/vertexai_client.py`

### Metrics & Reporting

Collection, aggregation, and output formatting.

- [Metrics Core](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/common_metrics.py): metrics · Per-request timing and throughput measurements such as TTFT, ITL, and E2E latency. · `src/llmperf/common_metrics.py`
- [Data Models](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/models.py): schema · Structured metric and result objects shared across the benchmark pipeline. · `src/llmperf/models.py`
- [Terminal Output](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/rich_output.py): presentation · Formatted tables and panels for interactive benchmark progress and summaries. · `src/llmperf/rich_output.py`
- [PDF Report](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/report.py): reporting · Generates the benchmark PDF report from aggregated results. · `src/llmperf/report.py`

### Shared Support

Prompt synthesis, tokenization, and shared models.

- [Prompt Setup](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/common.py): support · Shared request synthesis and prompt helpers. · `src/llmperf/common.py`
- [Tokenizer Factory](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/tokenizer_factory.py): support · Creates model-specific tokenizers for token accounting. · `src/llmperf/tokenizer_factory.py`
- [Sonnet Prompt](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/sonnet.txt): asset · Bundled prompt text used for request generation. · `src/llmperf/sonnet.txt`
- [Utilities](https://github.com/pranavvm26/llmperf/blob/main/src/llmperf/utils.py): support · General helper functions shared by the benchmark implementation. · `src/llmperf/utils.py`

## Connections

- Benchmark CLI → Request Launcher: drives · CLI entrypoint hands off benchmark lifecycle and concurrency sweep execution.
- Benchmark CLI → Prompt Setup: builds prompts · Driver assembles benchmark requests from shared prompt helpers.
- Benchmark CLI → Tokenizer Factory: counts tokens · Driver uses model-specific tokenization for request sizing and reporting.
- Benchmark CLI → Terminal Output: prints status · Driver emits formatted terminal progress and summary output.
- Request Launcher → Ray Client: dispatches · Launcher submits concurrent benchmark requests to the Ray worker client layer.
- Request Launcher → Metrics Core: collects · Launcher receives per-request timings and error outcomes for aggregation.
- Request Launcher → Data Models: materializes · Launcher builds structured per-step and per-run result objects.
- Ray Client → LiteLLM Client: uses · Worker client routes generic requests through LiteLLM when applicable.
- Ray Client → SageMaker Direct: uses · Worker client routes SageMaker direct and multimodal requests through boto3.
- Ray Client → SageMaker Client: uses · Worker client supports the legacy SageMaker invocation path.
- Ray Client → OpenAI Compat: uses · Worker client can target OpenAI-compatible chat completions endpoints.
- Ray Client → Vertex AI: uses · Worker client can target Vertex AI endpoints.
- Prompt Setup → Sonnet Prompt: loads · Prompt synthesis can use bundled text assets for request generation.
- Prompt Setup → Tokenizer Factory: depends on · Prompt setup relies on tokenizer selection for token accounting.
- Metrics Core → Data Models: fills · Measured latency and throughput values are stored in shared result models.
- Metrics Core → Terminal Output: summarizes to · Aggregated metrics feed terminal tables and percentile views.
- Metrics Core → PDF Report: exports to · Aggregated metrics are rendered into the PDF report.
- Data Models → PDF Report: feeds · Structured summary data is used to generate the final report artifacts.
- Benchmark CLI → PDF Report: writes · Driver triggers report generation into the chosen results directory.
- Benchmark CLI → Utilities: uses · Driver and helpers share common utility functions.