pranavvm26/llmperf

1. What this repo is

This is a local benchmarking CLI, not a server. The main executable (sm-benchmarker) drives load tests against external LLM endpoints and collects latency/throughput metrics. The core runtime is Python + Ray, with AWS-focused adapters plus a few non-AWS backends.

2. Top-level flow

The benchmark flow is:

CLI args → prompt/token setup → optional warmup → power-of-2 concurrency sweep → Ray-dispatched requests → per-request metrics → aggregation → terminal tables + JSON + PDF report.

There is no durable service state; outputs are written only to the chosen results directory.

3. Orchestration layer

token_benchmark_ray.py is the central entry point for running benchmarks. It likely owns argument handling, concurrency stepping, and test lifecycle. requests_launcher.py appears to coordinate issuing batches of benchmark requests at each concurrency level, while ray_llm_client.py provides the worker-side client abstraction used by Ray tasks.

4. Execution model

Ray is the primary parallelism boundary. Each concurrent request is executed in a Ray worker, which isolates request execution, timing, and backend-specific client behavior. This makes the benchmark distributed across local Ray workers but still controlled by a single driver process.

5. Backend adapter boundary

The src/llmperf/ray_clients/ package contains the provider-specific adapters:

- litellm_client.py for LiteLLM-based access

- sagemaker_direct_client.py for direct boto3 SageMaker calls, including inference components and multimodal payloads

- sagemaker_client.py for the legacy SageMaker path

- openai_chat_completions_client.py for OpenAI-compatible APIs

- vertexai_client.py for Vertex AI

These adapters are the main integration boundary between the benchmark engine and external model providers.

6. Prompt generation and tokenization

common.py, tokenizer_factory.py, and sonnet.txt support request synthesis and token counting. The repo’s design requires a model-specific tokenizer via --tokenizer, so token accounting is intentionally decoupled from the endpoint itself. For vision mode, requests use a bundled image plus instructions rather than sampled text prompts.

7. Metrics and aggregation

common_metrics.py and models.py likely define the per-request and aggregate metric objects: TTFT, ITL, E2E latency, output throughput, token counts, and error summaries. These are collected per request and then rolled up per concurrency step and across the full sweep.

8. Reporting and UX

rich_output.py handles formatted terminal tables/panels, while report.py generates the PDF benchmark report. The README’s output structure implies the repo produces both machine-readable JSON summaries and a human-readable PDF, with the terminal view showing percentile distributions and a final concurrency comparison table.

9. AWS and multimodal specifics

The main architectural differentiator is the sagemaker_direct path. It bypasses LiteLLM when the endpoint needs Inference Components or vision/multimodal payloads, and it uses boto3 directly. That creates a clean boundary between generic benchmark orchestration and AWS-specific payload/serialization concerns.

10. Supporting artifacts

sample-output/ contains an example generated report, and analyze-token-benchmark-results.ipynb is an offline analysis notebook for post-processing benchmark output. llm_correctness.py appears to be a separate utility or experiment script, not part of the benchmark execution path.

11. External dependencies that shape the design

The architecture depends on Ray for concurrency, AWS SageMaker/Boto3 for direct invocation, LiteLLM for gateway-style access, and optional OpenAI/Vertex AI compatibility. These are not interchangeable implementation details; they define the transport and payload shape at the backend boundary.

12. Diagram guidance

A good architecture diagram would show:

- a single CLI/driver node,

- a Ray worker pool beneath it,

- a backend adapter layer separating request execution from providers,

- a metrics/aggregation layer,

- and output sinks for terminal, JSON files, and PDF reports.

The important conceptual split is between benchmark orchestration and provider-specific invocation logic.