How project-etalon/etalon works
Project Type and Purpose
• This is an open-source benchmarking framework for LLM inference systems (etalon).
• It provides CLI entrypoints (run_benchmark, prefill_profiler, explore_fixed_qps), core libraries for request generation, metrics collection, capacity-search, and pluggable LLM clients.
• Key use cases: measuring QPS, latency, prefill times, capacity under SLO deadlines across public and private LLM backends (OpenAI, vLLM, custom).
High-Level Architecture Overview
• The system is structured as a set of Python modules under the top-level etalon/ package, plus data/ (traces) and docs/.
• CLI entrypoints load configurations → build generators and clients → launch requests → collect metrics → output results.
• Core layers:
• Configuration layer (etalon/config, etalon/capacity_search/config)
• Request generation (etalon/request_generator) – interval + length generators
• LLM client abstraction (etalon/core/llm_clients)
• Request orchestrator (etalon/core/requests_launcher)
• Metrics collection (etalon/metrics)
• Benchmark runners (etalon/run_benchmark.py, etalon/prefill_profiler.py, etalon/explore_fixed_qps.py)
• Capacity search (etalon/capacity_search)
Main Components and Their Roles
• CLI Runner
• Parses arguments → reads config YAML/dataclasses → passes to Runner class
• Config Manager
• Loads default + user overrides → exposes RequestConfig, MetricsConfig, DeadlineConfig
• Request Generators
• IntervalGenerator (poisson, static, trace)
• LengthGenerator (fixed, uniform, zipf, trace)
• Read data/processed_traces for trace-based generation
• LLM Client Actors
• BaseLLMClient interface → implementations for OpenAI, vLLM, custom (ray-remote)
• Each client handles send_llm_request and returns metrics + output
• RequestsLauncher
• Spawns N clients → multiplexes concurrent requests per client → measures service-level metrics
• Metrics Store and Logging
• Request-level metrics, aggregate service metrics, CDF sketches → local files + optional wandb
• Capacity Search Module
• Orchestrates iterative runs under varying loads to find max QPS for given SLOs
• External Services
• OpenAI API or self-hosted vLLM (OpenAI-compatible server)
• Optional wandb for metrics visualization
Relationships and Data Flows
• User runs “python -m etalon.run_benchmark”
↓
• Runner reads configs → instantiates IntervalGenerator & LengthGenerator
↓
• Runner spins up ray-remote LLM clients via BaseLLMClient subclass
↓
• RequestsLauncher feeds generated RequestConfig objects to clients
↓
• Clients call external API (OpenAI/vLLM) → return latency, token counts, etc.
↓
• Metrics module aggregates and writes outputs (CSV, JSON, wandb)
↓
• Capacity Search (if enabled) loops over deadlines and QPS values, feeding results back to config
Architectural Patterns and Principles
• Modular design: separate config, generation, transport, metrics, search
• Pluggable adapters: BaseLLMClient → support new inference systems
• Actor-based concurrency: ray-remote clients for isolation and parallelism
• Data-driven generative testing: using real traces for interval/length drawing
Technologies and Frameworks
• Python 3.10+, ray for remote actors, PyYAML/dataclasses for configs
• pandas/numpy (likely) for metrics computations, wandb for experiment logging
• CLI via argparse in entrypoint scripts
• Build/test: setup.py, Makefile, GitHub Actions for linting and docs deploy