# project-etalon/etalon architecture

> Architecture diagram of the GitHub repository project-etalon/etalon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/project-etalon/etalon
- Repository: https://github.com/project-etalon/etalon

## Overview

##### Project Type and Purpose
- This is an open-source benchmarking framework for LLM inference systems (etalon).
- It provides CLI entrypoints (run_benchmark, prefill_profiler, explore_fixed_qps), core libraries for request generation, metrics collection, capacity-search, and pluggable LLM clients.
- Key use cases: measuring QPS, latency, prefill times, capacity under SLO deadlines across public and private LLM backends (OpenAI, vLLM, custom).

##### High-Level Architecture Overview
- The system is structured as a set of Python modules under the top-level etalon/ package, plus data/ (traces) and docs/.
- CLI entrypoints load configurations → build generators and clients → launch requests → collect metrics → output results.
- Core layers:
- Configuration layer (etalon/config, etalon/capacity_search/config)
- Request generation (etalon/request_generator) – interval + length generators
- LLM client abstraction (etalon/core/llm_clients)
- Request orchestrator (etalon/core/requests_launcher)
- Metrics collection (etalon/metrics)
- Benchmark runners (etalon/run_benchmark.py, etalon/prefill_profiler.py, etalon/explore_fixed_qps.py)
- Capacity search (etalon/capacity_search)

##### Main Components and Their Roles
- CLI Runner
- Parses arguments → reads config YAML/dataclasses → passes to Runner class
- Config Manager
- Loads default + user overrides → exposes RequestConfig, MetricsConfig, DeadlineConfig
- Request Generators
- IntervalGenerator (poisson, static, trace)
- LengthGenerator (fixed, uniform, zipf, trace)
- Read data/processed_traces for trace-based generation
- LLM Client Actors
- BaseLLMClient interface → implementations for OpenAI, vLLM, custom (ray-remote)
- Each client handles send_llm_request and returns metrics + output
- RequestsLauncher
- Spawns N clients → multiplexes concurrent requests per client → measures service-level metrics
- Metrics Store and Logging
- Request-level metrics, aggregate service metrics, CDF sketches → local files + optional wandb
- Capacity Search Module
- Orchestrates iterative runs under varying loads to find max QPS for given SLOs
- External Services
- OpenAI API or self-hosted vLLM (OpenAI-compatible server)
- Optional wandb for metrics visualization

##### Relationships and Data Flows
- User runs “python -m etalon.run_benchmark”
↓
- Runner reads configs → instantiates IntervalGenerator & LengthGenerator
↓
- Runner spins up ray-remote LLM clients via BaseLLMClient subclass
↓
- RequestsLauncher feeds generated RequestConfig objects to clients
↓
- Clients call external API (OpenAI/vLLM) → return latency, token counts, etc.
↓
- Metrics module aggregates and writes outputs (CSV, JSON, wandb)
↓
- Capacity Search (if enabled) loops over deadlines and QPS values, feeding results back to config

##### Architectural Patterns and Principles
- Modular design: separate config, generation, transport, metrics, search
- Pluggable adapters: BaseLLMClient → support new inference systems
- Actor-based concurrency: ray-remote clients for isolation and parallelism
- Data-driven generative testing: using real traces for interval/length drawing

##### Technologies and Frameworks
- Python 3.10+, ray for remote actors, PyYAML/dataclasses for configs
- pandas/numpy (likely) for metrics computations, wandb for experiment logging
- CLI via argparse in entrypoint scripts
- Build/test: setup.py, Makefile, GitHub Actions for linting and docs deploy

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "CLI Entrypoints"
        direction TB
        RB["run_benchmark"]:::cli
        PP["prefill_profiler"]:::cli
        EFQ["explore_fixed_qps"]:::cli
    end

    subgraph "Configuration Layer"
        direction TB
        GC["General Config Files"]:::config
        CSCC["Capacity Search Config Files"]:::config
    end

    subgraph "Request Generators"
        direction TB
        IG["IntervalGenerators"]:::generation
        LG["LengthGenerators"]:::generation
    end

    subgraph "LLM Client Abstraction"
        direction TB
        BLC["BaseLLMClient"]:::client
        OAC["OpenAIChatClient"]:::client
    end

    RL["RequestsLauncher"]:::client

    subgraph "Metrics Collection"
        direction TB
        MS["MetricStore"]:::metrics
        RLM["RequestLevelMetrics"]:::metrics
        SM["ServiceMetrics"]:::metrics
        CDF["CDFSketch"]:::metrics
    end

    subgraph "Capacity Search"
        direction TB
        CS["CapacitySearch"]:::search
        SMGR["SearchManager"]:::search
        BW["BenchmarkWrapper"]:::search
        CSM["CapSearchMain"]:::search
    end

    subgraph "Data Store"
        direction TB
        TR["Processed Traces (.csv)"]:::storage
    end

    subgraph "External Services"
        direction TB
        OA["OpenAI API"]:::external
        VL["vLLM Server"]:::external
        WB["wandb"]:::external
    end

    %% Flow
    RB -->|"loads config"| GC
    RB -->|"loads config"| CSCC
    PP -->|"loads config"| GC
    EFQ -->|"loads config"| GC

    GC -->|"provides configs"| IG
    GC -->|"provides configs"| LG

    IG -->|"generates intervals"| RL
    LG -->|"generates lengths"| RL

    RL -->|"spawns ray actors"| BLC
    RL -->|"invokes send_llm_request"| BLC

    BLC -->|"calls API"| OA
    BLC -->|"calls API"| VL

    BLC -->|"returns metrics"| MS
    MS --> SM
    MS --> RLM
    MS --> CDF

    MS -->|"optional"| WB

    CSCC -->|"configs for search"| CS
    CS --> SMGR
    SMGR --> BW
    BW --> RL
    RL --> MS

    TR --> IG
    TR --> LG

    %% Click events
    click RB "https://github.com/project-etalon/etalon/blob/main/etalon/run_benchmark.py"
    click PP "https://github.com/project-etalon/etalon/blob/main/etalon/prefill_profiler.py"
    click EFQ "https://github.com/project-etalon/etalon/blob/main/etalon/explore_fixed_qps.py"
    click GC "https://github.com/project-etalon/etalon/blob/main/etalon/config/config.py"
    click GC "https://github.com/project-etalon/etalon/blob/main/etalon/config/base_fixed_config.py"
    click GC "https://github.com/project-etalon/etalon/blob/main/etalon/config/base_poly_config.py"
    click GC "https://github.com/project-etalon/etalon/blob/main/etalon/config/flat_dataclass.py"
    click GC "https://github.com/project-etalon/etalon/blob/main/etalon/config/utils.py"
    click CSCC "https://github.com/project-etalon/etalon/blob/main/etalon/capacity_search/config/config.py"
    click CSCC "https://github.com/project-etalon/etalon/blob/main/etalon/capacity_search/config/default.yml"
    click CSCC "https://github.com/project-etalon/etalon/blob/main/etalon/capacity_search/config/llama_70b.yml"
    click CSCC "https://github.com/project-etalon/etalon/blob/main/etalon/capacity_search/config/llama_8b.yml"
    click CSCC "https://github.com/project-etalon/etalon/blob/main/etalon/capacity_search/config/mixtral.yml"
    click IG "https://github.com/project-etalon/etalon/blob/main/etalon/request_generator/interval_generator/base_generator.py"
    click IG "https://github.com/project-etalon/etalon/blob/main/etalon/request_generator/interval_generator/poisson_generator.py"
    click IG "https://github.com/project-etalon/etalon/blob/main/etalon/request_generator/interval_generator/static_generator.py"
    click IG "https://github.com/project-etalon/etalon/blob/main/etalon/request_generator/interval_generator/gamma_generator.py"
    click IG "https://github.com/project-etalon/etalon/blob/main/etalon/request_generator/interval_generator/trace_generator.py"
    click IG "https://github.com/project-etalon/etalon/blob/main/etalon/request_generator/interval_generator/generator_registry.py"
    click LG "https://github.com/project-etalon/etalon/blob/main/etalon/request_generator/length_generator/base_generator.py"
    click LG "https://github.com/project-etalon/etalon/blob/main/etalon/request_generator/length_generator/fixed_generator.py"
    click LG "https://github.com/project-etalon/etalon/blob/main/etalon/request_generator/length_generator/uniform_generator.py"
    click LG "https://github.com/project-etalon/etalon/blob/main/etalon/request_generator/length_generator/zipf_generator.py"
    click LG "https://github.com/project-etalon/etalon/blob/main/etalon/request_generator/length_generator/trace_generator.py"
    click LG "https://github.com/project-etalon/etalon/blob/main/etalon/request_generator/length_generator/generator_registry.py"
    click BLC "https://github.com/project-etalon/etalon/blob/main/etalon/core/llm_clients/base_llm_client.py"
    click OAC "https://github.com/project-etalon/etalon/blob/main/etalon/core/llm_clients/openai_chat_completions_client.py"
    click RL "https://github.com/project-etalon/etalon/blob/main/etalon/core/requests_launcher.py"
    click MS "https://github.com/project-etalon/etalon/blob/main/etalon/metrics/metric_store.py"
    click RLM "https://github.com/project-etalon/etalon/blob/main/etalon/metrics/request_level_metrics.py"
    click SM "https://github.com/project-etalon/etalon/blob/main/etalon/metrics/service_metrics.py"
    click CDF "https://github.com/project-etalon/etalon/blob/main/etalon/metrics/cdf_sketch.py"
    click CS "https://github.com/project-etalon/etalon/blob/main/etalon/capacity_search/capacity_search.py"
    click SMGR "https://github.com/project-etalon/etalon/blob/main/etalon/capacity_search/search_manager.py"
    click BW "https://github.com/project-etalon/etalon/blob/main/etalon/capacity_search/benchmark_wrapper.py"
    click CSM "https://github.com/project-etalon/etalon/blob/main/etalon/capacity_search/main.py"
    click TR "https://github.com/project-etalon/etalon/blob/main/data/processed_traces/arxiv_summarization_filtered_stats_llama2_tokenizer.csv"

    %% Styles
    classDef cli fill:#E8F1F2,stroke:#333,stroke-width:1px
    classDef config fill:#D0E6FA,stroke:#333,stroke-width:1px
    classDef generation fill:#D6F5D6,stroke:#333,stroke-width:1px
    classDef client fill:#FFE5B4,stroke:#333,stroke-width:1px
    classDef metrics fill:#E9D6F5,stroke:#333,stroke-width:1px
    classDef search fill:#F5D6D6,stroke:#333,stroke-width:1px
    classDef storage fill:#C2C2C2,stroke:#333,stroke-width:1px
    classDef external fill:#F0F0F0,stroke:#333,stroke-width:1px
```
