# ayushi-uwc/elo-benchmark architecture

> Architecture diagram of the GitHub repository ayushi-uwc/elo-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayushi-uwc/elo-benchmark
- Repository: https://github.com/ayushi-uwc/elo-benchmark

## Overview

##### Project type and purpose
- UNER is a modular Python-based evaluation framework (CLI/daemon) for large-scale generative AI models. It orchestrates an eight-stage pipeline—model registration, pairing, prompt generation, solution generation, peer evaluation, scoring, rating propagation, and logging—while integrating with external LLM providers and a MongoDB backend.

##### File structure
- Top-level scripts and modules:
- main.py – entry point, parses CLI args and kicks off evaluation cycles
- tournament.py – orchestrates match lifecycles and stage sequencing
- matches.py – houses prompt templates, match logic, and peer-judging flows
- models.py & model_definitions.py – model abstractions and cost/pricing metadata
- database.py – MongoDB interaction layer
- config.py, litellm_config.py, env.template – environment and framework configuration
- logger_config.py – centralized logging setup
- check_models.py, matches.py, tournament.py – utility and test scaffolding
- Supporting files: requirements.txt, LICENSE, .gitignore, .gradio (for optional Gradio UI), README.md

##### Key insights from README
- Eight-stage dynamic evaluation pipeline
- Dual-track Elo ratings (raw and cost-adjusted)
- External dependencies: multiple LLM providers (OpenAI, Anthropic, Google Gemini, HuggingFace, Groq, etc.)
- MongoDB for immutable, auditable logs and leaderboards
- CLI options for batching, stats, log levels, dependency checks

##### Main components
a. Orchestrator Layer
– CLI / main.py (parses arguments, starts/stops runs)
– tournament.py (pipeline coordinator)
1. Model Registration (model_definitions.py → metadata loader)
2. Fair Matching (Swiss pairing, stratified sampling)
3. Challenge Creation (prompt generation via matches.py)
4. Solution Generation (models.py + external LLM APIs)
5. Peer Evaluation (weighted voting logic in matches.py)
6. Dual Scoring & Elo Updates (Elo formulae in tournament.py)
7. Rating Propagation (Gaussian kernel smoothing)
8. Logging & Dissemination (database.py → MongoDB, JSON logs, leaderboards)
c. Infrastructure & Supporting Services
– External LLM Providers (OpenAI, Anthropic, Google, etc.)
– MongoDB cluster (persistent storage of metadata, logs, leaderboards)
– Logging sink (local files, or external log aggregation)
– (Optional) Gradio UI component loading .gradio/certificate.pem
d. Configuration & Utilities
– config.py / litellm_config.py / env variables
– logger_config.py

##### Relationships and interactions
- CLI → Orchestrator → Pipeline stages
- Pipeline modules ↔ Configuration
- Pipeline modules → External LLM APIs (requests/responses with cost & latency)
- Pipeline modules → MongoDB (write artifacts: prompts, responses, votes, Elo deltas)
- Logging module → file system / log aggregator
- Orchestrator → Gradio UI (if enabled)
- Directionality:
- Bidirectional with config and DB

##### Architectural patterns & design principles
- Modular pipeline: each stage is an independent component called by a central orchestrator
- Separation of concerns:
- Model abstraction vs. match logic vs. data persistence
- Configuration isolated from code (env.template → config.py)
- Extensibility: new models added via model_definitions.py, new providers pluggable in models.py
- Immutable logging for auditability (append-only JSON + MongoDB)
- Weighted voting and dual-track Elo encapsulated in domain modules

##### Technologies & frameworks
- Python 3.8+
- Pymongo (MongoDB driver)
- LLM client libraries (OpenAI, Anthropic, HuggingFace, etc.)
- Logging (standard logging module via logger_config.py)
- Gradio (optional UI)
- YAML or JSON for model definitions/config

##### Detailed capture of essential elements
- Peer-federated judging as a subflow where models call back into matches.py
- Immutable logging to MongoDB as final stage

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestrator Layer
    subgraph "Orchestrator Layer"
        CLI["Orchestrator (CLI Entry)"]:::orchestrator
        Coordinator["Pipeline Coordinator"]:::orchestrator
    end

    CLI -->|"start evaluation"| Coordinator

    %% Evaluation Pipeline
    subgraph "Evaluation Pipeline" 
        direction TB
        Stage1["Model Registration"]:::pipeline
        Stage2["Fair Matching"]:::pipeline
        Stage3["Challenge Creation"]:::pipeline
        Stage4["Solution Generation"]:::pipeline
        Stage5["Peer Evaluation"]:::pipeline
        Stage6["Dual Scoring & Elo Updates"]:::pipeline
        Stage7["Rating Propagation"]:::pipeline
        Stage8["Logging & Dissemination"]:::pipeline
    end

    Coordinator -->|sequences| Stage1
    Stage1 --> Stage2
    Stage2 --> Stage3
    Stage3 --> Stage4
    Stage4 --> Stage5
    Stage5 --> Stage6
    Stage6 --> Stage7
    Stage7 --> Stage8

    %% External Services
    subgraph "External Services"
        LLM["LLM Providers"]:::external
        MongoDB["MongoDB Cluster"]:::datastore
        LogSink["Logging Sink"]:::logging
        GradioUI["Gradio UI (Optional)"]:::external
    end

    Stage4 -->|"request/response"| LLM
    Stage3 -->|write prompts| MongoDB
    Stage4 -->|log responses| MongoDB
    Stage5 -->|write votes| MongoDB
    Stage6 -->|write Elo deltas| MongoDB
    Stage7 -->|update ratings| MongoDB
    Stage8 -->|persist logs| MongoDB
    Stage8 -->|stream logs| LogSink
    CLI -->|if enabled| GradioUI

    %% Configuration & Utilities
    subgraph "Configuration & Utilities"
        ConfigCore["Core Configuration & Env Management"]:::config
        LoggerCfg["Logging Configuration"]:::logging
    end

    ConfigCore ---|provides settings| Stage1
    ConfigCore ---|provides settings| Stage2
    ConfigCore ---|provides settings| Stage3
    ConfigCore ---|provides settings| Stage4
    ConfigCore ---|provides settings| Stage5
    ConfigCore ---|provides settings| Stage6
    ConfigCore ---|provides settings| Stage7
    ConfigCore ---|provides settings| Stage8
    ConfigCore ---|provides settings| Coordinator
    LoggerCfg ---|configures| LogSink

    %% Click Events
    click CLI "https://github.com/ayushi-uwc/elo-benchmark/blob/main/main.py"
    click Coordinator "https://github.com/ayushi-uwc/elo-benchmark/blob/main/tournament.py"
    click Stage1 "https://github.com/ayushi-uwc/elo-benchmark/blob/main/model_definitions.py"
    click Stage2 "https://github.com/ayushi-uwc/elo-benchmark/blob/main/matches.py"
    click Stage3 "https://github.com/ayushi-uwc/elo-benchmark/blob/main/matches.py"
    click Stage4 "https://github.com/ayushi-uwc/elo-benchmark/blob/main/models.py"
    click Stage5 "https://github.com/ayushi-uwc/elo-benchmark/blob/main/matches.py"
    click Stage6 "https://github.com/ayushi-uwc/elo-benchmark/blob/main/tournament.py"
    click Stage7 "https://github.com/ayushi-uwc/elo-benchmark/blob/main/tournament.py"
    click Stage8 "https://github.com/ayushi-uwc/elo-benchmark/blob/main/database.py"
    click LoggerCfg "https://github.com/ayushi-uwc/elo-benchmark/blob/main/logger_config.py"
    click ConfigCore "https://github.com/ayushi-uwc/elo-benchmark/blob/main/config.py"
    click ConfigCore "https://github.com/ayushi-uwc/elo-benchmark/blob/main/litellm_config.py"
    click ConfigCore "https://github.com/ayushi-uwc/elo-benchmark/blob/main/env.template"
    click GradioUI "https://github.com/ayushi-uwc/elo-benchmark/blob/main/.gradio/certificate.pem"

    %% Styles
    classDef orchestrator fill:#D6EAF8,stroke:#21618C,color:#1B4F72
    classDef pipeline fill:#AED6F1,stroke:#1B4F72,color:#0B5345
    classDef external fill:#ABEBC6,stroke:#117A65,color:#0E6655
    classDef datastore fill:#F9E79F,stroke:#B9770E,color:#7E5109
    classDef config fill:#D5DBDB,stroke:#7B7D7D,color:#1C2833
    classDef logging fill:#F5B7B1,stroke:#C0392B,color:#641E16
```
