# v-xchen-v/elobench architecture

> Architecture diagram of the GitHub repository v-xchen-v/elobench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/v-xchen-v/elobench
- Repository: https://github.com/v-xchen-v/elobench

## Overview

##### Project Type and Purpose
- Type: Open-source Python framework/tool for evaluating and ranking large language models (LLMs) via an Elo rating system.
- Purpose: Automate pairwise “battles” between LLMs on real-world QA tasks, use GPT-4 as a judge, compute and analyze Elo ratings, and produce leaderboards and visualizations.

##### File Structure
- Top-level modules:
- apps/ – Standalone utilities and GUI-style monitors.
- data/ – Question pools, preparation scripts, and arena data.
- models/, openai_chat.py, judger/ – LLM interface implementations (local HF models, OpenAI API, GPT-4 judger).
- pipe/ – Battle orchestration pipelines (battle_pipe.py, iterative_battle_pipe.py).
- elo_rating/ – Elo rating core logic and simulation.
- battle_outcome_analysis/ & run_analysis.py – Post-battle analysis and plotting.
- run_experiment.py – End-to-end execution script (question selection, answer generation, judging, rating updates).
- tempcache/ & results/ – Caching layer and persistent storage for experiments.
- datamodel/ – Data classes for battles, records, ratings, model collections.
- scripts/ & exploratory_analysis/ & plots/ – Ad-hoc exploratory notebooks and scripts for research.

##### Additional Insights from README
- Workflows:
- Experiment setup (register models, questions, directories).
- Running battles (run_experiment.py).
- Analyzing results (run_analysis.py, battle_outcome_analysis).
- Technologies: Python 3.9+, NumPy, Torch, Transformers, Datasets, Accelerate, OpenAI API.

##### Components
a. Main Components
1. CLI Orchestrator (run_experiment.py)
2. Question Pool (data/)
3. Model Integration Layer
– HF Local Models (models/hf_local_lm.py, accelerate)
– OpenAI API Models (models/gpt_online_lm.py, openai_chat.py)
4. Battle Pipeline (pipe/battle_pipe.py, pipe/iterative_battle_pipe.py)
5. Caching Layer (tempcache/)
6. Judging Service (judger/gpt_judger.py using GPT-4)
7. Elo Engine (elo_rating/)
8. Persistent Store (results/)
9. Analysis & Visualization (battle_outcome_analysis/, plots/, run_analysis.py)
10. Monitoring & Apps (apps/monitors, apps/*.py)

b. Relationships and Data Flows
1. CLI Orchestrator reads model list and question list → feeds to Battle Pipeline.
2. Battle Pipeline invokes Model Integration to generate answers → writes to cache.
3. Cached answers + cached judgments → passed to Judging Service → outputs battle outcome.
4. Outcome sent to Elo Engine → updates ratings → persisted in results/.
5. run_analysis.py & battle_outcome_analysis read results/ → compute stats and plots.
6. apps/monitoring scripts subscribe to results/ and cache to display live metrics.

c. Architectural Patterns & Principles
- Pipeline Architecture: clear separation between data ingestion (questions), processing (answers, judging), and post-processing (rating, analysis).
- Modular LLM Interface: abstraction layer for plugging new models or endpoints.
- Caching for Idempotence: avoid repeated invocations of expensive LLM or GPT-4 calls.
- Data-centric design: uses datamodel classes to enforce consistent record formats.

d. Key Technologies and Frameworks
- Python ecosystem: NumPy, pandas (implicit in scripts/notebooks), PyTorch + Transformers + Accelerate.
- OpenAI API for GPT-4 judging.
- Structured storage: CSV/JSON for questions, battles, ratings.
- Jupyter notebooks for exploratory analysis and plots (Matplotlib/Seaborn likely).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layers and Components
    subgraph "Orchestration"
        CLI["CLI Orchestrator"]:::compute
    end

    subgraph "Data Layer"
        QPool["Question Pool"]:::store
        DM["Data Models"]:::compute
    end

    subgraph "Model Integration Layer"
        HFLM["HF Local Models"]:::compute
        OAIM["OpenAI API Models"]:::compute
    end

    subgraph "Pipeline Layer"
        BP["Battle Pipeline"]:::compute
        IBP["Iterative Pipeline"]:::compute
    end

    subgraph "Caching Layer"
        Cache["Cache"]:::store
    end

    subgraph "Judging Service"
        Judger["GPT-4 Judging Service"]:::external
    end

    subgraph "Rating Layer"
        Elo["Elo Engine"]:::compute
    end

    subgraph "Persistent Store"
        Results["Results Store"]:::store
    end

    subgraph "Analysis & Visualization"
        Analysis["Analysis & Visualization"]:::compute
    end

    subgraph "Monitoring & Apps" 
        Apps["Monitoring & Apps"]:::optional
    end

    subgraph "Scripts & Helpers"
        Scripts["Experiment Helpers"]:::compute
    end

    %% Data Flows
    CLI -->|"start experiment"| QPool
    CLI -->|uses| DM
    CLI -->|"invoke pipeline"| BP

    QPool -->|questions| BP
    DM -->|schemas| BP

    BP -->|"generate answers"| HFLM
    BP -->|"generate answers"| OAIM

    HFLM -->|"answers"| Cache
    OAIM -->|"answers"| Cache

    BP -->|"fetch Q&A"| Cache
    BP -->|"send to judge"| Judger

    Judger -->|"battle outcome"| Elo
    BP -->|"battle outcome"| Elo

    Elo -->|"update ratings"| Results
    BP -->|"store battle records"| Results

    Results -->|"read results"| Analysis
    Results -->|"subscribe"| Apps
    Cache -->|"subscribe"| Apps

    Scripts -->|"register models/questions"| CLI
    Scripts -->|"auxiliary tasks"| BP

    %% Click Events
    click CLI "https://github.com/v-xchen-v/elobench/blob/main/run_experiment.py"
    click QPool "https://github.com/v-xchen-v/elobench/tree/main/data/"
    click HFLM "https://github.com/v-xchen-v/elobench/blob/main/models/hf_local_lm.py"
    click OAIM "https://github.com/v-xchen-v/elobench/blob/main/models/gpt_online_lm.py"
    click OAIM "https://github.com/v-xchen-v/elobench/blob/main/openai_chat.py"
    click BP "https://github.com/v-xchen-v/elobench/blob/main/pipe/battle_pipe.py"
    click IBP "https://github.com/v-xchen-v/elobench/blob/main/pipe/iterative_battle_pipe.py"
    click Cache "https://github.com/v-xchen-v/elobench/tree/main/tempcache/"
    click Judger "https://github.com/v-xchen-v/elobench/blob/main/judger/gpt_judger.py"
    click Elo "https://github.com/v-xchen-v/elobench/tree/main/elo_rating/"
    click Results "https://github.com/v-xchen-v/elobench/tree/main/results/"
    click Analysis "https://github.com/v-xchen-v/elobench/tree/main/battle_outcome_analysis/"
    click Analysis "https://github.com/v-xchen-v/elobench/tree/main/plots/"
    click Analysis "https://github.com/v-xchen-v/elobench/blob/main/run_analysis.py"
    click Apps "https://github.com/v-xchen-v/elobench/tree/main/apps/"
    click Apps "https://github.com/v-xchen-v/elobench/tree/main/apps/monitors/"
    click DM "https://github.com/v-xchen-v/elobench/tree/main/datamodel/"
    click Scripts "https://github.com/v-xchen-v/elobench/tree/main/scripts/"

    %% Styles
    classDef compute fill:#E0F7FA,stroke:#006064,color:#006064;
    classDef store fill:#E8F5E9,stroke:#2E7D32,color:#2E7D32,stroke-width:2px;
    classDef external fill:#FFF3E0,stroke:#EF6C00,color:#E65100,stroke-dasharray: 5 5;
    classDef optional fill:#F3E5F5,stroke:#7B1FA2,color:#4A148C,stroke-dasharray: 2 2;
```
