How v-xchen-v/elobench works
Project Type and Purpose
• Type: Open-source Python framework/tool for evaluating and ranking large language models (LLMs) via an Elo rating system.
• Purpose: Automate pairwise “battles” between LLMs on real-world QA tasks, use GPT-4 as a judge, compute and analyze Elo ratings, and produce leaderboards and visualizations.
File Structure
• Top-level modules:
• apps/ – Standalone utilities and GUI-style monitors.
• data/ – Question pools, preparation scripts, and arena data.
• models/, openai_chat.py, judger/ – LLM interface implementations (local HF models, OpenAI API, GPT-4 judger).
• pipe/ – Battle orchestration pipelines (battle_pipe.py, iterative_battle_pipe.py).
• elo_rating/ – Elo rating core logic and simulation.
• battle_outcome_analysis/ & run_analysis.py – Post-battle analysis and plotting.
• run_experiment.py – End-to-end execution script (question selection, answer generation, judging, rating updates).
• tempcache/ & results/ – Caching layer and persistent storage for experiments.
• datamodel/ – Data classes for battles, records, ratings, model collections.
• scripts/ & exploratory_analysis/ & plots/ – Ad-hoc exploratory notebooks and scripts for research.
Additional Insights from README
• Workflows:
• Experiment setup (register models, questions, directories).
• Running battles (run_experiment.py).
• Analyzing results (run_analysis.py, battle_outcome_analysis).
• Technologies: Python 3.9+, NumPy, Torch, Transformers, Datasets, Accelerate, OpenAI API.
Components
a. Main Components
1. CLI Orchestrator (run_experiment.py)
2. Question Pool (data/)
3. Model Integration Layer
– HF Local Models (models/hf_local_lm.py, accelerate)
– OpenAI API Models (models/gpt_online_lm.py, openai_chat.py)
4. Battle Pipeline (pipe/battle_pipe.py, pipe/iterative_battle_pipe.py)
5. Caching Layer (tempcache/)
6. Judging Service (judger/gpt_judger.py using GPT-4)
7. Elo Engine (elo_rating/)
8. Persistent Store (results/)
9. Analysis & Visualization (battle_outcome_analysis/, plots/, run_analysis.py)
10. Monitoring & Apps (apps/monitors, apps/*.py)
b. Relationships and Data Flows
1. CLI Orchestrator reads model list and question list → feeds to Battle Pipeline.
2. Battle Pipeline invokes Model Integration to generate answers → writes to cache.
3. Cached answers + cached judgments → passed to Judging Service → outputs battle outcome.
4. Outcome sent to Elo Engine → updates ratings → persisted in results/.
5. run_analysis.py & battle_outcome_analysis read results/ → compute stats and plots.
6. apps/monitoring scripts subscribe to results/ and cache to display live metrics.
c. Architectural Patterns & Principles
• Pipeline Architecture: clear separation between data ingestion (questions), processing (answers, judging), and post-processing (rating, analysis).
• Modular LLM Interface: abstraction layer for plugging new models or endpoints.
• Caching for Idempotence: avoid repeated invocations of expensive LLM or GPT-4 calls.
• Data-centric design: uses datamodel classes to enforce consistent record formats.
d. Key Technologies and Frameworks
• Python ecosystem: NumPy, pandas (implicit in scripts/notebooks), PyTorch + Transformers + Accelerate.
• OpenAI API for GPT-4 judging.
• Structured storage: CSV/JSON for questions, battles, ratings.
• Jupyter notebooks for exploratory analysis and plots (Matplotlib/Seaborn likely).