How ayushi-uwc/elo-benchmark works
Project type and purpose
• UNER is a modular Python-based evaluation framework (CLI/daemon) for large-scale generative AI models. It orchestrates an eight-stage pipeline—model registration, pairing, prompt generation, solution generation, peer evaluation, scoring, rating propagation, and logging—while integrating with external LLM providers and a MongoDB backend.
File structure
• Top-level scripts and modules:
• main.py – entry point, parses CLI args and kicks off evaluation cycles
• tournament.py – orchestrates match lifecycles and stage sequencing
• matches.py – houses prompt templates, match logic, and peer-judging flows
• models.py & model_definitions.py – model abstractions and cost/pricing metadata
• database.py – MongoDB interaction layer
• config.py, litellm_config.py, env.template – environment and framework configuration
• logger_config.py – centralized logging setup
• check_models.py, matches.py, tournament.py – utility and test scaffolding
• Supporting files: requirements.txt, LICENSE, .gitignore, .gradio (for optional Gradio UI), README.md
Key insights from README
• Eight-stage dynamic evaluation pipeline
• Dual-track Elo ratings (raw and cost-adjusted)
• External dependencies: multiple LLM providers (OpenAI, Anthropic, Google Gemini, HuggingFace, Groq, etc.)
• MongoDB for immutable, auditable logs and leaderboards
• CLI options for batching, stats, log levels, dependency checks
Main components
a. Orchestrator Layer
– CLI / main.py (parses arguments, starts/stops runs)
– tournament.py (pipeline coordinator)
1. Model Registration (model_definitions.py → metadata loader)
2. Fair Matching (Swiss pairing, stratified sampling)
3. Challenge Creation (prompt generation via matches.py)
4. Solution Generation (models.py + external LLM APIs)
5. Peer Evaluation (weighted voting logic in matches.py)
6. Dual Scoring & Elo Updates (Elo formulae in tournament.py)
7. Rating Propagation (Gaussian kernel smoothing)
8. Logging & Dissemination (database.py → MongoDB, JSON logs, leaderboards)
c. Infrastructure & Supporting Services
– External LLM Providers (OpenAI, Anthropic, Google, etc.)
– MongoDB cluster (persistent storage of metadata, logs, leaderboards)
– Logging sink (local files, or external log aggregation)
– (Optional) Gradio UI component loading .gradio/certificate.pem
d. Configuration & Utilities
– config.py / litellm_config.py / env variables
– logger_config.py
Relationships and interactions
• CLI → Orchestrator → Pipeline stages
• Pipeline modules ↔ Configuration
• Pipeline modules → External LLM APIs (requests/responses with cost & latency)
• Pipeline modules → MongoDB (write artifacts: prompts, responses, votes, Elo deltas)
• Logging module → file system / log aggregator
• Orchestrator → Gradio UI (if enabled)
• Directionality:
• Bidirectional with config and DB
Architectural patterns & design principles
• Modular pipeline: each stage is an independent component called by a central orchestrator
• Separation of concerns:
• Model abstraction vs. match logic vs. data persistence
• Configuration isolated from code (env.template → config.py)
• Extensibility: new models added via model_definitions.py, new providers pluggable in models.py
• Immutable logging for auditability (append-only JSON + MongoDB)
• Weighted voting and dual-track Elo encapsulated in domain modules
Technologies & frameworks
• Python 3.8+
• Pymongo (MongoDB driver)
• LLM client libraries (OpenAI, Anthropic, HuggingFace, etc.)
• Logging (standard logging module via logger_config.py)
• Gradio (optional UI)
• YAML or JSON for model definitions/config
Detailed capture of essential elements
• Peer-federated judging as a subflow where models call back into matches.py
• Immutable logging to MongoDB as final stage