How wandb/llm-leaderboard works
Project Type and Purpose
• The repository is an evaluation platform for large language models (LLMs) – Nejumi Leaderboard 3 – that benchmarks general language performance as well as alignment and other nuanced evaluation metrics.
• The README describes various evaluation metrics (e.g., translation, summarization, reasoning, toxicity, truthfulness) and details how different datasets (jaster, MT-Bench, LCTG, JBBQ, etc.) are utilized.
• The project provides configuration files (in the configs directory) for diverse evaluation scenarios and models (both API-based and locally hosted) and uses chat templates to control the evaluation input structure.
File Structure
• Top-Level Components:
• “chat_templates”: Contains various Jinja templates to format prompts and conversation scripts for different models. These dictate how interactions with LLMs are structured.
• “configs”: Contains YAML configuration files that set up evaluation parameters, model details, and dataset paths. The existence of a “base_config.yaml” guides uniformity across model-specific settings.
• “scripts”: Organized into several submodules:
• “data_uploader”: Modules that handle downloading/uploading datasets.
• “evaluator”: Contains scripts to calculate metrics, aggregate results, and execute various evaluation routines (e.g., MTbench, Jaster, toxicity, etc.).
• “fastchat”: Implements functions related to conversation management, model judgment, and interaction pipelines.
• Utility and support modules (e.g., blend_run.py, llm_inference_adapter.py) that enable running different parts of the system.
• Supporting files such as “requirements.txt”, .gitignore, and various blend_run_configs indicate the integration with third-party services like WandB, as well as custom environment scripts.
a. Main Components of the System:
• User Interface / Entry-Point: Although not a formal UI, the central evaluation runner (scripts/run_eval.py) acts as the orchestrator.
• Configuration Management:
• Base and Model-specific Configurations (configs directory)
• Environment Variable Setup (as explained in the README)
• Data Preparation and Upload:
• Data Uploader Modules (scripts/data_uploader)
• Dataset artifacts (jaster, MT-Bench, etc.)
• Evaluation Engine:
• Evaluator Modules (scripts/evaluator) that load datasets, run evaluation scripts, and aggregate metrics.
• Metrics Computing Components (e.g., lctg_utils, toxicity, aggregate scripts)
• Chat Engine:
• Chat Templates (chat_templates) that structure the input and prompt construction.
• FastChat Components (scripts/fastchat) for managing conversations and asynchronous LLM processing.
• Model Integration:
• Model adapters (found in scripts/fastchat/model, llm_inference_adapter.py)
• API configurations for external models (in configs/ and documented in the README)
• Logging and Monitoring:
• Integration with WandB for reporting runs and artifact management (indicated in config files and the README)
b. Relationships and Interactions:
• Configuration files provide the parameters to the Evaluation Engine, Data Uploader, and Chat Engine.
• The Evaluation Runner (scripts/run_eval.py) invokes evaluator modules which in turn fetch dataset inputs (using scripts/data_uploader) and format these requests via chat templates.
• The fastchat module works with LLM model adapters to invoke either API-based models or VLLM models (depending on configuration). The results then flow back to the evaluator.
• Results and logs are forwarded to WandB as defined in the config files (base_config.yaml and model-specific configs).
c. Architectural Patterns and Design Principles:
• Modular Design: Each major functionality (data upload, evaluation, chat interface) is encapsulated in its own module for clarity and ease of extension.
• Configuration-Driven Setup: Separation of base configurations and model-specific YAML files allows flexible runtime behavior.
• Extensibility: The system’s use of Jinja-based templates and separate evaluator scripts enhances the ability to add new evaluation tasks or model integrations.
• Pipeline Orchestration: The system is structured as a pipeline where input data flows through configuration, formatting (via chat templates), evaluation, and logging to WandB.
d. Relevant Technologies / Frameworks:
• Python for core scripting and evaluation logic.
• YAML for configuration management.
• Jinja templating for building chat templates.
• WandB (Weights & Biases) for experiment tracking.
• API integrations (OpenAI, Anthropic, Google, etc.) and vLLM for model inference.
Elements
• “Config Manager” (configs, environment variables)
• “Data Uploader” (scripts/data_uploader)
• “Evaluator Engine” (scripts/evaluator and submodules)
• “Chat Template Manager” (chat_templates)
• “FastChat Module” (scripts/fastchat)
• “Model Adapter / Inference” (scripts/fastchat/model, llm_inference_adapter)
• “Result Logger / WandB Connector”