# wandb/llm-leaderboard architecture

> Architecture diagram of the GitHub repository wandb/llm-leaderboard, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wandb/llm-leaderboard
- Repository: https://github.com/wandb/llm-leaderboard

## Overview

##### Project Type and Purpose
- The repository is an evaluation platform for large language models (LLMs) – Nejumi Leaderboard 3 – that benchmarks general language performance as well as alignment and other nuanced evaluation metrics.
- The README describes various evaluation metrics (e.g., translation, summarization, reasoning, toxicity, truthfulness) and details how different datasets (jaster, MT-Bench, LCTG, JBBQ, etc.) are utilized.
- The project provides configuration files (in the configs directory) for diverse evaluation scenarios and models (both API-based and locally hosted) and uses chat templates to control the evaluation input structure.

##### File Structure
- Top-Level Components:
- “chat_templates”: Contains various Jinja templates to format prompts and conversation scripts for different models. These dictate how interactions with LLMs are structured.
- “configs”: Contains YAML configuration files that set up evaluation parameters, model details, and dataset paths. The existence of a “base_config.yaml” guides uniformity across model-specific settings.
- “scripts”: Organized into several submodules:
- “data_uploader”: Modules that handle downloading/uploading datasets.
- “evaluator”: Contains scripts to calculate metrics, aggregate results, and execute various evaluation routines (e.g., MTbench, Jaster, toxicity, etc.).
- “fastchat”: Implements functions related to conversation management, model judgment, and interaction pipelines.
- Utility and support modules (e.g., blend_run.py, llm_inference_adapter.py) that enable running different parts of the system.
- Supporting files such as “requirements.txt”, .gitignore, and various blend_run_configs indicate the integration with third-party services like WandB, as well as custom environment scripts.

a. Main Components of the System:
- User Interface / Entry-Point: Although not a formal UI, the central evaluation runner (scripts/run_eval.py) acts as the orchestrator.
- Configuration Management:
- Base and Model-specific Configurations (configs directory)
- Environment Variable Setup (as explained in the README)
- Data Preparation and Upload:
- Data Uploader Modules (scripts/data_uploader)
- Dataset artifacts (jaster, MT-Bench, etc.)
- Evaluation Engine:
- Evaluator Modules (scripts/evaluator) that load datasets, run evaluation scripts, and aggregate metrics.
- Metrics Computing Components (e.g., lctg_utils, toxicity, aggregate scripts)
- Chat Engine:
- Chat Templates (chat_templates) that structure the input and prompt construction.
- FastChat Components (scripts/fastchat) for managing conversations and asynchronous LLM processing.
- Model Integration:
- Model adapters (found in scripts/fastchat/model, llm_inference_adapter.py)
- API configurations for external models (in configs/ and documented in the README)
- Logging and Monitoring:
- Integration with WandB for reporting runs and artifact management (indicated in config files and the README)

b. Relationships and Interactions:
- Configuration files provide the parameters to the Evaluation Engine, Data Uploader, and Chat Engine.
- The Evaluation Runner (scripts/run_eval.py) invokes evaluator modules which in turn fetch dataset inputs (using scripts/data_uploader) and format these requests via chat templates.
- The fastchat module works with LLM model adapters to invoke either API-based models or VLLM models (depending on configuration). The results then flow back to the evaluator.
- Results and logs are forwarded to WandB as defined in the config files (base_config.yaml and model-specific configs).

c. Architectural Patterns and Design Principles:
- Modular Design: Each major functionality (data upload, evaluation, chat interface) is encapsulated in its own module for clarity and ease of extension.
- Configuration-Driven Setup: Separation of base configurations and model-specific YAML files allows flexible runtime behavior.
- Extensibility: The system’s use of Jinja-based templates and separate evaluator scripts enhances the ability to add new evaluation tasks or model integrations.
- Pipeline Orchestration: The system is structured as a pipeline where input data flows through configuration, formatting (via chat templates), evaluation, and logging to WandB.

d. Relevant Technologies / Frameworks:
- Python for core scripting and evaluation logic.
- YAML for configuration management.
- Jinja templating for building chat templates.
- WandB (Weights & Biases) for experiment tracking.
- API integrations (OpenAI, Anthropic, Google, etc.) and vLLM for model inference.

##### Elements
- “Config Manager” (configs, environment variables)
- “Data Uploader” (scripts/data_uploader)
- “Evaluator Engine” (scripts/evaluator and submodules)
- “Chat Template Manager” (chat_templates)
- “FastChat Module” (scripts/fastchat)
- “Model Adapter / Inference” (scripts/fastchat/model, llm_inference_adapter)
- “Result Logger / WandB Connector”

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input/Configuration Layer
    subgraph "Input/Configuration"
        UI["User Interface / Entry-Point"]:::input
        Config["Config Manager"]:::input
        ChatTemp["Chat Template Manager"]:::input
    end

    %% Processing Layer
    subgraph "Processing"
        DataUpload["Data Uploader"]:::processing
        EvalEngine["Evaluator Engine"]:::processing
        FastChat["FastChat Module"]:::processing
        subgraph "Model Integration"
            FastChatAdapter["FastChat Adapter"]:::processing
            LLMAdapter["LLM Inference Adapter"]:::processing
        end
    end

    %% Output/Monitoring Layer
    subgraph "Output/Monitoring"
        ResultLogger["Result Logger / WandB Connector"]:::output
    end

    %% Data Flow
    UI -->|"triggers"| EvalEngine
    Config -->|"configures"| EvalEngine
    Config -->|"configures"| DataUpload
    Config -->|"configures"| FastChat
    DataUpload -->|"feeds data"| EvalEngine
    ChatTemp -->|"formats prompts"| FastChat
    FastChat -->|"invokes"| FastChatAdapter
    FastChat -->|"invokes"| LLMAdapter
    FastChatAdapter -->|"returns results"| EvalEngine
    LLMAdapter -->|"returns results"| EvalEngine
    EvalEngine -->|"logs outputs"| ResultLogger

    %% Click Events
    click UI "https://github.com/wandb/llm-leaderboard/blob/main/scripts/run_eval.py"
    click Config "https://github.com/wandb/llm-leaderboard/tree/main/configs"
    click DataUpload "https://github.com/wandb/llm-leaderboard/tree/main/scripts/data_uploader"
    click EvalEngine "https://github.com/wandb/llm-leaderboard/tree/main/scripts/evaluator"
    click ChatTemp "https://github.com/wandb/llm-leaderboard/tree/main/chat_templates"
    click FastChat "https://github.com/wandb/llm-leaderboard/tree/main/scripts/fastchat"
    click FastChatAdapter "https://github.com/wandb/llm-leaderboard/tree/main/scripts/fastchat/model"
    click LLMAdapter "https://github.com/wandb/llm-leaderboard/blob/main/scripts/llm_inference_adapter.py"

    %% Styles
    classDef input fill:#AEDFF7,stroke:#1F77B4,stroke-width:2px;
    classDef processing fill:#C1F0C1,stroke:#1F7A1F,stroke-width:2px;
    classDef output fill:#FAD4A5,stroke:#D17A2B,stroke-width:2px;
```
