# openai/simple-evals architecture

> Architecture diagram of the GitHub repository openai/simple-evals, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openai/simple-evals
- Repository: https://github.com/openai/simple-evals

## Overview

##### Project type and purpose
- This is an open-source, command-line Python library for running standard benchmarks (“evals”) against large language models and reporting accuracy.
- Purpose: Provide a lightweight, extensible framework to run zero-shot and chain-of-thought evaluations (MMLU, MATH, GPQA, etc.) via different model APIs (OpenAI, Claude).

##### File structure
- Top-level Python modules (e.g., mmlu_eval.py, math_eval.py, …): one module per benchmark.
- simple_evals.py (entry point): discovers evals, parses CLI args, orchestrates runs.
- common.py, types.py: shared types, data-structures, helper functions.
- sampler/ (package): abstraction layer for model sampling, with concrete classes for OpenAI chat completion, Claude, etc.
- healthbench_scripts/: Jupyter notebook for analysis of HealthBench results (ancillary).
- Configuration: .gitignore, LICENSE, README.md.

##### README for additional insights
- CLI usage: `python -m simple-evals.simple_evals --model <name> --examples <n>`
- Supports injecting different samplers via environment variables (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY).
- Evals discover benchmark definitions, load prompt templates, call sampler to get model outputs, compute metrics (accuracy, F1).

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "CLI Layer"
        CLI["CLI Interface\n(simple_evals.py)"]:::core
    end

    subgraph "Core Framework"
        Orch["Evaluation Manager / Orchestrator\n(common.py)"]:::core
        Types["Shared Types & Data Structures\n(types.py)"]:::core
    end

    subgraph "Benchmark Plugins"
        MMLU["MMLU Benchmark\n(mmlu_eval.py)"]:::plugin
        MATH["MATH Benchmark\n(math_eval.py)"]:::plugin
        GPQA["GPQA Benchmark\n(gpqa_eval.py)"]:::plugin
        DROP["DROP Benchmark\n(drop_eval.py)"]:::plugin
        HE["HumanEval Benchmark\n(humaneval_eval.py)"]:::plugin
        MGSM["MGSM Benchmark\n(mgsm_eval.py)"]:::plugin
        BC["BrowseComp Benchmark\n(browsecomp_eval.py)"]:::plugin
        SQA["SimpleQA Benchmark\n(simpleqa_eval.py)"]:::plugin
        HB["HealthBench Benchmark\n(healthbench_eval.py)"]:::plugin
        HBM["HealthBench Meta-Eval\n(healthbench_meta_eval.py)"]:::plugin
    end

    subgraph "Sampler Plugins"
        Base["Base Chat Completion Sampler\n(chat_completion_sampler.py)"]:::sampler
        OpenAI["OpenAI Chat Completion Sampler\n(o_chat_completion_sampler.py)"]:::sampler
        Claude["Anthropic Claude Sampler\n(claude_sampler.py)"]:::sampler
        Resp["Response-Based Sampler\n(responses_sampler.py)"]:::sampler
    end

    subgraph "External APIs"
        OpenAIAPI["OpenAI API"]:::external
        ClaudeAPI["Anthropic Claude API"]:::external
    end

    Analysis["HealthBench Analysis Notebook\n(healthbench_scripts/healthbench_analysis.ipynb)"]:::analysis

    CLI -->|"parses arguments"| Orch
    CLI -->|"uses types"| Types
    Orch -->|"loads eval plugins"| MMLU
    Orch --> MATH
    Orch --> GPQA
    Orch --> DROP
    Orch --> HE
    Orch --> MGSM
    Orch --> BC
    Orch --> SQA
    Orch --> HB
    Orch --> HBM
    Orch -->|"uses types"| Types
    MMLU -->|"calls sampler"| Base
    MATH --> Base
    GPQA --> Base
    DROP --> Base
    HE --> Base
    MGSM --> Base
    BC --> Base
    SQA --> Base
    HB --> Base
    HBM --> Base
    Base -->|"extended by"| OpenAI
    Base -->|"extended by"| Claude
    Base -->|"extended by"| Resp
    OpenAI -->|"API request"| OpenAIAPI
    Claude -->|"API request"| ClaudeAPI
    MMLU -->|"returns metrics"| Orch
    MATH --> Orch
    GPQA --> Orch
    DROP --> Orch
    HE --> Orch
    MGSM --> Orch
    BC --> Orch
    SQA --> Orch
    HB --> Orch
    HBM --> Orch
    Orch -->|"aggregates results"| CLI
    CLI -->|"outputs reports"| CLI
    CLI -->|"optional analysis"| Analysis

    click CLI "https://github.com/openai/simple-evals/blob/main/simple_evals.py"
    click Orch "https://github.com/openai/simple-evals/blob/main/common.py"
    click Types "https://github.com/openai/simple-evals/blob/main/types.py"
    click MMLU "https://github.com/openai/simple-evals/blob/main/mmlu_eval.py"
    click MATH "https://github.com/openai/simple-evals/blob/main/math_eval.py"
    click GPQA "https://github.com/openai/simple-evals/blob/main/gpqa_eval.py"
    click DROP "https://github.com/openai/simple-evals/blob/main/drop_eval.py"
    click HE "https://github.com/openai/simple-evals/blob/main/humaneval_eval.py"
    click MGSM "https://github.com/openai/simple-evals/blob/main/mgsm_eval.py"
    click BC "https://github.com/openai/simple-evals/blob/main/browsecomp_eval.py"
    click SQA "https://github.com/openai/simple-evals/blob/main/simpleqa_eval.py"
    click HB "https://github.com/openai/simple-evals/blob/main/healthbench_eval.py"
    click HBM "https://github.com/openai/simple-evals/blob/main/healthbench_meta_eval.py"
    click Base "https://github.com/openai/simple-evals/blob/main/sampler/chat_completion_sampler.py"
    click OpenAI "https://github.com/openai/simple-evals/blob/main/sampler/o_chat_completion_sampler.py"
    click Claude "https://github.com/openai/simple-evals/blob/main/sampler/claude_sampler.py"
    click Resp "https://github.com/openai/simple-evals/blob/main/sampler/responses_sampler.py"
    click Analysis "https://github.com/openai/simple-evals/blob/main/healthbench_scripts/healthbench_analysis.ipynb"

    classDef core fill:#CCE5FF,stroke:#3399FF,color:#003366,stroke-width:2px;
    classDef plugin fill:#E6FFE6,stroke:#33CC33,color:#006600,stroke-width:2px;
    classDef sampler fill:#FFF5CC,stroke:#FFCC00,color:#665500,stroke-width:2px;
    classDef external fill:#FFE6E6,stroke:#FF6666,color:#660000,stroke-width:2px;
    classDef analysis fill:#F0E6FF,stroke:#9933CC,color:#330066,stroke-width:2px;
```
