How openai/simple-evals works
Project type and purpose
• This is an open-source, command-line Python library for running standard benchmarks (“evals”) against large language models and reporting accuracy.
• Purpose: Provide a lightweight, extensible framework to run zero-shot and chain-of-thought evaluations (MMLU, MATH, GPQA, etc.) via different model APIs (OpenAI, Claude).
File structure
• Top-level Python modules (e.g., mmlu_eval.py, math_eval.py, …): one module per benchmark.
• simple_evals.py (entry point): discovers evals, parses CLI args, orchestrates runs.
• common.py, types.py: shared types, data-structures, helper functions.
• sampler/ (package): abstraction layer for model sampling, with concrete classes for OpenAI chat completion, Claude, etc.
• healthbench_scripts/: Jupyter notebook for analysis of HealthBench results (ancillary).
• Configuration: .gitignore, LICENSE, README.md.
README for additional insights
• CLI usage: python -m simple-evals.simple_evals --model <name> --examples <n>
• Supports injecting different samplers via environment variables (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY).
• Evals discover benchmark definitions, load prompt templates, call sampler to get model outputs, compute metrics (accuracy, F1).