How eleutherai/lm-evaluation-harness works
LM Evaluation Harness runs language-model benchmarks through a configurable CLI or Python evaluation entry point. Users select tasks and a model backend; task definitions load dataset documents, create prompts and requests, and the evaluator obtains model responses, applies task-specific processing and metrics, then returns results and optionally samples. The tree and README document broad model, task, filter, plugin, caching, decontamination, and logging capabilities; sampled excerpts confirm portions of task loading, evaluation, model inference, and result tracking, but not every subsystem’s internal wiring.
Main entry points: evaluator.py Evaluation engine · task.py Task definitions · run.py Run command · manager.py Task manager
Components 21
Entry and orchestration
- CLI
lm_eval/__main__.py - Run command
lm_eval/_cli/run.py - Evaluation engine
lm_eval/evaluator.py
Task definitions
- Task manager
lm_eval/tasks/manager.py - Task factory
lm_eval/tasks/_factory.py - Task index
lm_eval/tasks/_index.py - Task definitions
lm_eval/api/task.py - Benchmark library
lm_eval/tasks - Prompt templates
lm_eval/prompts/__init__.py
Model inference
- Model interface
lm_eval/api/model.py - Model backends
lm_eval/models
Evaluation processing
- Response filters
lm_eval/filters/__init__.py - Metrics
lm_eval/api/metrics.py - Decontamination
lm_eval/decontamination/decontaminate.py - Request cache
lm_eval/caching/cache.py
Results and integrations
- Evaluation tracker
lm_eval/loggers/evaluation_tracker.py - Tracking integrations
lm_eval/loggers - Evaluation results
lm_eval/result_schema.py - Plugin registry
lm_eval/api/registry.py
Other components
- Evaluator
- Model service
Connections 21
- Evaluator → CLI: invokes
- CLI → Run command: dispatches
- Run command → Evaluation engine: starts evaluation
- Evaluation engine → Task manager: loads tasks
- Task manager → Task index: looks up
- Task manager → Task factory: builds tasks
- Task factory → Task definitions: instantiates
- Task definitions → Benchmark library: uses definitions
- Evaluation engine → Task definitions: builds requests
- Task definitions → Prompt templates: formats prompts
- Evaluation engine → Model interface: submits requests
- Model interface → Model backends: dispatches inference
- Model backends → Model service: queries
- Evaluation engine → Response filters: applies filters
- Evaluation engine → Metrics: computes metrics
- Evaluation engine → Decontamination: checks overlap
- Model interface → Request cache: reads and writes
- Evaluation engine → Evaluation tracker: records evaluation
- Evaluation tracker → Tracking integrations: exports to
- Evaluation engine → Evaluation results: produces
- Run command → Plugin registry: discovers plugins