# eleutherai/lm-evaluation-harness architecture

> Architecture diagram of the GitHub repository eleutherai/lm-evaluation-harness, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eleutherai/lm-evaluation-harness
- Repository: https://github.com/eleutherai/lm-evaluation-harness

## Overview

LM Evaluation Harness runs language-model benchmarks through a configurable CLI or Python evaluation entry point. Users select tasks and a model backend; task definitions load dataset documents, create prompts and requests, and the evaluator obtains model responses, applies task-specific processing and metrics, then returns results and optionally samples. The tree and README document broad model, task, filter, plugin, caching, decontamination, and logging capabilities; sampled excerpts confirm portions of task loading, evaluation, model inference, and result tracking, but not every subsystem’s internal wiring.

Main entry points: [`lm_eval/evaluator.py`](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/evaluator.py) (Evaluation engine), [`lm_eval/api/task.py`](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/api/task.py) (Task definitions), [`lm_eval/_cli/run.py`](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/_cli/run.py) (Run command), [`lm_eval/tasks/manager.py`](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/tasks/manager.py) (Task manager)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry and orchestration"]
  node_cli["CLI<br/>[__main__.py]"]
  node_run["Run command<br/>[run.py]"]
  node_evaluator["Evaluation engine<br/>[evaluator.py]"]
end

subgraph group_tasks["Task definitions"]
  node_manager["Task manager<br/>[manager.py]"]
  node_factory["Task factory<br/>[_factory.py]"]
  node_index["Task index<br/>[_index.py]"]
  node_task["Task definitions<br/>[task.py]"]
  node_tasklibrary["Benchmark library"]
  node_prompts["Prompt templates<br/>[__init__.py]"]
end

subgraph group_inference["Model inference"]
  node_modelapi["Model interface<br/>[model.py]"]
  node_backends["Model backends"]
end

subgraph group_processing["Evaluation processing"]
  node_filter["Response filters<br/>[__init__.py]"]
  node_metrics["Metrics<br/>[metrics.py]"]
  node_decontam["Decontamination<br/>[decontaminate.py]"]
  node_cache[("Request cache<br/>[cache.py]")]
end

subgraph group_results["Results and integrations"]
  node_tracker["Evaluation tracker"]
  node_trackers["Tracking integrations"]
  node_result["Evaluation results<br/>[result_schema.py]"]
  node_plugins["Plugin registry<br/>[registry.py]"]
end

node_user(("Evaluator"))
node_externalmodel(("Model service"))

node_user -->|"invokes"| node_cli
node_cli -->|"dispatches"| node_run
node_run -->|"starts evaluation"| node_evaluator
node_evaluator -->|"loads tasks"| node_manager
node_manager -->|"looks up"| node_index
node_manager -->|"builds tasks"| node_factory
node_factory -->|"instantiates"| node_task
node_task -->|"uses definitions"| node_tasklibrary
node_evaluator -->|"builds requests"| node_task
node_task -->|"formats prompts"| node_prompts
node_evaluator -->|"submits requests"| node_modelapi
node_modelapi -->|"dispatches inference"| node_backends
node_backends -.->|"queries"| node_externalmodel
node_evaluator -->|"applies filters"| node_filter
node_evaluator -->|"computes metrics"| node_metrics
node_evaluator -->|"checks overlap"| node_decontam
node_modelapi -->|"reads and writes"| node_cache
node_evaluator -->|"records evaluation"| node_tracker
node_tracker -->|"exports to"| node_trackers
node_evaluator -->|"produces"| node_result
node_run -.->|"discovers plugins"| node_plugins

click node_cli "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/__main__.py"
click node_run "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/_cli/run.py"
click node_evaluator "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/evaluator.py"
click node_manager "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/tasks/manager.py"
click node_factory "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/tasks/_factory.py"
click node_index "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/tasks/_index.py"
click node_task "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/api/task.py"
click node_tasklibrary "https://github.com/eleutherai/lm-evaluation-harness/tree/main/lm_eval/tasks"
click node_prompts "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/prompts/__init__.py"
click node_modelapi "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/api/model.py"
click node_backends "https://github.com/eleutherai/lm-evaluation-harness/tree/main/lm_eval/models"
click node_filter "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/filters/__init__.py"
click node_metrics "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/api/metrics.py"
click node_decontam "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/decontamination/decontaminate.py"
click node_cache "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/caching/cache.py"
click node_tracker "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/loggers/evaluation_tracker.py"
click node_trackers "https://github.com/eleutherai/lm-evaluation-harness/tree/main/lm_eval/loggers"
click node_result "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/result_schema.py"
click node_plugins "https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/api/registry.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_run,node_evaluator toneBlue
class node_manager,node_factory,node_index,node_task,node_tasklibrary,node_prompts toneAmber
class node_modelapi,node_backends toneMint
class node_filter,node_metrics,node_decontam,node_cache toneRose
class node_tracker,node_trackers,node_result,node_plugins,node_user,node_externalmodel toneIndigo
```

## Components

### Entry and orchestration

- [CLI](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/__main__.py): `lm_eval/__main__.py`
- [Run command](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/_cli/run.py): `lm_eval/_cli/run.py`
- [Evaluation engine](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/evaluator.py): `lm_eval/evaluator.py`

### Task definitions

- [Task manager](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/tasks/manager.py): `lm_eval/tasks/manager.py`
- [Task factory](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/tasks/_factory.py): `lm_eval/tasks/_factory.py`
- [Task index](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/tasks/_index.py): `lm_eval/tasks/_index.py`
- [Task definitions](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/api/task.py): `lm_eval/api/task.py`
- [Benchmark library](https://github.com/eleutherai/lm-evaluation-harness/tree/main/lm_eval/tasks): `lm_eval/tasks`
- [Prompt templates](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/prompts/__init__.py): `lm_eval/prompts/__init__.py`

### Model inference

- [Model interface](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/api/model.py): `lm_eval/api/model.py`
- [Model backends](https://github.com/eleutherai/lm-evaluation-harness/tree/main/lm_eval/models): `lm_eval/models`

### Evaluation processing

- [Response filters](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/filters/__init__.py): `lm_eval/filters/__init__.py`
- [Metrics](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/api/metrics.py): `lm_eval/api/metrics.py`
- [Decontamination](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/decontamination/decontaminate.py): `lm_eval/decontamination/decontaminate.py`
- [Request cache](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/caching/cache.py): `lm_eval/caching/cache.py`

### Results and integrations

- [Evaluation tracker](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/loggers/evaluation_tracker.py): `lm_eval/loggers/evaluation_tracker.py`
- [Tracking integrations](https://github.com/eleutherai/lm-evaluation-harness/tree/main/lm_eval/loggers): `lm_eval/loggers`
- [Evaluation results](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/result_schema.py): `lm_eval/result_schema.py`
- [Plugin registry](https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/api/registry.py): `lm_eval/api/registry.py`

### Other components

- **Evaluator**
- **Model service**

## Connections

- Evaluator → CLI: invokes
- CLI → Run command: dispatches
- Run command → Evaluation engine: starts evaluation
- Evaluation engine → Task manager: loads tasks
- Task manager → Task index: looks up
- Task manager → Task factory: builds tasks
- Task factory → Task definitions: instantiates
- Task definitions → Benchmark library: uses definitions
- Evaluation engine → Task definitions: builds requests
- Task definitions → Prompt templates: formats prompts
- Evaluation engine → Model interface: submits requests
- Model interface → Model backends: dispatches inference
- Model backends → Model service: queries
- Evaluation engine → Response filters: applies filters
- Evaluation engine → Metrics: computes metrics
- Evaluation engine → Decontamination: checks overlap
- Model interface → Request cache: reads and writes
- Evaluation engine → Evaluation tracker: records evaluation
- Evaluation tracker → Tracking integrations: exports to
- Evaluation engine → Evaluation results: produces
- Run command → Plugin registry: discovers plugins