# evolvinglmms-lab/lmms-eval architecture

> Architecture diagram of the GitHub repository evolvinglmms-lab/lmms-eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/evolvinglmms-lab/lmms-eval
- Repository: https://github.com/evolvinglmms-lab/lmms-eval

## Overview

LMMs-Eval is a multimodal model evaluation toolkit: users select a model backend and benchmark tasks, run inference over task examples, score the outputs, and inspect logged results. The central evaluation path is represented alongside model/task discovery, caching, and the documented HTTP, web UI, and MCP service interfaces. The README documents broad audio, video, and statistical-evaluation capabilities; the tree shows their likely implementation areas, but most of those sources were not sampled, so their internal wiring is not asserted here.

Main entry points: [`lmms_eval/evaluator.py`](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/evaluator.py) (Evaluation Engine), [`lmms_eval/entrypoints/job_scheduler.py`](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/entrypoints/job_scheduler.py) (Job Scheduler), [`lmms_eval/models`](https://github.com/evolvinglmms-lab/lmms-eval/tree/main/lmms_eval/models) (Model Backends), [`lmms_eval/cli/dispatch.py`](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/cli/dispatch.py) (CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_run["Evaluation Runtime"]
  node_cli["CLI<br/>[dispatch.py]"]
  node_engine["Evaluation Engine<br/>[evaluator.py]"]
  node_cache[("Response Cache<br/>[response_cache.py]")]
end

subgraph group_catalog["Models and Tasks"]
  node_task_api["Task API<br/>[task.py]"]
  node_tasks["Benchmark Tasks"]
  node_model_api["Model API<br/>[model.py]"]
  node_model_registry["Model Registry<br/>[registry_v2.py]"]
  node_model_backends["Model Backends"]
end

subgraph group_results["Scoring and Results"]
  node_filtering["Response Filters"]
  node_metrics["Metrics<br/>[metrics.py]"]
  node_verification["Verification<br/>[pipeline.py]"]
  node_logger["Result Logging"]
end

subgraph group_services["Serving Interfaces"]
  node_http_client["HTTP Client<br/>[client.py]"]
  node_http_server["HTTP Server<br/>[http_server.py]"]
  node_scheduler["Job Scheduler<br/>[job_scheduler.py]"]
  node_ui["Web UI<br/>[App.tsx]"]
  node_ui_backend["UI Backend<br/>[server.py]"]
  node_mcp_client["MCP Client<br/>[client.py]"]
  node_mcp_server["MCP Server<br/>[server.py]"]
  node_mcp_tools["MCP Tools<br/>[tools.py]"]
end

node_user(("Evaluator"))
node_model_service{{"Model Service"}}

node_user -->|"selects run"| node_cli
node_cli -->|"starts evaluation"| node_engine
node_engine -->|"loads tasks"| node_task_api
node_task_api -->|"resolves benchmark"| node_tasks
node_engine -->|"resolves backend"| node_model_registry
node_model_registry -->|"selects implementation"| node_model_backends
node_engine -->|"requests inference"| node_model_api
node_model_api -->|"dispatches generation"| node_model_backends
node_model_backends -.->|"requests inference"| node_model_service
node_engine -->|"reads and writes"| node_cache
node_engine -->|"processes responses"| node_filtering
node_engine -->|"computes scores"| node_metrics
node_engine -->|"records results"| node_logger
node_engine -.->|"verifies outputs"| node_verification
node_user -->|"submits job"| node_http_client
node_http_client -->|"sends requests"| node_http_server
node_http_server -->|"queues jobs"| node_scheduler
node_scheduler -->|"runs evaluation"| node_engine
node_user -->|"configures evaluation"| node_ui
node_ui -->|"calls API"| node_ui_backend
node_ui_backend -->|"manages jobs"| node_scheduler
node_user -->|"requests tool"| node_mcp_client
node_mcp_client -->|"calls tool"| node_mcp_server
node_mcp_server -->|"registers tools"| node_mcp_tools
node_mcp_tools -->|"submits and tracks"| node_scheduler

click node_cli "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/cli/dispatch.py"
click node_engine "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/evaluator.py"
click node_task_api "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/api/task.py"
click node_tasks "https://github.com/evolvinglmms-lab/lmms-eval/tree/main/lmms_eval/tasks"
click node_model_api "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/api/model.py"
click node_model_registry "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/models/registry_v2.py"
click node_model_backends "https://github.com/evolvinglmms-lab/lmms-eval/tree/main/lmms_eval/models"
click node_filtering "https://github.com/evolvinglmms-lab/lmms-eval/tree/main/lmms_eval/filters"
click node_metrics "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/api/metrics.py"
click node_verification "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/verifiers/pipeline.py"
click node_logger "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/loggers/evaluation_tracker.py"
click node_cache "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/caching/response_cache.py"
click node_http_client "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/entrypoints/client.py"
click node_http_server "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/entrypoints/http_server.py"
click node_scheduler "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/entrypoints/job_scheduler.py"
click node_ui "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/tui/web/src/App.tsx"
click node_ui_backend "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/tui/server.py"
click node_mcp_client "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/mcp/client.py"
click node_mcp_server "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/mcp/server.py"
click node_mcp_tools "https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/mcp/tools.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_engine,node_cache toneBlue
class node_task_api,node_tasks,node_model_api,node_model_registry,node_model_backends toneAmber
class node_filtering,node_metrics,node_verification,node_logger toneMint
class node_http_client,node_http_server,node_scheduler,node_ui,node_ui_backend,node_mcp_client,node_mcp_server,node_mcp_tools toneRose
class node_user,node_model_service toneIndigo
```

## Components

### Evaluation Runtime

- [CLI](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/cli/dispatch.py): `lmms_eval/cli/dispatch.py`
- [Evaluation Engine](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/evaluator.py): `lmms_eval/evaluator.py`
- [Response Cache](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/caching/response_cache.py): `lmms_eval/caching/response_cache.py`

### Models and Tasks

- [Task API](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/api/task.py): `lmms_eval/api/task.py`
- [Benchmark Tasks](https://github.com/evolvinglmms-lab/lmms-eval/tree/main/lmms_eval/tasks): `lmms_eval/tasks`
- [Model API](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/api/model.py): `lmms_eval/api/model.py`
- [Model Registry](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/models/registry_v2.py): `lmms_eval/models/registry_v2.py`
- [Model Backends](https://github.com/evolvinglmms-lab/lmms-eval/tree/main/lmms_eval/models): `lmms_eval/models`

### Scoring and Results

- [Response Filters](https://github.com/evolvinglmms-lab/lmms-eval/tree/main/lmms_eval/filters): `lmms_eval/filters`
- [Metrics](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/api/metrics.py): `lmms_eval/api/metrics.py`
- [Verification](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/verifiers/pipeline.py): `lmms_eval/verifiers/pipeline.py`
- [Result Logging](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/loggers/evaluation_tracker.py): `lmms_eval/loggers/evaluation_tracker.py`

### Serving Interfaces

- [HTTP Client](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/entrypoints/client.py): `lmms_eval/entrypoints/client.py`
- [HTTP Server](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/entrypoints/http_server.py): `lmms_eval/entrypoints/http_server.py`
- [Job Scheduler](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/entrypoints/job_scheduler.py): `lmms_eval/entrypoints/job_scheduler.py`
- [Web UI](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/tui/web/src/App.tsx): `lmms_eval/tui/web/src/App.tsx`
- [UI Backend](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/tui/server.py): `lmms_eval/tui/server.py`
- [MCP Client](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/mcp/client.py): `lmms_eval/mcp/client.py`
- [MCP Server](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/mcp/server.py): `lmms_eval/mcp/server.py`
- [MCP Tools](https://github.com/evolvinglmms-lab/lmms-eval/blob/main/lmms_eval/mcp/tools.py): `lmms_eval/mcp/tools.py`

### Other components

- **Evaluator**
- **Model Service**

## Connections

- Evaluator → CLI: selects run
- CLI → Evaluation Engine: starts evaluation
- Evaluation Engine → Task API: loads tasks
- Task API → Benchmark Tasks: resolves benchmark
- Evaluation Engine → Model Registry: resolves backend
- Model Registry → Model Backends: selects implementation
- Evaluation Engine → Model API: requests inference
- Model API → Model Backends: dispatches generation
- Model Backends → Model Service: requests inference
- Evaluation Engine → Response Cache: reads and writes
- Evaluation Engine → Response Filters: processes responses
- Evaluation Engine → Metrics: computes scores
- Evaluation Engine → Result Logging: records results
- Evaluation Engine → Verification: verifies outputs
- Evaluator → HTTP Client: submits job
- HTTP Client → HTTP Server: sends requests
- HTTP Server → Job Scheduler: queues jobs
- Job Scheduler → Evaluation Engine: runs evaluation
- Evaluator → Web UI: configures evaluation
- Web UI → UI Backend: calls API
- UI Backend → Job Scheduler: manages jobs
- Evaluator → MCP Client: requests tool
- MCP Client → MCP Server: calls tool
- MCP Server → MCP Tools: registers tools
- MCP Tools → Job Scheduler: submits and tracks