How evolvinglmms-lab/lmms-eval works
LMMs-Eval is a multimodal model evaluation toolkit: users select a model backend and benchmark tasks, run inference over task examples, score the outputs, and inspect logged results. The central evaluation path is represented alongside model/task discovery, caching, and the documented HTTP, web UI, and MCP service interfaces. The README documents broad audio, video, and statistical-evaluation capabilities; the tree shows their likely implementation areas, but most of those sources were not sampled, so their internal wiring is not asserted here.
Main entry points: evaluator.py Evaluation Engine · job_scheduler.py Job Scheduler · models Model Backends · dispatch.py CLI
Components 22
Evaluation Runtime
- CLI
lmms_eval/cli/dispatch.py - Evaluation Engine
lmms_eval/evaluator.py - Response Cache
lmms_eval/caching/response_cache.py
Models and Tasks
- Task API
lmms_eval/api/task.py - Benchmark Tasks
lmms_eval/tasks - Model API
lmms_eval/api/model.py - Model Registry
lmms_eval/models/registry_v2.py - Model Backends
lmms_eval/models
Scoring and Results
- Response Filters
lmms_eval/filters - Metrics
lmms_eval/api/metrics.py - Verification
lmms_eval/verifiers/pipeline.py - Result Logging
lmms_eval/loggers/evaluation_tracker.py
Serving Interfaces
- HTTP Client
lmms_eval/entrypoints/client.py - HTTP Server
lmms_eval/entrypoints/http_server.py - Job Scheduler
lmms_eval/entrypoints/job_scheduler.py - Web UI
lmms_eval/tui/web/src/App.tsx - UI Backend
lmms_eval/tui/server.py - MCP Client
lmms_eval/mcp/client.py - MCP Server
lmms_eval/mcp/server.py - MCP Tools
lmms_eval/mcp/tools.py
Other components
- Evaluator
- Model Service
Connections 25
- Evaluator → CLI: selects run
- CLI → Evaluation Engine: starts evaluation
- Evaluation Engine → Task API: loads tasks
- Task API → Benchmark Tasks: resolves benchmark
- Evaluation Engine → Model Registry: resolves backend
- Model Registry → Model Backends: selects implementation
- Evaluation Engine → Model API: requests inference
- Model API → Model Backends: dispatches generation
- Model Backends → Model Service: requests inference
- Evaluation Engine → Response Cache: reads and writes
- Evaluation Engine → Response Filters: processes responses
- Evaluation Engine → Metrics: computes scores
- Evaluation Engine → Result Logging: records results
- Evaluation Engine → Verification: verifies outputs
- Evaluator → HTTP Client: submits job
- HTTP Client → HTTP Server: sends requests
- HTTP Server → Job Scheduler: queues jobs
- Job Scheduler → Evaluation Engine: runs evaluation
- Evaluator → Web UI: configures evaluation
- Web UI → UI Backend: calls API
- UI Backend → Job Scheduler: manages jobs
- Evaluator → MCP Client: requests tool
- MCP Client → MCP Server: calls tool
- MCP Server → MCP Tools: registers tools
- MCP Tools → Job Scheduler: submits and tracks