How embeddings-benchmark/mteb works
Main Components
• Core Library (mteb package): At the heart is the “mteb” package. Break this into subcomponents:
– Task Definitions: Located under mteb/tasks and its many subdirectories (e.g., Classification, Clustering, Retrieval, STS, etc.). These modules define the various benchmarking tasks.
– Evaluation Framework: Under mteb/evaluation and mteb/evaluation/evaluators, which include evaluators specialized for different modalities (text, image, instruction retrieval, etc.).
– Model Integration: Found in mteb/models and wrappers (e.g., get_model interface, encoder_interface) for integrating external models (like SentenceTransformers) and custom models.
– Benchmark and Aggregation: Modules under mteb/benchmarks and aggregated_tasks that orchestrate running benchmarks across tasks.
– Leaderboard & Results Loading: The “mteb/leaderboard” and “mteb/load_results” modules are used for compiling, displaying, and comparing benchmark results.
– Caching & Logging: Components such as mteb/caching.py and mteb/logging.py deal with caching embeddings and logging run details.
Peripheral and Orchestration Components
• CLI Interface: The “mteb/cli.py” and “mteb/__main__.py” provide command-line functionalities for listing tasks, running evaluations, and integrating with external distributed resources.
• Scripts and Utility Tools: The scripts directory contains multiple utilities for data preparation (e.g., creating datasets, generating metadata, and task selection), which help in preparing the inputs for evaluations.
• Testing and CI/CD: The tests directory and the .github workflows show that automated integration tests and continuous integration are integral parts of the project.
Relationships and Data Flow
• User Inputs through CLI or Scripts initiate model evaluations.
• The CLI front-end calls into the core library functions which load the appropriate task definitions and evaluation routines.
• Benchmark routines (from evaluation and task modules) interact with the model interface (via mteb/models) to retrieve embeddings, run task-specific scoring, and produce results.
• Results are then aggregated and possibly cached (via caching modules) before being loaded by the leaderboard component to produce visualizations and reports.
• The system uses configuration files (Makefile, GitHub workflows, pre-commit, etc.) to automate builds, tests, and deployment.
Architectural Patterns and Design Principles
• Modular Design: The project is separated into clearly defined modules (tasks, evaluation, models) following separation of concerns.
• Plug-and-Play Integration: Interfaces to load models and tasks allow for extensibility (e.g., custom tasks and model integrations).
• Reproducibility: The presence of scripts, test suites, and CI/CD workflows shows an emphasis on reproducible research and robust benchmarking.
• Scalability: The architecture supports distributed evaluation (multi-GPU strategies) as indicated in the README where custom encoding functions can be provided for parallelization.