# embeddings-benchmark/mteb architecture

> Architecture diagram of the GitHub repository embeddings-benchmark/mteb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/embeddings-benchmark/mteb
- Repository: https://github.com/embeddings-benchmark/mteb

## Overview

##### Main Components
- Core Library (mteb package): At the heart is the “mteb” package. Break this into subcomponents:
– Task Definitions: Located under mteb/tasks and its many subdirectories (e.g., Classification, Clustering, Retrieval, STS, etc.). These modules define the various benchmarking tasks.
– Evaluation Framework: Under mteb/evaluation and mteb/evaluation/evaluators, which include evaluators specialized for different modalities (text, image, instruction retrieval, etc.).
– Model Integration: Found in mteb/models and wrappers (e.g., get_model interface, encoder_interface) for integrating external models (like SentenceTransformers) and custom models.
– Benchmark and Aggregation: Modules under mteb/benchmarks and aggregated_tasks that orchestrate running benchmarks across tasks.
– Leaderboard & Results Loading: The “mteb/leaderboard” and “mteb/load_results” modules are used for compiling, displaying, and comparing benchmark results.
– Caching & Logging: Components such as mteb/caching.py and mteb/logging.py deal with caching embeddings and logging run details.

##### Peripheral and Orchestration Components
- CLI Interface: The “mteb/cli.py” and “mteb/__main__.py” provide command-line functionalities for listing tasks, running evaluations, and integrating with external distributed resources.
- Scripts and Utility Tools: The scripts directory contains multiple utilities for data preparation (e.g., creating datasets, generating metadata, and task selection), which help in preparing the inputs for evaluations.
- Testing and CI/CD: The tests directory and the .github workflows show that automated integration tests and continuous integration are integral parts of the project.

##### Relationships and Data Flow
- User Inputs through CLI or Scripts initiate model evaluations.
- The CLI front-end calls into the core library functions which load the appropriate task definitions and evaluation routines.
- Benchmark routines (from evaluation and task modules) interact with the model interface (via mteb/models) to retrieve embeddings, run task-specific scoring, and produce results.
- Results are then aggregated and possibly cached (via caching modules) before being loaded by the leaderboard component to produce visualizations and reports.
- The system uses configuration files (Makefile, GitHub workflows, pre-commit, etc.) to automate builds, tests, and deployment.

##### Architectural Patterns and Design Principles
- Modular Design: The project is separated into clearly defined modules (tasks, evaluation, models) following separation of concerns.
- Plug-and-Play Integration: Interfaces to load models and tasks allow for extensibility (e.g., custom tasks and model integrations).
- Reproducibility: The presence of scripts, test suites, and CI/CD workflows shows an emphasis on reproducible research and robust benchmarking.
- Scalability: The architecture supports distributed evaluation (multi-GPU strategies) as indicated in the README where custom encoding functions can be provided for parallelization.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction
    U1["User Commands"]:::core
    CLI["CLI Interface"]:::core

    %% Core Library Components
    T["Task Definitions (Classification, Clustering, Retrieval, STS, ...)"]:::core
    E["Evaluation Framework (Evaluators, etc.)"]:::core
    M["Model Integration (Wrappers, SentenceTransformer integration)"]:::core
    B["Benchmark & Aggregation (Orchestration of benchmarks)"]:::core
    L["Leaderboard & Results Loading"]:::core
    C["Caching & Logging"]:::core

    %% External Dependency
    Ext["External: SentenceTransformer"]:::external

    %% Peripheral Components Subgraph
    subgraph "Peripheral Components"
        S["Scripts & Utility Tools"]:::peripheral
        Tc["Testing & CI/CD"]:::peripheral
        D["Documentation"]:::peripheral
        subgraph "Configuration Files"
            CF1["Makefile"]:::config
            CF2[".gitignore"]:::config
            CF3[".pre-commit-config.yaml"]:::config
        end
    end

    %% Data Flow
    U1 --> CLI
    CLI --> T
    T --> E
    E --> M
    M --> B
    B --> L

    %% Caching & Logging branch from Evaluation and Benchmark
    E --> C
    B --> C

    %% External dependency from Model Integration
    M --> Ext

    %% Link to Peripheral Components
    L --> S
    L --> Tc
    L --> D

    %% Styling Classes
    classDef core fill:#E0F7FA,stroke:#006064,stroke-width:2px;
    classDef peripheral fill:#E8F5E9,stroke:#2E7D32,stroke-width:2px;
    classDef external fill:#FFF9C4,stroke:#F9A825,stroke-width:2px;
    classDef config fill:#F3E5F5,stroke:#6A1B9A,stroke-width:2px;

    %% Click Events for Core Components
    click T "https://github.com/embeddings-benchmark/mteb/tree/main/mteb/tasks/"
    click E "https://github.com/embeddings-benchmark/mteb/tree/main/mteb/evaluation/"
    click M "https://github.com/embeddings-benchmark/mteb/tree/main/mteb/models/"
    click B "https://github.com/embeddings-benchmark/mteb/tree/main/mteb/benchmarks/"
    click L "https://github.com/embeddings-benchmark/mteb/tree/main/mteb/leaderboard/"
    click C "https://github.com/embeddings-benchmark/mteb/blob/main/mteb/caching.py"
    click C "https://github.com/embeddings-benchmark/mteb/blob/main/mteb/logging.py"
    click CLI "https://github.com/embeddings-benchmark/mteb/blob/main/mteb/cli.py"
    click CLI "https://github.com/embeddings-benchmark/mteb/blob/main/mteb/__main__.py"

    %% Click Events for Peripheral Components
    click S "https://github.com/embeddings-benchmark/mteb/tree/main/scripts/"
    click Tc "https://github.com/embeddings-benchmark/mteb/tree/main/tests/"
    click Tc "https://github.com/embeddings-benchmark/mteb/tree/main/.github/"
    click D "https://github.com/embeddings-benchmark/mteb/tree/main/docs/"

    %% Click Events for Configuration Files
    click CF1 "https://github.com/embeddings-benchmark/mteb/tree/main/Makefile"
    click CF2 "https://github.com/embeddings-benchmark/mteb/blob/main/.gitignore"
    click CF3 "https://github.com/embeddings-benchmark/mteb/blob/main/.pre-commit-config.yaml"
```
