# blaizzy/mlx-embeddings architecture

> Architecture diagram of the GitHub repository blaizzy/mlx-embeddings, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/blaizzy/mlx-embeddings
- Repository: https://github.com/blaizzy/mlx-embeddings

## Overview

##### Project Type and Purpose
- This is an open-source Python library (“mlx-embeddings”) that provides a unified API to load Vision and Language embedding models via the MLX runtime.
- Users import `mlx_embeddings.utils.load` (and other helpers) to fetch pretrained models/tokenizers, run inference, and obtain embeddings for text, masked-language tasks, images, or mixed image-text inputs.

##### File Structure Analysis
- Top-level: packaging & project metadata (`pyproject.toml`, `requirements.txt`, `LICENSE`, `README.md`, CI/CD workflows under `.github/`, documentation config `mkdocs.yml`)
- Core library package: `mlx_embeddings/`
- `utils.py` – public entrypoint (`load`, model-loader)
- `tokenizer_utils.py` – helper to build tokenizers/processors
- `convert.py` – model conversion utilities
- `version.py` – package versioning
- `models/` – model classes implementing `BaseModel` for BERT, ModernBERT, XLM-RoBERTa, SigLIP
- `tests/` – unit & smoke tests verifying loader, model outputs
- Assets: `images/` for examples, docs.

##### Key Components and Interactions
a. User Application (external)
– Imports `load` from `mlx_embeddings.utils` or direct calls to model classes.
– Supplies text or image inputs.

b. Public API Layer (`utils.py`)
– Parses model identifier string (e.g. `"mlx-community/all-MiniLM-L6-v2-4bit"`).
– Invokes `tokenizer_utils` to instantiate proper tokenizer or vision processor.
– Instantiates the correct subclass of `models.BaseModel` based on architecture.

c. Tokenizer / Processor Layer (`tokenizer_utils.py`)
– Creates HuggingFace-style tokenizers or ViT processors, wrapping MLX’s low-level APIs.
– Provides batching, encoding, masking.

d. Model Implementation Layer (`models/`)
– `base.py`: defines interface (forward, pooling, normalization).
– `bert.py`, `modernbert.py`, `xlm_roberta.py`, `siglip.py`: each encapsulates model‐specific forward pass, pooling strategy, image/text pairing.
– Internally uses `mlx.core` (MLX runtime) for tensor ops, inference kernels.

e. Conversion Utilities (`convert.py`)
– Scripts to convert third-party model checkpoints to MLX format.

f. External Dependencies
– MLX Runtime (`mlx.core`): actual tensor library, device execution.
– Pretrained model hub (remote storage from “mlx-community/...”).
– Optional: HuggingFace tokenizers, PIL, scikit-learn for demos.

g. CI/CD & Documentation
– GitHub Actions: test suite, docs build, PyPI publish workflows.
– `mkdocs.yml`: documentation site config.

##### Architectural Patterns & Principles
- Modular plugin pattern: each model architecture in its own module, all inheriting a common base.
- Single-responsibility: tokenization, model wrapping, conversion, and public API each in separate modules.
- Dependency inversion: user code depends on high-level `utils.load`, not on concrete model classes.
- Extensibility: adding a new model only requires adding a new subclass under `models/` and updating the loader.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Layer 1: User"
        User["User Application\n(Notebook/CLI)"]:::external
    end

    subgraph "Layer 2: Public API"
        Utils["utils.load()"]:::publicAPI
        Version["version.py"]:::internal
    end

    subgraph "Layer 3a: Tokenizer/Processor"
        Tokenizer["tokenizer_utils"]:::internal
    end

    subgraph "Layer 3b: Conversion Utilities"
        Convert["convert"]:::internal
    end

    subgraph "Layer 4: Model Implementations"
        Base["BaseModel\n(base.py)"]:::internal
        Bert["BERT Model\n(bert.py)"]:::internal
        Modern["ModernBERT Model\n(modernbert.py)"]:::internal
        XLM["XLM-RoBERTa Model\n(xlm_roberta.py)"]:::internal
        Siglip["SigLIP Model\n(siglip.py)"]:::internal
    end

    subgraph "Layer 5: External Dependencies"
        MLX["MLX Core Runtime\n(mlx.core)"]:::external
        Hub["Remote Model Hub"]:::external
    end

    subgraph "Layer 6: Hardware"
        HW["CPU/GPU Hardware"]:::external
    end

    subgraph "CI/CD & Docs"
        Tests["Unit & Smoke Tests"]:::ci
        CI["CI/CD Workflows"]:::ci
        Docs["mkdocs.yml\nDocs Config"]:::ci
        Metadata["Project Metadata\n(pyproject.toml, README.md, LICENSE)"]:::ci
    end

    User -->|"call load(model_name)"| Utils
    Utils -->|"build tokenizer"| Tokenizer
    Utils -->|"instantiate BaseModel"| Base
    Utils -->|"instantiate BERT"| Bert
    Utils -->|"instantiate ModernBERT"| Modern
    Utils -->|"instantiate XLM-RoBERTa"| XLM
    Utils -->|"instantiate SigLIP"| Siglip
    Base -->|"forward pass"| MLX
    Bert -->|"forward pass"| MLX
    Modern -->|"forward pass"| MLX
    XLM -->|"forward pass"| MLX
    Siglip -->|"forward pass"| MLX
    Tokenizer -->|"encode inputs"| MLX
    Utils -->|"download checkpoint"| Hub
    Convert -->|"fetch/convert checkpoint"| Hub
    MLX -->|"execute on"| HW
    MLX -->|"return tensors"| Utils
    Utils -->|"return embeddings"| User

    Tests --> Utils
    Tests --> Tokenizer
    Tests --> Convert
    Tests --> Base
    Tests --> Bert
    Tests --> Modern
    Tests --> XLM
    Tests --> Siglip
    CI --> Tests
    CI --> Docs
    Docs --> Metadata

    classDef publicAPI fill:#aedff7,stroke:#333,stroke-width:1px
    classDef internal fill:#b2f7b2,stroke:#333,stroke-width:1px
    classDef external fill:#f7d2b2,stroke:#333,stroke-width:1px
    classDef ci fill:#d3d3d3,stroke:#333,stroke-width:1px

    click Utils "https://github.com/blaizzy/mlx-embeddings/blob/main/mlx_embeddings/utils.py"
    click Tokenizer "https://github.com/blaizzy/mlx-embeddings/blob/main/mlx_embeddings/tokenizer_utils.py"
    click Convert "https://github.com/blaizzy/mlx-embeddings/blob/main/mlx_embeddings/convert.py"
    click Base "https://github.com/blaizzy/mlx-embeddings/blob/main/mlx_embeddings/models/base.py"
    click Bert "https://github.com/blaizzy/mlx-embeddings/blob/main/mlx_embeddings/models/bert.py"
    click Modern "https://github.com/blaizzy/mlx-embeddings/blob/main/mlx_embeddings/models/modernbert.py"
    click XLM "https://github.com/blaizzy/mlx-embeddings/blob/main/mlx_embeddings/models/xlm_roberta.py"
    click Siglip "https://github.com/blaizzy/mlx-embeddings/blob/main/mlx_embeddings/models/siglip.py"
    click Version "https://github.com/blaizzy/mlx-embeddings/blob/main/mlx_embeddings/version.py"
    click Tests "https://github.com/blaizzy/mlx-embeddings/tree/main/mlx_embeddings/tests/"
    click CI "https://github.com/blaizzy/mlx-embeddings/tree/main/.github/workflows/"
    click Docs "https://github.com/blaizzy/mlx-embeddings/blob/main/mkdocs.yml"
    click Metadata "https://github.com/blaizzy/mlx-embeddings/blob/main/pyproject.toml"
```
