How blaizzy/mlx-embeddings works
Project Type and Purpose
• This is an open-source Python library (“mlx-embeddings”) that provides a unified API to load Vision and Language embedding models via the MLX runtime.
• Users import mlx_embeddings.utils.load (and other helpers) to fetch pretrained models/tokenizers, run inference, and obtain embeddings for text, masked-language tasks, images, or mixed image-text inputs.
File Structure Analysis
• Top-level: packaging & project metadata (pyproject.toml, requirements.txt, LICENSE, README.md, CI/CD workflows under .github/, documentation config mkdocs.yml)
• Core library package: mlx_embeddings/
• utils.py – public entrypoint (load, model-loader)
• tokenizer_utils.py – helper to build tokenizers/processors
• convert.py – model conversion utilities
• version.py – package versioning
• models/ – model classes implementing BaseModel for BERT, ModernBERT, XLM-RoBERTa, SigLIP
• tests/ – unit & smoke tests verifying loader, model outputs
• Assets: images/ for examples, docs.
Key Components and Interactions
a. User Application (external)
– Imports load from mlx_embeddings.utils or direct calls to model classes.
– Supplies text or image inputs.
b. Public API Layer (utils.py)
– Parses model identifier string (e.g. "mlx-community/all-MiniLM-L6-v2-4bit").
– Invokes tokenizer_utils to instantiate proper tokenizer or vision processor.
– Instantiates the correct subclass of models.BaseModel based on architecture.
c. Tokenizer / Processor Layer (tokenizer_utils.py)
– Creates HuggingFace-style tokenizers or ViT processors, wrapping MLX’s low-level APIs.
– Provides batching, encoding, masking.
d. Model Implementation Layer (models/)
– base.py: defines interface (forward, pooling, normalization).
– bert.py, modernbert.py, xlm_roberta.py, siglip.py: each encapsulates model‐specific forward pass, pooling strategy, image/text pairing.
– Internally uses mlx.core (MLX runtime) for tensor ops, inference kernels.
e. Conversion Utilities (convert.py)
– Scripts to convert third-party model checkpoints to MLX format.
f. External Dependencies
– MLX Runtime (mlx.core): actual tensor library, device execution.
– Pretrained model hub (remote storage from “mlx-community/...”).
– Optional: HuggingFace tokenizers, PIL, scikit-learn for demos.
g. CI/CD & Documentation
– GitHub Actions: test suite, docs build, PyPI publish workflows.
– mkdocs.yml: documentation site config.
Architectural Patterns & Principles
• Modular plugin pattern: each model architecture in its own module, all inheriting a common base.
• Single-responsibility: tokenization, model wrapping, conversion, and public API each in separate modules.
• Dependency inversion: user code depends on high-level utils.load, not on concrete model classes.
• Extensibility: adding a new model only requires adding a new subclass under models/ and updating the loader.