How blaizzy/mlx-vlm works
Project Type and Purpose
• This is an open-source Python library and toolchain (MLX-VLM) for inference and fine-tuning of Vision-Language Models (VLMs).
• It provides multiple client interfaces (CLI, Python API scripts, Gradio Chat UI, FastAPI server), core model-loading and inference logic, model wrappers for many VLM backends, and a fine-tuning subsystem based on LoRA/QLoRA.
High-Level Components
a. Clients / Entry Points
• CLI: mlx_vlm.generate, mlx_vlm.chat, mlx_vlm.video_generate
• Gradio Chat UI: mlx_vlm.chat_ui
• FastAPI Server: mlx_vlm.server
• Python API: call into mlx_vlm.load, mlx_vlm.generate, mlx_vlm.prompt_utils, etc.
• Example scripts: examples/*.ipynb, computer_use/*.py
b. Core Inference Engine
• Model loader: mlx_vlm/load.py (invokes MLX core to load model + processor)
• Prompt & Sampling utilities: prompt_utils.py, sample_utils.py, tokenizer_utils.py
• Inference drivers: generate.py, chat.py, video_generate.py
c. Model Wrappers
• Directory mlx_vlm/models/*: one subpackage per model family (e.g., qwen2_vl, llava, idefics2), each with language- and vision-specific adapters.
• Base model interfaces: models/base.py, caching: models/cache.py
d. Fine-tuning Subsystem
• Package mlx_vlm.trainer: orchestrates LoRA/QLoRA-based fine-tuning (trainer.py, lora.py, utilities).
• Documentation: LORA.MD
e. Utilities & Configuration
• Global config loader: mlx_vlm/utils.py
• Project metadata: setup.py, version.py
• Testing suite: mlx_vlm/tests/*, pytest.ini
f. Ancillary Examples
• examples/: Jupyter notebooks demonstrating typical use cases (multi-image, OCR, video).
• computer_use/: GUI‐based autonomous agent demos with audio/visual assets.
Interactions & Data Flow
• User invokes a client → calls into core driver (e.g., generate.py) → uses load() to get model & processor → prompt_utils formats prompt → image(s)/video frames go through vision processor → tokens go through language model → output sequences return to client.
• Server wraps this flow behind a JSON REST API; it caches single loaded model to optimize repeated requests.
• Fine-tuning path: user supplies dataset → trainer/trainer.py loads base model & LoRA config → applies update steps → writes LoRA weights.
Architectural Patterns
• Plugin pattern for model families: each model registers its own language.py and vision.py logic under a uniform interface.
• Layered architecture:
• Presentation (CLI / Gradio / Server)
• Application logic (drivers + prompt utils)
• Domain (model wrappers + MLX core)
• Infrastructure (cache, trainer, I/O)
• Single-model cache in server: ensures only one model is resident in GPU/CPU at a time.
• Separation of concerns: inference vs. fine-tuning.