# blaizzy/mlx-vlm architecture

> Architecture diagram of the GitHub repository blaizzy/mlx-vlm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/blaizzy/mlx-vlm
- Repository: https://github.com/blaizzy/mlx-vlm

## Overview

##### Project Type and Purpose
- This is an open-source Python library and toolchain (MLX-VLM) for inference and fine-tuning of Vision-Language Models (VLMs).
- It provides multiple client interfaces (CLI, Python API scripts, Gradio Chat UI, FastAPI server), core model-loading and inference logic, model wrappers for many VLM backends, and a fine-tuning subsystem based on LoRA/QLoRA.

##### High-Level Components
a. Clients / Entry Points
- CLI: `mlx_vlm.generate`, `mlx_vlm.chat`, `mlx_vlm.video_generate`
- Gradio Chat UI: `mlx_vlm.chat_ui`
- FastAPI Server: `mlx_vlm.server`
- Python API: call into `mlx_vlm.load`, `mlx_vlm.generate`, `mlx_vlm.prompt_utils`, etc.
- Example scripts: `examples/*.ipynb`, `computer_use/*.py`
b. Core Inference Engine
- Model loader: `mlx_vlm/load.py` (invokes MLX core to load model + processor)
- Prompt & Sampling utilities: `prompt_utils.py`, `sample_utils.py`, `tokenizer_utils.py`
- Inference drivers: `generate.py`, `chat.py`, `video_generate.py`
c. Model Wrappers
- Directory `mlx_vlm/models/*`: one subpackage per model family (e.g., qwen2_vl, llava, idefics2), each with language- and vision-specific adapters.
- Base model interfaces: `models/base.py`, caching: `models/cache.py`
d. Fine-tuning Subsystem
- Package `mlx_vlm.trainer`: orchestrates LoRA/QLoRA-based fine-tuning (`trainer.py`, `lora.py`, utilities).
- Documentation: `LORA.MD`
e. Utilities & Configuration
- Global config loader: `mlx_vlm/utils.py`
- Project metadata: `setup.py`, `version.py`
- Testing suite: `mlx_vlm/tests/*`, `pytest.ini`
f. Ancillary Examples
- `examples/`: Jupyter notebooks demonstrating typical use cases (multi-image, OCR, video).
- `computer_use/`: GUI‐based autonomous agent demos with audio/visual assets.

##### Interactions & Data Flow
- User invokes a client → calls into core driver (e.g., `generate.py`) → uses `load()` to get model & processor → `prompt_utils` formats prompt → image(s)/video frames go through vision processor → tokens go through language model → output sequences return to client.
- Server wraps this flow behind a JSON REST API; it caches single loaded model to optimize repeated requests.
- Fine-tuning path: user supplies dataset → `trainer/trainer.py` loads base model & LoRA config → applies update steps → writes LoRA weights.

##### Architectural Patterns
- Plugin pattern for model families: each model registers its own `language.py` and `vision.py` logic under a uniform interface.
- Layered architecture:
- Presentation (CLI / Gradio / Server)
- Application logic (drivers + prompt utils)
- Domain (model wrappers + MLX core)
- Infrastructure (cache, trainer, I/O)
- Single-model cache in server: ensures only one model is resident in GPU/CPU at a time.
- Separation of concerns: inference vs. fine-tuning.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Clients Layer
    subgraph "Clients" 
        direction TB
        CLI_Generate["CLI: generate"]:::clients
        CLI_Chat["CLI: chat"]:::clients
        CLI_Video["CLI: video_generate"]:::clients
        CLI_SMOL["CLI: smolvlm_video_generate"]:::clients
        Python_API["Python API"]:::clients
        Gradio_UI["Gradio Chat UI"]:::clients
        FastAPI_Server["FastAPI Server"]:::clients
    end

    %% Application Layer
    subgraph "Application Logic" 
        direction TB
        Generate_Module["generate.py"]:::app
        Chat_Module["chat.py"]:::app
        Video_Module["video_generate.py"]:::app
        Convert_Module["convert.py"]:::app
        PromptUtils["prompt_utils.py"]:::app
        SampleUtils["sample_utils.py"]:::app
        TokenizerUtils["tokenizer_utils.py"]:::app
    end

    %% Model Loading & Registry
    subgraph "Model Infrastructure"
        direction TB
        Model_Loader["load.py / convert.py"]:::model
        Base_Model["base.py"]:::model
        Model_Cache["cache.py"]:::model
        subgraph "Model Registry"
            direction TB
            aya_vision["aya_vision"]:::model
            deepseek_vl_v2["deepseek_vl_v2"]:::model
            florence2["florence2"]:::model
            gemma3["gemma3"]:::model
            idefics2["idefics2"]:::model
            idefics3["idefics3"]:::model
            internvl_chat["internvl_chat"]:::model
            kimi_vl["kimi_vl"]:::model
            llama4["llama4"]:::model
            llava["llava"]:::model
            llava_bunny["llava_bunny"]:::model
            llava_next["llava_next"]:::model
            mistral3["mistral3"]:::model
            mllama["mllama"]:::model
            molmo["molmo"]:::model
            multi_modality["multi_modality"]:::model
            paligemma["paligemma"]:::model
            phi3_v["phi3_v"]:::model
            pixtral["pixtral"]:::model
            qwen2_5_vl["qwen2_5_vl"]:::model
            qwen2_vl["qwen2_vl"]:::model
            smolvlm["smolvlm"]:::model
        end
        MLX_Core["MLX Core"]:::model
    end

    %% Fine-tuning Subsystem
    subgraph "Fine-tuning Subsystem"
        direction TB
        Trainer["trainer.py"]:::infra
        LoRA["lora.py"]:::infra
        Trainer_Utils["utils.py"]:::infra
    end

    %% Examples
    subgraph "Ancillary Examples"
        direction TB
        Examples_NB["Jupyter Notebooks"]:::examples
        Examples_Scripts["computer_use demos"]:::examples
    end

    %% Data Flow
    CLI_Generate -->|calls| Generate_Module
    CLI_Chat -->|calls| Chat_Module
    CLI_Video -->|calls| Video_Module
    CLI_SMOL -->|calls| Video_Module
    Python_API -->|imports| Generate_Module
    Python_API -->|imports| Chat_Module
    Python_API -->|imports| Video_Module
    Python_API -->|imports| Convert_Module
    Gradio_UI -->|requests| Chat_Module
    FastAPI_Server -->|REST API| Chat_Module
    FastAPI_Server -->|REST API| Generate_Module
    FastAPI_Server -->|REST API| Video_Module

    Generate_Module -->|format prompt| PromptUtils
    Chat_Module -->|format prompt| PromptUtils
    Video_Module -->|format prompt| PromptUtils

    PromptUtils -->|process| TokenizerUtils
    Generate_Module -->|sample| SampleUtils

    Generate_Module -->|load/infer| Model_Loader
    Chat_Module -->|load/infer| Model_Loader
    Video_Module -->|load/infer| Model_Loader

    Model_Loader -->|uses| Base_Model
    Model_Loader -->|uses| Model_Cache
    Model_Cache -->|stores| Base_Model

    Model_Loader -->|loads| MLX_Core
    Model_Loader -->|selects| aya_vision & deepseek_vl_v2 & florence2 & gemma3 & idefics2 & idefics3 & internvl_chat & kimi_vl & llama4 & llava & llava_bunny & llava_next & mistral3 & mllama & molmo & multi_modality & paligemma & phi3_v & pixtral & qwen2_5_vl & qwen2_vl & smolvlm

    %% Fine-tuning Flow
    Examples_NB -->|dataset| Trainer
    Trainer -->|update| LoRA
    LoRA -->|write weights| Model_Cache

    %% Click Events
    click CLI_Generate "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py"
    click CLI_Chat "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/chat.py"
    click CLI_Video "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/video_generate.py"
    click CLI_SMOL "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/smolvlm_video_generate.py"
    click Gradio_UI "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/chat_ui.py"
    click FastAPI_Server "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py"
    click PromptUtils "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/prompt_utils.py"
    click SampleUtils "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/sample_utils.py"
    click TokenizerUtils "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/tokenizer_utils.py"
    click Convert_Module "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/convert.py"
    click Base_Model "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/models/base.py"
    click Model_Cache "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/models/cache.py"
    click Trainer "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/trainer/trainer.py"
    click LoRA "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/trainer/lora.py"
    click Trainer_Utils "https://github.com/blaizzy/mlx-vlm/blob/main/mlx_vlm/trainer/utils.py"
    click Examples_NB "https://github.com/blaizzy/mlx-vlm/tree/main/examples/"
    click Examples_Scripts "https://github.com/blaizzy/mlx-vlm/tree/main/computer_use/"

    %% Styles
    classDef clients fill:#cce5ff,stroke:#3399ff,color:#000;
    classDef app fill:#d4edda,stroke:#28a745,color:#000;
    classDef model fill:#fff3cd,stroke:#ffc107,color:#000;
    classDef infra fill:#e2e3e5,stroke:#6c757d,color:#000;
    classDef examples fill:#f0f0f0,stroke:#adb5bd,color:#000;
```
