# apple/ml-fastvlm architecture

> Architecture diagram of the GitHub repository apple/ml-fastvlm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apple/ml-fastvlm
- Repository: https://github.com/apple/ml-fastvlm

## Overview

##### Project type and purpose
- This is a multi-component research codebase for FastVLM (an efficient vision encoder), combining:
– A Python “backend” for training, exporting, and serving vision-language models (the `llava/`, `model_export/`, `predict.py`, training scripts, etc.).
– An iOS SwiftUI “frontend” demo app (under `app/FastVLM App/` and supporting Swift libraries in `app/FastVLM/`).
– Shell scripts for setup and model downloads.
- Purpose: train/finetune FastVLM variants, export them for Apple Silicon, serve them via CLI or web, and demonstrate on an iOS device.

##### File structure
Top-level directories:
- `llava/` – core Python package for model building, multimodal encoding, training (`train/`), and serving (`serve/`).
- `model_export/` – Python scripts to convert PyTorch checkpoints to Apple Silicon–compatible formats.
- `app/` – Xcode project containing:
– `FastVLM App/` – SwiftUI code and assets.
– `FastVLM/` – Swift bindings (Objective-C header + Swift extension) to call the exported model.
– `Video/` – camera capture and UI components.
- Root scripts: `predict.py` (CLI inference), `get_models.sh`, `get_pretrained_mlx_model.sh`.
- Configs: `pyproject.toml`, Xcode configs, Conda instructions in README.

##### Extract architectural insights from the README
- Training is built on LLaVA.
- Inference can be run via `predict.py` or the `llava.serve` Gradio/CLI modules.
- Models are exported for Apple Silicon in `model_export/` and then consumed by the iOS app.
- The demo iOS app uses Core ML (via Swift wrappers) to run inference directly on device.

##### Main components
A. Model Training Pipeline (Python)
– Input: datasets & config (`llava/train/...`, flash-attn patches)
– Process: build multimodal encoder + LLM adapter → fine-tune → produce PyTorch checkpoints in `checkpoints/`.
B. Model Export Pipeline
– Input: PyTorch checkpoints → using `model_export/export_vision_encoder.py` and patch scripts → Output: Apple Silicon `.mlmodel` or quantized weights.
C. Model Serving Layer (Python)
– `predict.py` (CLI)
– `llava.serve`
- `gradio_web_server.py` (Web UI)
- `controller.py`, `model_worker.py`, `register_worker.py`
- Outputs inference results to user.
D. iOS Demonstration App (Swift/SwiftUI)
– Camera & UI: `CameraController.swift`, `CameraControlsView.swift`, `VideoFrameView.swift`
– Model wrapper: `FastVLM.swift`, `FastVLM.h`, `MediaProcessingExtensions.swift`, `FastVLMModel.swift`
– Assets & app entry: `FastVLMApp.swift`, `ContentView.swift`, `InfoView.swift`
– Loads the exported `.mlmodel` and performs on-device inference.
E. External services & dependencies
– PyTorch, Conda environment, LLaVA codebase (Git submodule)
– Apple frameworks: Core ML, AVFoundation, SwiftUI
– Hardware: GPU (training/serving), Apple Silicon (inference on device)

##### Relationships and data flow
1. Training → produces PyTorch checkpoint files.
2. Model Export → consumes checkpoint → emits Apple-compatible model artifacts.
3. Python Serving or CLI (`predict.py`) → loads checkpoint or exported model → takes image+prompt → returns text output.
4. iOS App → uses exported `.mlmodel` → camera frames → inference → UI displays results.
5. Shell scripts (`get_models.sh`) orchestrate model downloads into `checkpoints/`.

##### Architectural patterns & principles
- Clear separation of concerns: training, export, serving, and client UI are modular.
- Polyglot layering: Python backend vs. Swift frontend.
- Pipeline pattern: data flows from raw training data → model → export → inference.
- Extensibility: `llava.model` and `llava.multimodal_encoder` are plug-and-play components for new architectures.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Training Pipeline
    subgraph "Model Training Pipeline" 
        direction TB
        LL_TRAIN_TRAIN["train.py"]:::training
        LL_TRAIN_TRAINR["llava_trainer.py"]:::training
        LL_PATCH1["llama_flash_attn_monkey_patch.py"]:::training
        LL_PATCH2["llama_xformers_attn_monkey_patch.py"]:::training
        LL_TRAIN_MEM["train_mem.py"]:::training
        LL_TRAIN_QWEN["train_qwen.py"]:::training
        LL_TRAIN_XFORMERS["train_xformers.py"]:::training
        Checkpoints["PyTorch Checkpoints\n(checkpoints/)"]:::artifact
    end

    %% Model Export Pipeline
    subgraph "Model Export Pipeline"
        direction TB
        EXPORT_SCRIPT["export_vision_encoder.py"]:::export
        PATCH["fastvlm_mlx-vlm.patch"]:::export
        MD_EXPORT["README.md"]:::export
        ML_MODEL["Apple Silicon .mlmodel"]:::artifact
    end

    %% Model Serving Layer
    subgraph "Model Serving Layer"
        direction TB
        PREDICT["predict.py"]:::serve
        CLI_SERVE["cli.py"]:::serve
        GRADIO_UI["gradio_web_server.py"]:::serve
        CONTROLLER["controller.py"]:::serve
        WORKER["model_worker.py"]:::serve
        REG_WORKER["register_worker.py"]:::serve
        SLANG_WORKER["sglang_worker.py"]:::serve
    end

    %% iOS Demonstration App
    subgraph "iOS Demonstration App"
        direction TB
        subgraph "SwiftUI UI & Assets"
            direction TB
            APP_ENTRY["FastVLMApp.swift"]:::ios
            CONTENT["ContentView.swift"]:::ios
            INFO_VIEW["InfoView.swift"]:::ios
            MODEL_VIEW["FastVLMModel.swift"]:::ios
            ASSETS["Assets.xcassets"]:::ios
            INFO_PLIST["Info.plist"]:::ios
        end
        subgraph "Model Wrapper"
            direction TB
            WRAP_SWIFT["FastVLM.swift"]:::ios
            WRAP_HEADER["FastVLM.h"]:::ios
            MEDIA_EXT["MediaProcessingExtensions.swift"]:::ios
        end
        subgraph "Camera & Video Components"
            direction TB
            CAM_CTRL["CameraController.swift"]:::ios
            CAM_CONTROLS["CameraControlsView.swift"]:::ios
            CAM_TYPE["CameraType.swift"]:::ios
            VID_FRAME["VideoFrameView.swift"]:::ios
            VID_HDR["Video.h"]:::ios
        end
    end

    %% Shell Scripts
    subgraph "Setup & Download Scripts"
        direction TB
        GET_MODELS["get_models.sh"]:::script
        GET_PRETRAINED["get_pretrained_mlx_model.sh"]:::script
        APP_README["app/README.md"]:::script
    end

    %% Core Python Package
    subgraph "Core Python Package (llava)"
        direction TB
        MODEL_BUILDER["builder.py\napply_delta.py\nconsolidate.py\nmake_delta.py"]:::core
        MULTI_ENCODER["multimodal_encoder/"]:::core
        LANG_MODEL["language_model/"]:::core
        CONST_UTILS["constants.py\nmm_utils.py\nconversation.py\nutils.py"]:::core
    end

    %% Flows
    LL_TRAIN_TRAIN -->|"produces .pt files"| Checkpoints
    LL_TRAIN_TRAINR --> Checkpoints
    Checkpoints -->|".pt files"| EXPORT_SCRIPT
    Checkpoints -->|".pt files"| PREDICT
    EXPORT_SCRIPT --> ML_MODEL
    EXPORT_SCRIPT --> PATCH
    ML_MODEL --> APP_ENTRY
    ML_MODEL --> WRAP_SWIFT
    ML_MODEL --> MEDIA_EXT
    PREDICT -->|"CLI inference"| CONTROLLER
    PREDICT -->|"CLI inference"| GRADIO_UI
    CONTROLLER --> WORKER
    CONTROLLER --> REG_WORKER
    WORKER --> SLANG_WORKER

    APP_ENTRY --> CONTENT
    CONTENT --> INFO_VIEW
    CONTENT --> MODEL_VIEW
    WRAP_SWIFT --> WRAP_HEADER
    WRAP_SWIFT --> MEDIA_EXT
    CAM_CTRL --> CAM_CONTROLS
    CAM_CTRL --> CAM_TYPE
    CAM_CTRL --> VID_FRAME
    CAM_CTRL --> VID_HDR

    GET_MODELS --> Checkpoints
    GET_PRETRAINED --> Checkpoints

    %% Click Events
    click LL_TRAIN_TRAIN "https://github.com/apple/ml-fastvlm/blob/main/llava/train/train.py"
    click LL_TRAIN_TRAINR "https://github.com/apple/ml-fastvlm/blob/main/llava/train/llava_trainer.py"
    click LL_PATCH1 "https://github.com/apple/ml-fastvlm/blob/main/llava/train/llama_flash_attn_monkey_patch.py"
    click LL_PATCH2 "https://github.com/apple/ml-fastvlm/blob/main/llava/train/llama_xformers_attn_monkey_patch.py"
    click LL_TRAIN_MEM "https://github.com/apple/ml-fastvlm/blob/main/llava/train/train_mem.py"
    click LL_TRAIN_QWEN "https://github.com/apple/ml-fastvlm/blob/main/llava/train/train_qwen.py"
    click LL_TRAIN_XFORMERS "https://github.com/apple/ml-fastvlm/blob/main/llava/train/train_xformers.py"

    click EXPORT_SCRIPT "https://github.com/apple/ml-fastvlm/blob/main/model_export/export_vision_encoder.py"
    click PATCH "https://github.com/apple/ml-fastvlm/blob/main/model_export/fastvlm_mlx-vlm.patch"
    click MD_EXPORT "https://github.com/apple/ml-fastvlm/blob/main/model_export/README.md"

    click PREDICT "https://github.com/apple/ml-fastvlm/blob/main/predict.py"
    click CLI_SERVE "https://github.com/apple/ml-fastvlm/blob/main/llava/serve/cli.py"
    click GRADIO_UI "https://github.com/apple/ml-fastvlm/blob/main/llava/serve/gradio_web_server.py"
    click CONTROLLER "https://github.com/apple/ml-fastvlm/blob/main/llava/serve/controller.py"
    click WORKER "https://github.com/apple/ml-fastvlm/blob/main/llava/serve/model_worker.py"
    click REG_WORKER "https://github.com/apple/ml-fastvlm/blob/main/llava/serve/register_worker.py"
    click SLANG_WORKER "https://github.com/apple/ml-fastvlm/blob/main/llava/serve/sglang_worker.py"

    click APP_ENTRY "https://github.com/apple/ml-fastvlm/blob/main/app/FastVLM App/FastVLMApp.swift"
    click CONTENT "https://github.com/apple/ml-fastvlm/blob/main/app/FastVLM App/ContentView.swift"
    click INFO_VIEW "https://github.com/apple/ml-fastvlm/blob/main/app/FastVLM App/InfoView.swift"
    click MODEL_VIEW "https://github.com/apple/ml-fastvlm/blob/main/app/FastVLM App/FastVLMModel.swift"
    click ASSETS "https://github.com/apple/ml-fastvlm/tree/main/app/FastVLM App/Assets.xcassets/"
    click INFO_PLIST "https://github.com/apple/ml-fastvlm/blob/main/app/FastVLM App/Info.plist"

    click WRAP_SWIFT "https://github.com/apple/ml-fastvlm/blob/main/app/FastVLM/FastVLM.swift"
    click WRAP_HEADER "https://github.com/apple/ml-fastvlm/blob/main/app/FastVLM/FastVLM.h"
    click MEDIA_EXT "https://github.com/apple/ml-fastvlm/blob/main/app/FastVLM/MediaProcessingExtensions.swift"

    click CAM_CTRL "https://github.com/apple/ml-fastvlm/blob/main/app/Video/CameraController.swift"
    click CAM_CONTROLS "https://github.com/apple/ml-fastvlm/blob/main/app/Video/CameraControlsView.swift"
    click CAM_TYPE "https://github.com/apple/ml-fastvlm/blob/main/app/Video/CameraType.swift"
    click VID_FRAME "https://github.com/apple/ml-fastvlm/blob/main/app/Video/VideoFrameView.swift"
    click VID_HDR "https://github.com/apple/ml-fastvlm/blob/main/app/Video/Video.h"

    click GET_MODELS "https://github.com/apple/ml-fastvlm/blob/main/get_models.sh"
    click GET_PRETRAINED "https://github.com/apple/ml-fastvlm/blob/main/app/get_pretrained_mlx_model.sh"
    click APP_README "https://github.com/apple/ml-fastvlm/blob/main/app/README.md"

    %% Styles
    classDef training fill:#D0E7FF,stroke:#0284C7,color:#000
    classDef export fill:#D0E7FF,stroke:#0369A1,color:#000
    classDef serve fill:#D1FAE5,stroke:#059669,color:#000
    classDef ios fill:#FFE7D9,stroke:#FB923C,color:#000
    classDef script fill:#F5F3FF,stroke:#8B5CF6,color:#000
    classDef core fill:#F0F9FF,stroke:#0284C7,color:#000
    classDef artifact fill:#E5E7EB,stroke:#6B7280,color:#000
```
