How apple/ml-fastvlm works
Project type and purpose
• This is a multi-component research codebase for FastVLM (an efficient vision encoder), combining:
– A Python “backend” for training, exporting, and serving vision-language models (the llava/, model_export/, predict.py, training scripts, etc.).
– An iOS SwiftUI “frontend” demo app (under app/FastVLM App/ and supporting Swift libraries in app/FastVLM/).
– Shell scripts for setup and model downloads.
• Purpose: train/finetune FastVLM variants, export them for Apple Silicon, serve them via CLI or web, and demonstrate on an iOS device.
File structure
Top-level directories:
• llava/ – core Python package for model building, multimodal encoding, training (train/), and serving (serve/).
• model_export/ – Python scripts to convert PyTorch checkpoints to Apple Silicon–compatible formats.
• app/ – Xcode project containing:
– FastVLM App/ – SwiftUI code and assets.
– FastVLM/ – Swift bindings (Objective-C header + Swift extension) to call the exported model.
– Video/ – camera capture and UI components.
• Root scripts: predict.py (CLI inference), get_models.sh, get_pretrained_mlx_model.sh.
• Configs: pyproject.toml, Xcode configs, Conda instructions in README.
Extract architectural insights from the README
• Training is built on LLaVA.
• Inference can be run via predict.py or the llava.serve Gradio/CLI modules.
• Models are exported for Apple Silicon in model_export/ and then consumed by the iOS app.
• The demo iOS app uses Core ML (via Swift wrappers) to run inference directly on device.
Main components
A. Model Training Pipeline (Python)
– Input: datasets & config (llava/train/..., flash-attn patches)
– Process: build multimodal encoder + LLM adapter → fine-tune → produce PyTorch checkpoints in checkpoints/.
B. Model Export Pipeline
– Input: PyTorch checkpoints → using model_export/export_vision_encoder.py and patch scripts → Output: Apple Silicon .mlmodel or quantized weights.
C. Model Serving Layer (Python)
– predict.py (CLI)
– llava.serve
• gradio_web_server.py (Web UI)
• controller.py, model_worker.py, register_worker.py
• Outputs inference results to user.
D. iOS Demonstration App (Swift/SwiftUI)
– Camera & UI: CameraController.swift, CameraControlsView.swift, VideoFrameView.swift
– Model wrapper: FastVLM.swift, FastVLM.h, MediaProcessingExtensions.swift, FastVLMModel.swift
– Assets & app entry: FastVLMApp.swift, ContentView.swift, InfoView.swift
– Loads the exported .mlmodel and performs on-device inference.
E. External services & dependencies
– PyTorch, Conda environment, LLaVA codebase (Git submodule)
– Apple frameworks: Core ML, AVFoundation, SwiftUI
– Hardware: GPU (training/serving), Apple Silicon (inference on device)
Relationships and data flow
1. Training → produces PyTorch checkpoint files.
2. Model Export → consumes checkpoint → emits Apple-compatible model artifacts.
3. Python Serving or CLI (predict.py) → loads checkpoint or exported model → takes image+prompt → returns text output.
4. iOS App → uses exported .mlmodel → camera frames → inference → UI displays results.
5. Shell scripts (get_models.sh) orchestrate model downloads into checkpoints/.
Architectural patterns & principles
• Clear separation of concerns: training, export, serving, and client UI are modular.
• Polyglot layering: Python backend vs. Swift frontend.
• Pipeline pattern: data flows from raw training data → model → export → inference.
• Extensibility: llava.model and llava.multimodal_encoder are plug-and-play components for new architectures.