How amshaker/mobile-o works
1. Top-level shape
The repo is split into three architecture planes: a Python research/training stack, an iOS on-device app, and an evaluation harness. The product goal is a single multimodal model family that can do text chat, image understanding, image generation, and image editing locally on mobile hardware.
2. Core model boundary
The unified model is organized around three functional blocks: a vision-language backbone, a diffusion-based image generator, and a small connector that passes multimodal context between them. This is the main architectural boundary to show in a diagram because it separates “understanding” from “generation” while still allowing joint conditioning.
3. Python model implementation
The root mobileo/ package contains the reference implementation of the model stack: language_model, multimodal_llava_encoder, multimodal_llava_projector, multimodal_decoder, and mobile_block. The naming indicates a LLaVA-like multimodal design with a MobileCLIP-style visual encoder, a compact LLM, and a diffusion decoder.
4. Training pipeline
Training is staged in three phases: pre-training, supervised fine-tuning, and post-training. The scripts in scripts/Mobile-O-0.5B and scripts/Mobile-O-1.5B drive those stages, while the Python code under mobileo/train/ contains the trainer variants and attention/memory adaptations used to make training practical.
5. Parameter adaptation flow
Post-training produces LoRA adapters for the language model rather than only a monolithic checkpoint. mobileo/merge_lora.py is the handoff point that merges those adapters back into the base weights to create the final inference checkpoint.
6. Inference/export boundary
The small top-level scripts infer_image_understanding.py, infer_image_generation.py, and infer_image_editing.py are the local inference entry points for the three main user-facing tasks. export.py and the Hugging Face checkpoint download flow indicate that deployment artifacts are created outside the app and then consumed by the mobile runtime.
7. iOS application architecture
Mobile-O-App/app/MobileO is a separate SwiftUI application target, not a thin wrapper around the Python code. Its major subsystems are UI views, view models, camera/video capture, model download services, and shared utilities.
8. Mobile runtime data flow
User input enters through chat, image picker, or camera views, is coordinated by ChatViewModel / SettingsViewModel, and then sent into the local model runtime. The response path is symmetric: text replies render in chat views, while generated images can be saved to the gallery through shared actions.
9. Device-side model lifecycle
The app is built to run fully on-device and relies on local model assets rather than cloud inference. ModelDownloadManager, NetworkMonitor, the download gate/progress views, and the bundled config files (preprocessor_config.json, scheduler_config.json) together define the lifecycle from “not installed” to “ready for inference.”
10. Media and performance layer
CameraManager, CameraPreviewView, and media-processing extensions support live camera understanding, while VAEKernels.metal suggests custom GPU acceleration for image-generation components. This is an important runtime boundary because the app’s performance claims depend on local CoreML/MLX-style execution and Metal kernels.
11. Evaluation subsystem
eval/lmms-eval is a vendored evaluation framework with a custom mobileo model adapter and many benchmark task definitions. It is separate from training and deployment, and exists to measure the model across understanding, OCR, reasoning, and video/image benchmarks.
12. Data and infrastructure dependencies
The repo depends heavily on Hugging Face for model and dataset distribution, and the README’s dataset stages map directly onto the training scripts. The rest of the infrastructure is mostly local/open-source tooling: Python packaging, Deepspeed config for large-model training, and Xcode for the app build.
13. How to draw the diagram
The cleanest diagram is: datasets and checkpoints on the left, Python training/export in the middle, and two deployment consumers on the right: the iOS app and the evaluation harness. Inside the model box, show the encoder → connector/projector → LLM/diffusion-decoder path, with the mobile app feeding text/image/camera inputs into the local inference path and returning either text or images.
Main entry points: mobile_block.py Mobile block · ChatViewModel.swift Chat VM · ModelDownloadManager.swift Download mgr · train.py Train code
Components 23
Data & Artifacts
Training data, checkpoints, and exported deployment assets.
- HF assets · External datasets, pretrained weights, and downloaded model artifacts.
Python Stack
Reference model, training, inference, and export code.
- Train stages · Scripted pre-training, supervised fine-tuning, and post-training entrypoints.
scripts/Mobile-O-0.5B/pre_train.sh - Train code · Core training implementations and memory/attention adaptations.
mobileo/train/train.py - LoRA merge · Merges post-training adapters back into base weights for inference.
mobileo/merge_lora.py - Export · Creates deployable artifacts consumed by the mobile runtime.
- Understand infer · Local text and image-understanding entrypoint.
infer_image_understanding.py - Generate infer · Local image generation entrypoint.
infer_image_generation.py - Edit infer · Local image editing entrypoint.
infer_image_editing.py - Vision encoder · MobileCLIP-style visual encoder for image and video context.
mobileo/model/multimodal_llava_encoder/builder.py - Projector · Maps visual features into the language-model token space.
mobileo/model/multimodal_llava_projector/builder.py - LLM backbone · Compact text model used for chat and multimodal reasoning.
mobileo/model/language_model/mobileo.py - Diffusion decoder · Image synthesis and editing decoder path.
mobileo/model/multimodal_decoder/builder.py - Mobile block · Shared architectural block tying understanding and generation together.
mobileo/model/mobile_block.py
iOS App
SwiftUI on-device application and runtime services.
- MobileO app · Standalone on-device iOS application target.
Mobile-O-App/app/MobileO/App/MobileOApp.swift - Chat UI · Conversation and message rendering surface.
Mobile-O-App/app/MobileO/Views/Chat/ConversationView.swift - Chat VM · Coordinates prompt submission, responses, and generation state.
Mobile-O-App/app/MobileO/ViewModels/ChatViewModel.swift - Download mgr · Manages model availability, download progress, and readiness state.
Mobile-O-App/app/MobileO/Services/ModelDownloadManager.swift - Camera stack · Camera preview and capture path for live visual input.
Mobile-O-App/app/MobileO/Video/CameraManager.swift - Understanding rt · Mobile runtime wrapper for text and image understanding.
Mobile-O-App/app/MobileO/Models/Understanding/ImageUnderstandingModel.swift - Generation rt · Mobile runtime wrapper for image generation and editing.
Mobile-O-App/app/MobileO/Models/Generation/MobileOModel.swift
Evaluation
Vendored benchmark harness and MobileO adapter.
- MobileO adapter · Evaluation harness adapter for the MobileO model family.
eval/lmms-eval/lmms_eval/models/mobileo.py - Eval harness · Task registry, metrics, and evaluation execution framework.
eval/lmms-eval/lmms_eval/evaluator.py - Bench tasks · Representative benchmark definitions spanning multimodal understanding.
eval/lmms-eval/lmms_eval/tasks/mmbench/mmbench.yaml
Connections 23
- HF assets → Train stages: data/weights · Training consumes datasets and pretrained sources.
- Train stages → Train code: drives · Phase scripts invoke the trainer stack.
- Train code → Mobile block: builds · Training updates the shared multimodal core.
- Mobile block → Vision encoder: uses · Shared core routes image features through the encoder.
- Mobile block → Projector: uses · Connector projects visual context into token space.
- Mobile block → LLM backbone: conditions · Language backbone handles chat and multimodal reasoning.
- Mobile block → Diffusion decoder: conditions · Generation path decodes image outputs.
- LoRA merge → Train code: adapter output · Post-training produces LoRA adapters from trainer output.
- LoRA merge → Export: merge · Adapters are merged into final inference weights.
- Export → Understand infer: feeds · Exported checkpoints power understanding inference.
- Export → Generate infer: feeds · Exported checkpoints power generation inference.
- Export → Edit infer: feeds · Exported checkpoints power editing inference.
- MobileO app → Download mgr: loads models · The app gates local inference on downloaded assets.
- MobileO app → Chat UI: hosts · App target contains the primary chat surface.
- Chat UI → Chat VM: binds · Views delegate coordination to the chat view model.
- Chat VM → Understanding rt: queries · Chat and image understanding calls flow into the local runtime.
- Chat VM → Generation rt: queries · Image generation and editing requests flow into the generation runtime.
- Camera stack → Understanding rt: captures input · Live camera frames are sent to the understanding path.
- Download mgr → Understanding rt: installs · Downloaded model assets enable understanding inference.
- Download mgr → Generation rt: installs · Downloaded model assets enable generation inference.
- MobileO adapter → Eval harness: registers · MobileO adapter plugs into the evaluation runner.
- Eval harness → Bench tasks: runs · The harness executes benchmark task suites.
- Bench tasks → MobileO adapter: scores · Benchmarks call the model adapter for predictions.