# amshaker/mobile-o architecture

> Architecture diagram of the GitHub repository amshaker/mobile-o, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amshaker/mobile-o
- Repository: https://github.com/amshaker/mobile-o

## Overview

1. **Top-level shape**
   The repo is split into three architecture planes: a Python research/training stack, an iOS on-device app, and an evaluation harness. The product goal is a single multimodal model family that can do text chat, image understanding, image generation, and image editing locally on mobile hardware.

2. **Core model boundary**
   The unified model is organized around three functional blocks: a vision-language backbone, a diffusion-based image generator, and a small connector that passes multimodal context between them. This is the main architectural boundary to show in a diagram because it separates “understanding” from “generation” while still allowing joint conditioning.

3. **Python model implementation**
   The root `mobileo/` package contains the reference implementation of the model stack: `language_model`, `multimodal_llava_encoder`, `multimodal_llava_projector`, `multimodal_decoder`, and `mobile_block`. The naming indicates a LLaVA-like multimodal design with a MobileCLIP-style visual encoder, a compact LLM, and a diffusion decoder.

4. **Training pipeline**
   Training is staged in three phases: pre-training, supervised fine-tuning, and post-training. The scripts in `scripts/Mobile-O-0.5B` and `scripts/Mobile-O-1.5B` drive those stages, while the Python code under `mobileo/train/` contains the trainer variants and attention/memory adaptations used to make training practical.

5. **Parameter adaptation flow**
   Post-training produces LoRA adapters for the language model rather than only a monolithic checkpoint. `mobileo/merge_lora.py` is the handoff point that merges those adapters back into the base weights to create the final inference checkpoint.

6. **Inference/export boundary**
   The small top-level scripts `infer_image_understanding.py`, `infer_image_generation.py`, and `infer_image_editing.py` are the local inference entry points for the three main user-facing tasks. `export.py` and the Hugging Face checkpoint download flow indicate that deployment artifacts are created outside the app and then consumed by the mobile runtime.

7. **iOS application architecture**
   `Mobile-O-App/app/MobileO` is a separate SwiftUI application target, not a thin wrapper around the Python code. Its major subsystems are UI views, view models, camera/video capture, model download services, and shared utilities.

8. **Mobile runtime data flow**
   User input enters through chat, image picker, or camera views, is coordinated by `ChatViewModel` / `SettingsViewModel`, and then sent into the local model runtime. The response path is symmetric: text replies render in chat views, while generated images can be saved to the gallery through shared actions.

9. **Device-side model lifecycle**
   The app is built to run fully on-device and relies on local model assets rather than cloud inference. `ModelDownloadManager`, `NetworkMonitor`, the download gate/progress views, and the bundled config files (`preprocessor_config.json`, `scheduler_config.json`) together define the lifecycle from “not installed” to “ready for inference.”

10. **Media and performance layer**
   `CameraManager`, `CameraPreviewView`, and media-processing extensions support live camera understanding, while `VAEKernels.metal` suggests custom GPU acceleration for image-generation components. This is an important runtime boundary because the app’s performance claims depend on local CoreML/MLX-style execution and Metal kernels.

11. **Evaluation subsystem**
   `eval/lmms-eval` is a vendored evaluation framework with a custom `mobileo` model adapter and many benchmark task definitions. It is separate from training and deployment, and exists to measure the model across understanding, OCR, reasoning, and video/image benchmarks.

12. **Data and infrastructure dependencies**
   The repo depends heavily on Hugging Face for model and dataset distribution, and the README’s dataset stages map directly onto the training scripts. The rest of the infrastructure is mostly local/open-source tooling: Python packaging, Deepspeed config for large-model training, and Xcode for the app build.

13. **How to draw the diagram**
   The cleanest diagram is: datasets and checkpoints on the left, Python training/export in the middle, and two deployment consumers on the right: the iOS app and the evaluation harness. Inside the model box, show the encoder → connector/projector → LLM/diffusion-decoder path, with the mobile app feeding text/image/camera inputs into the local inference path and returning either text or images.

Main entry points: [`mobileo/model/mobile_block.py`](https://github.com/amshaker/mobile-o/blob/main/mobileo/model/mobile_block.py) (Mobile block), [`Mobile-O-App/app/MobileO/ViewModels/ChatViewModel.swift`](https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/ViewModels/ChatViewModel.swift) (Chat VM), [`Mobile-O-App/app/MobileO/Services/ModelDownloadManager.swift`](https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/Services/ModelDownloadManager.swift) (Download mgr), [`mobileo/train/train.py`](https://github.com/amshaker/mobile-o/blob/main/mobileo/train/train.py) (Train code)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_artifacts["Data & Artifacts"]
  node_hf_assets[("HF assets<br/>data/checkpoints")]
end

subgraph group_python["Python Stack"]
  node_train_phase["Train stages<br/>pipeline/orchestration<br/>[pre_train.sh]"]
  node_train_core["Train code<br/>trainer stack<br/>[train.py]"]
  node_post_lora["LoRA merge<br/>adapter merge<br/>[merge_lora.py]"]
  node_export_artifact["Export"]
  node_infer_understand["Understand infer<br/>local inference"]
  node_infer_generate["Generate infer<br/>local inference"]
  node_infer_edit["Edit infer<br/>local inference"]
  node_vision_encoder["Vision encoder<br/>multimodal encoder<br/>[builder.py]"]
  node_projector["Projector<br/>multimodal bridge<br/>[builder.py]"]
  node_llm_backbone["LLM backbone<br/>language model<br/>[mobileo.py]"]
  node_diffusion_decoder["Diffusion decoder<br/>image generator<br/>[builder.py]"]
  node_mobile_block["Mobile block<br/>shared multimodal core<br/>[mobile_block.py]"]
end

subgraph group_ios["iOS App"]
  node_mobileo_app["MobileO app<br/>SwiftUI target<br/>[MobileOApp.swift]"]
  node_chat_ui["Chat UI<br/>chat views"]
  node_chat_vm["Chat VM<br/>view model"]
  node_download_mgr["Download mgr<br/>asset lifecycle"]
  node_camera_mgr["Camera stack<br/>capture pipeline"]
  node_understand_rt["Understanding rt<br/>on-device model"]
  node_generate_rt["Generation rt<br/>on-device model<br/>[MobileOModel.swift]"]
end

subgraph group_eval["Evaluation"]
  node_eval_adapter["MobileO adapter<br/>eval model adapter<br/>[mobileo.py]"]
  node_bench_harness["Eval harness<br/>benchmark runner<br/>[evaluator.py]"]
  node_bench_tasks["Bench tasks<br/>task suite<br/>[mmbench.yaml]"]
end

node_hf_assets -->|"data/weights"| node_train_phase
node_train_phase -->|"drives"| node_train_core
node_train_core -->|"builds"| node_mobile_block
node_mobile_block -->|"uses"| node_vision_encoder
node_mobile_block -->|"uses"| node_projector
node_mobile_block -->|"conditions"| node_llm_backbone
node_mobile_block -->|"conditions"| node_diffusion_decoder
node_post_lora -->|"adapter output"| node_train_core
node_post_lora -->|"merge"| node_export_artifact
node_export_artifact -->|"feeds"| node_infer_understand
node_export_artifact -->|"feeds"| node_infer_generate
node_export_artifact -->|"feeds"| node_infer_edit
node_mobileo_app -->|"loads models"| node_download_mgr
node_mobileo_app -->|"hosts"| node_chat_ui
node_chat_ui -->|"binds"| node_chat_vm
node_chat_vm -->|"queries"| node_understand_rt
node_chat_vm -->|"queries"| node_generate_rt
node_camera_mgr -->|"captures input"| node_understand_rt
node_download_mgr -->|"installs"| node_understand_rt
node_download_mgr -->|"installs"| node_generate_rt
node_eval_adapter -->|"registers"| node_bench_harness
node_bench_harness -->|"runs"| node_bench_tasks
node_bench_tasks -->|"scores"| node_eval_adapter

click node_train_phase "https://github.com/amshaker/mobile-o/blob/main/scripts/Mobile-O-0.5B/pre_train.sh"
click node_train_core "https://github.com/amshaker/mobile-o/blob/main/mobileo/train/train.py"
click node_post_lora "https://github.com/amshaker/mobile-o/blob/main/mobileo/merge_lora.py"
click node_infer_understand "https://github.com/amshaker/mobile-o/blob/main/infer_image_understanding.py"
click node_infer_generate "https://github.com/amshaker/mobile-o/blob/main/infer_image_generation.py"
click node_infer_edit "https://github.com/amshaker/mobile-o/blob/main/infer_image_editing.py"
click node_vision_encoder "https://github.com/amshaker/mobile-o/blob/main/mobileo/model/multimodal_llava_encoder/builder.py"
click node_projector "https://github.com/amshaker/mobile-o/blob/main/mobileo/model/multimodal_llava_projector/builder.py"
click node_llm_backbone "https://github.com/amshaker/mobile-o/blob/main/mobileo/model/language_model/mobileo.py"
click node_diffusion_decoder "https://github.com/amshaker/mobile-o/blob/main/mobileo/model/multimodal_decoder/builder.py"
click node_mobile_block "https://github.com/amshaker/mobile-o/blob/main/mobileo/model/mobile_block.py"
click node_mobileo_app "https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/App/MobileOApp.swift"
click node_chat_ui "https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/Views/Chat/ConversationView.swift"
click node_chat_vm "https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/ViewModels/ChatViewModel.swift"
click node_download_mgr "https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/Services/ModelDownloadManager.swift"
click node_camera_mgr "https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/Video/CameraManager.swift"
click node_understand_rt "https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/Models/Understanding/ImageUnderstandingModel.swift"
click node_generate_rt "https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/Models/Generation/MobileOModel.swift"
click node_eval_adapter "https://github.com/amshaker/mobile-o/blob/main/eval/lmms-eval/lmms_eval/models/mobileo.py"
click node_bench_harness "https://github.com/amshaker/mobile-o/blob/main/eval/lmms-eval/lmms_eval/evaluator.py"
click node_bench_tasks "https://github.com/amshaker/mobile-o/blob/main/eval/lmms-eval/lmms_eval/tasks/mmbench/mmbench.yaml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_hf_assets toneBlue
class node_train_phase,node_train_core,node_post_lora,node_export_artifact,node_infer_understand,node_infer_generate,node_infer_edit,node_vision_encoder,node_projector,node_llm_backbone,node_diffusion_decoder,node_mobile_block toneAmber
class node_mobileo_app,node_chat_ui,node_chat_vm,node_download_mgr,node_camera_mgr,node_understand_rt,node_generate_rt toneMint
class node_eval_adapter,node_bench_harness,node_bench_tasks toneRose
```

## Components

### Data & Artifacts

Training data, checkpoints, and exported deployment assets.

- **HF assets**: data/checkpoints · External datasets, pretrained weights, and downloaded model artifacts.

### Python Stack

Reference model, training, inference, and export code.

- [Train stages](https://github.com/amshaker/mobile-o/blob/main/scripts/Mobile-O-0.5B/pre_train.sh): pipeline/orchestration · Scripted pre-training, supervised fine-tuning, and post-training entrypoints. · `scripts/Mobile-O-0.5B/pre_train.sh`
- [Train code](https://github.com/amshaker/mobile-o/blob/main/mobileo/train/train.py): trainer stack · Core training implementations and memory/attention adaptations. · `mobileo/train/train.py`
- [LoRA merge](https://github.com/amshaker/mobile-o/blob/main/mobileo/merge_lora.py): adapter merge · Merges post-training adapters back into base weights for inference. · `mobileo/merge_lora.py`
- **Export**: checkpoint export · Creates deployable artifacts consumed by the mobile runtime.
- [Understand infer](https://github.com/amshaker/mobile-o/blob/main/infer_image_understanding.py): local inference · Local text and image-understanding entrypoint. · `infer_image_understanding.py`
- [Generate infer](https://github.com/amshaker/mobile-o/blob/main/infer_image_generation.py): local inference · Local image generation entrypoint. · `infer_image_generation.py`
- [Edit infer](https://github.com/amshaker/mobile-o/blob/main/infer_image_editing.py): local inference · Local image editing entrypoint. · `infer_image_editing.py`
- [Vision encoder](https://github.com/amshaker/mobile-o/blob/main/mobileo/model/multimodal_llava_encoder/builder.py): multimodal encoder · MobileCLIP-style visual encoder for image and video context. · `mobileo/model/multimodal_llava_encoder/builder.py`
- [Projector](https://github.com/amshaker/mobile-o/blob/main/mobileo/model/multimodal_llava_projector/builder.py): multimodal bridge · Maps visual features into the language-model token space. · `mobileo/model/multimodal_llava_projector/builder.py`
- [LLM backbone](https://github.com/amshaker/mobile-o/blob/main/mobileo/model/language_model/mobileo.py): language model · Compact text model used for chat and multimodal reasoning. · `mobileo/model/language_model/mobileo.py`
- [Diffusion decoder](https://github.com/amshaker/mobile-o/blob/main/mobileo/model/multimodal_decoder/builder.py): image generator · Image synthesis and editing decoder path. · `mobileo/model/multimodal_decoder/builder.py`
- [Mobile block](https://github.com/amshaker/mobile-o/blob/main/mobileo/model/mobile_block.py): shared multimodal core · Shared architectural block tying understanding and generation together. · `mobileo/model/mobile_block.py`

### iOS App

SwiftUI on-device application and runtime services.

- [MobileO app](https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/App/MobileOApp.swift): SwiftUI target · Standalone on-device iOS application target. · `Mobile-O-App/app/MobileO/App/MobileOApp.swift`
- [Chat UI](https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/Views/Chat/ConversationView.swift): chat views · Conversation and message rendering surface. · `Mobile-O-App/app/MobileO/Views/Chat/ConversationView.swift`
- [Chat VM](https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/ViewModels/ChatViewModel.swift): view model · Coordinates prompt submission, responses, and generation state. · `Mobile-O-App/app/MobileO/ViewModels/ChatViewModel.swift`
- [Download mgr](https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/Services/ModelDownloadManager.swift): asset lifecycle · Manages model availability, download progress, and readiness state. · `Mobile-O-App/app/MobileO/Services/ModelDownloadManager.swift`
- [Camera stack](https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/Video/CameraManager.swift): capture pipeline · Camera preview and capture path for live visual input. · `Mobile-O-App/app/MobileO/Video/CameraManager.swift`
- [Understanding rt](https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/Models/Understanding/ImageUnderstandingModel.swift): on-device model · Mobile runtime wrapper for text and image understanding. · `Mobile-O-App/app/MobileO/Models/Understanding/ImageUnderstandingModel.swift`
- [Generation rt](https://github.com/amshaker/mobile-o/blob/main/Mobile-O-App/app/MobileO/Models/Generation/MobileOModel.swift): on-device model · Mobile runtime wrapper for image generation and editing. · `Mobile-O-App/app/MobileO/Models/Generation/MobileOModel.swift`

### Evaluation

Vendored benchmark harness and MobileO adapter.

- [MobileO adapter](https://github.com/amshaker/mobile-o/blob/main/eval/lmms-eval/lmms_eval/models/mobileo.py): eval model adapter · Evaluation harness adapter for the MobileO model family. · `eval/lmms-eval/lmms_eval/models/mobileo.py`
- [Eval harness](https://github.com/amshaker/mobile-o/blob/main/eval/lmms-eval/lmms_eval/evaluator.py): benchmark runner · Task registry, metrics, and evaluation execution framework. · `eval/lmms-eval/lmms_eval/evaluator.py`
- [Bench tasks](https://github.com/amshaker/mobile-o/blob/main/eval/lmms-eval/lmms_eval/tasks/mmbench/mmbench.yaml): task suite · Representative benchmark definitions spanning multimodal understanding. · `eval/lmms-eval/lmms_eval/tasks/mmbench/mmbench.yaml`

## Connections

- HF assets → Train stages: data/weights · Training consumes datasets and pretrained sources.
- Train stages → Train code: drives · Phase scripts invoke the trainer stack.
- Train code → Mobile block: builds · Training updates the shared multimodal core.
- Mobile block → Vision encoder: uses · Shared core routes image features through the encoder.
- Mobile block → Projector: uses · Connector projects visual context into token space.
- Mobile block → LLM backbone: conditions · Language backbone handles chat and multimodal reasoning.
- Mobile block → Diffusion decoder: conditions · Generation path decodes image outputs.
- LoRA merge → Train code: adapter output · Post-training produces LoRA adapters from trainer output.
- LoRA merge → Export: merge · Adapters are merged into final inference weights.
- Export → Understand infer: feeds · Exported checkpoints power understanding inference.
- Export → Generate infer: feeds · Exported checkpoints power generation inference.
- Export → Edit infer: feeds · Exported checkpoints power editing inference.
- MobileO app → Download mgr: loads models · The app gates local inference on downloaded assets.
- MobileO app → Chat UI: hosts · App target contains the primary chat surface.
- Chat UI → Chat VM: binds · Views delegate coordination to the chat view model.
- Chat VM → Understanding rt: queries · Chat and image understanding calls flow into the local runtime.
- Chat VM → Generation rt: queries · Image generation and editing requests flow into the generation runtime.
- Camera stack → Understanding rt: captures input · Live camera frames are sent to the understanding path.
- Download mgr → Understanding rt: installs · Downloaded model assets enable understanding inference.
- Download mgr → Generation rt: installs · Downloaded model assets enable generation inference.
- MobileO adapter → Eval harness: registers · MobileO adapter plugs into the evaluation runner.
- Eval harness → Bench tasks: runs · The harness executes benchmark task suites.
- Bench tasks → MobileO adapter: scores · Benchmarks call the model adapter for predictions.
