# openbmb/minicpm-v-apps architecture

> Architecture diagram of the GitHub repository openbmb/minicpm-v-apps, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openbmb/minicpm-v-apps
- Repository: https://github.com/openbmb/minicpm-v-apps

## Overview

This repository is a three-platform demo for running MiniCPM multimodal, text-only, and speech-generation models locally. The main user workflow is selecting and downloading model files, then entering text or image/video input in the iOS chat, processing that input with the local model runtime, and viewing a response; camera-based real-time understanding and VoxCPM2 speech generation are separate capabilities. The tree and excerpts expose the iOS workflow in more detail than the Android and HarmonyOS app layers, so their native bridges are shown without inferring unseen UI wiring.

Main entry points: [`MiniCPM-V-demo/Sources/Home/ViewController/MBHomeViewController.swift`](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Home/ViewController/MBHomeViewController.swift) (Chat home), [`MiniCPM-V-demo/MTMDWrapper/MTMDWrapper.swift`](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/MTMDWrapper/MTMDWrapper.swift) (MTMD wrapper), [`llama.cpp-omni`](https://github.com/openbmb/minicpm-v-apps/blob/main/llama.cpp-omni) (llama.cpp runtime), [`MiniCPM-V-demo/Sources/Base/SwiftyCamera/ViewController/SwiftyCameraMainViewController.swift`](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/SwiftyCamera/ViewController/SwiftyCameraMainViewController.swift) (Camera capture)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ios_app["iOS app"]
  node_home["Chat home"]
  node_chat_model["Chat messages<br/>[MBChatModel.swift]"]
  node_tts_ui["TTS screen"]
end

subgraph group_input["Capture and input"]
  node_camera_ui["Camera capture"]
  node_camera_session["Camera session"]
  node_realtime["Realtime settings"]
  node_frame_queue["Video frame queue"]
  node_frame_extractor["Frame extraction"]
  node_media_preview["Media preview"]
end

subgraph group_models["Models and settings"]
  node_settings["Model settings"]
  node_model_detail["Model selection"]
  node_model_download["Model downloads"]
  node_artifact_store[("Model artifacts")]
end

subgraph group_runtime["Local inference"]
  node_wrapper["MTMD wrapper<br/>[MTMDWrapper.swift]"]
  node_native_tts["TTS bridge<br/>[TtsBridge.h]"]
  node_tts_engine["VoxCPM2 engine<br/>[TtsEngine.swift]"]
  node_llama_runtime["llama.cpp runtime<br/>[llama.cpp-omni]"]
end

subgraph group_other_platforms["Other platforms"]
  node_android_jni["Android JNI<br/>[omni_jni.cpp]"]
  node_harmony_napi["Harmony NAPI<br/>[omni_napi.cpp]"]
end

node_user(("User"))
node_remote_weights["Model hosting"]

node_user -->|"enters chat"| node_home
node_home -->|"updates messages"| node_chat_model
node_home -->|"runs inference"| node_wrapper
node_wrapper -->|"invokes"| node_llama_runtime
node_wrapper -->|"returns output"| node_home
node_home -->|"opens capture"| node_camera_ui
node_camera_ui -->|"uses session"| node_camera_session
node_camera_ui -->|"queues frames"| node_frame_queue
node_frame_queue -->|"adds frames"| node_wrapper
node_home -->|"extracts frames"| node_frame_extractor
node_home -->|"previews media"| node_media_preview
node_user -->|"chooses model"| node_settings
node_settings -->|"opens details"| node_model_detail
node_model_detail -->|"requests download"| node_model_download
node_model_download -->|"downloads files"| node_remote_weights
node_model_download -->|"stores artifacts"| node_artifact_store
node_user -->|"configures capture"| node_realtime
node_user -->|"requests speech"| node_tts_ui
node_tts_ui -->|"generates speech"| node_tts_engine
node_tts_engine -->|"calls native API"| node_native_tts
node_native_tts -->|"invokes"| node_llama_runtime
node_android_jni -->|"invokes"| node_llama_runtime
node_harmony_napi -->|"invokes"| node_llama_runtime

click node_home "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Home/ViewController/MBHomeViewController.swift"
click node_chat_model "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/Model/MBChatModel.swift"
click node_camera_ui "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/SwiftyCamera/ViewController/SwiftyCameraMainViewController.swift"
click node_camera_session "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/SwiftyCamera/SwiftyCameraSource/SwiftyCamViewController.swift"
click node_realtime "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Settings/RealTimeVC/MBRealtimeUnderstandingViewController.swift"
click node_frame_queue "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Home/VideoProcessQueue/MBVideoSerialProcessQueueManager.swift"
click node_frame_extractor "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/VideoFrameExtractor/MBVideoFrameExtractor.swift"
click node_media_preview "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/MBImageViewer/MBImageVideoViewController.swift"
click node_settings "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Settings/ViewControler/MBSettingsViewController.swift"
click node_model_detail "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Settings/ModelDetail/MBV4ModelDetailViewController.swift"
click node_model_download "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Settings/DownloadManager/MBV4ModelDownloadManager.swift"
click node_artifact_store "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/Utils/MBModelArtifactStore.swift"
click node_wrapper "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/MTMDWrapper/MTMDWrapper.swift"
click node_native_tts "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/MTMDWrapper/Bridge/TtsBridge.h"
click node_tts_engine "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/Model/TtsEngine.swift"
click node_tts_ui "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Home/ViewController/TtsViewController.swift"
click node_android_jni "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo-Android/app/src/main/cpp/omni_jni.cpp"
click node_harmony_napi "https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo-HarmonyOS/entry/src/main/cpp/omni_napi.cpp"
click node_llama_runtime "https://github.com/openbmb/minicpm-v-apps/blob/main/llama.cpp-omni"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_home,node_chat_model,node_tts_ui,node_user toneBlue
class node_camera_ui,node_camera_session,node_realtime,node_frame_queue,node_frame_extractor,node_media_preview toneAmber
class node_settings,node_model_detail,node_model_download,node_artifact_store toneMint
class node_wrapper,node_native_tts,node_tts_engine,node_llama_runtime toneRose
class node_android_jni,node_harmony_napi,node_remote_weights toneIndigo
```

## Components

### iOS app

- [Chat home](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Home/ViewController/MBHomeViewController.swift): `MiniCPM-V-demo/Sources/Home/ViewController/MBHomeViewController.swift`
- [Chat messages](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/Model/MBChatModel.swift): `MiniCPM-V-demo/Sources/Base/Model/MBChatModel.swift`
- [TTS screen](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Home/ViewController/TtsViewController.swift): `MiniCPM-V-demo/Sources/Home/ViewController/TtsViewController.swift`

### Capture and input

- [Camera capture](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/SwiftyCamera/ViewController/SwiftyCameraMainViewController.swift): `MiniCPM-V-demo/Sources/Base/SwiftyCamera/ViewController/SwiftyCameraMainViewController.swift`
- [Camera session](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/SwiftyCamera/SwiftyCameraSource/SwiftyCamViewController.swift): `MiniCPM-V-demo/Sources/Base/SwiftyCamera/SwiftyCameraSource/SwiftyCamViewController.swift`
- [Realtime settings](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Settings/RealTimeVC/MBRealtimeUnderstandingViewController.swift): `MiniCPM-V-demo/Sources/Settings/RealTimeVC/MBRealtimeUnderstandingViewController.swift`
- [Video frame queue](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Home/VideoProcessQueue/MBVideoSerialProcessQueueManager.swift): `MiniCPM-V-demo/Sources/Home/VideoProcessQueue/MBVideoSerialProcessQueueManager.swift`
- [Frame extraction](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/VideoFrameExtractor/MBVideoFrameExtractor.swift): `MiniCPM-V-demo/Sources/Base/VideoFrameExtractor/MBVideoFrameExtractor.swift`
- [Media preview](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/MBImageViewer/MBImageVideoViewController.swift): `MiniCPM-V-demo/Sources/Base/MBImageViewer/MBImageVideoViewController.swift`

### Models and settings

- [Model settings](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Settings/ViewControler/MBSettingsViewController.swift): `MiniCPM-V-demo/Sources/Settings/ViewControler/MBSettingsViewController.swift`
- [Model selection](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Settings/ModelDetail/MBV4ModelDetailViewController.swift): `MiniCPM-V-demo/Sources/Settings/ModelDetail/MBV4ModelDetailViewController.swift`
- [Model downloads](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Settings/DownloadManager/MBV4ModelDownloadManager.swift): `MiniCPM-V-demo/Sources/Settings/DownloadManager/MBV4ModelDownloadManager.swift`
- [Model artifacts](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/Utils/MBModelArtifactStore.swift): `MiniCPM-V-demo/Sources/Base/Utils/MBModelArtifactStore.swift`

### Local inference

- [MTMD wrapper](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/MTMDWrapper/MTMDWrapper.swift): `MiniCPM-V-demo/MTMDWrapper/MTMDWrapper.swift`
- [TTS bridge](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/MTMDWrapper/Bridge/TtsBridge.h): `MiniCPM-V-demo/MTMDWrapper/Bridge/TtsBridge.h`
- [VoxCPM2 engine](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo/Sources/Base/Model/TtsEngine.swift): `MiniCPM-V-demo/Sources/Base/Model/TtsEngine.swift`
- [llama.cpp runtime](https://github.com/openbmb/minicpm-v-apps/blob/main/llama.cpp-omni): `llama.cpp-omni`

### Other platforms

- [Android JNI](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo-Android/app/src/main/cpp/omni_jni.cpp): `MiniCPM-V-demo-Android/app/src/main/cpp/omni_jni.cpp`
- [Harmony NAPI](https://github.com/openbmb/minicpm-v-apps/blob/main/MiniCPM-V-demo-HarmonyOS/entry/src/main/cpp/omni_napi.cpp): `MiniCPM-V-demo-HarmonyOS/entry/src/main/cpp/omni_napi.cpp`

### Other components

- **User**
- **Model hosting**

## Connections

- User → Chat home: enters chat
- Chat home → Chat messages: updates messages
- Chat home → MTMD wrapper: runs inference
- MTMD wrapper → llama.cpp runtime: invokes
- MTMD wrapper → Chat home: returns output
- Chat home → Camera capture: opens capture
- Camera capture → Camera session: uses session
- Camera capture → Video frame queue: queues frames
- Video frame queue → MTMD wrapper: adds frames
- Chat home → Frame extraction: extracts frames
- Chat home → Media preview: previews media
- User → Model settings: chooses model
- Model settings → Model selection: opens details
- Model selection → Model downloads: requests download
- Model downloads → Model hosting: downloads files
- Model downloads → Model artifacts: stores artifacts
- User → Realtime settings: configures capture
- User → TTS screen: requests speech
- TTS screen → VoxCPM2 engine: generates speech
- VoxCPM2 engine → TTS bridge: calls native API
- TTS bridge → llama.cpp runtime: invokes
- Android JNI → llama.cpp runtime: invokes
- Harmony NAPI → llama.cpp runtime: invokes