How openbmb/minicpm-v-apps works
This repository is a three-platform demo for running MiniCPM multimodal, text-only, and speech-generation models locally. The main user workflow is selecting and downloading model files, then entering text or image/video input in the iOS chat, processing that input with the local model runtime, and viewing a response; camera-based real-time understanding and VoxCPM2 speech generation are separate capabilities. The tree and excerpts expose the iOS workflow in more detail than the Android and HarmonyOS app layers, so their native bridges are shown without inferring unseen UI wiring.
Main entry points: MBHomeViewController.swift Chat home · MTMDWrapper.swift MTMD wrapper · llama.cpp-omni llama.cpp runtime · SwiftyCameraMainViewController.swift Camera capture
Components 21
iOS app
- Chat home
MiniCPM-V-demo/Sources/Home/ViewController/MBHomeViewController.swift - Chat messages
MiniCPM-V-demo/Sources/Base/Model/MBChatModel.swift - TTS screen
MiniCPM-V-demo/Sources/Home/ViewController/TtsViewController.swift
Capture and input
- Camera capture
MiniCPM-V-demo/Sources/Base/SwiftyCamera/ViewController/SwiftyCameraMainViewController.swift - Camera session
MiniCPM-V-demo/Sources/Base/SwiftyCamera/SwiftyCameraSource/SwiftyCamViewController.swift - Realtime settings
MiniCPM-V-demo/Sources/Settings/RealTimeVC/MBRealtimeUnderstandingViewController.swift - Video frame queue
MiniCPM-V-demo/Sources/Home/VideoProcessQueue/MBVideoSerialProcessQueueManager.swift - Frame extraction
MiniCPM-V-demo/Sources/Base/VideoFrameExtractor/MBVideoFrameExtractor.swift - Media preview
MiniCPM-V-demo/Sources/Base/MBImageViewer/MBImageVideoViewController.swift
Models and settings
- Model settings
MiniCPM-V-demo/Sources/Settings/ViewControler/MBSettingsViewController.swift - Model selection
MiniCPM-V-demo/Sources/Settings/ModelDetail/MBV4ModelDetailViewController.swift - Model downloads
MiniCPM-V-demo/Sources/Settings/DownloadManager/MBV4ModelDownloadManager.swift - Model artifacts
MiniCPM-V-demo/Sources/Base/Utils/MBModelArtifactStore.swift
Local inference
- MTMD wrapper
MiniCPM-V-demo/MTMDWrapper/MTMDWrapper.swift - TTS bridge
MiniCPM-V-demo/MTMDWrapper/Bridge/TtsBridge.h - VoxCPM2 engine
MiniCPM-V-demo/Sources/Base/Model/TtsEngine.swift - llama.cpp runtime
llama.cpp-omni
Other platforms
- Android JNI
MiniCPM-V-demo-Android/app/src/main/cpp/omni_jni.cpp - Harmony NAPI
MiniCPM-V-demo-HarmonyOS/entry/src/main/cpp/omni_napi.cpp
Other components
- User
- Model hosting
Connections 23
- User → Chat home: enters chat
- Chat home → Chat messages: updates messages
- Chat home → MTMD wrapper: runs inference
- MTMD wrapper → llama.cpp runtime: invokes
- MTMD wrapper → Chat home: returns output
- Chat home → Camera capture: opens capture
- Camera capture → Camera session: uses session
- Camera capture → Video frame queue: queues frames
- Video frame queue → MTMD wrapper: adds frames
- Chat home → Frame extraction: extracts frames
- Chat home → Media preview: previews media
- User → Model settings: chooses model
- Model settings → Model selection: opens details
- Model selection → Model downloads: requests download
- Model downloads → Model hosting: downloads files
- Model downloads → Model artifacts: stores artifacts
- User → Realtime settings: configures capture
- User → TTS screen: requests speech
- TTS screen → VoxCPM2 engine: generates speech
- VoxCPM2 engine → TTS bridge: calls native API
- TTS bridge → llama.cpp runtime: invokes
- Android JNI → llama.cpp runtime: invokes
- Harmony NAPI → llama.cpp runtime: invokes