How 2bab/mediapiper works
Project Type and Purpose
• This is a Kotlin Multiplatform mobile sample app (Android + iOS) that demonstrates using Google’s MediaPipe tasks (LLM inference and object detection) in a shared codebase.
• The app module is a KMM library containing common business logic, UI definitions (Compose), and platform-specific implementations via expect/actual. The iosApp is a pure iOS Xcode project that consumes the generated KMM framework via CocoaPods.
File-Structure Highlights & Architectural Implications
• Top-level modules:
• app/: KMM library module (build.gradle.kts, src/commonMain, src/androidMain, src/iosMain)
• iosApp/: Xcode workspace & Pod integration, native Swift/SwiftUI entrypoints, MediaPipe pods
• commonMain: Contains core application entry (App.kt, Startup.kt), UI abstraction (Compose screens under llm/), view‐models (ChatViewModel), and the LLM operator interface (LLMOperator.kt).
• androidMain & iosMain: Provide platform-specific actualizations for Platform, LLMOperator, Theme, and App entrypoints.
• Pods & Native Frameworks: Under iosApp/Pods/MediaPipeTasksGenAI.xcframework and MediaPipeTasksGenAIC.xcframework, plus local .podspec for the KMM framework.
a. Shared KMM Core
• Business Logic: LLMOperator (expect/actual), ChatViewModel, AiRoute
• UI Layer: Compose UI screens (ChatScreen, LoadingScreen)
• Theme & Resource Layer
b. Android App
• Android entry (App.android.kt), Platform.android.kt
• Compose UI host, Gradle/Kotlin plugin
c. iOS App
• SwiftUI/Swift host (iosApp.swift, MediapipeLLMInferenceDelegate.swift)
• CocoaPods integration point, Pod‐installed MediaPipe frameworks
d. Native MediaPipe Frameworks
• MediaPipeTasksGenAI: ML & LLM inference engine
• MediaPipeTasksGenAIC: CPU/GPU optimized LLM cores, C headers
e. Device & Model Artifacts
• On-device LLM model files (pushed via adb on Android / folder ref on iOS)
• OS layers (Android SDK, iOS frameworks)
Component Relationships & Data Flow
1. User interacts with the App UI (Compose or SwiftUI)
2. UI invokes ChatViewModel → orchestrates LLMOperator
3. LLMOperator bridges to native MediaPipeTasksGenAI/C frameworks (via JNI on Android, via generated KMM framework + pods on iOS)
4. Native frameworks load local model files and perform inference
5. Inference results are returned back up through LLMOperator to the ViewModel → UI updates
Architectural Patterns & Design Principles
• Layered modularization: UI layer, ViewModel (MVVM), operator interface, native library
• Expect/Actual: abstraction of platform specifics behind a common API
• Single source of truth: all business logic and UI definitions reside in commonMain
• Dependency via CocoaPods (iOS) and Gradle (Android) for native libs