How homuler/mediapipeunityplugin works
Project Type and Purpose
• This project is a Unity native plugin that “ports the MediaPipe API (C++) one by one to C#.” It enables a Unity (2022.3 or later) project to call MediaPipe functions natively and even run official and custom solutions.
• The design makes it possible to run MediaPipe’s graph-based processing (calculators, graphs, packets) from Unity C# code. In brief, its purpose is to expose MediaPipe’s advanced multimedia processing (e.g., face detection, object detection, landmark detection) to Unity applications.
Key Components and Layers
a. Unity Application Layer:
– “Assets” in the file tree include examples, sample scenes, editor scripts, and UI components.
– Scripts in the Assets (e.g., HelloWorld.cs) illustrate how MonoBehaviour components instantiate and use classes like CalculatorGraph and PacketPoller.
– The plugin integrates into Unity’s build process and execution environment (both UnityEditor and runtime on different platforms).
b. C# API / Wrapper Layer:
– Located mainly in the “Packages” directory under “com.github.homuler.mediapipe”, this portion defines the C# wrappers and assemblies (and .asmdef files) for exposing MediaPipe functionalities.
– Here you have classes for CalculatorGraph, Packet, and various other APIs that mirror the native (C++) MediaPipe interfaces.
– This layer is critical for bridging Unity’s C# environment with the underlying native libraries.
c. Native / C++ / MediaPipe API Layer:
– The “mediapipe_api” directory contains the C++ code (with BUILD files, headers, and sources) that implement the port of MediaPipe’s calculators, graphs, formats, GPU helpers, etc.
– It leverages several third-party libraries and dependencies (e.g., absl, gl, protobuf) that are visible under “mediapipe_api/external” and “third_party”.
– This native layer is compiled (using Bazel and possibly Docker for cross-platform builds) into shared libraries that are then loaded at runtime using Unity’s native plugin mechanism.
d. Build and Integration Tools:
– Files such as WORKSPACE, .bazel* configuration files, build.py, and Docker folders indicate that there is a sophisticated build setup based on Bazel.
– These tools and configuration scripts generate the native binaries required by Unity.
– Additionally there are helper scripts (e.g., issue-template-version-updater.js) and documentation files in “docs” that support build and usage.
Relationships and Data Flow
• The primary flow begins in the Unity application (via a MonoBehaviour script such as HelloWorld.cs) that instantiates a CalculatorGraph.
• The C# API layer calls into the native layer (using DllImport and Unity native plugin interfaces) to create and manage MediaPipe graphs.
• MediaPipe’s native calculators process data (packets) through a graph; results are polled back into C# via the PacketPoller instance.
• The plugin also takes care of loading required assets (e.g., models or configuration files) via the StreamingAssets folder or local assets.
• The build system (Bazel, Docker, npm scripts) and directories for “mediapipe_api” and “third_party” show a clear separation: the native libraries are built independently and then integrated into the Unity project as a plugin.
Architectural Patterns and Design Principles
• Layered Architecture: A clear separation exists between the Unity (C#) presentation layer, the C# API/wrapper layer, and the native C++ implementation layer.
• Cross-platform Modular Design: The project is organized to support a range of platforms by abstracting platform-specific details (e.g., differences for Android, iOS, Windows, macOS, Linux).
• Build Integration: Utilizing Bazel, Docker and npm for packaging (tarballs/unitypackages) illustrates a systematic build and deployment pipeline.
Relevant Technologies and Frameworks
• Unity (C#) for the front-end/game engine layer.
• C# Native Plugins (via DllImport) for bridging to native C++ libraries.
• C++ with Bazel as a build tool for MediaPipe components.
• Third-party libraries such as absl, protobuf, OpenGL/Vulkan (for GPU, where applicable), and platform-specific libraries.
• Docker scripts and gradle templates for Android integration.