How asus4/tf-lite-unity-sample works
Project Type and Purpose
• This is a Unity‐based multi-platform sample project demonstrating TensorFlow Lite and MediaPipe integration in Unity. It’s organized as a set of UPM packages (com.github.asus4.tflite, com.github.asus4.tflite.common, com.github.asus4.mediapipe) plus example scenes under Assets/Samples. The goal is to show how to load TFLite models, run inference (vision, audio, NLP, pose/face/hand tracking), and render results in Unity.
File Structure Analysis
• Top level:
• Assets/ contains all Unity scenes, scripts, shaders, compute kernels, prefabs, sample data
• Packages/ contains three UPM packages:
– com.github.asus4.tflite: C# wrappers + native plugin binaries (.dll, .aar, .framework)
– com.github.asus4.tflite.common: utilities (texture↔tensor, camera feed, base predictor, filters)
– com.github.asus4.mediapipe: higher-level calculators (LowPassFilter, RectTransformationCalculator, etc.)
• StreamingAssets/ holds the .tflite model files loaded at runtime
• Scripts/ holds build and patch scripts for compiling TensorFlow Lite native libraries
• build_tflite.py: helper to compile TFLite for various platforms
• ProjectSettings/, Packages/manifest.json, .gitignore: project configuration
Key Insights from README
• Demonstrates port of TensorFlow Lite official examples and MediaPipe tasks
• Supports iOS/Android/macOS/Windows/Linux with different delegates (CPU, GPU, Metal, NNAPI)
• Advises how to include packages via UPM and how to build native libraries yourself
• Samples licensed under MIT, core under Apache 2.0
Main Components
a. Unity Engine & Editor
– The host environment for scenes and C# scripts
b. Sample Scenes (Assets/Samples/*)
– Each scene + sample controller script invoke inference for a specific task
c. Common Utilities Package (com.github.asus4.tflite.common)
– BaseVisionTask, BaseImagePredictor, TextureToTensor, CameraTextureBackground
d. Core TFLite Package (com.github.asus4.tflite)
– Interpreter C# wrappers → calls native TensorFlow Lite library → native binaries
– SignatureRunner, InterpreterOptions, delegates (GPU, NNAPI, Metal)
e. MediaPipe Package (com.github.asus4.mediapipe)
– Post-processing calculators (low-pass, rect transform, velocity filter) used by pose/mesh tasks
f. Native Plugins / Binaries
– Platform-specific .dll/.aar/.framework under Packages/com.github.asus4.tflite/Plugins
g. StreamingAssets (Model Files)
– .tflite models loaded at runtime
h. Build Scripts
– build_tflite.py & patch scripts to regenerate native libs
Relationships & Data Flow
• Sample Scene UI → Sample Controller (C#) → Common Utilities (pre-/post-processing)
→ TFLite Interpreter C# API → Native Plugin → Hardware (CPU/GPU/Metal/NNAPI)
→ Interpreter returns tensor output → Common Utilities (e.g., LabelToTexture.compute, drawers)
→ Unity Renderer draws results
• MediaPipe package components plug into the C# flow between Interpreter output and final rendering
Architectural Patterns & Design Principles
• Layered Plugin Architecture:
• Presentation Layer (Unity scenes, UI, drawing scripts)
• Business Logic Layer (sample controllers, post-processors)
• Service Layer (common utilities for tensor conversion, camera feed)
• Integration Layer (TFLite C# wrappers)
• Native Layer (TFLite native binaries)
• Separation of Concerns: samples are isolated from core packages, core packages are platform-agnostic, native binaries are swapped per platform
• UPM Modularization: each package (core, common, mediapipe) has its own asmdef for clear dependency management
Essential Elements to Capture
• UPM package boundaries and dependencies
• StreamingAssets model loading path
• Native plugin selection per platform delegate
• Separation between sample presentation code and core inference layers
• Build script role in regenerating native binaries