# asus4/tf-lite-unity-sample architecture

> Architecture diagram of the GitHub repository asus4/tf-lite-unity-sample, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/asus4/tf-lite-unity-sample
- Repository: https://github.com/asus4/tf-lite-unity-sample

## Overview

##### Project Type and Purpose
- This is a Unity‐based multi-platform sample project demonstrating TensorFlow Lite and MediaPipe integration in Unity. It’s organized as a set of UPM packages (com.github.asus4.tflite, com.github.asus4.tflite.common, com.github.asus4.mediapipe) plus example scenes under Assets/Samples. The goal is to show how to load TFLite models, run inference (vision, audio, NLP, pose/face/hand tracking), and render results in Unity.

##### File Structure Analysis
- Top level:
- Assets/ contains all Unity scenes, scripts, shaders, compute kernels, prefabs, sample data
- Packages/ contains three UPM packages:
– com.github.asus4.tflite: C# wrappers + native plugin binaries (.dll, .aar, .framework)
– com.github.asus4.tflite.common: utilities (texture↔tensor, camera feed, base predictor, filters)
– com.github.asus4.mediapipe: higher-level calculators (LowPassFilter, RectTransformationCalculator, etc.)
- StreamingAssets/ holds the .tflite model files loaded at runtime
- Scripts/ holds build and patch scripts for compiling TensorFlow Lite native libraries
- build_tflite.py: helper to compile TFLite for various platforms
- ProjectSettings/, Packages/manifest.json, .gitignore: project configuration

##### Key Insights from README
- Demonstrates port of TensorFlow Lite official examples and MediaPipe tasks
- Supports iOS/Android/macOS/Windows/Linux with different delegates (CPU, GPU, Metal, NNAPI)
- Advises how to include packages via UPM and how to build native libraries yourself
- Samples licensed under MIT, core under Apache 2.0

##### Main Components
a. Unity Engine & Editor
– The host environment for scenes and C# scripts
b. Sample Scenes (Assets/Samples/*)
– Each scene + sample controller script invoke inference for a specific task
c. Common Utilities Package (com.github.asus4.tflite.common)
– BaseVisionTask, BaseImagePredictor, TextureToTensor, CameraTextureBackground
d. Core TFLite Package (com.github.asus4.tflite)
– Interpreter C# wrappers → calls native TensorFlow Lite library → native binaries
– SignatureRunner, InterpreterOptions, delegates (GPU, NNAPI, Metal)
e. MediaPipe Package (com.github.asus4.mediapipe)
– Post-processing calculators (low-pass, rect transform, velocity filter) used by pose/mesh tasks
f. Native Plugins / Binaries
– Platform-specific .dll/.aar/.framework under Packages/com.github.asus4.tflite/Plugins
g. StreamingAssets (Model Files)
– .tflite models loaded at runtime
h. Build Scripts
– build_tflite.py & patch scripts to regenerate native libs

##### Relationships & Data Flow
- Sample Scene UI → Sample Controller (C#) → Common Utilities (pre-/post-processing)
→ TFLite Interpreter C# API → Native Plugin → Hardware (CPU/GPU/Metal/NNAPI)
→ Interpreter returns tensor output → Common Utilities (e.g., LabelToTexture.compute, drawers)
→ Unity Renderer draws results
- MediaPipe package components plug into the C# flow between Interpreter output and final rendering

##### Architectural Patterns & Design Principles
- Layered Plugin Architecture:
- Presentation Layer (Unity scenes, UI, drawing scripts)
- Business Logic Layer (sample controllers, post-processors)
- Service Layer (common utilities for tensor conversion, camera feed)
- Integration Layer (TFLite C# wrappers)
- Native Layer (TFLite native binaries)
- Separation of Concerns: samples are isolated from core packages, core packages are platform-agnostic, native binaries are swapped per platform
- UPM Modularization: each package (core, common, mediapipe) has its own asmdef for clear dependency management

##### Essential Elements to Capture
- UPM package boundaries and dependencies
- StreamingAssets model loading path
- Native plugin selection per platform delegate
- Separation between sample presentation code and core inference layers
- Build script role in regenerating native binaries

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Unity Engine & Editor"
        UE["Unity Engine & Editor"]:::unity
    end

    subgraph "Presentation Layer"
        SS["Sample Scenes\n(Assets/Samples/)"]:::presentation
    end

    subgraph "Service Layer"
        CU["Common Utilities\n(com.github.asus4.tflite.common)"]:::common
    end

    subgraph "Integration Layer"
        TI["Interpreter.cs\nC# API"]:::core
        Del["Delegates\n(GPU/NNAPI/Metal)"]:::core
    end

    subgraph "Post-Processing Layer"
        MP["MediaPipe Calculators\n(com.github.asus4.mediapipe)"]:::mediapipe
    end

    subgraph "Native Layer"
        NP["Native Plugins & Binaries"]:::native
    end

    subgraph "Models Repository"
        SA["StreamingAssets\n(.tflite models)"]:::models
    end

    subgraph "Build Scripts"
        BS["build_tflite.py & Scripts/"]:::build
    end

    UE -->|"runs Scene"| SS
    SS -->|"pre-process Texture2D"| CU
    CU -->|"Invoke Interpreter"| TI
    TI -->|"use delegates"| Del
    Del -->|"native call"| NP
    NP -->|"exec on hardware"| UE
    TI -->|"output tensors"| MP
    MP -->|"post-process annotations"| SS
    SA -->|"Load model"| TI
    BS -->|"regenerate native libs"| NP

    click UE "https://github.com/asus4/tf-lite-unity-sample/tree/master/ProjectSettings/"
    click UE "https://github.com/asus4/tf-lite-unity-sample/blob/master/Packages/manifest.json"
    click SS "https://github.com/asus4/tf-lite-unity-sample/tree/master/Assets/Samples/"
    click CU "https://github.com/asus4/tf-lite-unity-sample/tree/master/Packages/com.github.asus4.tflite.common/Runtime/"
    click TI "https://github.com/asus4/tf-lite-unity-sample/blob/master/Packages/com.github.asus4.tflite/Runtime/Interpreter.cs"
    click Del "https://github.com/asus4/tf-lite-unity-sample/tree/master/Packages/com.github.asus4.tflite/Runtime/Delegates/"
    click NP "https://github.com/asus4/tf-lite-unity-sample/tree/master/Packages/com.github.asus4.tflite/Plugins/"
    click MP "https://github.com/asus4/tf-lite-unity-sample/tree/master/Packages/com.github.asus4.mediapipe/Runtime/"
    click SA "https://github.com/asus4/tf-lite-unity-sample/tree/master/Assets/StreamingAssets/"
    click BS "https://github.com/asus4/tf-lite-unity-sample/blob/master/build_tflite.py"
    click BS "https://github.com/asus4/tf-lite-unity-sample/tree/master/Scripts/"

    classDef unity fill:#1E90FF,stroke:#000,color:#fff
    classDef presentation fill:#32CD32,stroke:#000,color:#000
    classDef common fill:#87CEFA,stroke:#000,color:#000
    classDef core fill:#00008B,stroke:#000,color:#fff
    classDef mediapipe fill:#008080,stroke:#000,color:#fff
    classDef native fill:#FFA500,stroke:#000,color:#000
    classDef models fill:#800080,stroke:#000,color:#fff
    classDef build fill:#A9A9A9,stroke:#000,color:#000
```
