# oculus-samples/unity-passthroughcameraapisamples architecture

> Architecture diagram of the GitHub repository oculus-samples/unity-passthroughcameraapisamples, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/oculus-samples/unity-passthroughcameraapisamples
- Repository: https://github.com/oculus-samples/unity-passthroughcameraapisamples

## Overview

##### Main Components
- Game Engine & Scene Management:
- Unity runtime is the overall host.
- Main Scenes/Prefabs: StartScene (menu), sample scenes like CameraViewer, CameraToWorld, BrightnessEstimation, MultiObjectDetection, and ShaderSample.
- Camera & Data Acquisition Module:
- WebCamTextureManager: Handles initialization, permission requests, starting/stopping the camera feed.
- PassthroughCameraUtils: Converts 2D coordinates to 3D rays, retrieves camera intrinsics and pose in world space.
- Permission handling using Unity’s Android.Permission (for both android.permission.CAMERA and horizontos.permission.HEADSET_CAMERA).
- Sample Implementations:
- Each sample folder (inside Assets/PassthroughCameraApiSamples) demonstrates a different use case. For example, CameraViewer displays the camera feed, while BrightnessEstimation shows environmental adaptation; MultiObjectDetection integrates an on-device ML model (via Unity Sentis) for object detection; ShaderSample demonstrates GPU-based effects.
- External Dependencies & Plugins:
- External libraries like Meta MR Utility Kit are used for building blocks (e.g., camera rig) and environment raycasting.
- Unity Sentis integrated in the MultiObjectDetection sample for running on-device ML models.
- Android plugins and manifest modifications to fulfill hardware and permission requirements.
- Project Settings & Configurations:
- Directories like ProjectSettings, Packages, and XR settings provide the runtime configuration and dependency management for XR, Android specifics, and Unity version compatibility.

##### Relationships and Interactions
- Data Flow and Permission Handling:
- On startup, the PassthroughCameraPermissions component is triggered to request necessary permissions.
- After permissions are granted, WebCamTextureManager initializes the camera feed using Unity’s WebCamTexture.
- The camera feed is further processed:
– The main samples then get access to the texture data. For instance, the CameraViewer assigns it to a UI RawImage.
– The CameraToWorld sample uses PassthroughCameraUtils to transform 2D screen coordinates into 3D rays.
– BrightnessEstimation processes the texture to determine appropriate lighting or environmental parameters.
– MultiObjectDetection samples pass the camera feed through the Unity Sentis module to perform object classification, with resulting markers rendered in the world.
– ShaderSample sends the texture to a GPU shader to apply custom visual effects.
- Integration with External Tools:
- The Meta MR Utility Kit provides building blocks (such as camera rig prefabs) and environment raycast management (for real-world feature alignment).
- Unity Sentis is an external dependency that processes image data on-device and returns detections used by the MultiObjectDetection sample.

##### Important Architectural Patterns and Principles
- Modularity & Separation of Concerns:
- UI / Scene components are separated from the low-level camera feed initialization and permission management.
- Each sample encapsulates a specific feature (camera preview, coordinate conversion, brightness estimation, object detection, shader effects).
- Reusability:
- The helper classes (WebCamTextureManager, PassthroughCameraUtils) are used by multiple sample implementations to allow consistent processing of camera data.
- Real-time Data Flow:
- Emphasis on asynchronous handling (e.g., camera initiation, permission processing) and efficient data propagation from camera hardware through the Unity engine to UI elements.
- Integration of External Modules:
- The architecture shows clear integration points for external SDKs (Meta MR Utility Kit for XR and Unity Sentis for machine learning), ensuring that the samples can be tested and extended independently.

##### Elements
- Clear Labels: Mark out each major component – “Unity Runtime/Scene Manager”, “WebCamTextureManager”, “PassthroughCameraUtils”, “Permission Handler”, “Camera Samples (CameraViewer, CameraToWorld, etc.)”, “Meta MR Utility Kit”, “Unity Sentis Module”, “Android Native APIs/Manifest”.
- Annotations: Include any critical annotations regarding permission management, asynchronous processing, or integration with Android low-level APIs.

- Unity Scene/Frontend Layer (StartScene, Sample Scenes)
- Core Camera Data Management (WebCamTextureManager + PassthroughCameraUtils)
- Permission & Android API Integration (Permission Handler, AndroidManifest considerations)
- Sample-Specific Modules (CameraViewer, CameraToWorld, BrightnessEstimation, MultiObjectDetection, ShaderSample)
- External Integrations (Meta MR Utility Kit, Unity Sentis)

## Diagram (Mermaid)

```mermaid
graph TD
    %% Unity Engine
    U["Unity Runtime/Scene Manager"]:::engine

    %% Frontend Layer
    subgraph "Frontend Layer"
        A1["StartScene"]:::frontend
        A2["CameraViewer Sample"]:::frontend
        A3["CameraToWorld Sample"]:::frontend
        A4["BrightnessEstimation Sample"]:::frontend
        A5["MultiObjectDetection Sample"]:::frontend
        A6["ShaderSample"]:::frontend
    end

    %% Core Camera Data Management
    subgraph "Core Camera Data Management"
        B1["WebCamTextureManager"]:::core
        B2["PassthroughCameraUtils"]:::core
    end

    %% Permission & Android API Integration
    subgraph "Permission & Android API Integration"
        C1["PassthroughCameraPermissions"]:::permission
        C2["Android Manifest"]:::permission
    end

    %% Sample-Specific Modules
    subgraph "Sample-Specific Modules"
        D1["CameraViewer Manager"]:::sample
        D2["CameraToWorldManager"]:::sample
        D3["CameraToWorldRayRenderer"]:::sample
        D4["BrightnessEstimationManager"]:::sample
        D5["BrightnessEstimationDebugger"]:::sample
        D6["DetectionManager"]:::sample
        D7["SentisInferenceRunManager"]:::sample
        D8["SentisInferenceUiManager"]:::sample
        D9["SentisObjectDetectedUiManager"]:::sample
        D10["ShaderSampleManager"]:::sample
    end

    %% External Integrations & Configurations
    subgraph "External Integrations & Configurations"
        E1["Meta MR Utility Kit"]:::external
        E2["Unity Sentis Module"]:::external
        E3["XR Settings"]:::external
        E4["Project Settings"]:::external
    end

    %% Relationships from Unity Engine to Frontend and Configurations
    U -->|"load_scene"| A1
    U -->|"load_scene"| A2
    U -->|"load_scene"| A3
    U -->|"load_scene"| A4
    U -->|"load_scene"| A5
    U -->|"load_scene"| A6
    U -->|"XR_config"| E3
    U -->|"proj_config"| E4

    %% Permission Handling and Camera Initialization
    C2 -->|"perm_config"| C1
    C1 -->|"init_camera"| B1

    %% Distribution of Camera Feed to Sample Modules
    B1 -->|"feed_data"| D1
    B1 -->|"feed_data"| D2
    B2 -->|"coord_utils"| D2
    D2 -->|"gen_rays"| D3
    B1 -->|"feed_data"| D4
    B1 -->|"feed_data"| D6
    B1 -->|"feed_data"| D10

    %% Video Feed linked to UI Scenes in Frontend
    A2 -->|"UI_control"| D1
    A3 -->|"UI_control"| D2
    A3 -->|"UI_control"| D3
    A4 -->|"UI_control"| D4
    A4 -->|"UI_control"| D5
    A5 -->|"UI_control"| D6
    A5 -->|"UI_control"| D7
    A5 -->|"UI_control"| D8
    A5 -->|"UI_control"| D9
    A6 -->|"UI_control"| D10

    %% MultiObjectDetection Sample - Inference Pipeline
    D6 -->|"run_inference"| D7
    D6 -->|"display_UI"| D8
    D6 -->|"display_det"| D9

    %% External Integrations supporting modules
    E1 -->|"support_feed"| B1
    E2 -->|"ML_integration"| D7
    E2 -->|"ML_integration"| D8
    E2 -->|"ML_integration"| D9

    %% Class definitions for styling
    classDef frontend fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef core fill:#AED6F1,stroke:#2471A3,stroke-width:2px;
    classDef permission fill:#A9DFBF,stroke:#27AE60,stroke-width:2px;
    classDef sample fill:#D2B4DE,stroke:#8E44AD,stroke-width:2px;
    classDef external fill:#F9E79F,stroke:#D4AC0D,stroke-width:2px;
    classDef engine fill:#A3E4D7,stroke:#16A085,stroke-width:2px;

    %% Click Events for Frontend Layer
    click A1 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/StartScene/StartScene.unity"
    click A2 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/CameraViewer/CameraViewer.unity"
    click A3 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/CameraToWorld/CameraToWorld.unity"
    click A4 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/BrightnessEstimation/BrightnessEstimation.unity"
    click A5 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/MultiObjectDetection/MultiObjectDetection.unity"
    click A6 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/ShaderSample/ShaderSample.unity"

    %% Click Events for Core Camera Data Management
    click B1 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/PassthroughCamera/Scripts/WebCamTextureManager.cs"
    click B2 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/PassthroughCamera/Scripts/PassthroughCameraUtils.cs"

    %% Click Events for Permission & Android API Integration
    click C1 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/PassthroughCamera/Scripts/PassthroughCameraPermissions.cs"
    click C2 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/Plugins/Android/AndroidManifest.xml"

    %% Click Events for Sample-Specific Modules
    click D1 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/CameraViewer/Scripts/CameraViewerManager.cs"
    click D2 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/CameraToWorld/Scripts/CameraToWorldManager.cs"
    click D3 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/CameraToWorld/Scripts/CameraToWorldRayRenderer.cs"
    click D4 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/BrightnessEstimation/Scripts/BrightnessEstimationManager.cs"
    click D5 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/BrightnessEstimation/Scripts/BrightnessEstimationDebugger.cs"
    click D6 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/MultiObjectDetection/DetectionManager/Scripts/DetectionManager.cs"
    click D7 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/MultiObjectDetection/SentisInference/Scripts/SentisInferenceRunManager.cs"
    click D8 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/MultiObjectDetection/SentisInference/Scripts/SentisInferenceUiManager.cs"
    click D9 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/MultiObjectDetection/SentisInference/Scripts/SentisObjectDetectedUiManager.cs"
    click D10 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/PassthroughCameraApiSamples/ShaderSample/Scripts/ShaderSampleManager.cs"

    %% Click Events for External Integrations & Configurations
    click E1 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/MetaXR/MetaXRProjectSettings.asset"
    click E3 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/blob/main/Assets/XR/Settings/OculusSettings.asset"
    click E4 "https://github.com/oculus-samples/unity-passthroughcameraapisamples/tree/main/ProjectSettings"
```
