How oculus-samples/unity-passthroughcameraapisamples works
Main Components
• Game Engine & Scene Management:
• Unity runtime is the overall host.
• Main Scenes/Prefabs: StartScene (menu), sample scenes like CameraViewer, CameraToWorld, BrightnessEstimation, MultiObjectDetection, and ShaderSample.
• Camera & Data Acquisition Module:
• WebCamTextureManager: Handles initialization, permission requests, starting/stopping the camera feed.
• PassthroughCameraUtils: Converts 2D coordinates to 3D rays, retrieves camera intrinsics and pose in world space.
• Permission handling using Unity’s Android.Permission (for both android.permission.CAMERA and horizontos.permission.HEADSET_CAMERA).
• Sample Implementations:
• Each sample folder (inside Assets/PassthroughCameraApiSamples) demonstrates a different use case. For example, CameraViewer displays the camera feed, while BrightnessEstimation shows environmental adaptation; MultiObjectDetection integrates an on-device ML model (via Unity Sentis) for object detection; ShaderSample demonstrates GPU-based effects.
• External Dependencies & Plugins:
• External libraries like Meta MR Utility Kit are used for building blocks (e.g., camera rig) and environment raycasting.
• Unity Sentis integrated in the MultiObjectDetection sample for running on-device ML models.
• Android plugins and manifest modifications to fulfill hardware and permission requirements.
• Project Settings & Configurations:
• Directories like ProjectSettings, Packages, and XR settings provide the runtime configuration and dependency management for XR, Android specifics, and Unity version compatibility.
Relationships and Interactions
• Data Flow and Permission Handling:
• On startup, the PassthroughCameraPermissions component is triggered to request necessary permissions.
• After permissions are granted, WebCamTextureManager initializes the camera feed using Unity’s WebCamTexture.
• The camera feed is further processed:
– The main samples then get access to the texture data. For instance, the CameraViewer assigns it to a UI RawImage.
– The CameraToWorld sample uses PassthroughCameraUtils to transform 2D screen coordinates into 3D rays.
– BrightnessEstimation processes the texture to determine appropriate lighting or environmental parameters.
– MultiObjectDetection samples pass the camera feed through the Unity Sentis module to perform object classification, with resulting markers rendered in the world.
– ShaderSample sends the texture to a GPU shader to apply custom visual effects.
• Integration with External Tools:
• The Meta MR Utility Kit provides building blocks (such as camera rig prefabs) and environment raycast management (for real-world feature alignment).
• Unity Sentis is an external dependency that processes image data on-device and returns detections used by the MultiObjectDetection sample.
Important Architectural Patterns and Principles
• Modularity & Separation of Concerns:
• UI / Scene components are separated from the low-level camera feed initialization and permission management.
• Each sample encapsulates a specific feature (camera preview, coordinate conversion, brightness estimation, object detection, shader effects).
• Reusability:
• The helper classes (WebCamTextureManager, PassthroughCameraUtils) are used by multiple sample implementations to allow consistent processing of camera data.
• Real-time Data Flow:
• Emphasis on asynchronous handling (e.g., camera initiation, permission processing) and efficient data propagation from camera hardware through the Unity engine to UI elements.
• Integration of External Modules:
• The architecture shows clear integration points for external SDKs (Meta MR Utility Kit for XR and Unity Sentis for machine learning), ensuring that the samples can be tested and extended independently.
Elements
• Clear Labels: Mark out each major component – “Unity Runtime/Scene Manager”, “WebCamTextureManager”, “PassthroughCameraUtils”, “Permission Handler”, “Camera Samples (CameraViewer, CameraToWorld, etc.)”, “Meta MR Utility Kit”, “Unity Sentis Module”, “Android Native APIs/Manifest”.
• Annotations: Include any critical annotations regarding permission management, asynchronous processing, or integration with Android low-level APIs.
• Unity Scene/Frontend Layer (StartScene, Sample Scenes)
• Core Camera Data Management (WebCamTextureManager + PassthroughCameraUtils)
• Permission & Android API Integration (Permission Handler, AndroidManifest considerations)
• Sample-Specific Modules (CameraViewer, CameraToWorld, BrightnessEstimation, MultiObjectDetection, ShaderSample)
• External Integrations (Meta MR Utility Kit, Unity Sentis)