How google/xr-objects works
Key components to capture include:
User Input & Sensor Acquisition
• “Camera/Video Input Module” that receives a live video stream either from the smartphone’s built–in camera (for the Android AR implementation) or from an external webcam (for the headset “DemoHeadset” version).
• That on the headset implementation the project uses UVC4UnityAndroid to receive the video stream.
Gemini LLM Integration for Metadata
• A “Gemini Integration” component that is invoked by the system (for example, through the ImageQuery.cs script) to fetch metadata about an identified object.
• That the Gemini API key (configured in the project) is used to call the multimodal LLM, which enriches object data with context, summaries, or answers.
AR/Spatial Interaction & UI Module
• This module—the AR Cursor, the scene UI panels, and context menu scripts (located in Assets/XRObjects/Scripts and related UI folders)—responds to user gestures (tap/click) to reveal actionable digital content.
• That the context menus are spatially anchored to the real–world objects as detected by the system.
Cross–Platform XR Integration
• Since the project supports both Android–handheld devices (using ARCore and AR Foundation) and XR headsets (like Quest 3 or Vision Pro, via the “DemoHeadset” template), illustrate that the project contains two build configurations.
• For Android, the camera feed comes from the device in conjunction with ARCore, while the headset uses a dedicated “DemoHeadset” scene along with an external camera.
• This branch can be represented as separate deployment pipelines that eventually connect to the common object detection and UI logic.
Supporting Libraries & Third–Party Components
• In a lower level detail (or as annotations), mention that the system depends on various third–party components such as the MediaPipe Unity Plugin, Unity Android Speech Recognizer, and UVC4UnityAndroid.
• These may appear as “external libraries” feeding into the object detection or the UI integration modules.
Data Flow & Interaction Patterns
• The camera module streaming video to the MediaPipe object detection module.
• On tapping an identified object, the UI system sends a query via the Gemini integration module for additional metadata.
• The resulting information then feeds into the display logic of the context menu.
• That, once objects are detected, there is an option to stop the continuous detection to improve application performance while maintaining the set of previously identified objects.