# google/xr-objects architecture

> Architecture diagram of the GitHub repository google/xr-objects, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google/xr-objects
- Repository: https://github.com/google/xr-objects

## Overview

Key components to capture include:

##### User Input & Sensor Acquisition
- “Camera/Video Input Module” that receives a live video stream either from the smartphone’s built–in camera (for the Android AR implementation) or from an external webcam (for the headset “DemoHeadset” version).
- That on the headset implementation the project uses UVC4UnityAndroid to receive the video stream.

##### Gemini LLM Integration for Metadata
- A “Gemini Integration” component that is invoked by the system (for example, through the ImageQuery.cs script) to fetch metadata about an identified object.
- That the Gemini API key (configured in the project) is used to call the multimodal LLM, which enriches object data with context, summaries, or answers.

##### AR/Spatial Interaction & UI Module
- This module—the AR Cursor, the scene UI panels, and context menu scripts (located in Assets/XRObjects/Scripts and related UI folders)—responds to user gestures (tap/click) to reveal actionable digital content.
- That the context menus are spatially anchored to the real–world objects as detected by the system.

##### Cross–Platform XR Integration
- Since the project supports both Android–handheld devices (using ARCore and AR Foundation) and XR headsets (like Quest 3 or Vision Pro, via the “DemoHeadset” template), illustrate that the project contains two build configurations.
- For Android, the camera feed comes from the device in conjunction with ARCore, while the headset uses a dedicated “DemoHeadset” scene along with an external camera.
- This branch can be represented as separate deployment pipelines that eventually connect to the common object detection and UI logic.

##### Supporting Libraries & Third–Party Components
- In a lower level detail (or as annotations), mention that the system depends on various third–party components such as the MediaPipe Unity Plugin, Unity Android Speech Recognizer, and UVC4UnityAndroid.
- These may appear as “external libraries” feeding into the object detection or the UI integration modules.

##### Data Flow & Interaction Patterns
- The camera module streaming video to the MediaPipe object detection module.
- On tapping an identified object, the UI system sends a query via the Gemini integration module for additional metadata.
- The resulting information then feeds into the display logic of the context menu.
- That, once objects are detected, there is an option to stop the continuous detection to improve application performance while maintaining the set of previously identified objects.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Deployment & Input Subgraph
    subgraph "Deployment & Input"
        A["Cross–Platform XR Integration"]:::external
        F["UVC4UnityAndroid (External Camera Plugin for Headset)"]:::external
        B["Camera/Video Input Module"]:::sensor
    end

    %% Processing Subgraph
    subgraph "Processing"
        C["Object Detection & MediaPipe Processing"]:::processing
        H["Supporting Third-Party Components & Utilities"]:::external
    end

    %% Decision & Integration Subgraph
    subgraph "Decision & Integration"
        E["AR/Spatial Interaction & UI Module"]:::ui
        D["Gemini LLM Integration for Metadata"]:::processing
        G["Unity Android Speech Recognizer"]:::external
    end

    %% Connections between Subgraphs
    A -->|"feeds"| B
    F -->|"enables"| B
    B -->|"streams"| C
    H -->|"dependency"| C
    C -->|"sendsDetection"| E
    E <-->|"queryResult"| D
    G -->|"voiceCommand"| E

    %% Styles
    classDef sensor fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef processing fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef ui fill:#FFA500,stroke:#000,stroke-width:2px;
    classDef external fill:#D3D3D3,stroke:#000,stroke-width:2px;

    %% Click Events
    click A "https://github.com/google/xr-objects/tree/main/XRObjects/Assets/XR"
    click B "https://github.com/google/xr-objects/blob/main/XRObjects/Assets/MediaPipeUnity/Samples/Common/Scripts/ImageSource/WebCamSource.cs"
    click C "https://github.com/google/xr-objects/blob/main/Packages/com.github.homuler.mediapipe"
    click D "https://github.com/google/xr-objects/blob/main/XRObjects/Assets/XRObjects/Scripts/ImageQuery.cs"
    click E "https://github.com/google/xr-objects/tree/main/XRObjects/Assets/XRObjects/Scripts"
    click F "https://github.com/google/xr-objects/tree/main/DemoHeadset/Assets/UVC4UnityAndroidPlugin"
    click G "https://github.com/google/xr-objects/tree/main/XRObjects/Assets/Plugins/SpeechRecognizer"
    click H "https://github.com/google/xr-objects/tree/main/third_party"
```
