# dionyichia/gappler architecture

> Architecture diagram of the GitHub repository dionyichia/gappler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dionyichia/gappler
- Repository: https://github.com/dionyichia/gappler

## Overview

This repository is a ROS 2 robotics stack spanning Project Aria sensing, object perception, grasp planning, and mobile navigation. The principal intended workflow is to acquire visual/gaze data, identify a prompted object, and make perception available to grasp and robot-control components. The sampled Aria application has an explicit runtime orchestrator; links between separate ROS packages are shown only where the supplied source index or excerpts support them. Several package-level capabilities are visible in the tree but their runtime wiring was not sampled.

Main entry points: [`aria/aria_app/main.py`](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/main.py) (Aria application), [`aria/aria_app/services/object_recognition/object_recognition_pipeline.py`](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/object_recognition/object_recognition_pipeline.py) (Object recognition), [`shared/gappler_common.py`](https://github.com/dionyichia/gappler/blob/dev/shared/gappler_common.py) (Shared configuration), [`launchers/grasp_orchestrator.py`](https://github.com/dionyichia/gappler/blob/dev/launchers/grasp_orchestrator.py) (Grasp orchestrator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_aria["Aria sensing"]
  node_aria_app["Aria application<br/>[main.py]"]
  node_device["Aria device control"]
  node_image_pipeline["Image pipeline"]
  node_audio_pipeline["Audio pipeline"]
  node_pose_pipeline["VIO pose pipeline"]
  node_playback["Recording playback"]
  node_prompt["Prompt extraction"]
end

subgraph group_perception["Perception"]
  node_recognition["Object recognition"]
  node_sam3["SAM 3 model<br/>[sam3_model.py]"]
  node_feature_match["Feature matching"]
  node_mask_viz["Mask visualization"]
  node_ros_viz["ROS visualization feeds<br/>[ros_manager.py]"]
end

subgraph group_grasp["Grasp execution"]
  node_grasp_orchestrator["Grasp orchestrator"]
  node_grasp_pipeline["Grasp pipeline launcher"]
  node_sam_ros["SAM ROS node<br/>[sam3_ros_node.py]"]
  node_anygrasp["AnyGrasp detection<br/>[anygrasp_node.py]"]
  node_grasp_machine["Grasp state machine"]
  node_mtc["MTC planner<br/>[mtc_planner.cpp]"]
  node_arm_bringup["Arm bringup"]
  node_estop["Emergency stop<br/>[estop.py]"]
end

subgraph group_navigation["Navigation"]
  node_navigation["Navigation launch"]
  node_slam["SLAM localization"]
  node_approach["Object approach"]
  node_goto_gaze["Navigate to glasses<br/>[goto_glasses.py]"]
end

subgraph group_shared["Shared runtime"]
  node_common["Shared configuration<br/>[gappler_common.py]"]
end

node_operator(("Operator"))

node_operator -.->|"starts"| node_aria_app
node_aria_app -->|"starts stream"| node_device
node_aria_app -->|"starts audio"| node_audio_pipeline
node_aria_app -->|"configures"| node_recognition
node_aria_app -->|"configures"| node_feature_match
node_aria_app -->|"starts playback"| node_playback
node_aria_app -->|"starts visual feed"| node_image_pipeline
node_aria_app -->|"starts pose feed"| node_pose_pipeline
node_audio_pipeline -->|"extracts intent"| node_prompt
node_recognition -->|"uses matching"| node_feature_match
node_recognition -->|"uses renderer"| node_mask_viz
node_ros_viz -.->|"delivers visualization"| node_operator
node_sam_ros -->|"uses model"| node_sam3
node_sam_ros -->|"uses renderer"| node_mask_viz
node_grasp_machine -->|"uses planner"| node_mtc
node_grasp_orchestrator -->|"reads paths"| node_common
node_grasp_pipeline -->|"reads paths"| node_common
node_slam -->|"reads paths"| node_common
node_navigation -.->|"launches localization"| node_slam
node_grasp_orchestrator -.->|"coordinates"| node_grasp_pipeline

click node_aria_app "https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/main.py"
click node_device "https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/aria_device/aria_device_controller.py"
click node_image_pipeline "https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/aria_device/stream/image_streaming_pipeline.py"
click node_audio_pipeline "https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/aria_device/stream/audio_streaming_pipeline.py"
click node_pose_pipeline "https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/aria_device/stream/pose_streaming_pipeline.py"
click node_playback "https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/playback_controller.py"
click node_prompt "https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/prompt_extractor.py"
click node_recognition "https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/object_recognition/object_recognition_pipeline.py"
click node_sam3 "https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/object_recognition/sam3_model.py"
click node_feature_match "https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/feature_matching.py"
click node_mask_viz "https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/visualizer/renderers/object_mask_visualizer.py"
click node_ros_viz "https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/visualizer/ros_manager.py"
click node_grasp_orchestrator "https://github.com/dionyichia/gappler/blob/dev/launchers/grasp_orchestrator.py"
click node_grasp_pipeline "https://github.com/dionyichia/gappler/blob/dev/launchers/start_grasp_pipeline.py"
click node_sam_ros "https://github.com/dionyichia/gappler/blob/dev/grasp/segmentation/sam3_ros_node.py"
click node_anygrasp "https://github.com/dionyichia/gappler/blob/dev/grasp/anygrasp_node/anygrasp_node.py"
click node_grasp_machine "https://github.com/dionyichia/gappler/blob/dev/grasp/grasp_state_machine/src/grasp_state_machine.cpp"
click node_mtc "https://github.com/dionyichia/gappler/blob/dev/grasp/grasp_state_machine/src/mtc_planner.cpp"
click node_arm_bringup "https://github.com/dionyichia/gappler/blob/dev/arm/arm_bringup/launch/arm_bringup.launch.py"
click node_estop "https://github.com/dionyichia/gappler/blob/dev/arm/estop/estop.py"
click node_navigation "https://github.com/dionyichia/gappler/blob/dev/nav/robot_navigation/launch/navigation.launch.py"
click node_slam "https://github.com/dionyichia/gappler/blob/dev/nav/robot_slam/launch/slam_localization.launch.py"
click node_approach "https://github.com/dionyichia/gappler/blob/dev/nav/object_approach/object_approach_node.py"
click node_goto_gaze "https://github.com/dionyichia/gappler/blob/dev/nav/goto_glasses/goto_glasses.py"
click node_common "https://github.com/dionyichia/gappler/blob/dev/shared/gappler_common.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_aria_app,node_device,node_image_pipeline,node_audio_pipeline,node_pose_pipeline,node_playback,node_prompt toneBlue
class node_recognition,node_sam3,node_feature_match,node_mask_viz,node_ros_viz toneAmber
class node_grasp_orchestrator,node_grasp_pipeline,node_sam_ros,node_anygrasp,node_grasp_machine,node_mtc,node_arm_bringup,node_estop toneMint
class node_navigation,node_slam,node_approach,node_goto_gaze toneRose
class node_common,node_operator toneIndigo
```

## Components

### Aria sensing

- [Aria application](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/main.py): `aria/aria_app/main.py`
- [Aria device control](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/aria_device/aria_device_controller.py): `aria/aria_app/services/aria_device/aria_device_controller.py`
- [Image pipeline](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/aria_device/stream/image_streaming_pipeline.py): `aria/aria_app/services/aria_device/stream/image_streaming_pipeline.py`
- [Audio pipeline](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/aria_device/stream/audio_streaming_pipeline.py): `aria/aria_app/services/aria_device/stream/audio_streaming_pipeline.py`
- [VIO pose pipeline](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/aria_device/stream/pose_streaming_pipeline.py): `aria/aria_app/services/aria_device/stream/pose_streaming_pipeline.py`
- [Recording playback](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/playback_controller.py): `aria/aria_app/services/playback_controller.py`
- [Prompt extraction](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/prompt_extractor.py): `aria/aria_app/services/prompt_extractor.py`

### Perception

- [Object recognition](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/object_recognition/object_recognition_pipeline.py): `aria/aria_app/services/object_recognition/object_recognition_pipeline.py`
- [SAM 3 model](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/object_recognition/sam3_model.py): `aria/aria_app/services/object_recognition/sam3_model.py`
- [Feature matching](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/feature_matching.py): `aria/aria_app/services/feature_matching.py`
- [Mask visualization](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/visualizer/renderers/object_mask_visualizer.py): `aria/aria_app/services/visualizer/renderers/object_mask_visualizer.py`
- [ROS visualization feeds](https://github.com/dionyichia/gappler/blob/dev/aria/aria_app/services/visualizer/ros_manager.py): `aria/aria_app/services/visualizer/ros_manager.py`

### Grasp execution

- [Grasp orchestrator](https://github.com/dionyichia/gappler/blob/dev/launchers/grasp_orchestrator.py): `launchers/grasp_orchestrator.py`
- [Grasp pipeline launcher](https://github.com/dionyichia/gappler/blob/dev/launchers/start_grasp_pipeline.py): `launchers/start_grasp_pipeline.py`
- [SAM ROS node](https://github.com/dionyichia/gappler/blob/dev/grasp/segmentation/sam3_ros_node.py): `grasp/segmentation/sam3_ros_node.py`
- [AnyGrasp detection](https://github.com/dionyichia/gappler/blob/dev/grasp/anygrasp_node/anygrasp_node.py): `grasp/anygrasp_node/anygrasp_node.py`
- [Grasp state machine](https://github.com/dionyichia/gappler/blob/dev/grasp/grasp_state_machine/src/grasp_state_machine.cpp): `grasp/grasp_state_machine/src/grasp_state_machine.cpp`
- [MTC planner](https://github.com/dionyichia/gappler/blob/dev/grasp/grasp_state_machine/src/mtc_planner.cpp): `grasp/grasp_state_machine/src/mtc_planner.cpp`
- [Arm bringup](https://github.com/dionyichia/gappler/blob/dev/arm/arm_bringup/launch/arm_bringup.launch.py): `arm/arm_bringup/launch/arm_bringup.launch.py`
- [Emergency stop](https://github.com/dionyichia/gappler/blob/dev/arm/estop/estop.py): `arm/estop/estop.py`

### Navigation

- [Navigation launch](https://github.com/dionyichia/gappler/blob/dev/nav/robot_navigation/launch/navigation.launch.py): `nav/robot_navigation/launch/navigation.launch.py`
- [SLAM localization](https://github.com/dionyichia/gappler/blob/dev/nav/robot_slam/launch/slam_localization.launch.py): `nav/robot_slam/launch/slam_localization.launch.py`
- [Object approach](https://github.com/dionyichia/gappler/blob/dev/nav/object_approach/object_approach_node.py): `nav/object_approach/object_approach_node.py`
- [Navigate to glasses](https://github.com/dionyichia/gappler/blob/dev/nav/goto_glasses/goto_glasses.py): `nav/goto_glasses/goto_glasses.py`

### Shared runtime

- [Shared configuration](https://github.com/dionyichia/gappler/blob/dev/shared/gappler_common.py): `shared/gappler_common.py`

### Other components

- **Operator**

## Connections

- Operator → Aria application: starts
- Aria application → Aria device control: starts stream (evidence: `aria/aria_app/main.py`)
- Aria application → Audio pipeline: starts audio (evidence: `aria/aria_app/main.py`)
- Aria application → Object recognition: configures (evidence: `aria/aria_app/main.py`)
- Aria application → Feature matching: configures (evidence: `aria/aria_app/main.py`)
- Aria application → Recording playback: starts playback (evidence: `aria/aria_app/main.py`)
- Aria application → Image pipeline: starts visual feed (evidence: `aria/aria_app/main.py`)
- Aria application → VIO pose pipeline: starts pose feed (evidence: `aria/aria_app/main.py`)
- Audio pipeline → Prompt extraction: extracts intent (evidence: `aria/aria_app/services/aria_device/stream/audio_streaming_pipeline.py`)
- Object recognition → Feature matching: uses matching (evidence: `aria/aria_app/services/object_recognition/object_recognition_pipeline.py`)
- Object recognition → Mask visualization: uses renderer (evidence: `aria/aria_app/services/object_recognition/object_recognition_pipeline.py`)
- ROS visualization feeds → Operator: delivers visualization
- SAM ROS node → SAM 3 model: uses model (evidence: `grasp/segmentation/sam3_ros_node.py`)
- SAM ROS node → Mask visualization: uses renderer (evidence: `grasp/segmentation/sam3_ros_node.py`)
- Grasp state machine → MTC planner: uses planner (evidence: `grasp/grasp_state_machine/src/grasp_state_machine.cpp`)
- Grasp orchestrator → Shared configuration: reads paths (evidence: `launchers/grasp_orchestrator.py`)
- Grasp pipeline launcher → Shared configuration: reads paths (evidence: `launchers/start_grasp_pipeline.py`)
- SLAM localization → Shared configuration: reads paths (evidence: `nav/robot_slam/launch/slam_localization.launch.py`)
- Navigation launch → SLAM localization: launches localization
- Grasp orchestrator → Grasp pipeline launcher: coordinates
