# rivindu02/dovsg architecture

> Architecture diagram of the GitHub repository rivindu02/dovsg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rivindu02/dovsg
- Repository: https://github.com/rivindu02/dovsg

## Overview

DovSG is a long-term language-guided mobile-manipulation system. Its principal workflow collects RGB-D observations, estimates camera poses, builds a relocalization model and view dataset, extracts object instances and a dynamic 3D scene graph, plans tasks from language, then navigates and manipulates while updating scene memory. The graph keeps those domain stages distinct and includes the robot-side command server. Edges follow the README’s documented workflow and the sampled imports; many implementation details were not sampled, so the graph does not infer finer control flow.

Main entry points: [`dovsg/controller.py`](https://github.com/rivindu02/dovsg/blob/master/dovsg/controller.py) (Robot task controller), [`dovsg/memory/view_dataset.py`](https://github.com/rivindu02/dovsg/blob/master/dovsg/memory/view_dataset.py) (View dataset), [`dovsg/memory/scene_graph/scene_graph_processer.py`](https://github.com/rivindu02/dovsg/blob/master/dovsg/memory/scene_graph/scene_graph_processer.py) (Dynamic scene graph), [`hardcode/robot_control.py`](https://github.com/rivindu02/dovsg/blob/master/hardcode/robot_control.py) (Robot control)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_perception["Sensing &amp; Mapping"]
  node_capture["RGB-D capture"]
  node_pose["Pose estimation<br/>[pose_estimation.py]"]
  node_ace["ACE relocalization<br/>[ace_network.py]"]
  node_features["Feature matching"]
end

subgraph group_memory["Scene Memory"]
  node_views[("View dataset<br/>[view_dataset.py]")]
  node_detector["Open-vocabulary detection"]
  node_instances["Instance processing"]
  node_scenegraph[("Dynamic scene graph")]
end

subgraph group_planning["Task &amp; Motion"]
  node_planner["Language task planning"]
  node_navigation["Path planning<br/>[pathplanning.py]"]
  node_localizer["Instance localization"]
  node_manipulation["Object handling<br/>[objecthandler.py]"]
  node_controller["Robot task controller<br/>[controller.py]"]
end

subgraph group_robot["Robot Runtime"]
  node_socket["Command server<br/>[server.py]"]
  node_robotcontrol["Robot control<br/>[robot_control.py]"]
  node_arm["xArm arm<br/>[xarm6.py]"]
  node_base["Ranger base<br/>[ranger_mini_3.py]"]
end

node_user(("Task requester"))
node_demo["Demo entry<br/>[demo.py]"]
node_hardware{{"Mobile robot"}}

node_user -->|"requests task"| node_demo
node_demo -->|"starts workflow"| node_controller
node_controller -->|"collects RGB-D"| node_capture
node_capture -->|"provides frames"| node_pose
node_pose -->|"supports training"| node_ace
node_ace -->|"supports relocalization"| node_views
node_controller -->|"builds dataset"| node_views
node_views -->|"supplies observations"| node_instances
node_detector -->|"provides detections"| node_instances
node_instances -->|"builds objects"| node_scenegraph
node_views -->|"provides 3D points"| node_scenegraph
node_scenegraph -->|"informs planning"| node_planner
node_planner -->|"returns task plan"| node_controller
node_controller -->|"relocalizes"| node_features
node_views -->|"supplies views"| node_features
node_controller -->|"plans movement"| node_navigation
node_controller -->|"localizes targets"| node_localizer
node_controller -->|"executes object handling"| node_manipulation
node_controller -->|"updates memory"| node_scenegraph
node_controller -->|"sends robot commands"| node_socket
node_socket -->|"dispatches commands"| node_robotcontrol
node_robotcontrol -->|"controls arm"| node_arm
node_robotcontrol -->|"controls base"| node_base
node_arm -.->|"actuates"| node_hardware
node_base -.->|"moves"| node_hardware
node_controller -.->|"requests observations"| node_capture

click node_demo "https://github.com/rivindu02/dovsg/blob/master/demo.py"
click node_capture "https://github.com/rivindu02/dovsg/blob/master/dovsg/scripts/realsense_recorder.py"
click node_pose "https://github.com/rivindu02/dovsg/blob/master/dovsg/scripts/pose_estimation.py"
click node_ace "https://github.com/rivindu02/dovsg/blob/master/ace/ace_network.py"
click node_features "https://github.com/rivindu02/dovsg/blob/master/dovsg/scripts/rgb_feature_match.py"
click node_views "https://github.com/rivindu02/dovsg/blob/master/dovsg/memory/view_dataset.py"
click node_detector "https://github.com/rivindu02/dovsg/blob/master/dovsg/perception/models/mygroundingdinosam2.py"
click node_instances "https://github.com/rivindu02/dovsg/blob/master/dovsg/memory/instances/instance_process.py"
click node_scenegraph "https://github.com/rivindu02/dovsg/blob/master/dovsg/memory/scene_graph/scene_graph_processer.py"
click node_planner "https://github.com/rivindu02/dovsg/blob/master/dovsg/task_planning/gpt_task_planning.py"
click node_navigation "https://github.com/rivindu02/dovsg/blob/master/dovsg/navigation/pathplanning.py"
click node_localizer "https://github.com/rivindu02/dovsg/blob/master/dovsg/navigation/instances_localizer.py"
click node_manipulation "https://github.com/rivindu02/dovsg/blob/master/dovsg/manipulation/objecthandler.py"
click node_controller "https://github.com/rivindu02/dovsg/blob/master/dovsg/controller.py"
click node_socket "https://github.com/rivindu02/dovsg/blob/master/hardcode/server.py"
click node_robotcontrol "https://github.com/rivindu02/dovsg/blob/master/hardcode/robot_control.py"
click node_arm "https://github.com/rivindu02/dovsg/blob/master/hardcode/arm/xarm6.py"
click node_base "https://github.com/rivindu02/dovsg/blob/master/hardcode/base/ranger_mini_3.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_capture,node_pose,node_ace,node_features toneBlue
class node_views,node_detector,node_instances,node_scenegraph toneAmber
class node_planner,node_navigation,node_localizer,node_manipulation,node_controller toneMint
class node_socket,node_robotcontrol,node_arm,node_base,node_user toneRose
class node_demo toneTeal
class node_hardware toneIndigo
```

## Components

### Sensing & Mapping

- [RGB-D capture](https://github.com/rivindu02/dovsg/blob/master/dovsg/scripts/realsense_recorder.py): `dovsg/scripts/realsense_recorder.py`
- [Pose estimation](https://github.com/rivindu02/dovsg/blob/master/dovsg/scripts/pose_estimation.py): `dovsg/scripts/pose_estimation.py`
- [ACE relocalization](https://github.com/rivindu02/dovsg/blob/master/ace/ace_network.py): `ace/ace_network.py`
- [Feature matching](https://github.com/rivindu02/dovsg/blob/master/dovsg/scripts/rgb_feature_match.py): `dovsg/scripts/rgb_feature_match.py`

### Scene Memory

- [View dataset](https://github.com/rivindu02/dovsg/blob/master/dovsg/memory/view_dataset.py): `dovsg/memory/view_dataset.py`
- [Open-vocabulary detection](https://github.com/rivindu02/dovsg/blob/master/dovsg/perception/models/mygroundingdinosam2.py): `dovsg/perception/models/mygroundingdinosam2.py`
- [Instance processing](https://github.com/rivindu02/dovsg/blob/master/dovsg/memory/instances/instance_process.py): `dovsg/memory/instances/instance_process.py`
- [Dynamic scene graph](https://github.com/rivindu02/dovsg/blob/master/dovsg/memory/scene_graph/scene_graph_processer.py): `dovsg/memory/scene_graph/scene_graph_processer.py`

### Task & Motion

- [Language task planning](https://github.com/rivindu02/dovsg/blob/master/dovsg/task_planning/gpt_task_planning.py): `dovsg/task_planning/gpt_task_planning.py`
- [Path planning](https://github.com/rivindu02/dovsg/blob/master/dovsg/navigation/pathplanning.py): `dovsg/navigation/pathplanning.py`
- [Instance localization](https://github.com/rivindu02/dovsg/blob/master/dovsg/navigation/instances_localizer.py): `dovsg/navigation/instances_localizer.py`
- [Object handling](https://github.com/rivindu02/dovsg/blob/master/dovsg/manipulation/objecthandler.py): `dovsg/manipulation/objecthandler.py`
- [Robot task controller](https://github.com/rivindu02/dovsg/blob/master/dovsg/controller.py): `dovsg/controller.py`

### Robot Runtime

- [Command server](https://github.com/rivindu02/dovsg/blob/master/hardcode/server.py): `hardcode/server.py`
- [Robot control](https://github.com/rivindu02/dovsg/blob/master/hardcode/robot_control.py): `hardcode/robot_control.py`
- [xArm arm](https://github.com/rivindu02/dovsg/blob/master/hardcode/arm/xarm6.py): `hardcode/arm/xarm6.py`
- [Ranger base](https://github.com/rivindu02/dovsg/blob/master/hardcode/base/ranger_mini_3.py): `hardcode/base/ranger_mini_3.py`

### Other components

- **Task requester**
- [Demo entry](https://github.com/rivindu02/dovsg/blob/master/demo.py): `demo.py`
- **Mobile robot**

## Connections

- Task requester → Demo entry: requests task
- Demo entry → Robot task controller: starts workflow (evidence: `demo.py`)
- Robot task controller → RGB-D capture: collects RGB-D (evidence: `dovsg/controller.py`)
- RGB-D capture → Pose estimation: provides frames
- Pose estimation → ACE relocalization: supports training
- ACE relocalization → View dataset: supports relocalization
- Robot task controller → View dataset: builds dataset (evidence: `dovsg/controller.py`)
- View dataset → Instance processing: supplies observations (evidence: `dovsg/memory/instances/instance_process.py`)
- Open-vocabulary detection → Instance processing: provides detections (evidence: `dovsg/controller.py`)
- Instance processing → Dynamic scene graph: builds objects (evidence: `dovsg/memory/scene_graph/scene_graph_processer.py`)
- View dataset → Dynamic scene graph: provides 3D points (evidence: `dovsg/memory/scene_graph/scene_graph_processer.py`)
- Dynamic scene graph → Language task planning: informs planning (evidence: `dovsg/task_planning/gpt_task_planning.py`)
- Language task planning → Robot task controller: returns task plan (evidence: `dovsg/controller.py`)
- Robot task controller → Feature matching: relocalizes (evidence: `dovsg/controller.py`)
- View dataset → Feature matching: supplies views (evidence: `dovsg/scripts/rgb_feature_match.py`)
- Robot task controller → Path planning: plans movement (evidence: `dovsg/controller.py`)
- Robot task controller → Instance localization: localizes targets (evidence: `dovsg/controller.py`)
- Robot task controller → Object handling: executes object handling (evidence: `dovsg/controller.py`)
- Robot task controller → Dynamic scene graph: updates memory (evidence: `dovsg/controller.py`)
- Robot task controller → Command server: sends robot commands (evidence: `dovsg/controller.py`)
- Command server → Robot control: dispatches commands (evidence: `hardcode/server.py`)
- Robot control → xArm arm: controls arm (evidence: `hardcode/robot_control.py`)
- Robot control → Ranger base: controls base (evidence: `hardcode/robot_control.py`)
- xArm arm → Mobile robot: actuates
- Ranger base → Mobile robot: moves
- Robot task controller → RGB-D capture: requests observations (evidence: `dovsg/controller.py`)
