How rivindu02/dovsg works
DovSG is a long-term language-guided mobile-manipulation system. Its principal workflow collects RGB-D observations, estimates camera poses, builds a relocalization model and view dataset, extracts object instances and a dynamic 3D scene graph, plans tasks from language, then navigates and manipulates while updating scene memory. The graph keeps those domain stages distinct and includes the robot-side command server. Edges follow the README’s documented workflow and the sampled imports; many implementation details were not sampled, so the graph does not infer finer control flow.
Main entry points: controller.py Robot task controller · view_dataset.py View dataset · scene_graph_processer.py Dynamic scene graph · robot_control.py Robot control
Components 20
Sensing & Mapping
- RGB-D capture
dovsg/scripts/realsense_recorder.py - Pose estimation
dovsg/scripts/pose_estimation.py - ACE relocalization
ace/ace_network.py - Feature matching
dovsg/scripts/rgb_feature_match.py
Scene Memory
- View dataset
dovsg/memory/view_dataset.py - Open-vocabulary detection
dovsg/perception/models/mygroundingdinosam2.py - Instance processing
dovsg/memory/instances/instance_process.py - Dynamic scene graph
dovsg/memory/scene_graph/scene_graph_processer.py
Task & Motion
- Language task planning
dovsg/task_planning/gpt_task_planning.py - Path planning
dovsg/navigation/pathplanning.py - Instance localization
dovsg/navigation/instances_localizer.py - Object handling
dovsg/manipulation/objecthandler.py - Robot task controller
dovsg/controller.py
Robot Runtime
- Command server
hardcode/server.py - Robot control
hardcode/robot_control.py - xArm arm
hardcode/arm/xarm6.py - Ranger base
hardcode/base/ranger_mini_3.py
Other components
- Task requester
- Demo entry
demo.py - Mobile robot
Connections 26
- Task requester → Demo entry: requests task
- Demo entry → Robot task controller: starts workflow
demo.py - Robot task controller → RGB-D capture: collects RGB-D
dovsg/controller.py - RGB-D capture → Pose estimation: provides frames
- Pose estimation → ACE relocalization: supports training
- ACE relocalization → View dataset: supports relocalization
- Robot task controller → View dataset: builds dataset
dovsg/controller.py - View dataset → Instance processing: supplies observations
dovsg/memory/instances/instance_process.py - Open-vocabulary detection → Instance processing: provides detections
dovsg/controller.py - Instance processing → Dynamic scene graph: builds objects
dovsg/memory/scene_graph/scene_graph_processer.py - View dataset → Dynamic scene graph: provides 3D points
dovsg/memory/scene_graph/scene_graph_processer.py - Dynamic scene graph → Language task planning: informs planning
dovsg/task_planning/gpt_task_planning.py - Language task planning → Robot task controller: returns task plan
dovsg/controller.py - Robot task controller → Feature matching: relocalizes
dovsg/controller.py - View dataset → Feature matching: supplies views
dovsg/scripts/rgb_feature_match.py - Robot task controller → Path planning: plans movement
dovsg/controller.py - Robot task controller → Instance localization: localizes targets
dovsg/controller.py - Robot task controller → Object handling: executes object handling
dovsg/controller.py - Robot task controller → Dynamic scene graph: updates memory
dovsg/controller.py - Robot task controller → Command server: sends robot commands
dovsg/controller.py - Command server → Robot control: dispatches commands
hardcode/server.py - Robot control → xArm arm: controls arm
hardcode/robot_control.py - Robot control → Ranger base: controls base
hardcode/robot_control.py - xArm arm → Mobile robot: actuates
- Ranger base → Mobile robot: moves
- Robot task controller → RGB-D capture: requests observations
dovsg/controller.py