How mit-spark/hydra works
Project Type and Purpose
• Hydra is an open‐source robotics system/tool for real‐time 3D Scene Graph construction and optimization. It’s written primarily in C++ with Python bindings and integrates with ROS.
• It incrementally processes sensor data (camera/lidar), builds a volumetric/graph‐based map, performs loop closure via GNNs, labels rooms and objects, and outputs a hierarchical 3D scene graph.
File Structure Analysis
• Top‐level organization:
• cmake, CMakeLists.txt, package.xml, python/ and src/ – build & source code
• /hydra & src/… – C++ modular pipeline code
• python/src/hydra_python – Python CLI & bindings
• eval/ – separate evaluation library & tools
• config/ – YAML configs for datasets, label spaces, loop closure
• models/lcd – ONNX GNN models
• scripts/, notebooks/ – utilities and debugging
• install/ – ROS rosinstall manifests
• Directory patterns correspond to core modules:
• input, frontend, backend, loop_closure, odometry, places, reconstruction, rooms, gnn, utils, active_window
• Each module has headers (include/hydra/…) and sources (src/…)
README Insights
• Confirms Hydra’s pipeline stages (input→front end→loop closure→back end→scene graph)
• Notes external dependencies: ROS, GTSAM, ONNX Runtime, semantic_inference, Spatial-Hash
• Mentions Python bindings (hydra_python CLI) and evaluation suite
• Sensor Inputs: Camera, Lidar
• Input Module: input_conversion, sensor_input_packet
• Pipeline Manager: common/hydra_pipeline, message_queue
• Frontend Modules: graph_builder, mesh_segmenter, GVD integrator, view_selector
• GNN Loop Closure: ONNX models (models/lcd/*.onnx), gnn_interface
• Backend Optimizer: merge_tracker, update_agents/buildings/rooms functors, PGMO config
• Reconstruction Engine: TSDF integrator, mesh_integrator, marching_cubes
• Room & Object Labeling: rooms/graph_clustering, label_remapper
• ROS Interface: hydra_ros launch files, zmq_interfaces
• Python CLI & Bindings: hydra_python commands, python/bindings
• Config & Models: config/*.yaml, models/*.onnx
• Evaluation Suite: eval/python & eval/src tools
1. Sensors → Input Module
2. Input Module → Pipeline Manager
3. Pipeline → Frontend Modules (parallel fetch/extraction)
4. Frontend output → Shared DSG (Distributed Scene Graph)
5. Shared DSG → Loop Closure (descriptor extraction → ONNX → match)
6. Loop Closure results → Backend Optimizer (factor graph, PGMO)
7. Optimizer → Updated DSG → Reconstruction Engine → 3D Mesh & TSDF
8. Updated DSG → Room/Object Labeling
9. Final Scene Graph → ROS Topics / Python Export / Visualization
10. Config & Models feed into modules at startup
11. Evaluation Suite subscribes to outputs & computes metrics
c. Architectural Patterns & Principles
• Modular Pipeline: each subsystem (input, frontend, loop_closure, backend, reconstruction, rooms) is a pluggable module.
• Producer–Consumer Queues: common/pipeline_queues and message_queue for inter‐module communication.
• Factor Graph Optimization: backend uses PGMO (Kimera-PGMO) for pose graph adjustments.
• Data‐Driven Configuration: YAML configs for datasets, label spaces, loop closure parameters.
• Language Bindings: C++ core with Python wrappers via pybind11 (python/bindings).
d. Key Technologies & Frameworks
• CMake/Catkin (CMakeLists.txt, rosinstall)
• ROS Noetic (roslaunch, topics, srv)
• GTSAM (graph optimization)
• ONNX Runtime (GNN inference)
• Python & pybind11 for CLI and scripting
• YAML config utilities (MIT-SPARK/config_utilities)
• Spatial-Hash TSDF (voxblox replacement)