# mit-spark/hydra architecture

> Architecture diagram of the GitHub repository mit-spark/hydra, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mit-spark/hydra
- Repository: https://github.com/mit-spark/hydra

## Overview

##### Project Type and Purpose
- Hydra is an open‐source robotics system/tool for real‐time 3D Scene Graph construction and optimization. It’s written primarily in C++ with Python bindings and integrates with ROS.
- It incrementally processes sensor data (camera/lidar), builds a volumetric/graph‐based map, performs loop closure via GNNs, labels rooms and objects, and outputs a hierarchical 3D scene graph.

##### File­ Structure Analysis
- Top‐level organization:
- cmake, CMakeLists.txt, package.xml, python/ and src/ – build & source code
- /hydra & src/… – C++ modular pipeline code
- python/src/hydra_python – Python CLI & bindings
- eval/ – separate evaluation library & tools
- config/ – YAML configs for datasets, label spaces, loop closure
- models/lcd – ONNX GNN models
- scripts/, notebooks/ – utilities and debugging
- install/ – ROS rosinstall manifests
- Directory patterns correspond to core modules:
- input, frontend, backend, loop_closure, odometry, places, reconstruction, rooms, gnn, utils, active_window
- Each module has headers (include/hydra/…) and sources (src/…)

##### README Insights
- Confirms Hydra’s pipeline stages (input→front end→loop closure→back end→scene graph)
- Notes external dependencies: ROS, GTSAM, ONNX Runtime, semantic_inference, Spatial-Hash
- Mentions Python bindings (hydra_python CLI) and evaluation suite

- Sensor Inputs: Camera, Lidar
- Input Module: input_conversion, sensor_input_packet
- Pipeline Manager: common/hydra_pipeline, message_queue
- Frontend Modules: graph_builder, mesh_segmenter, GVD integrator, view_selector
- GNN Loop Closure: ONNX models (models/lcd/*.onnx), gnn_interface
- Backend Optimizer: merge_tracker, update_agents/buildings/rooms functors, PGMO config
- Reconstruction Engine: TSDF integrator, mesh_integrator, marching_cubes
- Room & Object Labeling: rooms/graph_clustering, label_remapper
- ROS Interface: hydra_ros launch files, zmq_interfaces
- Python CLI & Bindings: hydra_python commands, python/bindings
- Config & Models: config/*.yaml, models/*.onnx
- Evaluation Suite: eval/python & eval/src tools

1. Sensors → Input Module
2. Input Module → Pipeline Manager
3. Pipeline → Frontend Modules (parallel fetch/extraction)
4. Frontend output → Shared DSG (Distributed Scene Graph)
5. Shared DSG → Loop Closure (descriptor extraction → ONNX → match)
6. Loop Closure results → Backend Optimizer (factor graph, PGMO)
7. Optimizer → Updated DSG → Reconstruction Engine → 3D Mesh & TSDF
8. Updated DSG → Room/Object Labeling
9. Final Scene Graph → ROS Topics / Python Export / Visualization
10. Config & Models feed into modules at startup
11. Evaluation Suite subscribes to outputs & computes metrics

c. Architectural Patterns & Principles
- Modular Pipeline: each subsystem (input, frontend, loop_closure, backend, reconstruction, rooms) is a pluggable module.
- Producer–Consumer Queues: common/pipeline_queues and message_queue for inter‐module communication.
- Factor Graph Optimization: backend uses PGMO (Kimera-PGMO) for pose graph adjustments.
- Data‐Driven Configuration: YAML configs for datasets, label spaces, loop closure parameters.
- Language Bindings: C++ core with Python wrappers via pybind11 (python/bindings).

d. Key Technologies & Frameworks
- CMake/Catkin (CMakeLists.txt, rosinstall)
- ROS Noetic (roslaunch, topics, srv)
- GTSAM (graph optimization)
- ONNX Runtime (GNN inference)
- Python & pybind11 for CLI and scripting
- YAML config utilities (MIT-SPARK/config_utilities)
- Spatial-Hash TSDF (voxblox replacement)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Sensors
    subgraph "Sensors"
        Camera["Camera"]:::core
        Lidar["LiDAR"]:::core
    end

    %% Input Module
    subgraph "Input Module"
        Conversion["input_conversion"]:::core
        Packet["sensor_input_packet"]:::core
    end

    %% Pipeline Manager
    subgraph "Pipeline Manager"
        HydraPipeline["hydra_pipeline"]:::core
        MessageQueue["message_queue"]:::core
    end

    %% Frontend Modules
    subgraph "Frontend"
        GraphBuilder["graph_builder"]:::core
        MeshSegmenter["mesh_segmenter"]:::core
        GVDExtractor["gvd_place_extractor"]:::core
        ViewSelector["view_selector"]:::core
    end

    %% Loop Closure
    subgraph "Loop Closure"
        LoopClosureModule["loop_closure_module"]:::core
        DescriptorMatching["descriptor_matching"]:::core
        GNNInterface["gnn_interface"]:::core
        ONNXModels["ONNX Models"]:::model
    end

    %% Backend Optimizer
    subgraph "Backend Optimizer"
        BackendModule["backend_module"]:::core
        MergeTracker["merge_tracker"]:::core
        PGMOConfigs["pgmo_configs"]:::core
    end

    %% Reconstruction Engine
    subgraph "Reconstruction"
        VolumetricMap["volumetric_map"]:::core
        MeshIntegrator["mesh_integrator"]:::core
        MarchingCubes["marching_cubes"]:::core
    end

    %% Room & Object Labeling
    subgraph "Labeling"
        GraphClustering["graph_clustering"]:::core
        RoomFinder["room_finder"]:::core
        LabelRemapper["label_remapper"]:::core
    end

    %% Interfaces
    subgraph "Interfaces"
        ROSInterface["ROS Interface"]:::interface
        ZMQInterfaces["zmq_interfaces"]:::core
    end

    %% Python CLI & Bindings
    subgraph "Python CLI & Bindings"
        PythonPipeline["python_pipeline"]:::interface
        RunCmd["run.py"]:::interface
        VisualizeCmd["visualize.py"]:::interface
    end

    %% Config & Models
    subgraph "Config & Models"
        ConfigYAML["YAML Configs"]:::config
        ONNXModelFile["place_gnn.onnx"]:::model
    end

    %% Evaluation Suite
    subgraph "Evaluation Suite"
        PlaceEvaluator["place_evaluator"]:::eval
        TimingCmd["timing.py"]:::eval
    end

    %% Data Flow
    Camera -->|"Image Data"| Conversion
    Lidar -->|"PointCloud"| Conversion
    Conversion -->|"SensorPacket"| Packet
    Packet -->|"Packet"| HydraPipeline
    HydraPipeline -->|"Tasks"| MessageQueue
    MessageQueue -->|"Data"| GraphBuilder
    MessageQueue -->|"Data"| MeshSegmenter
    MessageQueue -->|"Data"| GVDExtractor
    MessageQueue -->|"Data"| ViewSelector
    GraphBuilder -->|"FrontEnd Output"| ONNXModels
    MeshSegmenter --> ONNXModels
    GVDExtractor --> ONNXModels
    ViewSelector --> ONNXModels
    ONNXModels -->|"Descriptors"| DescriptorMatching
    DescriptorMatching -->|"Matches"| LoopClosureModule
    LoopClosureModule -->|"Loop Constraints"| BackendModule
    BackendModule -->|"Factor Graph"| PGMOConfigs
    PGMOConfigs -->|"Optimized DSG"| VolumetricMap
    VolumetricMap -->|"TSDF"| MeshIntegrator
    MeshIntegrator -->|"Mesh"| MarchingCubes
    MarchingCubes -->|"MeshOutput"| GraphClustering
    GraphClustering --> RoomFinder
    RoomFinder --> LabelRemapper
    LabelRemapper --> ROSInterface
    ROSInterface -->|"ROS Topics"| ROSInterface
    PythonPipeline --> RunCmd
    PythonPipeline --> VisualizeCmd
    ConfigYAML --> GraphBuilder
    ConfigYAML --> LoopClosureModule
    ConfigYAML --> BackendModule
    ONNXModelFile --> ONNXModels
    ROSInterface --> PlaceEvaluator
    RunCmd --> PlaceEvaluator
    VisualizeCmd --> TimingCmd

    %% Click Events
    click Camera "https://github.com/mit-spark/hydra/blob/main/include/hydra/input/camera.h"
    click Lidar "https://github.com/mit-spark/hydra/blob/main/include/hydra/input/lidar.h"
    click Conversion "https://github.com/mit-spark/hydra/blob/main/include/hydra/input/input_conversion.h"
    click Packet "https://github.com/mit-spark/hydra/blob/main/include/hydra/input/sensor_input_packet.h"
    click HydraPipeline "https://github.com/mit-spark/hydra/blob/main/include/hydra/common/hydra_pipeline.h"
    click MessageQueue "https://github.com/mit-spark/hydra/blob/main/include/hydra/common/message_queue.h"
    click GraphBuilder "https://github.com/mit-spark/hydra/blob/main/include/hydra/frontend/graph_builder.h"
    click MeshSegmenter "https://github.com/mit-spark/hydra/blob/main/include/hydra/frontend/mesh_segmenter.h"
    click GVDExtractor "https://github.com/mit-spark/hydra/blob/main/include/hydra/frontend/gvd_place_extractor.h"
    click ViewSelector "https://github.com/mit-spark/hydra/blob/main/include/hydra/frontend/view_selector.h"
    click LoopClosureModule "https://github.com/mit-spark/hydra/blob/main/include/hydra/loop_closure/loop_closure_module.h"
    click DescriptorMatching "https://github.com/mit-spark/hydra/blob/main/include/hydra/loop_closure/descriptor_matching.h"
    click GNNInterface "https://github.com/mit-spark/hydra/blob/main/include/hydra/gnn/gnn_interface.h"
    click BackendModule "https://github.com/mit-spark/hydra/blob/main/include/hydra/backend/backend_module.h"
    click MergeTracker "https://github.com/mit-spark/hydra/blob/main/include/hydra/backend/merge_tracker.h"
    click PGMOConfigs "https://github.com/mit-spark/hydra/blob/main/include/hydra/backend/pgmo_configs.h"
    click VolumetricMap "https://github.com/mit-spark/hydra/blob/main/include/hydra/reconstruction/volumetric_map.h"
    click MeshIntegrator "https://github.com/mit-spark/hydra/blob/main/include/hydra/reconstruction/mesh_integrator.h"
    click MarchingCubes "https://github.com/mit-spark/hydra/blob/main/include/hydra/reconstruction/marching_cubes.h"
    click GraphClustering "https://github.com/mit-spark/hydra/blob/main/include/hydra/rooms/graph_clustering.h"
    click RoomFinder "https://github.com/mit-spark/hydra/blob/main/include/hydra/rooms/room_finder.h"
    click LabelRemapper "https://github.com/mit-spark/hydra/blob/main/include/hydra/common/label_remapper.h"
    click ZMQInterfaces "https://github.com/mit-spark/hydra/blob/main/include/hydra/backend/zmq_interfaces.h"
    click PythonPipeline "https://github.com/mit-spark/hydra/blob/main/python/bindings/include/hydra/bindings/python_pipeline.h"
    click RunCmd "https://github.com/mit-spark/hydra/blob/main/python/src/hydra_python/commands/run.py"
    click VisualizeCmd "https://github.com/mit-spark/hydra/blob/main/python/src/hydra_python/commands/visualize.py"
    click ConfigYAML "https://github.com/mit-spark/hydra/blob/main/config/lcd/default.yaml"
    click ONNXModelFile "https://github.com/mit-spark/hydra/blob/main/models/lcd/place_gnn.onnx"
    click PlaceEvaluator "https://github.com/mit-spark/hydra/blob/main/eval/include/hydra/eval/place_evaluator.h"
    click TimingCmd "https://github.com/mit-spark/hydra/blob/main/eval/python/hydra_eval/commands/timing.py"

    %% Styles
    classDef core fill:#AED6F1,stroke:#1B4F72,color:#1B2631
    classDef interface fill:#A9DFBF,stroke:#186A3B,color:#0E6655
    classDef config fill:#F9E79F,stroke:#B7950B,color:#7D6608
    classDef model fill:#D2B4DE,stroke:#6C3483,color:#4A235A
    classDef eval fill:#D5DBDB,stroke:#566573,color:#17202A
```
