How mit-spark/khronos works
Purpose and Runtime
Khronos is a ROS 2 Jazzy C++ system for online metric-semantic SLAM in dynamic environments. It distinguishes short-term moving objects from long-term scene changes and maintains a queryable spatio-temporal map. It is built as colcon packages: core khronos, ROS integration khronos_ros, messages khronos_msgs, and evaluation khronos_eval.
Operational Entry Points
The mapping executable is khronos_ros/app/khronos_node.cpp, typically started through khronos_ros/launch/khronos.launch.yaml or dataset launches such as khronos_ros/launch/uhumans2_khronos.launch.yaml and khronos_ros/launch/jackal_khronos.launch.yaml. The offline 4D map viewer starts at khronos_ros/app/spatio_temporal_visualizer_node.cpp via khronos_ros/launch/4d_visualizer.launch.yaml.
ROS Pipeline Boundary
khronos_ros/src/khronos_pipeline.cpp is the orchestration boundary between ROS inputs/outputs and the core library. It converts ROS sensor, pose, semantic, mesh, and graph data through khronos_ros/src/utils/ros_conversions.cpp, manages experiment persistence, and drives the core map pipeline. ROS-facing visualization is separate in khronos_ros/src/visualization/active_window_visualizer.cpp and khronos_ros/src/visualization/spatio_temporal_visualizer.cpp.
Inputs can use ground-truth/pre-recorded semantic topics or the optional external semantic_inference package; real-data playback may use recorded odometry transformed by khronos_ros/scripts/odom_to_tf.py.
Active-Window Reconstruction and Dynamics
The short-horizon processing stage is khronos/src/active_window/active_window.cpp. It retains recent frame state using khronos/src/active_window/data/frame_data_buffer.cpp, then:
• derives object hypotheses from semantics/instances in khronos/src/active_window/object_detection/connected_semantics.cpp and khronos/src/active_window/object_detection/instance_forwarding.cpp;
• extracts object meshes asynchronously through khronos/src/active_window/object_extraction/mesh_object_extractor.cpp and khronos/src/active_window/object_extraction/object_worker_pool.cpp;
• associates detections into tracks using khronos/src/active_window/tracking/max_iou_tracker.cpp or khronos/src/active_window/tracking/external_tracker.cpp;
• detects motion against free space in khronos/src/active_window/motion_detection/free_space_motion_detector.cpp.
Tracking and object integration bridge this transient window into longer-lived map state: khronos/src/active_window/integration/tracking_integrator.cpp and khronos/src/active_window/integration/object_integrator.cpp.
Persistent Spatio-Temporal Map
khronos/src/spatio_temporal_map/spatio_temporal_map.cpp owns the persistent representation of scene state across times, while khronos/src/spatio_temporal_map/incremental_mesh.cpp maintains the background mesh incrementally. This is the boundary between recent observations/tracks and historical scene reconstruction; the saved map is later consumed by the 4D visualizer.
Backend Change Detection and Reconciliation
khronos/src/backend/backend.cpp performs longer-horizon processing when revisits or map updates expose disagreement with prior state. Change detection is layered: sequential coordination in khronos/src/backend/change_detection/sequential_change_detector.cpp, shared ray verification in khronos/src/backend/change_detection/ray_verificator.cpp, and specialized background/object ray detectors in khronos/src/backend/change_detection/background/ray_background_change_detector.cpp and khronos/src/backend/change_detection/objects/ray_object_change_detector.cpp.
Detected changes update explicit state in khronos/src/backend/change_state.cpp, then reconciliation applies mesh-level merging or overwrite policies through khronos/src/backend/reconciliation/reconciler.cpp, khronos/src/backend/reconciliation/mesh/mesh_merger.cpp, and khronos/src/backend/reconciliation/mesh/overwrite_mesh.cpp.
Messages, Evaluation, and External Dependencies
ROS contracts are defined in khronos_msgs/msg/Changes.msg, khronos_msgs/msg/ObjectChange.msg, and visualization state/services under khronos_msgs. Evaluation is a separate ROS package centered on khronos_eval/app/exp_pipeline.cpp and khronos_eval/src/pipeline_evaluator.cpp, with visualizer nodes for change detection and results.
Material dependencies include ROS 2, GTSAM, glog, nlohmann-json, RViz, rosbag playback, and source dependencies imported from install/ssh.rosinstall or install/https.rosinstall.
Main entry points: spatio_temporal_map.cpp Spatio-temporal map · khronos_pipeline.cpp ROS pipeline · active_window.cpp Active window · connected_semantics.cpp Object hypotheses
Components 23
ROS Integration
ROS entry points, orchestration, and visualization.
- Mapping launches · Starts mapping for generic and dataset workflows.
khronos_ros/launch/khronos.launch.yaml - Khronos node
khronos_ros/app/khronos_node.cpp - ROS pipeline · Bridges ROS I/O, persistence, and core processing.
khronos_ros/src/khronos_pipeline.cpp - ROS conversions · Converts sensor, pose, semantic, mesh, and graph data.
khronos_ros/src/utils/ros_conversions.cpp - Semantic inputs · Uses recorded or ground-truth topics, optionally semantic_inference.
- Active-window visualizer
khronos_ros/src/visualization/active_window_visualizer.cpp - 4D map viewer · Loads and visualizes saved spatio-temporal maps.
khronos_ros/app/spatio_temporal_visualizer_node.cpp - ROS change messages
khronos_msgs/msg/Changes.msg
Active Window
Short-horizon reconstruction, object processing, and motion analysis.
- Frame data buffer
khronos/src/active_window/data/frame_data_buffer.cpp - Active window · Reconstructs recent scene state and dynamics.
khronos/src/active_window/active_window.cpp - Object hypotheses · Derives objects from connected semantics or forwarded instances.
khronos/src/active_window/object_detection/connected_semantics.cpp - Async object extraction · Builds object meshes asynchronously.
khronos/src/active_window/object_extraction/object_worker_pool.cpp - Object tracking · Associates detections with IoU or external tracking.
khronos/src/active_window/tracking/max_iou_tracker.cpp - Free-space motion detection
khronos/src/active_window/motion_detection/free_space_motion_detector.cpp - Tracking integration
khronos/src/active_window/integration/tracking_integrator.cpp - Object integration
khronos/src/active_window/integration/object_integrator.cpp
Map and Backend
Persistent scene state plus long-horizon change reconciliation.
- Spatio-temporal map · Owns historical metric-semantic scene state.
khronos/src/spatio_temporal_map/spatio_temporal_map.cpp - Incremental background mesh
khronos/src/spatio_temporal_map/incremental_mesh.cpp - Backend · Processes revisits and disagreement with prior state.
khronos/src/backend/backend.cpp - Sequential change detection · Coordinates shared and specialized background/object ray checks.
khronos/src/backend/change_detection/sequential_change_detector.cpp - Change state
khronos/src/backend/change_state.cpp - Mesh reconciliation · Applies mesh-level merge or overwrite updates.
khronos/src/backend/reconciliation/reconciler.cpp
Other components
- Evaluation pipeline
khronos_eval/app/exp_pipeline.cpp
Connections 26
- Mapping launches → Khronos node: starts
- Khronos node → ROS pipeline: runs
- Semantic inputs → ROS conversions: semantic data
- ROS conversions → ROS pipeline: converted ROS data
- ROS pipeline → Active window: drives
- Active window → Frame data buffer: retains frames
- Active window → Object hypotheses: derives
- Object hypotheses → Async object extraction: object candidates
- Object hypotheses → Object tracking: detections
- Async object extraction → Object tracking: object meshes
- Active window → Free-space motion detection: free-space evidence
- Object tracking → Tracking integration: tracks
- Async object extraction → Object integration: object meshes
- Free-space motion detection → Object integration: motion state
- Tracking integration → Spatio-temporal map: long-lived tracks
- Object integration → Spatio-temporal map: object updates
- Spatio-temporal map → Incremental background mesh: background updates
- Spatio-temporal map → Backend: revisits and map updates
- Backend → Sequential change detection: coordinates
- Sequential change detection → Change state: detected changes
- Change state → Mesh reconciliation: reconciliation input
- Mesh reconciliation → Spatio-temporal map: reconciled scene updates
- ROS pipeline → Active-window visualizer: live visualization
- ROS pipeline → ROS change messages: publishes changes
- Spatio-temporal map → 4D map viewer: saved map
- Spatio-temporal map → Evaluation pipeline: map results