# mit-spark/khronos architecture

> Architecture diagram of the GitHub repository mit-spark/khronos, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mit-spark/khronos
- Repository: https://github.com/mit-spark/khronos

## Overview

#### Purpose and Runtime

Khronos is a ROS 2 Jazzy C++ system for online metric-semantic SLAM in dynamic environments. It distinguishes short-term moving objects from long-term scene changes and maintains a queryable spatio-temporal map. It is built as colcon packages: core `khronos`, ROS integration `khronos_ros`, messages `khronos_msgs`, and evaluation `khronos_eval`.

#### Operational Entry Points

The mapping executable is `khronos_ros/app/khronos_node.cpp`, typically started through `khronos_ros/launch/khronos.launch.yaml` or dataset launches such as `khronos_ros/launch/uhumans2_khronos.launch.yaml` and `khronos_ros/launch/jackal_khronos.launch.yaml`. The offline 4D map viewer starts at `khronos_ros/app/spatio_temporal_visualizer_node.cpp` via `khronos_ros/launch/4d_visualizer.launch.yaml`.

#### ROS Pipeline Boundary

`khronos_ros/src/khronos_pipeline.cpp` is the orchestration boundary between ROS inputs/outputs and the core library. It converts ROS sensor, pose, semantic, mesh, and graph data through `khronos_ros/src/utils/ros_conversions.cpp`, manages experiment persistence, and drives the core map pipeline. ROS-facing visualization is separate in `khronos_ros/src/visualization/active_window_visualizer.cpp` and `khronos_ros/src/visualization/spatio_temporal_visualizer.cpp`.

Inputs can use ground-truth/pre-recorded semantic topics or the optional external `semantic_inference` package; real-data playback may use recorded odometry transformed by `khronos_ros/scripts/odom_to_tf.py`.

#### Active-Window Reconstruction and Dynamics

The short-horizon processing stage is `khronos/src/active_window/active_window.cpp`. It retains recent frame state using `khronos/src/active_window/data/frame_data_buffer.cpp`, then:
- derives object hypotheses from semantics/instances in `khronos/src/active_window/object_detection/connected_semantics.cpp` and `khronos/src/active_window/object_detection/instance_forwarding.cpp`;
- extracts object meshes asynchronously through `khronos/src/active_window/object_extraction/mesh_object_extractor.cpp` and `khronos/src/active_window/object_extraction/object_worker_pool.cpp`;
- associates detections into tracks using `khronos/src/active_window/tracking/max_iou_tracker.cpp` or `khronos/src/active_window/tracking/external_tracker.cpp`;
- detects motion against free space in `khronos/src/active_window/motion_detection/free_space_motion_detector.cpp`.

Tracking and object integration bridge this transient window into longer-lived map state: `khronos/src/active_window/integration/tracking_integrator.cpp` and `khronos/src/active_window/integration/object_integrator.cpp`.

#### Persistent Spatio-Temporal Map

`khronos/src/spatio_temporal_map/spatio_temporal_map.cpp` owns the persistent representation of scene state across times, while `khronos/src/spatio_temporal_map/incremental_mesh.cpp` maintains the background mesh incrementally. This is the boundary between recent observations/tracks and historical scene reconstruction; the saved map is later consumed by the 4D visualizer.

#### Backend Change Detection and Reconciliation

`khronos/src/backend/backend.cpp` performs longer-horizon processing when revisits or map updates expose disagreement with prior state. Change detection is layered: sequential coordination in `khronos/src/backend/change_detection/sequential_change_detector.cpp`, shared ray verification in `khronos/src/backend/change_detection/ray_verificator.cpp`, and specialized background/object ray detectors in `khronos/src/backend/change_detection/background/ray_background_change_detector.cpp` and `khronos/src/backend/change_detection/objects/ray_object_change_detector.cpp`.

Detected changes update explicit state in `khronos/src/backend/change_state.cpp`, then reconciliation applies mesh-level merging or overwrite policies through `khronos/src/backend/reconciliation/reconciler.cpp`, `khronos/src/backend/reconciliation/mesh/mesh_merger.cpp`, and `khronos/src/backend/reconciliation/mesh/overwrite_mesh.cpp`.

#### Messages, Evaluation, and External Dependencies

ROS contracts are defined in `khronos_msgs/msg/Changes.msg`, `khronos_msgs/msg/ObjectChange.msg`, and visualization state/services under `khronos_msgs`. Evaluation is a separate ROS package centered on `khronos_eval/app/exp_pipeline.cpp` and `khronos_eval/src/pipeline_evaluator.cpp`, with visualizer nodes for change detection and results.

Material dependencies include ROS 2, GTSAM, glog, nlohmann-json, RViz, rosbag playback, and source dependencies imported from `install/ssh.rosinstall` or `install/https.rosinstall`.

Main entry points: [`khronos/src/spatio_temporal_map/spatio_temporal_map.cpp`](https://github.com/mit-spark/khronos/blob/main/khronos/src/spatio_temporal_map/spatio_temporal_map.cpp) (Spatio-temporal map), [`khronos_ros/src/khronos_pipeline.cpp`](https://github.com/mit-spark/khronos/blob/main/khronos_ros/src/khronos_pipeline.cpp) (ROS pipeline), [`khronos/src/active_window/active_window.cpp`](https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/active_window.cpp) (Active window), [`khronos/src/active_window/object_detection/connected_semantics.cpp`](https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/object_detection/connected_semantics.cpp) (Object hypotheses)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ros["ROS Integration"]
  node_mapping_launch["Mapping launches<br/>ROS launch"]
  node_mapping_node["Khronos node<br/>ROS executable<br/>[khronos_node.cpp]"]
  node_ros_pipeline["ROS pipeline<br/>orchestrator"]
  node_ros_conversions["ROS conversions<br/>adapter"]
  node_semantic_source["Semantic inputs"]
  node_active_visualizer["Active-window visualizer<br/>ROS visualization"]
  node_map_viewer["4D map viewer<br/>ROS executable"]
  node_messages["ROS change messages<br/>ROS contracts<br/>[Changes.msg]"]
end

subgraph group_active["Active Window"]
  node_frame_buffer[("Frame data buffer<br/>recent-state store")]
  node_active_window["Active window<br/>short-horizon pipeline<br/>[active_window.cpp]"]
  node_object_hypotheses["Object hypotheses<br/>semantic / instance detection"]
  node_mesh_extraction["Async object extraction<br/>mesh extraction workers"]
  node_tracking["Object tracking<br/>track association"]
  node_motion_detection["Free-space motion detection<br/>dynamic-object analysis"]
  node_tracking_integration["Tracking integration<br/>map bridge"]
  node_object_integration["Object integration<br/>map bridge"]
end

subgraph group_map_backend["Map and Backend"]
  node_spatiotemporal_map[("Spatio-temporal map<br/>persistent scene state")]
  node_incremental_mesh["Incremental background mesh<br/>mesh maintenance"]
  node_backend["Backend<br/>long-horizon processor<br/>[backend.cpp]"]
  node_change_detection["Sequential change detection<br/>ray-verification coordinator"]
  node_change_state[("Change state<br/>[change_state.cpp]")]
  node_reconciliation["Mesh reconciliation<br/>merge / overwrite policy<br/>[reconciler.cpp]"]
end

node_evaluation["Evaluation pipeline<br/>ROS evaluation package<br/>[exp_pipeline.cpp]"]

node_mapping_launch -->|"starts"| node_mapping_node
node_mapping_node -->|"runs"| node_ros_pipeline
node_semantic_source -->|"semantic data"| node_ros_conversions
node_ros_conversions -->|"converted ROS data"| node_ros_pipeline
node_ros_pipeline -->|"drives"| node_active_window
node_active_window -->|"retains frames"| node_frame_buffer
node_active_window -->|"derives"| node_object_hypotheses
node_object_hypotheses -->|"object candidates"| node_mesh_extraction
node_object_hypotheses -->|"detections"| node_tracking
node_mesh_extraction -->|"object meshes"| node_tracking
node_active_window -->|"free-space evidence"| node_motion_detection
node_tracking -->|"tracks"| node_tracking_integration
node_mesh_extraction -->|"object meshes"| node_object_integration
node_motion_detection -->|"motion state"| node_object_integration
node_tracking_integration -->|"long-lived tracks"| node_spatiotemporal_map
node_object_integration -->|"object updates"| node_spatiotemporal_map
node_spatiotemporal_map -->|"background updates"| node_incremental_mesh
node_spatiotemporal_map -->|"revisits and map updates"| node_backend
node_backend -->|"coordinates"| node_change_detection
node_change_detection -->|"detected changes"| node_change_state
node_change_state -->|"reconciliation input"| node_reconciliation
node_reconciliation -->|"reconciled scene updates"| node_spatiotemporal_map
node_ros_pipeline -->|"live visualization"| node_active_visualizer
node_ros_pipeline -->|"publishes changes"| node_messages
node_spatiotemporal_map -->|"saved map"| node_map_viewer
node_spatiotemporal_map -.->|"map results"| node_evaluation

click node_mapping_launch "https://github.com/mit-spark/khronos/blob/main/khronos_ros/launch/khronos.launch.yaml"
click node_mapping_node "https://github.com/mit-spark/khronos/blob/main/khronos_ros/app/khronos_node.cpp"
click node_ros_pipeline "https://github.com/mit-spark/khronos/blob/main/khronos_ros/src/khronos_pipeline.cpp"
click node_ros_conversions "https://github.com/mit-spark/khronos/blob/main/khronos_ros/src/utils/ros_conversions.cpp"
click node_active_visualizer "https://github.com/mit-spark/khronos/blob/main/khronos_ros/src/visualization/active_window_visualizer.cpp"
click node_frame_buffer "https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/data/frame_data_buffer.cpp"
click node_active_window "https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/active_window.cpp"
click node_object_hypotheses "https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/object_detection/connected_semantics.cpp"
click node_mesh_extraction "https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/object_extraction/object_worker_pool.cpp"
click node_tracking "https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/tracking/max_iou_tracker.cpp"
click node_motion_detection "https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/motion_detection/free_space_motion_detector.cpp"
click node_tracking_integration "https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/integration/tracking_integrator.cpp"
click node_object_integration "https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/integration/object_integrator.cpp"
click node_spatiotemporal_map "https://github.com/mit-spark/khronos/blob/main/khronos/src/spatio_temporal_map/spatio_temporal_map.cpp"
click node_incremental_mesh "https://github.com/mit-spark/khronos/blob/main/khronos/src/spatio_temporal_map/incremental_mesh.cpp"
click node_backend "https://github.com/mit-spark/khronos/blob/main/khronos/src/backend/backend.cpp"
click node_change_detection "https://github.com/mit-spark/khronos/blob/main/khronos/src/backend/change_detection/sequential_change_detector.cpp"
click node_change_state "https://github.com/mit-spark/khronos/blob/main/khronos/src/backend/change_state.cpp"
click node_reconciliation "https://github.com/mit-spark/khronos/blob/main/khronos/src/backend/reconciliation/reconciler.cpp"
click node_map_viewer "https://github.com/mit-spark/khronos/blob/main/khronos_ros/app/spatio_temporal_visualizer_node.cpp"
click node_evaluation "https://github.com/mit-spark/khronos/blob/main/khronos_eval/app/exp_pipeline.cpp"
click node_messages "https://github.com/mit-spark/khronos/blob/main/khronos_msgs/msg/Changes.msg"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mapping_launch,node_mapping_node,node_ros_pipeline,node_ros_conversions,node_semantic_source,node_active_visualizer,node_map_viewer,node_messages toneBlue
class node_frame_buffer,node_active_window,node_object_hypotheses,node_mesh_extraction,node_tracking,node_motion_detection,node_tracking_integration,node_object_integration toneAmber
class node_spatiotemporal_map,node_incremental_mesh,node_backend,node_change_detection,node_change_state,node_reconciliation toneMint
class node_evaluation toneNeutral
```

## Components

### ROS Integration

ROS entry points, orchestration, and visualization.

- [Mapping launches](https://github.com/mit-spark/khronos/blob/main/khronos_ros/launch/khronos.launch.yaml): ROS launch · Starts mapping for generic and dataset workflows. · `khronos_ros/launch/khronos.launch.yaml`
- [Khronos node](https://github.com/mit-spark/khronos/blob/main/khronos_ros/app/khronos_node.cpp): ROS executable · `khronos_ros/app/khronos_node.cpp`
- [ROS pipeline](https://github.com/mit-spark/khronos/blob/main/khronos_ros/src/khronos_pipeline.cpp): orchestrator · Bridges ROS I/O, persistence, and core processing. · `khronos_ros/src/khronos_pipeline.cpp`
- [ROS conversions](https://github.com/mit-spark/khronos/blob/main/khronos_ros/src/utils/ros_conversions.cpp): adapter · Converts sensor, pose, semantic, mesh, and graph data. · `khronos_ros/src/utils/ros_conversions.cpp`
- **Semantic inputs**: ROS topics / optional inference · Uses recorded or ground-truth topics, optionally semantic_inference.
- [Active-window visualizer](https://github.com/mit-spark/khronos/blob/main/khronos_ros/src/visualization/active_window_visualizer.cpp): ROS visualization · `khronos_ros/src/visualization/active_window_visualizer.cpp`
- [4D map viewer](https://github.com/mit-spark/khronos/blob/main/khronos_ros/app/spatio_temporal_visualizer_node.cpp): ROS executable · Loads and visualizes saved spatio-temporal maps. · `khronos_ros/app/spatio_temporal_visualizer_node.cpp`
- [ROS change messages](https://github.com/mit-spark/khronos/blob/main/khronos_msgs/msg/Changes.msg): ROS contracts · `khronos_msgs/msg/Changes.msg`

### Active Window

Short-horizon reconstruction, object processing, and motion analysis.

- [Frame data buffer](https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/data/frame_data_buffer.cpp): recent-state store · `khronos/src/active_window/data/frame_data_buffer.cpp`
- [Active window](https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/active_window.cpp): short-horizon pipeline · Reconstructs recent scene state and dynamics. · `khronos/src/active_window/active_window.cpp`
- [Object hypotheses](https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/object_detection/connected_semantics.cpp): semantic / instance detection · Derives objects from connected semantics or forwarded instances. · `khronos/src/active_window/object_detection/connected_semantics.cpp`
- [Async object extraction](https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/object_extraction/object_worker_pool.cpp): mesh extraction workers · Builds object meshes asynchronously. · `khronos/src/active_window/object_extraction/object_worker_pool.cpp`
- [Object tracking](https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/tracking/max_iou_tracker.cpp): track association · Associates detections with IoU or external tracking. · `khronos/src/active_window/tracking/max_iou_tracker.cpp`
- [Free-space motion detection](https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/motion_detection/free_space_motion_detector.cpp): dynamic-object analysis · `khronos/src/active_window/motion_detection/free_space_motion_detector.cpp`
- [Tracking integration](https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/integration/tracking_integrator.cpp): map bridge · `khronos/src/active_window/integration/tracking_integrator.cpp`
- [Object integration](https://github.com/mit-spark/khronos/blob/main/khronos/src/active_window/integration/object_integrator.cpp): map bridge · `khronos/src/active_window/integration/object_integrator.cpp`

### Map and Backend

Persistent scene state plus long-horizon change reconciliation.

- [Spatio-temporal map](https://github.com/mit-spark/khronos/blob/main/khronos/src/spatio_temporal_map/spatio_temporal_map.cpp): persistent scene state · Owns historical metric-semantic scene state. · `khronos/src/spatio_temporal_map/spatio_temporal_map.cpp`
- [Incremental background mesh](https://github.com/mit-spark/khronos/blob/main/khronos/src/spatio_temporal_map/incremental_mesh.cpp): mesh maintenance · `khronos/src/spatio_temporal_map/incremental_mesh.cpp`
- [Backend](https://github.com/mit-spark/khronos/blob/main/khronos/src/backend/backend.cpp): long-horizon processor · Processes revisits and disagreement with prior state. · `khronos/src/backend/backend.cpp`
- [Sequential change detection](https://github.com/mit-spark/khronos/blob/main/khronos/src/backend/change_detection/sequential_change_detector.cpp): ray-verification coordinator · Coordinates shared and specialized background/object ray checks. · `khronos/src/backend/change_detection/sequential_change_detector.cpp`
- [Change state](https://github.com/mit-spark/khronos/blob/main/khronos/src/backend/change_state.cpp): detected-change state · `khronos/src/backend/change_state.cpp`
- [Mesh reconciliation](https://github.com/mit-spark/khronos/blob/main/khronos/src/backend/reconciliation/reconciler.cpp): merge / overwrite policy · Applies mesh-level merge or overwrite updates. · `khronos/src/backend/reconciliation/reconciler.cpp`

### Other components

- [Evaluation pipeline](https://github.com/mit-spark/khronos/blob/main/khronos_eval/app/exp_pipeline.cpp): ROS evaluation package · `khronos_eval/app/exp_pipeline.cpp`

## Connections

- Mapping launches → Khronos node: starts
- Khronos node → ROS pipeline: runs
- Semantic inputs → ROS conversions: semantic data
- ROS conversions → ROS pipeline: converted ROS data
- ROS pipeline → Active window: drives
- Active window → Frame data buffer: retains frames
- Active window → Object hypotheses: derives
- Object hypotheses → Async object extraction: object candidates
- Object hypotheses → Object tracking: detections
- Async object extraction → Object tracking: object meshes
- Active window → Free-space motion detection: free-space evidence
- Object tracking → Tracking integration: tracks
- Async object extraction → Object integration: object meshes
- Free-space motion detection → Object integration: motion state
- Tracking integration → Spatio-temporal map: long-lived tracks
- Object integration → Spatio-temporal map: object updates
- Spatio-temporal map → Incremental background mesh: background updates
- Spatio-temporal map → Backend: revisits and map updates
- Backend → Sequential change detection: coordinates
- Sequential change detection → Change state: detected changes
- Change state → Mesh reconciliation: reconciliation input
- Mesh reconciliation → Spatio-temporal map: reconciled scene updates
- ROS pipeline → Active-window visualizer: live visualization
- ROS pipeline → ROS change messages: publishes changes
- Spatio-temporal map → 4D map viewer: saved map
- Spatio-temporal map → Evaluation pipeline: map results
