How mmcza/semantic-mapping-for-autonomous-vehicles works
This repository builds semantic point-cloud maps for autonomous-vehicle scenes, with a ROS2 workflow that synchronizes camera images and point clouds, runs ONNX segmentation, projects labels using camera/LiDAR transforms, filters and colors points, then publishes a cloud for OctoMap and RViz. Related capabilities include model training and evaluation, point-cloud analysis, and GroundingDINO/SAM-assisted annotation. ROS relationships are documented in the README; sampled source also shows the annotation app’s HTTP request and the LangSAM model calls. Other tool relationships are not inferred from names alone.
Main entry points: segmentation_node.cpp Segmentation node · lang_sam.py Text-guided segmentation · segmentation_node_launch.py Mapping launch · server.py Inference server
Components 31
Semantic mapping
- Segmentation node
segmentation_node/src/segmentation_node.cpp - Node state and interfaces
segmentation_node/include/segmentation_node/segmentation_node.hpp - Mapping launch
segmentation_node/launch/segmentation_node_launch.py - OctoMap server
- Semantic map saver
segmentation_node/src/semantic_octomap_saver.cpp - RViz
Segmentation models
- ONNX models
HF_segmentation/ONNX_trained - WandB training
HF_segmentation/main_wandb.py - DeepLab models
utils/deeplab3.py - DeepLab architecture
utils/model_deeplab.py - PIDNet model
utils/pidnet_model.py - Multi-encoder UNet
utils/unet_multiple_encoders.py - Mask creation
HF_segmentation/create_masks_b5.py
Annotation tools
- Text-guided segmentation
utils/segmentation_with_sam/lang_sam/lang_sam.py - GroundingDINO
utils/segmentation_with_sam/lang_sam/models/gdino.py - SAM masks
utils/segmentation_with_sam/lang_sam/models/sam.py - Inference server
utils/segmentation_with_sam/lang_sam/server.py - Annotation web app
utils/segmentation_with_sam/app.py - Training mask generator
utils/segmentation_with_sam/generate_masks_for_training.py - Batch mask generator
utils/segmentation_with_sam/generate_masks_for_training_with_batch_queries.py
Data and analysis
- Segmentation viewer
HF_segmentation/simple_viewer.py - Point-cloud viewer
utils/viewer.py - Point-cloud processing
point_cloud_processor.py - Inference benchmarks
utils/inference_time_calculation.py - Cloud accuracy
utils/pointcloud_accuracy_calculation.py - Cloud statistics
utils/pointcloud_statistics_calculation.py - Bag image extraction
utils/ros2_bag_to_png.py
Other components
- Operator
- RGB camera
- LiDAR point cloud
- TF transforms
Connections 18
- Operator → Mapping launch: launches
README.md - Mapping launch → Segmentation node: starts
README.md - Mapping launch → OctoMap server: starts
README.md - Mapping launch → RViz: starts
README.md - RGB camera → Segmentation node: provides images
README.md - LiDAR point cloud → Segmentation node: provides cloud
README.md - TF transforms → Segmentation node: provides transforms
README.md - Segmentation node → ONNX models: runs inference
README.md - Segmentation node → OctoMap server: publishes cloud
README.md - OctoMap server → RViz: visualizes map
README.md - Segmentation node → Node state and interfaces: uses interfaces
segmentation_node/src/segmentation_node.cpp - Operator → Annotation web app: submits prompt
- Annotation web app → Inference server: posts prediction
utils/segmentation_with_sam/app.py - Inference server → Text-guided segmentation: predicts with
utils/segmentation_with_sam/lang_sam/server.py - Text-guided segmentation → GroundingDINO: detects boxes
utils/segmentation_with_sam/lang_sam/lang_sam.py - Text-guided segmentation → SAM masks: predicts masks
utils/segmentation_with_sam/lang_sam/lang_sam.py - Training mask generator → Text-guided segmentation: generates annotations
utils/segmentation_with_sam/generate_masks_for_training.py - Batch mask generator → Text-guided segmentation: generates annotations
utils/segmentation_with_sam/generate_masks_for_training_with_batch_queries.py