How gabrielmanalu/edgedrive-perception works
1. Repository purpose
- EdgeDrive Perception is an end-to-end autonomous driving perception stack focused on edge deployment on Jetson Orin Nano Super.
- The repo is organized as a pipeline: dataset conversion and training → model export and quantization → sensor fusion → real-time C++ deployment → benchmarking and analysis.
2. Primary architectural split: Python research/training vs C++ deployment
- Most model development lives in Python: dataset conversion, YOLO training, quantization, LiDAR inference, fusion experiments, and demo scripts.
- The deployment boundary is a separate C++/CUDA/TensorRT runtime under deployment/, which consumes exported engines and runs the real-time pipeline on Jetson hardware.
3. Dataset and preprocessing flow
- nuScenes is the core dataset across the repo, especially nuScenes Mini for development and validation.
- training/convert_nuscenes_det.py and convert_nuscenes_seg.py reshape nuScenes annotations into YOLO-friendly formats.
- calibration.yaml is used for INT8 TensorRT calibration, tying the training/export stage to deployment constraints.
4. Camera perception subsystem
- The camera branch is built around YOLO models: train_yolo26n.py, train_yolo26n_seg.py, and train_yolov8n.py.
- The chosen deployment model is YOLO26n, selected for NMS-free inference and quantization robustness rather than only raw FP32 accuracy.
- Outputs include 2D detections and, for the segmentation variant, instance masks.
5. LiDAR perception subsystem
- The LiDAR branch centers on PointPillars, trained and evaluated in fusion/train_pointpillars.py and fusion/pointpillars_inference.py.
- This branch produces BEV-aligned 3D detections from LIDAR_TOP data, serving as the spatially precise modality in fusion.
6. Sensor fusion subsystem
- Fusion is implemented as late fusion in BEV rather than a single unified multimodal network.
- The Python fusion path (fusion/late_fusion.py, camera_to_bev.py, bev_visualization.py, fusion_evaluation.py) aligns camera detections into BEV and matches them to LiDAR detections.
- The readme describes class-aware nearest-neighbor matching with a 12 m threshold and weighted score blending: LiDAR contributes more to geometry, camera contributes semantics.
7. Deployment runtime architecture
- deployment/ is the Jetson-facing runtime, built with CMake and organized around a central executable in src/main.cpp.
- Key runtime modules are:
- camera_capture.cpp for live frame ingestion,
- preprocessor.cu for GPU-side preprocessing,
- trt_engine.cpp for TensorRT inference execution,
- decoder modules for model-specific outputs,
- visualization and analytics modules for BEV rendering and application logic.
- This is the execution boundary where Python-exported models become low-latency on-device inference.
8. Model-specific decoding and postprocessing
- The deployment layer separates TensorRT execution from model decoding.
- yolo26_decoder.cpp, yolov8_decoder.cpp, segmentation_decoder.cpp, pointpillars_decoder.cpp, and yolo26_decoder.hpp handle turning raw network outputs into boxes, masks, or BEV detections.
- This modularity lets the runtime support multiple perception heads without rewriting the inference engine.
9. Real-time analytics and application layer
- Beyond detection, the deployment runtime includes downstream perception utilities:
- object_counter.cpp
- speed_estimator.cpp
- heatmap_generator.cpp
- bev_visualizer.cpp
- These modules consume detector outputs and convert them into user-facing analytics, showing that the runtime is not just inference but a full perception application stack.
10. Export, quantization, and hardware optimization path
- training/export_all_formats.py, quantize.py, and prune.py form the model optimization pipeline.
- The repo emphasizes PTQ/QAT and export to ONNX, TFLite, and TensorRT, but the Jetson deployment path specifically depends on TensorRT engines built on target hardware.
- The design choice is hardware-driven: the same model is validated in Python, then specialized for NVIDIA deployment.
11. Demo and solution scripts as application examples
- solutions/ contains real-time analytics demos built on the Ultralytics Solutions API: heatmaps, object counting, speed estimation, segmentation, and simple analytics dashboards.
- These are not core infrastructure, but they show how the detection model is reused as a component in multiple edge applications.
12. Benchmarking, documentation, and development traceability
- benchmarks/, scripts/, and docs/ capture performance measurement, plotting, and analysis of architectural tradeoffs.
- notebooks/development_walkthrough.ipynb documents the investigative path and key design decisions, including model choice, fusion issues, and quantization results.
- These directories are analysis/support layers rather than runtime components, but they explain why the architecture is structured around edge constraints.
13. Overall data flow to diagram
- The highest-level flow is: nuScenes data → conversion/training → export/quantize → TensorRT engine build → Jetson C++ runtime.
- In parallel, camera and LiDAR inference branches produce modality-specific outputs that are aligned in BEV and optionally fused, then passed to visualization/analytics modules.
- The main architectural boundary is between offline Python experimentation and online C++ deployment on Jetson.
Main entry points: export_all_formats.py Model Opt · convert_nuscenes_det.py Dataset Convert · trt_engine.cpp TensorRT Core · main.cpp Main Runtime
Components 22
Offline Python
Dataset preparation, training, fusion experiments, export, and analysis.
- nuScenes Data · Core dataset source for detection, segmentation, and fusion workflows.
- Dataset Convert · Converts nuScenes annotations into YOLO-friendly detection and segmentation formats.
training/convert_nuscenes_det.py - YOLO Training · Trains camera perception models, including detection and segmentation variants.
training/train_yolo26n.py - YOLOv8 Train · Alternative camera model training path used for comparison and experimentation.
training/train_yolov8n.py - Segmentation Train · Trains YOLO segmentation models for instance masks and downstream deployment.
training/train_yolo26n_seg.py - Model Opt · Exports, prunes, and quantizes trained models for deployment formats and calibration.
training/export_all_formats.py - Calibration · INT8 calibration data and constraints used to specialize TensorRT deployment.
calibration.yaml - PointPillars Train · Trains the LiDAR perception branch for BEV-aligned 3D detection.
fusion/train_pointpillars.py - Fusion Research · Python late-fusion experiments, BEV projection, and evaluation.
fusion/late_fusion.py - Build Engine · Builds TensorRT engines for target hardware deployment.
scripts/build_engine.sh
Jetson Runtime
C++/CUDA/TensorRT deployment stack for real-time edge inference.
- Main Runtime · Central Jetson executable that orchestrates capture, inference, decoding, and analytics.
deployment/src/main.cpp - Capture + GPU Prep · Live frame ingestion and GPU-side preprocessing before inference.
deployment/src/camera_capture.cpp - TensorRT Core · Executes exported TensorRT engines on device.
deployment/src/trt_engine.cpp - Decoders · Model-specific output decoding for boxes, masks, and BEV detections.
deployment/src/yolo26_decoder.cpp - BEV Fusion · Runtime late fusion and BEV alignment of camera and LiDAR outputs.
deployment/src/late_fusion.cpp - Analytics · Object counting, speed estimation, heatmaps, and BEV visualization.
deployment/src/object_counter.cpp - LiDAR Decode · Transforms PointPillars network outputs into BEV detections.
deployment/src/pointpillars_decoder.cpp - Camera Decode · Transforms YOLO outputs into detections or masks for runtime use.
deployment/src/yolov8_decoder.cpp
Support
Documentation, benchmarks, demos, and notebooks that explain or exercise the stack.
- Benchmarks · Performance measurement, plotting, and report generation for the stack.
benchmarks/README.md - Solutions · Edge application demos built on perception outputs and analytics primitives.
solutions/README.md - Docs · Architecture notes, optimization logs, and tradeoff analysis.
docs/architecture.md - Walkthrough · Development notebook capturing key design decisions and investigation history.
notebooks/development_walkthrough.ipynb
Connections 23
- nuScenes Data → Dataset Convert: annotate · nuScenes annotations are converted into task-specific training formats.
- Dataset Convert → YOLO Training: feed · Converted detection data trains the camera detector.
- Dataset Convert → Segmentation Train: feed · Converted segmentation data trains the camera segmentation model.
- Dataset Convert → PointPillars Train: feed · Dataset preparation also supports LiDAR training workflows.
- YOLO Training → Model Opt: export · Trained models are exported and optimized for deployment.
- YOLOv8 Train → Model Opt: compare · YOLOv8 provides a comparison baseline for optimization and accuracy.
- Segmentation Train → Model Opt: export · Segmentation models are packaged into deployable artifacts.
- Model Opt → Calibration: calibrate · Quantization uses calibration data and INT8 constraints.
- Model Opt → Build Engine: build · Optimized artifacts are turned into target-device TensorRT engines.
- PointPillars Train → Fusion Research: evaluate · LiDAR detections are consumed by late-fusion experiments.
- Fusion Research → Build Engine: specialize · Fusion and model choices inform deployment engine creation.
- Build Engine → Main Runtime: deploy · Built engines are consumed by the Jetson runtime.
- Main Runtime → Capture + GPU Prep: capture · Main runtime orchestrates camera input and preprocessing.
- Main Runtime → TensorRT Core: infer · Main runtime invokes TensorRT execution.
- TensorRT Core → Decoders: decode · Raw outputs are routed to model-specific decoders.
- TensorRT Core → LiDAR Decode: decode · LiDAR network outputs are decoded into BEV detections.
- TensorRT Core → Camera Decode: decode · Camera network outputs are decoded into detections or masks.
- Decoders → BEV Fusion: align · Camera outputs are projected and matched in BEV fusion.
- LiDAR Decode → BEV Fusion: fuse · LiDAR outputs provide geometric anchors for late fusion.
- BEV Fusion → Analytics: analyze · Fused perception results feed analytics and visualization modules.
- Analytics → Solutions: demo · Analytics primitives are reused in packaged solution demos.
- Benchmarks → Docs: document · Benchmarks are summarized and interpreted in the documentation set.
- Walkthrough → Docs: inform · The walkthrough records decisions reflected in the documentation.