# gabrielmanalu/edgedrive-perception architecture

> Architecture diagram of the GitHub repository gabrielmanalu/edgedrive-perception, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gabrielmanalu/edgedrive-perception
- Repository: https://github.com/gabrielmanalu/edgedrive-perception

## Overview

1. **Repository purpose**
   - EdgeDrive Perception is an end-to-end autonomous driving perception stack focused on edge deployment on Jetson Orin Nano Super.
   - The repo is organized as a pipeline: dataset conversion and training → model export and quantization → sensor fusion → real-time C++ deployment → benchmarking and analysis.

2. **Primary architectural split: Python research/training vs C++ deployment**
   - Most model development lives in Python: dataset conversion, YOLO training, quantization, LiDAR inference, fusion experiments, and demo scripts.
   - The deployment boundary is a separate C++/CUDA/TensorRT runtime under `deployment/`, which consumes exported engines and runs the real-time pipeline on Jetson hardware.

3. **Dataset and preprocessing flow**
   - nuScenes is the core dataset across the repo, especially `nuScenes Mini` for development and validation.
   - `training/convert_nuscenes_det.py` and `convert_nuscenes_seg.py` reshape nuScenes annotations into YOLO-friendly formats.
   - `calibration.yaml` is used for INT8 TensorRT calibration, tying the training/export stage to deployment constraints.

4. **Camera perception subsystem**
   - The camera branch is built around YOLO models: `train_yolo26n.py`, `train_yolo26n_seg.py`, and `train_yolov8n.py`.
   - The chosen deployment model is YOLO26n, selected for NMS-free inference and quantization robustness rather than only raw FP32 accuracy.
   - Outputs include 2D detections and, for the segmentation variant, instance masks.

5. **LiDAR perception subsystem**
   - The LiDAR branch centers on PointPillars, trained and evaluated in `fusion/train_pointpillars.py` and `fusion/pointpillars_inference.py`.
   - This branch produces BEV-aligned 3D detections from `LIDAR_TOP` data, serving as the spatially precise modality in fusion.

6. **Sensor fusion subsystem**
   - Fusion is implemented as late fusion in BEV rather than a single unified multimodal network.
   - The Python fusion path (`fusion/late_fusion.py`, `camera_to_bev.py`, `bev_visualization.py`, `fusion_evaluation.py`) aligns camera detections into BEV and matches them to LiDAR detections.
   - The readme describes class-aware nearest-neighbor matching with a 12 m threshold and weighted score blending: LiDAR contributes more to geometry, camera contributes semantics.

7. **Deployment runtime architecture**
   - `deployment/` is the Jetson-facing runtime, built with CMake and organized around a central executable in `src/main.cpp`.
   - Key runtime modules are:
     - `camera_capture.cpp` for live frame ingestion,
     - `preprocessor.cu` for GPU-side preprocessing,
     - `trt_engine.cpp` for TensorRT inference execution,
     - decoder modules for model-specific outputs,
     - visualization and analytics modules for BEV rendering and application logic.
   - This is the execution boundary where Python-exported models become low-latency on-device inference.

8. **Model-specific decoding and postprocessing**
   - The deployment layer separates TensorRT execution from model decoding.
   - `yolo26_decoder.cpp`, `yolov8_decoder.cpp`, `segmentation_decoder.cpp`, `pointpillars_decoder.cpp`, and `yolo26_decoder.hpp` handle turning raw network outputs into boxes, masks, or BEV detections.
   - This modularity lets the runtime support multiple perception heads without rewriting the inference engine.

9. **Real-time analytics and application layer**
   - Beyond detection, the deployment runtime includes downstream perception utilities:
     - `object_counter.cpp`
     - `speed_estimator.cpp`
     - `heatmap_generator.cpp`
     - `bev_visualizer.cpp`
   - These modules consume detector outputs and convert them into user-facing analytics, showing that the runtime is not just inference but a full perception application stack.

10. **Export, quantization, and hardware optimization path**
   - `training/export_all_formats.py`, `quantize.py`, and `prune.py` form the model optimization pipeline.
   - The repo emphasizes PTQ/QAT and export to ONNX, TFLite, and TensorRT, but the Jetson deployment path specifically depends on TensorRT engines built on target hardware.
   - The design choice is hardware-driven: the same model is validated in Python, then specialized for NVIDIA deployment.

11. **Demo and solution scripts as application examples**
   - `solutions/` contains real-time analytics demos built on the Ultralytics Solutions API: heatmaps, object counting, speed estimation, segmentation, and simple analytics dashboards.
   - These are not core infrastructure, but they show how the detection model is reused as a component in multiple edge applications.

12. **Benchmarking, documentation, and development traceability**
   - `benchmarks/`, `scripts/`, and `docs/` capture performance measurement, plotting, and analysis of architectural tradeoffs.
   - `notebooks/development_walkthrough.ipynb` documents the investigative path and key design decisions, including model choice, fusion issues, and quantization results.
   - These directories are analysis/support layers rather than runtime components, but they explain why the architecture is structured around edge constraints.

13. **Overall data flow to diagram**
   - The highest-level flow is: nuScenes data → conversion/training → export/quantize → TensorRT engine build → Jetson C++ runtime.
   - In parallel, camera and LiDAR inference branches produce modality-specific outputs that are aligned in BEV and optionally fused, then passed to visualization/analytics modules.
   - The main architectural boundary is between offline Python experimentation and online C++ deployment on Jetson.

Main entry points: [`training/export_all_formats.py`](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/training/export_all_formats.py) (Model Opt), [`training/convert_nuscenes_det.py`](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/training/convert_nuscenes_det.py) (Dataset Convert), [`deployment/src/trt_engine.cpp`](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/trt_engine.cpp) (TensorRT Core), [`deployment/src/main.cpp`](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/main.cpp) (Main Runtime)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["Offline Python"]
  node_n1[("nuScenes Data<br/>dataset")]
  node_n2["Dataset Convert<br/>preprocess"]
  node_n3["YOLO Training<br/>[train_yolo26n.py]"]
  node_n4["YOLOv8 Train<br/>training<br/>[train_yolov8n.py]"]
  node_n5["Segmentation Train<br/>training"]
  node_n6["Model Opt<br/>export/quantize"]
  node_n7["Calibration<br/>int8 calib<br/>[calibration.yaml]"]
  node_n8["PointPillars Train<br/>lidar training"]
  node_n9["Fusion Research<br/>[late_fusion.py]"]
  node_n10["Build Engine<br/>build script<br/>[build_engine.sh]"]
end

subgraph group_g2["Jetson Runtime"]
  node_n11(("Main Runtime<br/>app entry<br/>[main.cpp]"))
  node_n12["Capture + GPU Prep<br/>io/preprocess<br/>[camera_capture.cpp]"]
  node_n13["TensorRT Core<br/>inference engine<br/>[trt_engine.cpp]"]
  node_n14["Decoders<br/>postprocess<br/>[yolo26_decoder.cpp]"]
  node_n15["BEV Fusion<br/>fusion runtime<br/>[late_fusion.cpp]"]
  node_n16["Analytics<br/>[object_counter.cpp]"]
  node_n17["LiDAR Decode<br/>postprocess"]
  node_n18["Camera Decode<br/>postprocess<br/>[yolov8_decoder.cpp]"]
end

subgraph group_g3["Support"]
  node_n19["Benchmarks<br/>analysis<br/>[README.md]"]
  node_n20["Solutions<br/>demos<br/>[README.md]"]
  node_n21["Docs<br/>documentation<br/>[architecture.md]"]
  node_n22["Walkthrough<br/>notebook"]
end

node_n1 -->|"annotate"| node_n2
node_n2 -->|"feed"| node_n3
node_n2 -->|"feed"| node_n5
node_n2 -->|"feed"| node_n8
node_n3 -->|"export"| node_n6
node_n4 -.->|"compare"| node_n6
node_n5 -->|"export"| node_n6
node_n6 -->|"calibrate"| node_n7
node_n6 -->|"build"| node_n10
node_n8 -->|"evaluate"| node_n9
node_n9 -.->|"specialize"| node_n10
node_n10 -->|"deploy"| node_n11
node_n11 -->|"capture"| node_n12
node_n11 -->|"infer"| node_n13
node_n13 -->|"decode"| node_n14
node_n13 -->|"decode"| node_n17
node_n13 -->|"decode"| node_n18
node_n14 -->|"align"| node_n15
node_n17 -->|"fuse"| node_n15
node_n15 -->|"analyze"| node_n16
node_n16 -.->|"demo"| node_n20
node_n19 -.->|"document"| node_n21
node_n22 -.->|"inform"| node_n21

click node_n2 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/training/convert_nuscenes_det.py"
click node_n3 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/training/train_yolo26n.py"
click node_n4 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/training/train_yolov8n.py"
click node_n5 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/training/train_yolo26n_seg.py"
click node_n6 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/training/export_all_formats.py"
click node_n7 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/calibration.yaml"
click node_n8 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/fusion/train_pointpillars.py"
click node_n9 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/fusion/late_fusion.py"
click node_n10 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/scripts/build_engine.sh"
click node_n11 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/main.cpp"
click node_n12 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/camera_capture.cpp"
click node_n13 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/trt_engine.cpp"
click node_n14 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/yolo26_decoder.cpp"
click node_n15 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/late_fusion.cpp"
click node_n16 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/object_counter.cpp"
click node_n17 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/pointpillars_decoder.cpp"
click node_n18 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/yolov8_decoder.cpp"
click node_n19 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/benchmarks/README.md"
click node_n20 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/solutions/README.md"
click node_n21 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/docs/architecture.md"
click node_n22 "https://github.com/gabrielmanalu/edgedrive-perception/blob/main/notebooks/development_walkthrough.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n1,node_n2,node_n3,node_n4,node_n5,node_n6,node_n7,node_n8,node_n9,node_n10 toneBlue
class node_n11,node_n12,node_n13,node_n14,node_n15,node_n16,node_n17,node_n18 toneAmber
class node_n19,node_n20,node_n21,node_n22 toneMint
```

## Components

### Offline Python

Dataset preparation, training, fusion experiments, export, and analysis.

- **nuScenes Data**: dataset · Core dataset source for detection, segmentation, and fusion workflows.
- [Dataset Convert](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/training/convert_nuscenes_det.py): preprocess · Converts nuScenes annotations into YOLO-friendly detection and segmentation formats. · `training/convert_nuscenes_det.py`
- [YOLO Training](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/training/train_yolo26n.py): training · Trains camera perception models, including detection and segmentation variants. · `training/train_yolo26n.py`
- [YOLOv8 Train](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/training/train_yolov8n.py): training · Alternative camera model training path used for comparison and experimentation. · `training/train_yolov8n.py`
- [Segmentation Train](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/training/train_yolo26n_seg.py): training · Trains YOLO segmentation models for instance masks and downstream deployment. · `training/train_yolo26n_seg.py`
- [Model Opt](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/training/export_all_formats.py): export/quantize · Exports, prunes, and quantizes trained models for deployment formats and calibration. · `training/export_all_formats.py`
- [Calibration](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/calibration.yaml): int8 calib · INT8 calibration data and constraints used to specialize TensorRT deployment. · `calibration.yaml`
- [PointPillars Train](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/fusion/train_pointpillars.py): lidar training · Trains the LiDAR perception branch for BEV-aligned 3D detection. · `fusion/train_pointpillars.py`
- [Fusion Research](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/fusion/late_fusion.py): fusion · Python late-fusion experiments, BEV projection, and evaluation. · `fusion/late_fusion.py`
- [Build Engine](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/scripts/build_engine.sh): build script · Builds TensorRT engines for target hardware deployment. · `scripts/build_engine.sh`

### Jetson Runtime

C++/CUDA/TensorRT deployment stack for real-time edge inference.

- [Main Runtime](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/main.cpp): app entry · Central Jetson executable that orchestrates capture, inference, decoding, and analytics. · `deployment/src/main.cpp`
- [Capture + GPU Prep](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/camera_capture.cpp): io/preprocess · Live frame ingestion and GPU-side preprocessing before inference. · `deployment/src/camera_capture.cpp`
- [TensorRT Core](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/trt_engine.cpp): inference engine · Executes exported TensorRT engines on device. · `deployment/src/trt_engine.cpp`
- [Decoders](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/yolo26_decoder.cpp): postprocess · Model-specific output decoding for boxes, masks, and BEV detections. · `deployment/src/yolo26_decoder.cpp`
- [BEV Fusion](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/late_fusion.cpp): fusion runtime · Runtime late fusion and BEV alignment of camera and LiDAR outputs. · `deployment/src/late_fusion.cpp`
- [Analytics](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/object_counter.cpp): application · Object counting, speed estimation, heatmaps, and BEV visualization. · `deployment/src/object_counter.cpp`
- [LiDAR Decode](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/pointpillars_decoder.cpp): postprocess · Transforms PointPillars network outputs into BEV detections. · `deployment/src/pointpillars_decoder.cpp`
- [Camera Decode](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/deployment/src/yolov8_decoder.cpp): postprocess · Transforms YOLO outputs into detections or masks for runtime use. · `deployment/src/yolov8_decoder.cpp`

### Support

Documentation, benchmarks, demos, and notebooks that explain or exercise the stack.

- [Benchmarks](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/benchmarks/README.md): analysis · Performance measurement, plotting, and report generation for the stack. · `benchmarks/README.md`
- [Solutions](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/solutions/README.md): demos · Edge application demos built on perception outputs and analytics primitives. · `solutions/README.md`
- [Docs](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/docs/architecture.md): documentation · Architecture notes, optimization logs, and tradeoff analysis. · `docs/architecture.md`
- [Walkthrough](https://github.com/gabrielmanalu/edgedrive-perception/blob/main/notebooks/development_walkthrough.ipynb): notebook · Development notebook capturing key design decisions and investigation history. · `notebooks/development_walkthrough.ipynb`

## Connections

- nuScenes Data → Dataset Convert: annotate · nuScenes annotations are converted into task-specific training formats.
- Dataset Convert → YOLO Training: feed · Converted detection data trains the camera detector.
- Dataset Convert → Segmentation Train: feed · Converted segmentation data trains the camera segmentation model.
- Dataset Convert → PointPillars Train: feed · Dataset preparation also supports LiDAR training workflows.
- YOLO Training → Model Opt: export · Trained models are exported and optimized for deployment.
- YOLOv8 Train → Model Opt: compare · YOLOv8 provides a comparison baseline for optimization and accuracy.
- Segmentation Train → Model Opt: export · Segmentation models are packaged into deployable artifacts.
- Model Opt → Calibration: calibrate · Quantization uses calibration data and INT8 constraints.
- Model Opt → Build Engine: build · Optimized artifacts are turned into target-device TensorRT engines.
- PointPillars Train → Fusion Research: evaluate · LiDAR detections are consumed by late-fusion experiments.
- Fusion Research → Build Engine: specialize · Fusion and model choices inform deployment engine creation.
- Build Engine → Main Runtime: deploy · Built engines are consumed by the Jetson runtime.
- Main Runtime → Capture + GPU Prep: capture · Main runtime orchestrates camera input and preprocessing.
- Main Runtime → TensorRT Core: infer · Main runtime invokes TensorRT execution.
- TensorRT Core → Decoders: decode · Raw outputs are routed to model-specific decoders.
- TensorRT Core → LiDAR Decode: decode · LiDAR network outputs are decoded into BEV detections.
- TensorRT Core → Camera Decode: decode · Camera network outputs are decoded into detections or masks.
- Decoders → BEV Fusion: align · Camera outputs are projected and matched in BEV fusion.
- LiDAR Decode → BEV Fusion: fuse · LiDAR outputs provide geometric anchors for late fusion.
- BEV Fusion → Analytics: analyze · Fused perception results feed analytics and visualization modules.
- Analytics → Solutions: demo · Analytics primitives are reused in packaged solution demos.
- Benchmarks → Docs: document · Benchmarks are summarized and interpreted in the documentation set.
- Walkthrough → Docs: inform · The walkthrough records decisions reflected in the documentation.