# feiyang2025/jetson-cuda-overlay architecture

> Architecture diagram of the GitHub repository feiyang2025/jetson-cuda-overlay, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/feiyang2025/jetson-cuda-overlay
- Repository: https://github.com/feiyang2025/jetson-cuda-overlay

## Overview

This repository is a Jetson CUDA/V4L2 overlay for openpilot-family forks. Its primary runtime path captures GMSL camera frames, converts them to NV12, and feeds model inference; model selection supports split engines and the merged BigCombo engine, with TensorRT support and documented fallback behavior. Separate capabilities cover camera calibration, optional CH347 IMU publishing, UI overlays, and cross-fork Panda protocol/firmware maintenance. Some subsystem links are documented only in the README; unsampled implementation details are not inferred.

Main entry points: [`panda/env.py`](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/panda/env.py) (Fork discovery), [`openpilot/system/camerad/webcam/camerad.py`](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/system/camerad/webcam/camerad.py) (Camera daemon), [`kits/modeld/modeld.py`](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/kits/modeld/modeld.py) (Model dispatcher), [`openpilot/system/camerad/webcam/v4l2_dmabuf_camera.py`](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/system/camerad/webcam/v4l2_dmabuf_camera.py) (V4L2 capture)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_camera["Camera pipeline"]
  node_capture["V4L2 capture"]
  node_camerad["Camera daemon<br/>[camerad.py]"]
  node_camera_cuda["CUDA conversion<br/>[camera_cuda.py]"]
end

subgraph group_inference["Model inference"]
  node_modeld["Model dispatcher<br/>[modeld.py]"]
  node_bigcombo["BigCombo inference<br/>[modeld_bigcombo.py]"]
  node_trt["TensorRT runner<br/>[tensorrt_runner.py]"]
  node_gpu_backend["GPU backend<br/>[backend.py]"]
  node_profiles["Model profiles<br/>[registry.py]"]
  node_gpu_state["GPU model state<br/>[gpu_model_state.py]"]
end

subgraph group_calibration["Calibration tools"]
  node_calibration_tools["Camera calibration"]
  node_calibration_ui["UI overlays"]
  node_speed_ui["Speed-limit UI<br/>[speed_limit.py]"]
end

subgraph group_sensors["Sensor integration"]
  node_ch347["CH347 IMU daemon<br/>[ch347t.cc]"]
  node_imu_stack["IMU consumers"]
end

subgraph group_panda["Panda maintenance"]
  node_panda_env["Fork discovery<br/>[env.py]"]
  node_panda_repair["Protocol repair<br/>[repair_protocol.py]"]
  node_panda_sync["Firmware sync<br/>[sync_firmware.py]"]
  node_panda_verify["Protocol verification<br/>[verify_protocol.py]"]
end

node_vehicle(("Camera hardware"))
node_params[("Calibration parameters")]
node_forks[("Openpilot forks")]

node_vehicle -.->|"supplies frames"| node_capture
node_capture -->|"provides frames"| node_camerad
node_camerad -.->|"converts frames"| node_camera_cuda
node_camerad -.->|"sends vision frames"| node_modeld
node_modeld -->|"dispatches BigCombo"| node_bigcombo
node_modeld -->|"loads TensorRT"| node_trt
node_bigcombo -->|"loads engine"| node_trt
node_gpu_backend -.->|"resolves models"| node_profiles
node_gpu_state -.->|"uses backend"| node_gpu_backend
node_calibration_tools -.->|"reads and writes"| node_params
node_ch347 -->|"publishes samples"| node_imu_stack
node_panda_repair -->|"discovers forks"| node_panda_env
node_panda_sync -->|"discovers forks"| node_panda_env
node_panda_verify -->|"discovers forks"| node_panda_env
node_panda_env -->|"scans forks"| node_forks

click node_capture "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/system/camerad/webcam/v4l2_dmabuf_camera.py"
click node_camerad "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/system/camerad/webcam/camerad.py"
click node_camera_cuda "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/system/camerad/webcam/camera_cuda.py"
click node_modeld "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/kits/modeld/modeld.py"
click node_bigcombo "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/kits/modeld/modeld_bigcombo.py"
click node_trt "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/selfdrive/modeld/runners/tensorrt_runner.py"
click node_gpu_backend "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/sunnypilot/modeld_v2/gpu_backend/backend.py"
click node_profiles "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/sunnypilot/modeld_v2/gpu_backend/models/registry.py"
click node_gpu_state "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/sunnypilot/modeld_v2/gpu_model_state.py"
click node_calibration_tools "https://github.com/feiyang2025/jetson-cuda-overlay/tree/main/openpilot/tools/calib"
click node_calibration_ui "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/kits/ui/calibration_panel.py"
click node_speed_ui "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/kits/ui/speed_limit.py"
click node_ch347 "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/system/sensord/ch347t.cc"
click node_panda_env "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/panda/env.py"
click node_panda_repair "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/panda/repair_protocol.py"
click node_panda_sync "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/panda/sync_firmware.py"
click node_panda_verify "https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/panda/verify_protocol.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_capture,node_camerad,node_camera_cuda toneBlue
class node_modeld,node_bigcombo,node_trt,node_gpu_backend,node_profiles,node_gpu_state,node_params,node_forks toneAmber
class node_calibration_tools,node_calibration_ui,node_speed_ui toneMint
class node_ch347,node_imu_stack toneRose
class node_panda_env,node_panda_repair,node_panda_sync,node_panda_verify,node_vehicle toneIndigo
```

## Components

### Camera pipeline

- [V4L2 capture](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/system/camerad/webcam/v4l2_dmabuf_camera.py): `openpilot/system/camerad/webcam/v4l2_dmabuf_camera.py`
- [Camera daemon](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/system/camerad/webcam/camerad.py): `openpilot/system/camerad/webcam/camerad.py`
- [CUDA conversion](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/system/camerad/webcam/camera_cuda.py): `openpilot/system/camerad/webcam/camera_cuda.py`

### Model inference

- [Model dispatcher](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/kits/modeld/modeld.py): `kits/modeld/modeld.py`
- [BigCombo inference](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/kits/modeld/modeld_bigcombo.py): `kits/modeld/modeld_bigcombo.py`
- [TensorRT runner](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/selfdrive/modeld/runners/tensorrt_runner.py): `openpilot/selfdrive/modeld/runners/tensorrt_runner.py`
- [GPU backend](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/sunnypilot/modeld_v2/gpu_backend/backend.py): `openpilot/sunnypilot/modeld_v2/gpu_backend/backend.py`
- [Model profiles](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/sunnypilot/modeld_v2/gpu_backend/models/registry.py): `openpilot/sunnypilot/modeld_v2/gpu_backend/models/registry.py`
- [GPU model state](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/sunnypilot/modeld_v2/gpu_model_state.py): `openpilot/sunnypilot/modeld_v2/gpu_model_state.py`

### Calibration tools

- [Camera calibration](https://github.com/feiyang2025/jetson-cuda-overlay/tree/main/openpilot/tools/calib): `openpilot/tools/calib`
- [UI overlays](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/kits/ui/calibration_panel.py): `kits/ui/calibration_panel.py`
- [Speed-limit UI](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/kits/ui/speed_limit.py): `kits/ui/speed_limit.py`

### Sensor integration

- [CH347 IMU daemon](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/openpilot/system/sensord/ch347t.cc): `openpilot/system/sensord/ch347t.cc`
- **IMU consumers**

### Panda maintenance

- [Fork discovery](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/panda/env.py): `panda/env.py`
- [Protocol repair](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/panda/repair_protocol.py): `panda/repair_protocol.py`
- [Firmware sync](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/panda/sync_firmware.py): `panda/sync_firmware.py`
- [Protocol verification](https://github.com/feiyang2025/jetson-cuda-overlay/blob/main/panda/verify_protocol.py): `panda/verify_protocol.py`

### Other components

- **Camera hardware**
- **Calibration parameters**
- **Openpilot forks**

## Connections

- Camera hardware → V4L2 capture: supplies frames
- V4L2 capture → Camera daemon: provides frames (evidence: `openpilot/system/camerad/webcam/camerad.py`)
- Camera daemon → CUDA conversion: converts frames
- Camera daemon → Model dispatcher: sends vision frames
- Model dispatcher → BigCombo inference: dispatches BigCombo (evidence: `kits/modeld/modeld.py`)
- Model dispatcher → TensorRT runner: loads TensorRT (evidence: `kits/modeld/modeld.py`)
- BigCombo inference → TensorRT runner: loads engine (evidence: `kits/modeld/modeld_bigcombo.py`)
- GPU backend → Model profiles: resolves models
- GPU model state → GPU backend: uses backend
- Camera calibration → Calibration parameters: reads and writes
- CH347 IMU daemon → IMU consumers: publishes samples
- Protocol repair → Fork discovery: discovers forks (evidence: `panda/repair_protocol.py`)
- Firmware sync → Fork discovery: discovers forks (evidence: `panda/sync_firmware.py`)
- Protocol verification → Fork discovery: discovers forks (evidence: `panda/verify_protocol.py`)
- Fork discovery → Openpilot forks: scans forks (evidence: `panda/env.py`)
