How derryhub/bevformer_tensorrt works
Project Type and Purpose
• This is a model‐deployment toolchain for Bird’s-Eye-View (BEV) 3D object detection models (e.g., BEVFormer, BEVDet) on NVIDIA TensorRT.
• It provides end-to-end scripts and libraries to convert PyTorch checkpoints → ONNX → TensorRT engines (FP32/FP16/INT8), implements custom TensorRT plugins (CUDA/C++) for performance-critical ops, and orchestrates inference/benchmarking.
High-Level Components
A. Data & Config
• data/ – symbolic links to COCO, NuScenes, CAN-bus datasets
• configs/ – model hyperparameters, dataset settings, schedules
B. Model Conversion & Quantization
• det2trt/convert – Python scripts for .pth→.onnx and .onnx→TensorRT
• det2trt/quantization – calibrator classes for PTQ (QD-Q, TR-Q)
C. Core Python API & Tools
• det2trt/models – backbones, heads, detectors mirroring BEV detection architectures
• tools/ – entry-point scripts for training, evaluation, ONNX conversion, quantization, visualization
D. Custom TensorRT Plugins
• TensorRT/common – helper utilities, serialization
• TensorRT/plugin/* – CUDA kernels + plugin wrappers for bev_pool_v2, grid_sampler, inverse, modulated_deformable_conv2d, multi_head_attn, multi_scale_deformable_attn, rotate
• CMakeLists, build/, install.sh – build system to compile and install .so libs into TensorRT plugin directory
E. Third-Party Extensions
• third_party/bev_mmdet3d – forked MMDetection3D code for dataset, model definitions, ops integration
F. Checkpoints & Samples
• checkpoints/ – storage for PyTorch, ONNX, TensorRT engine files
• samples/ – per-model shell scripts orchestrating conversion, calibration, evaluation workflows
G. Environment & CI
• docker/ – Dockerfile for consistent build (CUDA/CuDNN/TensorRT/Python)
• .github/workflows – CI for formatting
• requirements.txt – Python dependencies
Component Interactions & Data Flow
• Data Preparation
raw data → data/ symlinks → samples/bevformer/create_data.sh → formatted dataset
↓
• Model Export
PyTorch checkpoint (.pth) → det2trt/convert/pytorch2onnx.py → ONNX model
↓
• Engine Generation
ONNX model → det2trt/convert/onnx2tensorrt.py → TensorRT Builder API + custom plugins (.so) → TensorRT engine (.plan)
• optional calibrator (det2trt/quantization) for INT8 → calibration table → engine
• Inference & Benchmark
TensorRT engine + plugins → tools/*_evaluate_trt.py → inference on GPU → metrics
• (Alternate) Quantization Aware Training → det2trt/quantization → updated PyTorch checkpoint
Architectural Patterns & Principles
• Modular Plugin Architecture: each custom op encapsulated in its own CUDA/C++ folder, built by CMake, installed as a standalone TensorRT plugin.
• Pipeline Segmentation: clear separation between export (PyTorch→ONNX), build (ONNX→TRT engine), and runtime (inference scripts).
• Configuration-Driven: configs/ define model/data parameters; samples/*.sh parameterize pipeline per model/precision.
• Extensibility: third_party leverages existing MMDetection3D; det2trt/models mirrors and wraps core model logic for conversion.