How youngskkim/crn works
Based on our analysis of the file tree and README, the CRN project is a specialized multi-modal 3D perception system. It integrates camera images and radar data to generate a bird’s-eye-view (BEV) feature map for tasks like 3D object detection. The project builds on components from mmdetection3d, BEVDepth, and BEVFormer, and uses a series of pre-processing scripts for data (e.g., generating depth ground truth, radar projections, and annotations) as well as a complex deep learning architecture that spans backbones, feature transformation, multi-modal fusion, and detection heads.
And Label Main System Components
• Data Sources & Pre-processing:
– External Data: NuScenes dataset (camera images, radar blobs, LiDAR keyframes).
– Pre-processing Scripts: gen_info.py to create dataset information; gen_depth_gt.py to produce ground truth depth; gen_radar_bev.py and gen_radar_pv.py to process radar point clouds.
• Model Architecture:
– Camera Branch: A backbone network (e.g., ResNet variants) extracting image features.
– Radar Branch: Modules for processing sparse, accurate radar points.
– BEV Transformation: Components (possibly in “layers/backbones” and “models”) that transform perspective image features into a BEV space.
– Multi-modal Fusion: Fusion layers that use deformable attention (found in modules like “multimodal_deformable_cross_attention” and “multimodal_feature_aggregation”) to merge the camera and radar information.
– Detection/Prediction Head: Final heads that output 3D detections; examples include detectors defined in “exps/det/CRN_r18_256x704_128x128_4key.py” and “exps/det/CRN_r50_256x704_128x128_4key.py”.
• Training and Evaluation:
– Training Pipelines: Scripts and configuration files (under “exps”, “configs”) orchestrate training.
– Evaluation Modules: Evaluators in “evaluators/” and test scripts in “tests” for performance and inference measurements.
Data and Control Flows
– Start with data ingestion (show external source, e.g., nuScenes, with a note about symlinking or dataset download).
– Flow from pre-processed data into the training pipeline, which then feeds into two parallel extraction branches (camera and radar) that later merge in the BEV space.
Architectural Components and Design Patterns
– Mark clearly the separation of concerns between data pre-processing, feature extraction, transformation, fusion, and detection. This separation is a core architectural principle here.
– Indicate use of modular design: for instance, the “layers” for backbones, fuser, heads, and modules show MVC-like separation (feature extraction → transformation → prediction).
– Emphasize reusability: the project builds on mmdetection3d, so you may label a “Core 3D Detection Framework” module that encompasses many common operations (e.g., anchors, ROI heads, voxel encoders) reused for CRN’s purpose.
Incorporate Relevant Technologies and Dependencies
– Label key frameworks such as PyTorch, PyTorch Lightning, MMCV, and components from MMDetection3D.
– Optionally highlight the use of GPU acceleration and FP16 precision in both training and evaluation.