How nvlabs/focalformer3d works
Main Components
• Core Detector Models: There are two primary families of models present—FocalFormer3D (with various configurations, including TTA and multi-modal versions) and DeformFormer3D. Each of these is defined in separate configuration files under projects/configs/focalformer3d.
• Datasets and Pipeline: The datasets and pipelines for data preprocessing are integrated into the plugin (see projects/mmdet3d_plugin/datasets/pipelines).
• Tools for Training/Testing and Data Conversion: Tools in the tools/ directory provide scripts for analysis, training, testing, log analysis, data conversion, and launching distributed jobs. These are essential for the experimental and operational aspects of the project.
• Configuration & Experiment Setup: The config files within projects/configs/focalformer3d define how the models are to be trained, tested, and evaluated across different datasets (e.g., Waymo, nuScenes).
Relationships and Interactions
• How the input data from various sensors (LiDAR and, optionally, camera input in multi-modal variants) flows into the backbone network (defined in mmdet3d_plugin/models/backbones, e.g., the Swin backbone).
• Transition from feature extraction to the neck modules (focal_encoder, deepinteraction_encoder, etc.), which then feed into the detection heads (focal or deep interaction decoders).
• Training and inference pipelines from the tools scripts (dist_train.sh, dist_test.sh) that drive experiments based on the configuration files.
Architectural Patterns & Design Principles
• Modularity: The project is highly modular—with clear separation between configuration, model components, post-processing, and utility scripts.
• Plugin Architecture: The mmdet3d_plugin is an extension point, adding new functionalities (like specialized ops for BEV pooling and local attention) on top of the base detection framework.
• Integration with External Libraries: Note that the entire project builds on mmdetection3d and leverages other open-source projects (e.g., transfusion, DeepInteraction) to complement certain technical aspects.