How megvii-research/far3d works
Project Type and Purpose
• Type: Research framework for 3D object detection on surrounding views (especially long-range objects) using a novel sparse query-based approach.
• Purpose: Integrate high-quality 2D object priors and construct adaptive 3D queries to efficiently learn multi-view, multi-scale features. The design relies on modules from MMDetection3D and extends them with additional pipelines (e.g., perspective-aware aggregation and range-modulated 3D denoising).
Analyzing the File Structure
• The core of the project is under “projects/mmdet3d_plugin” which includes:
• core (apis, bbox handling, evaluation procedures)
• datasets (data loaders, augmentation pipelines, samplers for multi-view 3D data)
• models (organized into subdirectories for backbones, dense heads, detectors, depth predictor, necks, and utilities)
• Additional directories (such as “tools” and “docs”) provide scripts for training, testing, analysis, and visualization.
• The configuration file (e.g., projects/configs/far3d.py) ties this all together and is the entry point for running experiments.
Examining the README
• The README explains that Far3D implements a novel sparse query-based design, leveraging high-quality 2D object priors to guide 3D adaptive queries.
• It highlights key modules (such as the perspective-aware aggregation module and 3D denoising) and notes that the pipeline is based on StreamPETR.
• It includes instructions for training and testing along with citations to related works and comparisons.
a. Main Components to Include:
• Configuration & Setup: projects/configs/far3d.py (defines experiment settings, training parameters, network architecture choices)
• Data Pipeline:
• Datasets: Modules under projects/mmdet3d_plugin/datasets that load and preprocess data (e.g., Argoverse2, nuScenes variants)
• Pipelines: Data augmentation, transformation steps (custom_pipeline, formatting, transform_3d)
• Samplers: Managing distributed sampling and grouping.
• Model Architecture:
• Backbones: e.g., VoVnet variants found in models/backbones for feature extraction.
• Necks: Feature pyramid networks (FPN) as seen in projects/mmdet3d_plugin/models/necks.
• Detectors & Dense Heads: Modules in models/detectors and models/dense_heads that handle object detection; here, “far3d.py” is visible along with other heads.
• Depth Predictor: Including modules for depth estimation (helpful for 3D localization).
• Utility Modules: Attention, positional encoding, transformer utilities within models/utils.
• Query Modules: Elements implemented for sparse, adaptive 3D queries (implied by the training APIs and detection heads).
• Training and Evaluation:
• API Scripts: Located in core/apis (train.py, test.py, mmdet_train.py) to manage model training, inference, and evaluations.
• Tools: A suite of scripts under “tools” for distributed training/testing, visualizations, and checkpoint handling.
• Visualization & Debugging:
• Tools under “tools/visual” which provide notebooks and scripts for 2D/3D visualizations (e.g., vis_2d, vis_3dpred).
b. Relationships and Interactions:
• Data flows from the dataset loaders (and pipelines) into the training APIs.
• The configuration file (far3d.py) centrally connects settings to both the model architecture and data pipeline.
• The model components (backbones, necks, detectors, etc.) interact sequentially – raw data ➜ feature extraction ➜ feature aggregation and transformation ➜ prediction (via dense heads/detectors).
• External scripts (tools) manage orchestration (e.g., distributed training, evaluation, visualization) around these core components.
• Utility modules inform various parts of the model with additional algorithms (for attention, transformer designs, etc.).