How opendrivelab/occnet works
Main Components
• Data Sources and Input: Represent the multiple input modalities (e.g., cameras from the nuScenes dataset), and show how raw images and sensor meta information are ingested.
• Dataset and Configuration Management: Include components for handling the occupancy and flow dataset as well as the configuration folders (projects/configs and its subfolders holding dataset, model, and schedule configurations).
• Backbone networks (see projects/mmdet3d_plugin/models/backbones)
• Detection and occupancy heads (from projects/mmdet3d_plugin/models and the bevformer-specific detectors and dense_heads in the mmdet3d_plugin/bevformer folder)
• Transformer layers and attention modules (in projects/mmdet3d_plugin/bevformer/modules)
• Training and Inference Pipelines:
• The training scripts (tools/train.py, tools/dist_train.sh) and runner modules (projects/mmdet3d_plugin/bevformer/runner) should be shown.
• The inference pipeline and evaluation (tools/dist_test.sh, projects/mmdet3d_plugin/datasets for data pipelines).
• Evaluation and Metrics:
• A separate evaluation module for computing ray-based mIoU and occupancy flow metrics (projects/mmdet3d_plugin/datasets/ray_metrics.py, tools/ray_iou).
• Submission Conversion:
• Process of converting output predictions into the standardized submission format, detailing the scripts in tools/ray_iou and their integration with the rest of the pipeline.
• Utilities:
• Mention auxiliary scripts (like utils/vis.py for visualization, docs for getting started) and external dependencies (such as BEVFormer) that support the system.
Relationships and Interactions
• Data flow starting from raw input (camera images and nuScenes meta files) through preprocessing pipelines (projects/mmdet3d_plugin/datasets/pipelines) to the network’s forward pass.
• Bidirectional relationship between training/inference scripts (tools/ folder) and the model/plugins, highlighting how training scripts call the runner modules which in turn invoke model components.
• Evaluation module’s connection to both prediction outputs and ground-truth data, indicating the use of ray projection (tools/ray_iou/ray_casting.py) and metric aggregation.
• Submission module that packages final predictions into the required submission structure (as detailed in the README).
Architectural Patterns and Design Principles
• Modular Design: Emphasize that the project is split into modules by functionality (e.g., configurations, models, datasets, utilities) for flexibility and extensibility.
• Plugin Architecture: Show that custom plugins extend MMDetection3D’s capabilities (located in projects/mmdet3d_plugin) allowing new model heads, attention mechanisms, and evaluation routines.
• Separation of Concerns: Clearly separate training, inference, evaluation, and submission preparation into distinct pipelines.
• Config-driven Design: Stress how the base and specialized configuration files are used to parameterize and control the model’s behavior without hardcoding details.
• Clear labels for each component (e.g., “Image Input”, “Configuration Manager”, “Data Pipeline”, “Backbone Network”, “Transformer Encoder/Decoder”, “Detection Heads”, “Evaluation Module”, “Submission Formatter”).
• Processing modules (rectangles)