How mediabrain-sjtu/where2comm works
Main Components
• Data Pipeline – This includes the data utilities for loading and preprocessing. Look at the directories under opencood/data_utils (e.g., “augmentor,” “pre_processor,” “datasets,” and “post_processor”). These modules handle raw data augmentation, point cloud preprocessing (like voxel preprocessor and sp_voxel_preprocessor), and format the datasets.
• Core models such as point_pillar, point_pillar_disconet, point_pillar_v2vnet, etc.
• Fusion modules and communication modules in subdirectories (“comm_modules” and “fuse_modules”), which realize different collaborative strategies (for example, where2comm, when2com, and variants based on V2X-ViT approaches).
• Sub modules containing components like backbone networks and custom operators (e.g., base_bev_backbone, convgru, dcn_net, mswin, etc.).
• Loss Functions – Defined under opencood/loss, these include losses specific to 3D object detection like point pillar loss and point pillar disconet loss.
• Auxiliary Utilities – These include visualization tools (in opencood/visualization), pcdet utilities (for 3D detection specific functions like IoU calculations and pointnet2 operations found under opencood/pcdet_utils), and common utility functions (under opencood/utils).
• Configuration and Hyperparameters – The YAML files under opencood/hypes_yaml provide the configuration for different experimental setups and fusion strategies. These are essential for understanding system parameters and experimental setups.
• Training and Inference Modules – Found in opencood/tools, these modules (train.py, inference.py, and their utility scripts) manage the overall training, testing, and debugging workflows.
Relationships and Interactions
• Data flows from raw input to augmentation (data_utils/augmentor), then to preprocessing (data_utils/pre_processor) and dataset assembly (data_utils/datasets).
• The preprocessed data is fed into different deep learning models (opencood/models) where a backbone extracts features. Here, various fusion modules integrate information from multiple agents.
• After the model processes the data, post-processing steps (data_utils/post_processor) help refine the detection outputs.
• Loss functions are computed on the network outputs to guide training.
• The tools (train.py and inference.py) orchestrate the end-to-end procedure by loading configurations (from hypes_yaml) and chaining the above components.
• Visualization modules capture and display results, providing feedback for debugging and evaluation.
Architectural Patterns and Design Principles
• Modularity: The project is clearly divided into distinct modules for data handling, model definition, loss computation, and visualization. This separation facilitates the integration of multiple fusion strategies and easy replacement of components.
• Configurability: YAML files are used to specify different training regimes and experimental configurations (e.g., early fusion, late fusion, intermediate fusion), making the system highly configurable.
• Extensibility: By isolating the core functions (e.g., feature extraction, fusion, etc.) in separate submodules and utility libraries, the codebase allows for the easy addition of new communication strategies or backbone networks.