# mediabrain-sjtu/where2comm architecture

> Architecture diagram of the GitHub repository mediabrain-sjtu/where2comm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mediabrain-sjtu/where2comm
- Repository: https://github.com/mediabrain-sjtu/where2comm

## Overview

##### Main Components
- Data Pipeline – This includes the data utilities for loading and preprocessing. Look at the directories under opencood/data_utils (e.g., “augmentor,” “pre_processor,” “datasets,” and “post_processor”). These modules handle raw data augmentation, point cloud preprocessing (like voxel preprocessor and sp_voxel_preprocessor), and format the datasets.
- Core models such as point_pillar, point_pillar_disconet, point_pillar_v2vnet, etc.
- Fusion modules and communication modules in subdirectories (“comm_modules” and “fuse_modules”), which realize different collaborative strategies (for example, where2comm, when2com, and variants based on V2X-ViT approaches).
- Sub modules containing components like backbone networks and custom operators (e.g., base_bev_backbone, convgru, dcn_net, mswin, etc.).
- Loss Functions – Defined under opencood/loss, these include losses specific to 3D object detection like point pillar loss and point pillar disconet loss.
- Auxiliary Utilities – These include visualization tools (in opencood/visualization), pcdet utilities (for 3D detection specific functions like IoU calculations and pointnet2 operations found under opencood/pcdet_utils), and common utility functions (under opencood/utils).
- Configuration and Hyperparameters – The YAML files under opencood/hypes_yaml provide the configuration for different experimental setups and fusion strategies. These are essential for understanding system parameters and experimental setups.
- Training and Inference Modules – Found in opencood/tools, these modules (train.py, inference.py, and their utility scripts) manage the overall training, testing, and debugging workflows.

##### Relationships and Interactions
- Data flows from raw input to augmentation (data_utils/augmentor), then to preprocessing (data_utils/pre_processor) and dataset assembly (data_utils/datasets).
- The preprocessed data is fed into different deep learning models (opencood/models) where a backbone extracts features. Here, various fusion modules integrate information from multiple agents.
- After the model processes the data, post-processing steps (data_utils/post_processor) help refine the detection outputs.
- Loss functions are computed on the network outputs to guide training.
- The tools (train.py and inference.py) orchestrate the end-to-end procedure by loading configurations (from hypes_yaml) and chaining the above components.
- Visualization modules capture and display results, providing feedback for debugging and evaluation.

##### Architectural Patterns and Design Principles
- Modularity: The project is clearly divided into distinct modules for data handling, model definition, loss computation, and visualization. This separation facilitates the integration of multiple fusion strategies and easy replacement of components.
- Configurability: YAML files are used to specify different training regimes and experimental configurations (e.g., early fusion, late fusion, intermediate fusion), making the system highly configurable.
- Extensibility: By isolating the core functions (e.g., feature extraction, fusion, etc.) in separate submodules and utility libraries, the codebase allows for the easy addition of new communication strategies or backbone networks.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Pipeline Subgraph
    subgraph "Input Data Pipeline"
        A1["Augmentor"]:::datapipe
        A2["Preprocessor"]:::datapipe
        A3["Dataset Loader"]:::datapipe
        A4["Postprocessor"]:::datapipe
    end

    %% Model Architecture Subgraph
    subgraph "Model Architecture"
        M1["Point Pillar Model"]:::model
        M2["Point Pillar Disconet"]:::model
        M3["Point Pillar V2VNet"]:::model
        M4["Point Pillar V2XViT"]:::model

        M5["Where2comm Communication"]:::model
        M6["When2com Fusion"]:::model
        M7["Where2comm Attn Fusion"]:::model
        M8["V2V Fuse"]:::model

        M9["Base BEV Backbone"]:::model
        M10["ConvGRU"]:::model
        M11["DCN Net"]:::model
        M12["MSWIN Module"]:::model
    end

    %% Loss Functions Subgraph
    subgraph "Loss Computation"
        L1["Point Pillar Loss"]:::loss
        L2["Point Pillar Disconet Loss"]:::loss
    end

    %% Configuration Subgraph
    subgraph "Configuration & Hyperparameters"
        C1["YAML Config Files"]:::config
        C2["YAML Utils"]:::config
    end

    %% Training and Inference Subgraph
    subgraph "Training & Inference"
        T1["Training Script"]:::train
        T2["Inference Script"]:::train
        T3["Debug Tools"]:::train
    end

    %% Auxiliary Utilities Subgraph
    subgraph "Auxiliary Utilities"
        U1["Visualization Tools"]:::util
        U2["3D Detection Utils"]:::util
        U3["Common Utilities"]:::util
    end

    %% External Start Node
    R["Raw Data"]:::input

    %% Data Flow Connections
    R -->|"data"| A1
    A1 -->|"augments"| A2
    A2 -->|"preprocesses"| A3
    A3 -->|"feeds into"| M1
    A3 -->|"feeds into"| M2
    A3 -->|"feeds into"| M3
    A3 -->|"feeds into"| M4
    A4 -->|"refines"| M1
    A4 -->|"refines"| M2
    A4 -->|"refines"| M3
    A4 -->|"refines"| M4

    %% Model Architecture Internal Flow
    M1 -->|"features"| M9
    M2 -->|"features"| M9
    M3 -->|"features"| M9
    M4 -->|"features"| M9

    M9 -->|"passes to"| M5
    M9 -->|"passes to"| M6
    M9 -->|"passes to"| M7
    M9 -->|"passes to"| M8

    %% Loss Flow Connections
    M5 -->|"output"| L1
    M6 -->|"output"| L1
    M7 -->|"output"| L1
    M8 -->|"output"| L1
    M1 -->|"output"| L2
    M2 -->|"output"| L2

    %% Configuration feeds into Training & Inference
    C1 -->|"configures"| T1
    C2 -->|"assists"| T1

    %% Training to Inference Flow
    T1 -->|"trained model"| T2
    T1 -->|"debug info"| T3

    %% Auxiliary Utilities interactions
    M1 -->|"visualized by"| U1
    M2 -->|"visualized by"| U1
    L1 -->|"monitored by"| U2
    L2 -->|"monitored by"| U2
    T1 -->|"logs via"| U3
    T2 -->|"logs via"| U3

    %% External Dependency (PyTorch)
    P["PyTorch Library"]:::external
    M1 ---|"built on"| P
    M2 --- P
    M3 --- P
    M4 --- P

    %% Click Events for Data Pipeline
    click A1 "https://github.com/mediabrain-sjtu/where2comm/tree/main/opencood/data_utils/augmentor"
    click A2 "https://github.com/mediabrain-sjtu/where2comm/tree/main/opencood/data_utils/pre_processor"
    click A3 "https://github.com/mediabrain-sjtu/where2comm/tree/main/opencood/data_utils/datasets"
    click A4 "https://github.com/mediabrain-sjtu/where2comm/tree/main/opencood/data_utils/post_processor"

    %% Click Events for Model Architecture - Core Models
    click M1 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/models/point_pillar.py"
    click M2 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/models/point_pillar_disconet.py"
    click M3 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/models/point_pillar_v2vnet.py"
    click M4 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/models/point_pillar_v2xvit.py"

    %% Click Events for Model Architecture - Communication & Fusion Modules
    click M5 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/models/comm_modules/where2comm.py"
    click M6 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/models/fuse_modules/when2com.py"
    click M7 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/models/fuse_modules/where2comm_attn.py"
    click M8 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/models/fuse_modules/v2v_fuse.py"

    %% Click Events for Model Architecture - Sub-Modules
    click M9 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/models/sub_modules/base_bev_backbone.py"
    click M10 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/models/sub_modules/convgru.py"
    click M11 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/models/sub_modules/dcn_net.py"
    click M12 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/models/sub_modules/mswin.py"

    %% Click Events for Loss Functions
    click L1 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/loss/point_pillar_loss.py"
    click L2 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/loss/point_pillar_disconet_loss.py"

    %% Click Events for Auxiliary Utilities
    click U1 "https://github.com/mediabrain-sjtu/where2comm/tree/main/opencood/visualization"
    click U2 "https://github.com/mediabrain-sjtu/where2comm/tree/main/opencood/pcdet_utils"
    click U3 "https://github.com/mediabrain-sjtu/where2comm/tree/main/opencood/utils"

    %% Click Events for Configuration
    click C1 "https://github.com/mediabrain-sjtu/where2comm/tree/main/opencood/hypes_yaml/dair-v2x"
    click C2 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/hypes_yaml/yaml_utils.py"

    %% Click Events for Training and Inference
    click T1 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/tools/train.py"
    click T2 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/tools/inference.py"
    click T3 "https://github.com/mediabrain-sjtu/where2comm/blob/main/opencood/tools/debug_utils.py"

    %% Styles
    classDef datapipe fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef model fill:#98FB98,stroke:#000,stroke-width:2px;
    classDef loss fill:#FFCCCB,stroke:#000,stroke-width:2px;
    classDef config fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef train fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef util fill:#FFFACD,stroke:#000,stroke-width:2px;
    classDef input fill:#E6E6FA,stroke:#000,stroke-width:2px;
    classDef external fill:#F5DEB3,stroke:#000,stroke-width:2px;
```
