# hanchaa/meformer architecture

> Architecture diagram of the GitHub repository hanchaa/meformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hanchaa/meformer
- Repository: https://github.com/hanchaa/meformer

## Overview

##### Overall Project Overview
- Project Type: A research and development project for robust multimodal 3D object detection.
- Purpose: To detect 3D objects by integrating multiple modalities (e.g., LiDAR, camera inputs) using advanced custom modules and pipelines.
- Key technologies: Python 3.8/CUDA 11.1/PyTorch 1.10, mmcv, and custom implementations under “mmdet3d_plugin.”

##### Main Components
A. Data Preparation & Conversion:
- Flow: Represent the raw input data and how it is transformed and augmented via the pipelines (found in projects/mmdet3d_plugin/datasets/pipelines).

B. Configuration Files:
- “projects/configs”: Contains different experimental configurations (e.g., meformer_voxel0075_vov_1600x640_cbgs.py and variants). These files define hyperparameters, model architecture choices, training and evaluation settings.
- Flow: The configuration files feed into the training and model initialization steps.

C. Custom 3D Detection Plugin (projects/mmdet3d_plugin):
- Core Modules:
- "core" (e.g., bbox assigners, coders, match costs) – these modules handle object detection algorithms at a low level.
- “datasets” – custom dataset wrappers and pipelines (e.g., custom_nuscenes_dataset) that preprocess and augment input data.
- “mmcv_custom” – contains custom operations (especially voxel operations) and runner hooks (e.g., freeze_weight, optimizer hooks) that integrate with mmcv’s training loop.
- Model Components:
- "models": Contains components for backbones (e.g., VovNet), dense heads, detectors (the “meformer” detector), necks (e.g., cp_fpn) and utilities (e.g., attention modules, grid mask, various transformer implementations).

D. Training and Inference Procedures:
- Tools: In the “tools” directory, there are scripts for training (train.py, dist_train.sh) and inference/testing (dist_test.sh, test.py, test_speed.py).
- Flow: Illustrate how the system’s entry point (triggered via the command line using these scripts) utilizes configurations to call the custom modules, start model training, and then proceed to inference.

E. External Dependencies & Environment:

##### Relationships and Interactions
- Flow of raw data → processed via data pipelines → fed into model input.
- Model initialization uses configuration details from “projects/configs.”
- Training and inference flows that use “tools” to invoke various components from “mmdet3d_plugin.”
- Interconnections between custom operations (mmcv_custom) and the core model components.
- That the outputs (predictions, evaluation results) are routed back to evaluation modules or further visualization (refer to tools/visual_utils).

##### Design Principles and Architectural Patterns
- Modularity: The project is organized into clearly separated modules (configs, core modules, datasets, models, and tools).
- Extensibility: The use of plugin-like architecture in “mmdet3d_plugin” indicates the system’s design is extensible, where additional modules can be plugged in.
- Separation of Concerns: Distinct responsibilities for data preparation, model training, and inference.
- Customization: Custom mmcv operations and hook integration show a layered approach blended with framework extensions.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "External Dependencies & Environment"
       E1["requirements.txt"]:::external
       E2["README.md"]:::external
    end

    subgraph "Data Preparation & Conversion"
       D1["create_data.py"]:::dataPre
       D2["create_data.sh"]:::dataPre
       D3["nuscenes_converter.py"]:::dataPre
       D4["create_gt_database.py"]:::dataPre
       D5["open3d_vis_utils.py"]:::dataPre
       D6["visualize_utils.py"]:::dataPre
    end

    subgraph "Configuration Management"
       C1["Configuration Files"]:::config
    end

    subgraph "Custom 3D Detection Plugin"
       subgraph "Core Modules"
           P1["hungarian_assigner_3d.py"]:::plugin
           P2["multi_task_bbox_coder.py"]:::plugin
           P3["match_cost.py"]:::plugin
           P4["bbox/util.py"]:::plugin
       end
       subgraph "Datasets & Pipelines"
           P5["custom_nuscenes_dataset.py"]:::plugin
           P6["dbsampler.py"]:::plugin
           P7["transform_3d.py"]:::plugin
       end
       subgraph "Custom Ops (mmcv_custom)"
           P8["spconv_voxelize.py"]:::plugin
           P9["freeze_weight.py"]:::plugin
           P10["optimizer.py"]:::plugin
           P11["drop_gt_sampling.py"]:::plugin
       end
       subgraph "Model Components"
           subgraph "Backbones"
               M1["vovnet.py"]:::plugin
           end
           subgraph "Dense Heads"
               M2["meformer_head.py"]:::plugin
               M3["separate_task_head.py"]:::plugin
           end
           subgraph "Detectors"
               M4["meformer.py"]:::plugin
           end
           subgraph "Necks"
               M5["cp_fpn.py"]:::plugin
           end
           subgraph "Utility Modules"
               M6["attention.py"]:::plugin
               M7["grid_mask.py"]:::plugin
               M8["moad_transformer.py"]:::plugin
               M9["petr_transformer.py"]:::plugin
               M10["pme_transformer.py"]:::plugin
           end
       end
    end

    subgraph "Training & Inference Procedures"
       T1["train.py"]:::training
       T2["dist_train.sh"]:::training
       T3["test.py"]:::training
       T4["dist_test.sh"]:::training
       T5["test_speed.py"]:::training
    end

    %% Connections
    E1 -->|"dependency"| C1
    D3 -->|"pipeline"| P5
    D4 -->|"pipeline"| P5
    C1 -->|"configures"| M4
    T1 -->|"triggersTraining"| M2
    T1 -->|"triggersTraining"| P9
    T3 -->|"runsInference"| M4

    %% Styles
    classDef dataPre fill:#FFDDCC,stroke:#333,stroke-width:2px;
    classDef config fill:#DDEEFF,stroke:#333,stroke-width:2px;
    classDef plugin fill:#CCFFCC,stroke:#333,stroke-width:2px;
    classDef training fill:#FFE6E6,stroke:#333,stroke-width:2px;
    classDef external fill:#FFFFCC,stroke:#333,stroke-width:2px;

    %% Click Events for Data Preparation & Conversion
    click D1 "https://github.com/hanchaa/meformer/blob/main/tools/create_data.py"
    click D2 "https://github.com/hanchaa/meformer/blob/main/tools/create_data.sh"
    click D3 "https://github.com/hanchaa/meformer/blob/main/tools/data_converter/nuscenes_converter.py"
    click D4 "https://github.com/hanchaa/meformer/blob/main/tools/data_converter/create_gt_database.py"
    click D5 "https://github.com/hanchaa/meformer/blob/main/tools/visual_utils/open3d_vis_utils.py"
    click D6 "https://github.com/hanchaa/meformer/blob/main/tools/visual_utils/visualize_utils.py"

    %% Click Events for Configuration Files
    click C1 "https://github.com/hanchaa/meformer/tree/main/projects/configs"

    %% Click Events for Custom 3D Detection Plugin - Core Modules
    click P1 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/core/bbox/assigners/hungarian_assigner_3d.py"
    click P2 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/core/bbox/coders/multi_task_bbox_coder.py"
    click P3 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/core/bbox/match_costs/match_cost.py"
    click P4 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/core/bbox/util.py"

    %% Click Events for Custom 3D Detection Plugin - Datasets & Pipelines
    click P5 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/datasets/custom_nuscenes_dataset.py"
    click P6 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/datasets/pipelines/dbsampler.py"
    click P7 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/datasets/pipelines/transform_3d.py"

    %% Click Events for Custom 3D Detection Plugin - Custom Ops (mmcv_custom)
    click P8 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/mmcv_custom/ops/voxel/spconv_voxelize.py"
    click P9 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/mmcv_custom/runner/hooks/freeze_weight.py"
    click P10 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/mmcv_custom/runner/hooks/optimizer.py"
    click P11 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/mmcv_custom/runner/hooks/drop_gt_sampling.py"

    %% Click Events for Custom 3D Detection Plugin - Model Components - Backbones
    click M1 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/models/backbones/vovnet.py"
    %% Dense Heads
    click M2 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/models/dense_heads/meformer_head.py"
    click M3 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/models/dense_heads/separate_task_head.py"
    %% Detectors
    click M4 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/models/detectors/meformer.py"
    %% Necks
    click M5 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/models/necks/cp_fpn.py"
    %% Utility Modules
    click M6 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/models/utils/attention.py"
    click M7 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/models/utils/grid_mask.py"
    click M8 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/models/utils/moad_transformer.py"
    click M9 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/models/utils/petr_transformer.py"
    click M10 "https://github.com/hanchaa/meformer/blob/main/projects/mmdet3d_plugin/models/utils/pme_transformer.py"

    %% Click Events for Training & Inference Procedures
    click T1 "https://github.com/hanchaa/meformer/blob/main/tools/train.py"
    click T2 "https://github.com/hanchaa/meformer/blob/main/tools/dist_train.sh"
    click T3 "https://github.com/hanchaa/meformer/blob/main/tools/test.py"
    click T4 "https://github.com/hanchaa/meformer/blob/main/tools/dist_test.sh"
    click T5 "https://github.com/hanchaa/meformer/blob/main/tools/test_speed.py"

    %% Click Events for External Dependencies
    click E1 "https://github.com/hanchaa/meformer/blob/main/requirements.txt"
    click E2 "https://github.com/hanchaa/meformer/blob/main/README.md"
```
