# megvii-basedetection/bevdepth architecture

> Architecture diagram of the GitHub repository megvii-basedetection/bevdepth, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/megvii-basedetection/bevdepth
- Repository: https://github.com/megvii-basedetection/bevdepth

## Overview

##### Project type and purpose
- This is a research‐oriented Python library for multi‐view 3D object detection in Bird’s Eye View (BEV). It provides implementations of BEVDepth, BEVStereo and MatrixVT, builds on PyTorch and MMDetection3D, and is structured around experiment configurations, model components, data handling, custom operators, and evaluation.

##### File structure
- Top-level package “bevdepth/”: contains all code.
- callbacks/: training callbacks (e.g., EMA)
- datasets/: dataset definitions (nuScenes detector dataset)
- evaluators/: evaluation metrics and utilities
- exps/: experiment entrypoints and scripts (CLI for training/eval)
- layers/: model building blocks (backbones, heads)
- models/: full model definitions combining backbones & heads
- ops/: custom C++/CUDA operators for voxel pooling (train & inference)
- utils/: distributed training helpers
- scripts/: data‐prep and visualization scripts
- setup.py & requirements*.txt: packaging and dependencies
- test/: unit tests for datasets, layers, ops

##### README insights
- Describes three main detectors: BEVDepth, BEVStereo, MatrixVT
- Lists external dependencies: PyTorch, MMDetection3D, nuScenes dataset
- Shows high‐level workflow: install → prepare data → run exps

a. Main components
1. CLI / Experiment Runner (bevdepth/exps/base_cli.py)
2. Data Layer
– Raw nuScenes dataset (scripts/gen_info.py → infos.pkl)
– Dataset class (bevdepth/datasets/nusc_det_dataset.py)
– DataLoader (PyTorch)
3. Model Layer
– Backbone modules (bevdepth/layers/backbones/*)
– View Transformer (LSS, MatrixVT, fusion modules)
– Head (bevdepth/layers/heads/bev_depth_head.py)
– Model wrapper (bevdepth/models/*.py)
4. Custom Operators
– Voxel pooling inference & training (C++/CUDA in bevdepth/ops/…)
5. Training Loop
– Optimizer & scheduler (via MMDetection3D)
– Callbacks (EMA in bevdepth/callbacks/ema.py)
– Loss computation & backprop
6. Evaluation
– Evaluators (bevdepth/evaluators/det_evaluators.py)
– Output metrics & checkpoints
7. Scripts & Utilities
– Data prep (scripts/gen_info.py)
– Visualization (scripts/visualize_nusc.py)
b. Relationships & data flow
- User invokes CLI → loads config (EXP_PATH) → initializes dataset → model → optimizer → callbacks
- DataLoader provides batches to model forward pass
- Forward: images → backbone → view transformer → BEV feature map → detection head → predictions
- Custom ops called inside backbone or view transformer for voxel pooling
- Loss → backprop → optimizer step; EMA callback updates weights
- At checkpoints or eval mode: evaluator computes metrics, writes results
c. Architectural patterns & design principles
– Modular, plugin‐based (layers, models, ops all pluggable)
– Clean separation of concerns (data vs. model vs. training vs. evaluation)
– Extensible experiments framework (one file per configuration in exps/)
– Custom C++/CUDA operators wrapped for performance
d. Key technologies & frameworks
– Python 3, PyTorch, MMDetection3D
– nuScenes dataset
– C++/CUDA for high‐performance ops
– Hydra-like CLI via base_cli.py
– pytest for unit tests

##### Final advice
- Be exhaustive: break the model into backbone, transformer, head, ops
- Both the static structure (package layout) and dynamic flow (training → evaluation)
- Keep it readable: group related components within containers representing Python packages

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Frameworks
    subgraph "External Frameworks"
        style ExternalFrameworks fill:#f9f,stroke:#333,stroke-width:1px
        PyTorch[("PyTorch")]:::external
        MMDet3D[("MMDetection3D")]:::external
        nuScenesDB[(nuScenes Dataset)]:::external
    end

    %% Scripts & CLI
    subgraph "Scripts & CLI"
        direction TB
        CLI(("CLI / Experiment Runner")):::script
        DataPrep(("Data Prep Script")):::script
        Visualize(("Visualization Script")):::script
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DatasetClass[("Dataset Class<br/>(nuScenes Detector)")]:::data
        DataLoader[("PyTorch DataLoader")]:::data
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        subgraph "Backbones"
            direction TB
            BaseLSSFPN[("Base LSS FPN Backbone")]:::model
            StereoBackbone[("BEVStereo Backbone")]:::model
            FusionBackbone[("Fusion Backbone")]:::model
            MatrixVTBackbone[("MatrixVT Backbone")]:::model
        end
        ViewTransformer[("View Transformer<br/>(LSS/MatrixVT/Fusion)")]:::model
        DetectionHead[("Detection Head")]:::model
        ModelWrapper[("Model Wrapper")]:::model
    end

    %% Custom Operators
    subgraph "Custom Operators"
        direction TB
        subgraph "Inference Ops"
            direction TB
            VoxInfPy[("voxel_pooling_inference.py")]:::ops
            VoxInfCPP[("inference_forward.cpp")]:::ops
            VoxInfCUDA[("inference_forward_cuda.cu")]:::ops
        end
        subgraph "Training Ops"
            direction TB
            VoxTrainPy[("voxel_pooling_train.py")]:::ops
            VoxTrainCPP[("train_forward.cpp")]:::ops
            VoxTrainCUDA[("train_forward_cuda.cu")]:::ops
        end
    end

    %% Training Loop & Utilities
    subgraph "Training Loop"
        direction TB
        Optimizer[("Optimizer & Scheduler")]:::util
        Loss[("Loss Computation & Backprop")]:::util
        EMA[("EMA Callback")]:::util
        DistUtils[("Distributed Training Utils")]:::util
    end

    %% Evaluation
    subgraph "Evaluation"
        direction TB
        Evaluator[("Det Evaluator")]:::util
        Checkpoint[("Checkpoint & Metrics")]:::util
    end

    %% Data Flow & Connections
    PyTorch --> CLI
    MMDet3D --> CLI
    nuScenesDB --> DataPrep
    CLI -->|"invoke"| Config{{"Experiment Configs"}}:::script
    Config --> DatasetClass
    DataPrep --> DatasetClass
    DatasetClass --> DataLoader
    DataLoader -->|"Batch of camera images"| BaseLSSFPN
    BaseLSSFPN --> ViewTransformer
    StereoBackbone --> ViewTransformer
    FusionBackbone --> ViewTransformer
    MatrixVTBackbone --> ViewTransformer
    ViewTransformer --> DetectionHead
    DetectionHead --> ModelWrapper
    ModelWrapper -->|"predictions"| VoxInfPy
    VoxInfPy --> VoxInfCPP
    VoxInfCPP --> VoxInfCUDA
    ModelWrapper -->|"training pooling"| VoxTrainPy
    VoxTrainPy --> VoxTrainCPP
    VoxTrainCPP --> VoxTrainCUDA
    ModelWrapper --> Loss
    Loss --> Optimizer
    Optimizer --> ModelWrapper
    Optimizer --> EMA
    DistUtils --> Optimizer
    ModelWrapper --> Checkpoint
    Checkpoint --> Evaluator
    Evaluator -->|"metrics"| Checkpoint

    %% Scripts connections
    CLI --> DataPrep
    CLI --> Visualize
    Visualize --> nuScenesDB

    %% Click Events
    click CLI "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/exps/base_cli.py"
    click Config "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/exps/nuscenes/MatrixVT/matrixvt_bev_depth_lss_r50_256x704_128x128_24e_ema.py"
    click DataPrep "https://github.com/megvii-basedetection/bevdepth/blob/main/scripts/gen_info.py"
    click DatasetClass "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/datasets/nusc_det_dataset.py"
    click Evaluator "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/evaluators/det_evaluators.py"
    click EMA "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/callbacks/ema.py"
    click DistUtils "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/utils/torch_dist.py"
    click BaseLSSFPN "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/layers/backbones/base_lss_fpn.py"
    click StereoBackbone "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/layers/backbones/bevstereo_lss_fpn.py"
    click FusionBackbone "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/layers/backbones/fusion_lss_fpn.py"
    click MatrixVTBackbone "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/layers/backbones/matrixvt.py"
    click DetectionHead "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/layers/heads/bev_depth_head.py"
    click ModelWrapper "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/models/base_bev_depth.py"
    click VoxInfPy "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/ops/voxel_pooling_inference/voxel_pooling_inference.py"
    click VoxInfCPP "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/ops/voxel_pooling_inference/src/voxel_pooling_inference_forward.cpp"
    click VoxInfCUDA "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/ops/voxel_pooling_inference/src/voxel_pooling_inference_forward_cuda.cu"
    click VoxTrainPy "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/ops/voxel_pooling_train/voxel_pooling_train.py"
    click VoxTrainCPP "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/ops/voxel_pooling_train/src/voxel_pooling_train_forward.cpp"
    click VoxTrainCUDA "https://github.com/megvii-basedetection/bevdepth/blob/main/bevdepth/ops/voxel_pooling_train/src/voxel_pooling_train_forward_cuda.cu"
    click Visualize "https://github.com/megvii-basedetection/bevdepth/blob/main/scripts/visualize_nusc.py"

    %% Styles
    classDef data fill:#bbdefb,stroke:#1e88e5,stroke-width:1px
    classDef model fill:#c8e6c9,stroke:#43a047,stroke-width:1px
    classDef ops fill:#ffe0b2,stroke:#fb8c00,stroke-width:1px
    classDef util fill:#e1bee7,stroke:#8e24aa,stroke-width:1px
    classDef script fill:#b3e5fc,stroke:#0288d1,stroke-width:1px,stroke-dasharray: 5 5
    classDef external fill:#eeeeee,stroke:#666666,stroke-width:1px,stroke-dasharray: 2 2
```
