# mcg-nju/sparsebev architecture

> Architecture diagram of the GitHub repository mcg-nju/sparsebev, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mcg-nju/sparsebev
- Repository: https://github.com/mcg-nju/sparsebev

## Overview

##### Project type and purpose
- SparseBEV is an open-source PyTorch implementation of a 3D object detection algorithm from multi-camera video (ICCV2023).
- It comprises data preparation, model definition (backbones, heads, samplers), training/evaluation scripts, CUDA extensions, and visualization utilities.

##### File structure
- Top-level scripts: train.py (orchestrator for training), val.py (evaluation), timing.py (benchmark), viz_* (visualization).
- configs/: a catalog of .py files defining model architectures, data settings, training hyperparameters.
- loaders/: dataset builder and augmentation pipelines (nuscenes_dataset.py, pipelines/loading.py, transforms.py).
- models/:
- backbones/ (ViT-based EVA02, VoVNet, etc.)
- sparsebev.py, sparsebev_head.py, sparsebev_transformer.py (core model)
- bbox sub-packages (assigners, coders, match costs)
- csrc/ (custom CUDA samplers written in C++/CUDA + Python wrappers)
- utils.py, timing.py, assert files for logging, checkpoint loading, etc.

##### Extract key insights from README
- Underlying frameworks: PyTorch 2.0, MMCV, MMDetection3D.
- External dependencies: nuScenes dataset, pretrained backbones under pretrain/.
- Workflow steps: dataset prep → build DataLoader → instantiate model → GPU training/inference → output checkpoints/metrics → optional visualization.

##### Main components
a. Configuration management
– configs/ files driving every run.
– gen_sweep_info.py for data info generation.
b. Data ingestion & preprocessing (CPU)
– nuScenes raw data (images, sweeps, maps)
– loaders/builder.py → nuscenes_dataset.py
– loaders/pipelines/loading.py (file I/O)
– loaders/pipelines/transforms.py (augmentations)
c. Core model (GPU)
– models/backbones/** (feature extractors)
– models/sparsebev_transformer.py (temporal/spatial attention)
– models/sparsebev_head.py (detection head)
– models/csrc wrapper → custom CUDA kernels for sampling
d. Training / evaluation orchestration
– train.py / val.py using torchrun to launch multi-GPU
– timer & profiling via timing.py
– checkpoint.py for save/load
e. Visualization
– viz_bbox_predictions.py
– viz_sample_points.py
f. External services & storage
– Pretrained weights (pretrain/)
– nuScenes dataset (data/nuscenes/)
– Python package dependencies (PyTorch, MMCV, MMDet3D)

##### Relationships & interactions
- DataLoader (CPU) streams batches → GPU model forward
- Model calls into csrc CUDA kernels → feature sampling
- Loss computation → optimizer → backward → parameter update
- Checkpoints & logs written to disk
- val.py reuses same DataLoader+Model for inference → metrics output
- viz scripts load model+checkpoint → generate figures

##### Architectural patterns & design principles
- Modular pipelines (MMDetection style): clear separation of configs, dataset, model, training loop.
- Plugin-style extensibility: backbones, bbox coders, matchers, samplers can be swapped via configs.
- Hybrid Python/C++: heavy compute kernels offloaded to custom CUDA extensions.
- Leverage existing frameworks (MMCV, MMDet3D) for boilerplate.

##### Technologies & frameworks to annotate
- PyTorch + CUDA
- MMCV / MMDetection / MMDetection3D
- C++/CUDA (models/csrc)
- Python packaging (setup.py in csrc)

Layer 1 (Top): Scripts & Config
[train.py, val.py, viz_*] ← configs/*.py
Layer 2: Data layer
[nuScenes dataset] → [builder.py/nuscenes_dataset.py] → [transforms]
Layer 3: Model layer
[backbones] → [sparsebev_transformer & head] → [bbox coders/assigners]
↳ calls → [csrc CUDA kernels]
Layer 4: Compute
[GPU: forward/backward, optimizer]
Layer 5: Outputs
[checkpoints] → [evaluation metrics]
[visualizations]

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layer 1: Scripts & Config
    subgraph "Layer 1: Scripts & Config"
        configs["configs/"]:::cpu
        gen["gen_sweep_info.py"]:::cpu
        train["train.py"]:::orch
        val["val.py"]:::orch
        timing_script["timing.py"]:::orch
        viz1["viz_bbox_predictions.py"]:::orch
        viz2["viz_sample_points.py"]:::orch
    end

    %% Layer 2: Data Ingestion & Preprocessing
    subgraph "Layer 2: Data Ingestion & Preprocessing"
        nuscenes["nuScenes dataset"]:::external
        builder["builder.py"]:::cpu
        nus_dataset["nuscenes_dataset.py"]:::cpu
        loading["loading.py"]:::cpu
        transforms["transforms.py"]:::cpu
    end

    %% Layer 3: Core Model
    subgraph "Layer 3: Core Model"
        subgraph "Backbones"
            eva02["eva02/"]:::gpu
            vovnet["vovnet.py"]:::gpu
        end
        sparsebev_main["sparsebev.py"]:::gpu
        transformer["sparsebev_transformer.py"]:::gpu
        head["sparsebev_head.py"]:::gpu
        sampling_utils["sparsebev_sampling.py"]:::gpu
        subgraph "BBox Utilities"
            assigner["hungarian_assigner_3d.py"]:::gpu
            coder["nms_free_coder.py"]:::gpu
            match_cost["match_cost.py"]:::gpu
            bbox_utils["utils.py"]:::gpu
        end
        csrc_ext["models/csrc/"]:::external
    end

    %% Layer 4: Compute
    subgraph "Layer 4: Compute"
        gpu_compute["GPU: forward/backward/optimizer"]:::gpu
    end

    %% Layer 5: Outputs
    subgraph "Layer 5: Outputs"
        checkpoints["models/checkpoint.py"]:::external
        metrics["Evaluation metrics"]:::external
        visuals_out["Visualization outputs"]:::external
    end

    %% Connections
    configs -->|drives| train
    configs -->|drives| val
    gen -->|generates sweep info| train
    train -->|builds| builder
    builder -->|uses| nus_dataset
    nus_dataset -->|loads| loading
    loading -->|applies| transforms
    transforms -->|provides batches| train
    train -->|streams data| sparsebev_main
    eva02 -->|provides features| sparsebev_main
    vovnet -->|provides features| sparsebev_main
    sparsebev_main -->|builds attention| transformer
    transformer -->|passes queries| head
    head -->|samples points| sampling_utils
    assigner -->|assigns boxes| head
    coder -->|encodes boxes| head
    match_cost -->|computes cost| head
    bbox_utils -->|utility funcs| head
    transformer -->|calls| csrc_ext
    sparsebev_main -->|executes| gpu_compute
    gpu_compute -->|saves| checkpoints
    gpu_compute -->|outputs| metrics
    val -->|reuse pipeline| builder
    val -->|streams data| sparsebev_main
    val -->|executes| gpu_compute
    viz1 -->|loads model| sparsebev_main
    viz1 -->|loads ckpt| checkpoints
    viz1 -->|generates| visuals_out
    viz2 -->|loads model| sparsebev_main
    viz2 -->|loads ckpt| checkpoints
    viz2 -->|generates| visuals_out

    %% Click Events
    click configs "https://github.com/mcg-nju/sparsebev/tree/main/configs/"
    click gen "https://github.com/mcg-nju/sparsebev/blob/main/gen_sweep_info.py"
    click train "https://github.com/mcg-nju/sparsebev/blob/main/train.py"
    click val "https://github.com/mcg-nju/sparsebev/blob/main/val.py"
    click timing_script "https://github.com/mcg-nju/sparsebev/blob/main/timing.py"
    click viz1 "https://github.com/mcg-nju/sparsebev/blob/main/viz_bbox_predictions.py"
    click viz2 "https://github.com/mcg-nju/sparsebev/blob/main/viz_sample_points.py"
    click builder "https://github.com/mcg-nju/sparsebev/blob/main/loaders/builder.py"
    click nus_dataset "https://github.com/mcg-nju/sparsebev/blob/main/loaders/nuscenes_dataset.py"
    click loading "https://github.com/mcg-nju/sparsebev/blob/main/loaders/pipelines/loading.py"
    click transforms "https://github.com/mcg-nju/sparsebev/blob/main/loaders/pipelines/transforms.py"
    click eva02 "https://github.com/mcg-nju/sparsebev/tree/main/models/backbones/eva02/"
    click vovnet "https://github.com/mcg-nju/sparsebev/blob/main/models/backbones/vovnet.py"
    click sparsebev_main "https://github.com/mcg-nju/sparsebev/blob/main/models/sparsebev.py"
    click transformer "https://github.com/mcg-nju/sparsebev/blob/main/models/sparsebev_transformer.py"
    click head "https://github.com/mcg-nju/sparsebev/blob/main/models/sparsebev_head.py"
    click sampling_utils "https://github.com/mcg-nju/sparsebev/blob/main/models/sparsebev_sampling.py"
    click assigner "https://github.com/mcg-nju/sparsebev/blob/main/models/bbox/assigners/hungarian_assigner_3d.py"
    click coder "https://github.com/mcg-nju/sparsebev/blob/main/models/bbox/coders/nms_free_coder.py"
    click match_cost "https://github.com/mcg-nju/sparsebev/blob/main/models/bbox/match_costs/match_cost.py"
    click bbox_utils "https://github.com/mcg-nju/sparsebev/blob/main/models/bbox/utils.py"
    click csrc_ext "https://github.com/mcg-nju/sparsebev/tree/main/models/csrc/"

    %% Styles
    classDef cpu fill:#8ecae6,stroke:#000,stroke-width:1px
    classDef gpu fill:#ffa500,stroke:#000,stroke-width:1px
    classDef orch fill:#80ed99,stroke:#000,stroke-width:1px
    classDef external fill:#adb5bd,stroke:#000,stroke-width:1px
```
