How mcg-nju/sparsebev works
Project type and purpose
• SparseBEV is an open-source PyTorch implementation of a 3D object detection algorithm from multi-camera video (ICCV2023).
• It comprises data preparation, model definition (backbones, heads, samplers), training/evaluation scripts, CUDA extensions, and visualization utilities.
File structure
• Top-level scripts: train.py (orchestrator for training), val.py (evaluation), timing.py (benchmark), viz_* (visualization).
• configs/: a catalog of .py files defining model architectures, data settings, training hyperparameters.
• loaders/: dataset builder and augmentation pipelines (nuscenes_dataset.py, pipelines/loading.py, transforms.py).
• models/:
• backbones/ (ViT-based EVA02, VoVNet, etc.)
• sparsebev.py, sparsebev_head.py, sparsebev_transformer.py (core model)
• bbox sub-packages (assigners, coders, match costs)
• csrc/ (custom CUDA samplers written in C++/CUDA + Python wrappers)
• utils.py, timing.py, assert files for logging, checkpoint loading, etc.
Extract key insights from README
• Underlying frameworks: PyTorch 2.0, MMCV, MMDetection3D.
• External dependencies: nuScenes dataset, pretrained backbones under pretrain/.
• Workflow steps: dataset prep → build DataLoader → instantiate model → GPU training/inference → output checkpoints/metrics → optional visualization.
Main components
a. Configuration management
– configs/ files driving every run.
– gen_sweep_info.py for data info generation.
b. Data ingestion & preprocessing (CPU)
– nuScenes raw data (images, sweeps, maps)
– loaders/builder.py → nuscenes_dataset.py
– loaders/pipelines/loading.py (file I/O)
– loaders/pipelines/transforms.py (augmentations)
c. Core model (GPU)
– models/backbones/** (feature extractors)
– models/sparsebev_transformer.py (temporal/spatial attention)
– models/sparsebev_head.py (detection head)
– models/csrc wrapper → custom CUDA kernels for sampling
d. Training / evaluation orchestration
– train.py / val.py using torchrun to launch multi-GPU
– timer & profiling via timing.py
– checkpoint.py for save/load
e. Visualization
– viz_bbox_predictions.py
– viz_sample_points.py
f. External services & storage
– Pretrained weights (pretrain/)
– nuScenes dataset (data/nuscenes/)
– Python package dependencies (PyTorch, MMCV, MMDet3D)
Relationships & interactions
• DataLoader (CPU) streams batches → GPU model forward
• Model calls into csrc CUDA kernels → feature sampling
• Loss computation → optimizer → backward → parameter update
• Checkpoints & logs written to disk
• val.py reuses same DataLoader+Model for inference → metrics output
• viz scripts load model+checkpoint → generate figures
Architectural patterns & design principles
• Modular pipelines (MMDetection style): clear separation of configs, dataset, model, training loop.
• Plugin-style extensibility: backbones, bbox coders, matchers, samplers can be swapped via configs.
• Hybrid Python/C++: heavy compute kernels offloaded to custom CUDA extensions.
• Leverage existing frameworks (MMCV, MMDet3D) for boilerplate.
Technologies & frameworks to annotate
• PyTorch + CUDA
• MMCV / MMDetection / MMDetection3D
• C++/CUDA (models/csrc)
• Python packaging (setup.py in csrc)
Layer 1 (Top): Scripts & Config
[train.py, val.py, viz_*] ← configs/*.py
Layer 2: Data layer
[nuScenes dataset] → [builder.py/nuscenes_dataset.py] → [transforms]
Layer 3: Model layer
[backbones] → [sparsebev_transformer & head] → [bbox coders/assigners]
↳ calls → [csrc CUDA kernels]
Layer 4: Compute
[GPU: forward/backward, optimizer]
Layer 5: Outputs
[checkpoints] → [evaluation metrics]
[visualizations]