How aharley/simple_bev works
Project Type and Purpose
• Type: Research codebase for multi-sensor BEV (Bird’s-Eye-View) perception.
• Purpose: Train and evaluate deep-learning models that fuse camera and radar data into a BEV representation for perception tasks.
File Structure
Top-level components:
• Scripts and entry points
– train_nuscenes.py, train_lyft.py, eval_nuscenes.py, eval_lyft.py
– shell wrappers (train.sh, eval.sh, *_runner.sh, get_*_model.sh)
• Model definitions (nets/)
– bevformernet.py, liftnet.py (core architectures)
– ops/ (custom CUDA/CPU extensions for multi-scale deformable attention)
• Data loader
– nuscenesdataset.py (reads nuScenes data, applies augmentations)
• Utilities (utils/)
– basic I/O, geometric transforms, image processing, voxelization
• Checkpointing and logging
– saverloader.py
– log directories, checkpoints directory (via command-line flags)
• Visualization
– vis_nuscenes.py, vis_collage.py
Extract Insights from the README
• Training workflow: load data → preprocess → feed to model → compute loss/backprop → save checkpoints → log metrics
• Evaluation workflow: load checkpoint → load data → model inference → compute metrics (mean IoU) → save logs
• Tensor conventions: B,S,C,Z,Y,X axes; geometry transforms (cam_T_world, etc.)
• External dependencies: PyTorch, CUDA extensions, nuScenes dataset
Main Components and Their Relationships
A. Data Sources
• nuScenes raw data (images, RADAR sweeps, annotations)
B. Data Loader & Preprocessor
• nuscenesdataset.py → uses utils/geom.py, utils/improc.py, utils/vox.py
C. Model Core
1. Backbone Encoder (ResNet-based image encoders)
2. View Transformer / LiftNet (lift image features into voxel space)
3. BEVFormerNet (fusion of multi-sensor features in BEV)
4. Segmentation Head (predict BEV segmentation map)
D. Custom Ops Library
• nets/ops modules (ms_deform_attn) implemented in C++/CUDA → compiled via setup.py
E. Training Manager
• train_nuscenes.py orchestrates data loader, model instantiation, optimizer/scheduler, loss, backprop, saverloader
F. Evaluation Manager
• eval_nuscenes.py loads model checkpoint, runs inference, computes metrics
G. Checkpointing & Logging
• saverloader.py for saving/loading model states
• Command-line flags define log_dir, ckpt_dir; logs written to filesystem
H. Visualization Tools
• vis_nuscenes.py for qualitative outputs (collages, GIFs)
Architectural Patterns & Design Principles
• Modular separation: data loading, preprocessing, model definition, training, evaluation
• Custom C++/CUDA ops for performance-critical attention modules
• Clear tensor conventions for reproducibility
• Command-line configurability for experiments
– Blue for data components (loader, dataset)
– Green for model components (encoders, BEVFormerNet)
– Orange for scripts/orchestration (train, eval)
– Gray for utilities and custom ops