How megvii-research/motrv2 works
Project type and purpose
• This is a research‐grade deep‐learning pipeline for end-to-end multi-object tracking (MOTRv2), built on top of Deformable DETR and MOTR, with a bootstrap from a pretrained detector (YOLOX). Its primary purpose is to train, evaluate, and run inference for multi-object tracking on datasets like DanceTrack and BDD100K.
File structure
• Top‐level directories:
• configs/ – argument files for different experiments
• datasets/ – data loaders, transforms, samplers, evaluation modules
• models/ – PyTorch model definitions (backbone, transformer, matcher, MOTR core, and custom CUDA/C++ ops under ops/)
• tools/ – shell and Python scripts to launch training, inference, evaluation, distributed runs, visualization
• util/ – miscellaneous helpers (checkpointing, plotting, metrics)
• Key scripts:
• main.py – entry point for training/inference
• engine.py – training/inference loop orchestrator
• submit_dance.py – job submission for a specific challenge
• Build artifacts:
• models/ops/ – custom multi‐scale deformable attention layers implemented in CPU/CUDA, compiled via make.sh
Extract insights from the README
• The pipeline:
1. Load detection proposals from YOLOX as anchors
2. Pass image and anchors into Deformable Transformer (MOTR)
3. Jointly predict detection and track queries
4. Output tracking results
• Dependencies: PyTorch, torchvision, CUDA toolkit, custom ops
a. Main components:
1. Data Layer
– Dataset Loader (DanceTrack, CrowdHuman, BDD100K)
– Transforms & Sampler modules
2. Detection Prior Module
– Pretrained YOLOX detector (external service or precomputed proposals)
3. Core Model
– Backbone (ResNet or Deformable DETR’s CNN)
– Deformable Transformer (+ multi‐scale attention ops)
– MOTR Tracking Head (query generation, matching, track management)
4. Custom Ops
– CPU/CUDA ms_deform_attn kernels (built via models/ops)
5. Training & Inference Engine
– engine.py (loss computation, optimizer step, inference loop)
– main.py (argument parsing, setup, calls engine)
6. Utilities & Tools
– checkpoint saving/loading (util/checkpoint.py)
– evaluation metrics (util/evaluation.py, datasets/panoptic_eval.py, util/motdet_eval.py)
– orchestration scripts (tools/train.sh, tools/simple_inference.sh, tools/run_dist_launch.sh)
b. Relationships & interactions:
– Data Layer feeds images to the Detection Prior Module and Core Model
– Detection Prior Module outputs proposals consumed as anchor queries by the Core Model
– Core Model invokes Custom Ops for efficient attention computations
– Engine orchestrates forward/backward passes, interacts with Utilities for logging, checkpointing, and evaluation
– Tools wrap Engine invocations for single‐GPU, multi‐GPU, SLURM, or challenge submissions
c. Architectural patterns & principles:
– Modular separation: data / model / ops / engine / tools / utils
– Pipeline stages: proposal generation → feature extraction → transformer decoding → prediction → loss/eval
– Extension points: swap in different backbones, detection priors, or transformer variants
d. Key technologies/frameworks:
– PyTorch for model & training
– Custom C++/CUDA kernels for MultiScaleDeformableAttention
– Shell scripts for job management and reproducibility