# megvii-research/motrv2 architecture

> Architecture diagram of the GitHub repository megvii-research/motrv2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/megvii-research/motrv2
- Repository: https://github.com/megvii-research/motrv2

## Overview

##### Project type and purpose
- This is a research‐grade deep‐learning pipeline for end-to-end multi-object tracking (MOTRv2), built on top of Deformable DETR and MOTR, with a bootstrap from a pretrained detector (YOLOX). Its primary purpose is to train, evaluate, and run inference for multi-object tracking on datasets like DanceTrack and BDD100K.

##### File structure
- Top‐level directories:
- configs/ – argument files for different experiments
- datasets/ – data loaders, transforms, samplers, evaluation modules
- models/ – PyTorch model definitions (backbone, transformer, matcher, MOTR core, and custom CUDA/C++ ops under ops/)
- tools/ – shell and Python scripts to launch training, inference, evaluation, distributed runs, visualization
- util/ – miscellaneous helpers (checkpointing, plotting, metrics)
- Key scripts:
- main.py – entry point for training/inference
- engine.py – training/inference loop orchestrator
- submit_dance.py – job submission for a specific challenge
- Build artifacts:
- models/ops/ – custom multi‐scale deformable attention layers implemented in CPU/CUDA, compiled via make.sh

##### Extract insights from the README
- The pipeline:
1. Load detection proposals from YOLOX as anchors
2. Pass image and anchors into Deformable Transformer (MOTR)
3. Jointly predict detection and track queries
4. Output tracking results
- Dependencies: PyTorch, torchvision, CUDA toolkit, custom ops

a. Main components:
1. Data Layer
– Dataset Loader (DanceTrack, CrowdHuman, BDD100K)
– Transforms & Sampler modules
2. Detection Prior Module
– Pretrained YOLOX detector (external service or precomputed proposals)
3. Core Model
– Backbone (ResNet or Deformable DETR’s CNN)
– Deformable Transformer (+ multi‐scale attention ops)
– MOTR Tracking Head (query generation, matching, track management)
4. Custom Ops
– CPU/CUDA ms_deform_attn kernels (built via models/ops)
5. Training & Inference Engine
– engine.py (loss computation, optimizer step, inference loop)
– main.py (argument parsing, setup, calls engine)
6. Utilities & Tools
– checkpoint saving/loading (util/checkpoint.py)
– evaluation metrics (util/evaluation.py, datasets/panoptic_eval.py, util/motdet_eval.py)
– orchestration scripts (tools/train.sh, tools/simple_inference.sh, tools/run_dist_launch.sh)
b. Relationships & interactions:
– Data Layer feeds images to the Detection Prior Module and Core Model
– Detection Prior Module outputs proposals consumed as anchor queries by the Core Model
– Core Model invokes Custom Ops for efficient attention computations
– Engine orchestrates forward/backward passes, interacts with Utilities for logging, checkpointing, and evaluation
– Tools wrap Engine invocations for single‐GPU, multi‐GPU, SLURM, or challenge submissions
c. Architectural patterns & principles:
– Modular separation: data / model / ops / engine / tools / utils
– Pipeline stages: proposal generation → feature extraction → transformer decoding → prediction → loss/eval
– Extension points: swap in different backbones, detection priors, or transformer variants
d. Key technologies/frameworks:
– PyTorch for model & training
– Custom C++/CUDA kernels for MultiScaleDeformableAttention
– Shell scripts for job management and reproducibility

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Layer
    subgraph "Data Layer"
        DL1["DanceTrack Loader"]:::data
        DL2["Joint Loader"]:::data
        T1["Transforms"]:::data
        S1["Samplers"]:::data
        DP["Data Prefetcher"]:::data
        PE["Panoptic Evaluation"]:::data
    end

    %% Detection Prior Module
    subgraph "Detection Prior Module"
        MAKE_DETDB["DetDB Creator"]:::external
    end

    %% Core Model
    subgraph "Core Model"
        B["Backbone"]:::model
        DD["Deformable DETR"]:::model
        DT["Deformable Transformer+"]:::model
        QIM["Query Interaction Module"]:::model
        MOTR["MOTR Tracking Head"]:::model
        M["Matcher"]:::model
        PE_mod["Position Encoding"]:::model
        BOXES["Boxes Struct"]:::model
        INST["Instances Struct"]:::model
    end

    %% Custom Ops
    subgraph "Custom Ops"
        F["ms_deform_attn Func"]:::model
        M_OP["ms_deform_attn Module"]:::model
        MAKE_SH["make.sh"]:::model
        SETUP["setup.py"]:::model
        CPUCPP["ms_deform_attn_cpu.cpp"]:::model
        CPUH["ms_deform_attn_cpu.h"]:::model
        CUDACU["ms_deform_attn_cuda.cu"]:::model
        CUDAH["ms_deform_attn_cuda.cuh"]:::model
        IM2CUH["ms_deform_im2col_cuda.cuh"]:::model
        ATTNH["ms_deform_attn.h"]:::model
        VCPP["vision.cpp"]:::model
    end

    %% Training & Inference Engine
    subgraph "Engine"
        MAIN["main.py"]:::external
        ENGINE["engine.py"]:::external
        CONFIG["configs/motrv2.args"]:::external
    end

    %% Utilities & Tools
    subgraph "Utilities & Tools"
        CP["checkpoint.py"]:::util
        EVAL["evaluation.py"]:::util
        MOTE["motdet_eval.py"]:::util
        BOXOPS["box_ops.py"]:::util
        PLOT["plot_utils.py"]:::util
        VIS["visualize.py"]:::util
        MISC["misc.py"]:::util
        TOOL["tool.py"]:::util
        RUN_DIST["run_dist_launch.sh"]:::util
        SLURM["run_dist_slurm.sh"]:::util
        LAUNCH["launch.py"]:::util
        TRAIN_SH["train.sh"]:::util
        SIMPLE_INF["simple_inference.sh"]:::util
        EVAL_DANCE["eval_dance.sh"]:::util
        SUB_DANCE["submit_dance.py"]:::util
        DEBUG["debug.sh"]:::util
        RESUME["resume.sh"]:::util
        COPY["copy_back.sh"]:::util
        BATCH_DIFF["batch_diff.py"]:::util
        MERGE_SH["merge_dance_tracklets.sh"]:::util
        MERGE_PY["merge_dance_tracklets.py"]:::util
    end

    %% Data flow
    DL1 -->|loads images| B
    DL2 -->|loads images| B
    T1 -->|processes| B
    S1 -->|samples| B
    DP -->|prefetches| B
    DL1 --> MAKE_DETDB
    DL2 --> MAKE_DETDB
    PE -->|evaluation data| EVAL

    %% Detection to Core
    MAKE_DETDB -->|proposals| QIM

    %% Core internal
    B --> DD
    DD --> DT
    DT --> QIM
    QIM --> MOTR
    M --> MOTR
    PE_mod --> MOTR
    BOXES --> MOTR
    INST --> MOTR

    %% Custom Ops integration
    F --> DT
    M_OP --> DT
    CPUCPP --> F
    CPUH --> F
    CUDACU --> F
    CUDAH --> F
    IM2CUH --> F
    ATTNH --> F
    VCPP --> F
    MAKE_SH --> F
    SETUP --> F

    %% Engine orchestration
    MAIN --> ENGINE
    ENGINE --> B
    ENGINE --> DD
    ENGINE --> DT
    ENGINE --> QIM
    ENGINE --> MOTR
    ENGINE --> CP
    ENGINE --> EVAL
    ENGINE --> MOTE
    ENGINE --> BOXOPS
    ENGINE --> PLOT

    %% Tools invoke engine
    TRAIN_SH --> MAIN
    SIMPLE_INF --> MAIN
    EVAL_DANCE --> MAIN
    RUN_DIST --> MAIN
    SLURM --> MAIN
    LAUNCH --> MAIN
    SUB_DANCE --> MAIN
    DEBUG --> MAIN
    RESUME --> MAIN
    COPY --> MAIN
    BATCH_DIFF --> MAIN
    MERGE_SH --> MAIN
    MERGE_PY --> MAIN
    VIS --> MAIN

    %% Click Events
    click DL1 "https://github.com/megvii-research/motrv2/blob/main/datasets/dance.py"
    click DL2 "https://github.com/megvii-research/motrv2/blob/main/datasets/joint.py"
    click T1 "https://github.com/megvii-research/motrv2/blob/main/datasets/transforms.py"
    click S1 "https://github.com/megvii-research/motrv2/blob/main/datasets/samplers.py"
    click DP "https://github.com/megvii-research/motrv2/blob/main/datasets/data_prefetcher.py"
    click PE "https://github.com/megvii-research/motrv2/blob/main/datasets/panoptic_eval.py"
    click MAKE_DETDB "https://github.com/megvii-research/motrv2/blob/main/tools/make_detdb.py"
    click B "https://github.com/megvii-research/motrv2/blob/main/models/backbone.py"
    click DD "https://github.com/megvii-research/motrv2/blob/main/models/deformable_detr.py"
    click DT "https://github.com/megvii-research/motrv2/blob/main/models/deformable_transformer_plus.py"
    click QIM "https://github.com/megvii-research/motrv2/blob/main/models/qim.py"
    click MOTR "https://github.com/megvii-research/motrv2/blob/main/models/motr.py"
    click M "https://github.com/megvii-research/motrv2/blob/main/models/matcher.py"
    click PE_mod "https://github.com/megvii-research/motrv2/blob/main/models/position_encoding.py"
    click BOXES "https://github.com/megvii-research/motrv2/blob/main/models/structures/boxes.py"
    click INST "https://github.com/megvii-research/motrv2/blob/main/models/structures/instances.py"
    click F "https://github.com/megvii-research/motrv2/blob/main/models/ops/functions/ms_deform_attn_func.py"
    click M_OP "https://github.com/megvii-research/motrv2/blob/main/models/ops/modules/ms_deform_attn.py"
    click MAKE_SH "https://github.com/megvii-research/motrv2/blob/main/models/ops/make.sh"
    click SETUP "https://github.com/megvii-research/motrv2/blob/main/models/ops/setup.py"
    click CPUCPP "https://github.com/megvii-research/motrv2/blob/main/models/ops/src/cpu/ms_deform_attn_cpu.cpp"
    click CPUH "https://github.com/megvii-research/motrv2/blob/main/models/ops/src/cpu/ms_deform_attn_cpu.h"
    click CUDACU "https://github.com/megvii-research/motrv2/blob/main/models/ops/src/cuda/ms_deform_attn_cuda.cu"
    click CUDAH "https://github.com/megvii-research/motrv2/blob/main/models/ops/src/cuda/ms_deform_attn_cuda.cuh"
    click IM2CUH "https://github.com/megvii-research/motrv2/blob/main/models/ops/src/cuda/ms_deform_im2col_cuda.cuh"
    click ATTNH "https://github.com/megvii-research/motrv2/blob/main/models/ops/src/ms_deform_attn.h"
    click VCPP "https://github.com/megvii-research/motrv2/blob/main/models/ops/src/vision.cpp"
    click MAIN "https://github.com/megvii-research/motrv2/blob/main/main.py"
    click ENGINE "https://github.com/megvii-research/motrv2/blob/main/engine.py"
    click CONFIG "https://github.com/megvii-research/motrv2/blob/main/configs/motrv2.args"
    click CP "https://github.com/megvii-research/motrv2/blob/main/util/checkpoint.py"
    click EVAL "https://github.com/megvii-research/motrv2/blob/main/util/evaluation.py"
    click MOTE "https://github.com/megvii-research/motrv2/blob/main/util/motdet_eval.py"
    click BOXOPS "https://github.com/megvii-research/motrv2/blob/main/util/box_ops.py"
    click PLOT "https://github.com/megvii-research/motrv2/blob/main/util/plot_utils.py"
    click VIS "https://github.com/megvii-research/motrv2/blob/main/tools/visualize.py"
    click MISC "https://github.com/megvii-research/motrv2/blob/main/util/misc.py"
    click TOOL "https://github.com/megvii-research/motrv2/blob/main/util/tool.py"
    click RUN_DIST "https://github.com/megvii-research/motrv2/blob/main/tools/run_dist_launch.sh"
    click SLURM "https://github.com/megvii-research/motrv2/blob/main/tools/run_dist_slurm.sh"
    click LAUNCH "https://github.com/megvii-research/motrv2/blob/main/tools/launch.py"
    click TRAIN_SH "https://github.com/megvii-research/motrv2/blob/main/tools/train.sh"
    click SIMPLE_INF "https://github.com/megvii-research/motrv2/blob/main/tools/simple_inference.sh"
    click EVAL_DANCE "https://github.com/megvii-research/motrv2/blob/main/tools/eval_dance.sh"
    click SUB_DANCE "https://github.com/megvii-research/motrv2/blob/main/submit_dance.py"
    click DEBUG "https://github.com/megvii-research/motrv2/blob/main/tools/debug.sh"
    click RESUME "https://github.com/megvii-research/motrv2/blob/main/tools/resume.sh"
    click COPY "https://github.com/megvii-research/motrv2/blob/main/tools/copy_back.sh"
    click BATCH_DIFF "https://github.com/megvii-research/motrv2/blob/main/tools/batch_diff.py"
    click MERGE_SH "https://github.com/megvii-research/motrv2/blob/main/tools/merge_dance_tracklets.sh"
    click MERGE_PY "https://github.com/megvii-research/motrv2/blob/main/tools/merge_dance_tracklets.py"

    %% Styles
    classDef data fill:#bbdefb,stroke:#333,stroke-width:1px
    classDef model fill:#c8e6c9,stroke:#333,stroke-width:1px
    classDef external fill:#ffe0b2,stroke:#333,stroke-width:1px
    classDef util fill:#e0e0e0,stroke:#333,stroke-width:1px
```
