How mcg-nju/mixsort works
Project Type and Purpose
• MixSort is a full-stack, research-oriented multi-object tracking system.
• It integrates:
• A detection backbone (YOLOX)
• An appearance embedding network (MixFormer in MixViT)
• A data‐association algorithm (ByteTrack / OC-SORT)
• Data conversion tools, training pipelines, evaluation submodule (TrackEval)
• Deployment pipelines (DeepStream / ONNX / TensorRT / ncnn)
File Structure
Top-level modules:
• /yolox: detection model and trackers
• /MixViT: appearance-embedding training (MixFormer)
• /tools: orchestration scripts for training, tracking, conversion
• /deploy: multiple inference backends (DeepStream, ONNXRuntime, TensorRT, ncnn)
• /TrackEval: evaluation submodule
• /exps: experiment definitions for YOLOX trackers
• /assets, /videos, /README, /requirements: project metadata
Extract Architectural Insights from README
• Training stage (tools/train.py, MixViT/lib/train/run_training.py)
• Tracking stage (tools/track_mixsort.py, track_mixsort_oc.py)
• Evaluation via TrackEval submodule
Main Components & Their Interactions
a. Data Preparation & Conversion
• Raw video/frames → conversion tools (tools/convert_*.py) → COCO/TrackingNet datasets
• Configured by experiment scripts (exps/*.py)
b. Detection Module (YOLOX)
• Training: tools/train.py → exp configs → optimizer → checkpoint
• Inference: pretrained checkpoint → detect.py (via tools/track_*.py)
c. Appearance Embedding (MixFormer / MixViT)
• Training: MixViT/lib/train/** → data loader → mixformer model → save checkpoint
• Configs: MixViT/experiments/.yaml + MixViT/lib/config/
• Inference: track.yaml → feature extractor serves embedding for each detection
d. Association Module (MixSort wrapper around ByteTrack / OC-SORT)
• Inputs: detection bboxes + appearance embeddings
• Algorithm: match via IoU threshold + appearance affinity (alpha parameter)
• Implemented in tools/track_mixsort.py and yolox/byte_tracker or yolox/ocsort_tracker
e. Evaluation (TrackEval submodule)
• Inputs: tracker outputs → standardized TrackEval directories
• Scripts: TrackEval/scripts/run_mot_challenge.py or eval_mot.py
f. Deployment Backends
• DeepStream C++ plugin (deploy/DeepStream)
• ONNXRuntime (deploy/ONNXRuntime/onnx_inference.py)
• TensorRT C++ / Python (deploy/TensorRT)
• ncnn C++ (deploy/ncnn)
• Each backend: loads serialized model → performs detection → association → publishes results
Architectural Patterns & Principles
• Modular separation: Detection | Embedding | Association | Evaluation | Deployment
• Plugin-style backends for inference
• Config-driven experiments (exp scripts + YAML/py configs)
• Reuse of external code via submodules (TrackEval, PreciseRoIPooling, AR code)
• GPU-accelerated training/inference
Final Checklist
[ ] All components clearly labeled
[ ] Annotations of key technologies (PyTorch, ONNX, TensorRT, DeepStream, ncnn)
[ ] References to configuration locations (exps/, MixViT/experiments/, deploy/)