How foundationvision/bytetrack works
1. Repository purpose
- ByteTrack is a multi-object tracking system built around a YOLOX detector plus a motion/association tracker.
- The repo is split into three major paths: model training/evaluation in Python, demo/tracking entrypoints, and deployment-specific runtimes in C++/Python.
2. Core Python package: yolox/
- This is the main framework layer inherited from YOLOX.
- It contains model definitions, training loop, data loading/augmentation, evaluators, tracking utilities, and experiment configuration support.
- Most high-level scripts in tools/ call into this package rather than implementing logic themselves.
3. Detection model subsystem
- yolox/models/ defines the detector architecture: backbone/neck/head blocks and the final YOLOX model.
- yolox/exp/ and exps/ define experiment variants, hyperparameters, input sizes, and dataset wiring for specific training or evaluation setups.
- The detector produces per-frame bounding boxes and scores, which are the input to tracking.
4. Tracking subsystem
- yolox/tracker/ contains the ByteTrack implementation: track state management, Kalman filter motion model, and matching logic.
- The key architectural idea is that the tracker consumes both high-confidence and low-confidence detections, then associates them to existing tracks instead of discarding weak boxes.
- Other tracker variants are included as comparators or alternatives: sort_tracker/, deepsort_tracker/, and motdt_tracker/.
5. Data flow at inference time
- A frame is passed through the detector to obtain detection boxes and scores.
- The tracker updates active tracklets using motion prediction plus assignment between detections and existing tracks.
- The output is a set of tracked objects with persistent IDs per frame, used by demos, evaluation, and deployment wrappers.
6. Training and evaluation pipeline
- tools/train.py drives detector training using an experiment file from exps/.
- yolox/data/ provides dataset wrappers, mosaic augmentation, samplers, and dataloading utilities.
- yolox/evaluators/ and yolox/tracking_utils/ compute detection/tracking metrics, including MOTChallenge-style evaluation.
7. Dataset preparation and format conversion
- tools/convert_*.py scripts convert MOT17, MOT20, CrowdHuman, CityPersons, and ETHZ into COCO-like training data.
- tools/mix_data_*.py creates mixed training sets used by the model zoo experiments.
- This is an explicit preprocessing boundary: raw MOT-style datasets are not consumed directly by training.
8. Experiment definitions
- exps/default/ contains standard detector configs, while exps/example/mot/ contains MOT-specific tracker/detector training recipes.
- These files select model scale, dataset mix, image size, and tracker-related thresholds.
- The repo is designed so a new dataset or training regime is usually added by creating a new experiment file rather than modifying core code.
9. Demo and offline tracking tools
- tools/track.py, track_sort.py, track_deepsort.py, and track_motdt.py run different trackers on the same detector outputs.
- tools/demo_track.py, tools/convert_video.py, and tools/txt2video.py support video-based visualization and result rendering.
- tools/interpolation.py post-processes tracking outputs to improve trajectory continuity before MOTChallenge submission.
10. Deployment targets
- deploy/ONNXRuntime/ exports and runs the model via ONNXRuntime in Python.
- deploy/TensorRT/ provides both Python and C++ inference paths for NVIDIA acceleration.
- deploy/ncnn/ targets lightweight C++ inference.
- deploy/DeepStream/ integrates ByteTrack into NVIDIA DeepStream for streaming video analytics.
- These deployments share the same conceptual pipeline: detector inference first, then ByteTrack association.
11. C++ tracker implementations in deployment
- The C++ deployment folders include ports of BYTETracker, Kalman filter, linear assignment, and STrack logic.
- This separates inference runtime from tracking logic so the same tracking behavior can be embedded into GPU-accelerated or edge pipelines.
- NvMOTContext.cpp in DeepStream is the bridge into the DeepStream plugin/runtime model.
12. Tutorials as adapter examples
- tutorials/ contains integrations of ByteTrack with other detector/tracker ecosystems such as FairMOT, JDE, CenterTrack, TransTrack, QDTrack, CSTRack, and MOTR.
- These are not the core path of the repo; they show how the tracker logic can be reused behind different detector front ends.
- Architecturally, they demonstrate ByteTrack’s detector-agnostic boundary: any source of (x1, y1, x2, y2, score) detections can feed the tracker.
13. Operational boundaries and artifacts
- datasets/ holds data roots and split references; pretrained/ is expected for weights; YOLOX_outputs/ is used for logs/checkpoints.
- Dockerfile packages the full training/inference environment, while requirements.txt and setup.py define the Python runtime.
- videos/ and assets/ are example media and documentation artifacts, not core runtime components.
14. External dependencies that matter architecturally
- The system is built around PyTorch and COCO-style tooling for training/evaluation.
- MOTChallenge is the primary benchmark/output target.
- ONNXRuntime, TensorRT, ncnn, and DeepStream are the main runtime backends beyond Python.
Main entry points: byte_tracker.py ByteTrack · train.py Train · trainer.py Trainer · yolox.py YOLOX
Components 19
Core Python
Main YOLOX framework, detector, tracker, and evaluation stack.
- Train · Drives training using an experiment definition.
tools/train.py - Base Exp · Common experiment configuration and wiring layer.
yolox/exp/base_exp.py - Trainer · Orchestrates optimization, logging, and checkpointing.
yolox/core/trainer.py - YOLOX · Top-level detector that produces boxes and scores.
yolox/models/yolox.py - PAFPN · Feature aggregation neck used by the detector.
yolox/models/yolo_pafpn.py - Head · Prediction head for classification and box regression.
yolox/models/yolo_head.py - MOT Data · MOT-style dataset wrapper used for training and evaluation.
yolox/data/datasets/mot.py - Augment · Augmentation and sample transformation stage.
yolox/data/data_augment.py - MOT Eval · Measures MOT-style tracking and detection performance.
yolox/evaluators/mot_evaluator.py - ByteTrack · Associates detections into persistent track IDs.
yolox/tracker/byte_tracker.py - Matching · Assignment logic for detection-to-track matching.
yolox/tracker/matching.py - Kalman · Motion prediction for track state propagation.
yolox/tracker/kalman_filter.py - Track Tool · Runs detector plus tracker on images or video.
tools/track.py
Experiments
Training recipes and dataset conversion boundary.
- MOT Convert · Converts raw MOT data into COCO-like training data.
tools/convert_mot17_to_coco.py - MOT Exp · Example MOT recipe selecting model scale and thresholds.
exps/example/mot/yolox_x_mix_det.py
Deployment
Backend-specific inference and C++ tracker ports.
- ONNXRuntime · Python inference path for exported ONNX models.
deploy/ONNXRuntime/onnx_inference.py - TensorRT C++ · TensorRT inference backend with embedded tracking path.
deploy/TensorRT/cpp/src/bytetrack.cpp - DeepStream · DeepStream runtime bridge into ByteTrack tracking.
deploy/DeepStream/src/NvMOTContext.cpp
Adapters
Example integrations showing detector-agnostic tracking.
- FairMOT Adapter · Example of ByteTrack behind an alternate detector/tracker stack.
tutorials/fairmot/byte_tracker.py
Connections 19
- Train → Base Exp: loads · Training starts from an experiment definition.
- Base Exp → MOT Exp: specialized by · MOT recipes extend the base experiment layer.
- Train → Trainer: drives · The training entrypoint hands control to the trainer.
- Trainer → YOLOX: optimizes · Trainer updates detector weights.
- YOLOX → PAFPN: uses · Detector is built from backbone/neck/head blocks.
- PAFPN → Head: feeds · Aggregated features flow into prediction head.
- Train → MOT Data: reads · Training consumes MOT-style datasets.
- MOT Data → Augment: augmented by · Dataset samples pass through augmentation pipeline.
- MOT Convert → MOT Data: prepares · Conversion scripts create training-ready data.
- Trainer → MOT Eval: validates · Evaluation runs during training and benchmarking.
- Track Tool → YOLOX: runs · Tracking entrypoints first produce detections.
- Track Tool → ByteTrack: updates · Detector outputs are associated into tracks.
- ByteTrack → Matching: assigns · Tracker uses matching to link detections and tracks.
- ByteTrack → Kalman: predicts · Tracker uses Kalman motion prediction.
- ByteTrack → MOT Eval: scores · Tracking outputs are evaluated with MOT metrics.
- ONNXRuntime → ByteTrack: feeds · Exported model inference feeds the same tracker logic.
- TensorRT C++ → ByteTrack: embeds · TensorRT runtime includes a C++ ByteTrack port.
- DeepStream → TensorRT C++: bridges · DeepStream integrates the runtime/tracking pipeline.
- FairMOT Adapter → ByteTrack: adapts · Tutorial shows tracker reuse behind another detector stack.