# foundationvision/bytetrack architecture

> Architecture diagram of the GitHub repository foundationvision/bytetrack, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/foundationvision/bytetrack
- Repository: https://github.com/foundationvision/bytetrack

## Overview

1. **Repository purpose**
   - ByteTrack is a multi-object tracking system built around a YOLOX detector plus a motion/association tracker.
   - The repo is split into three major paths: model training/evaluation in Python, demo/tracking entrypoints, and deployment-specific runtimes in C++/Python.

2. **Core Python package: `yolox/`**
   - This is the main framework layer inherited from YOLOX.
   - It contains model definitions, training loop, data loading/augmentation, evaluators, tracking utilities, and experiment configuration support.
   - Most high-level scripts in `tools/` call into this package rather than implementing logic themselves.

3. **Detection model subsystem**
   - `yolox/models/` defines the detector architecture: backbone/neck/head blocks and the final YOLOX model.
   - `yolox/exp/` and `exps/` define experiment variants, hyperparameters, input sizes, and dataset wiring for specific training or evaluation setups.
   - The detector produces per-frame bounding boxes and scores, which are the input to tracking.

4. **Tracking subsystem**
   - `yolox/tracker/` contains the ByteTrack implementation: track state management, Kalman filter motion model, and matching logic.
   - The key architectural idea is that the tracker consumes both high-confidence and low-confidence detections, then associates them to existing tracks instead of discarding weak boxes.
   - Other tracker variants are included as comparators or alternatives: `sort_tracker/`, `deepsort_tracker/`, and `motdt_tracker/`.

5. **Data flow at inference time**
   - A frame is passed through the detector to obtain detection boxes and scores.
   - The tracker updates active tracklets using motion prediction plus assignment between detections and existing tracks.
   - The output is a set of tracked objects with persistent IDs per frame, used by demos, evaluation, and deployment wrappers.

6. **Training and evaluation pipeline**
   - `tools/train.py` drives detector training using an experiment file from `exps/`.
   - `yolox/data/` provides dataset wrappers, mosaic augmentation, samplers, and dataloading utilities.
   - `yolox/evaluators/` and `yolox/tracking_utils/` compute detection/tracking metrics, including MOTChallenge-style evaluation.

7. **Dataset preparation and format conversion**
   - `tools/convert_*.py` scripts convert MOT17, MOT20, CrowdHuman, CityPersons, and ETHZ into COCO-like training data.
   - `tools/mix_data_*.py` creates mixed training sets used by the model zoo experiments.
   - This is an explicit preprocessing boundary: raw MOT-style datasets are not consumed directly by training.

8. **Experiment definitions**
   - `exps/default/` contains standard detector configs, while `exps/example/mot/` contains MOT-specific tracker/detector training recipes.
   - These files select model scale, dataset mix, image size, and tracker-related thresholds.
   - The repo is designed so a new dataset or training regime is usually added by creating a new experiment file rather than modifying core code.

9. **Demo and offline tracking tools**
   - `tools/track.py`, `track_sort.py`, `track_deepsort.py`, and `track_motdt.py` run different trackers on the same detector outputs.
   - `tools/demo_track.py`, `tools/convert_video.py`, and `tools/txt2video.py` support video-based visualization and result rendering.
   - `tools/interpolation.py` post-processes tracking outputs to improve trajectory continuity before MOTChallenge submission.

10. **Deployment targets**
   - `deploy/ONNXRuntime/` exports and runs the model via ONNXRuntime in Python.
   - `deploy/TensorRT/` provides both Python and C++ inference paths for NVIDIA acceleration.
   - `deploy/ncnn/` targets lightweight C++ inference.
   - `deploy/DeepStream/` integrates ByteTrack into NVIDIA DeepStream for streaming video analytics.
   - These deployments share the same conceptual pipeline: detector inference first, then ByteTrack association.

11. **C++ tracker implementations in deployment**
   - The C++ deployment folders include ports of BYTETracker, Kalman filter, linear assignment, and STrack logic.
   - This separates inference runtime from tracking logic so the same tracking behavior can be embedded into GPU-accelerated or edge pipelines.
   - `NvMOTContext.cpp` in DeepStream is the bridge into the DeepStream plugin/runtime model.

12. **Tutorials as adapter examples**
   - `tutorials/` contains integrations of ByteTrack with other detector/tracker ecosystems such as FairMOT, JDE, CenterTrack, TransTrack, QDTrack, CSTRack, and MOTR.
   - These are not the core path of the repo; they show how the tracker logic can be reused behind different detector front ends.
   - Architecturally, they demonstrate ByteTrack’s detector-agnostic boundary: any source of `(x1, y1, x2, y2, score)` detections can feed the tracker.

13. **Operational boundaries and artifacts**
   - `datasets/` holds data roots and split references; `pretrained/` is expected for weights; `YOLOX_outputs/` is used for logs/checkpoints.
   - `Dockerfile` packages the full training/inference environment, while `requirements.txt` and `setup.py` define the Python runtime.
   - `videos/` and `assets/` are example media and documentation artifacts, not core runtime components.

14. **External dependencies that matter architecturally**
   - The system is built around PyTorch and COCO-style tooling for training/evaluation.
   - MOTChallenge is the primary benchmark/output target.
   - ONNXRuntime, TensorRT, ncnn, and DeepStream are the main runtime backends beyond Python.

Main entry points: [`yolox/tracker/byte_tracker.py`](https://github.com/foundationvision/bytetrack/blob/main/yolox/tracker/byte_tracker.py) (ByteTrack), [`tools/train.py`](https://github.com/foundationvision/bytetrack/blob/main/tools/train.py) (Train), [`yolox/core/trainer.py`](https://github.com/foundationvision/bytetrack/blob/main/yolox/core/trainer.py) (Trainer), [`yolox/models/yolox.py`](https://github.com/foundationvision/bytetrack/blob/main/yolox/models/yolox.py) (YOLOX)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core Python"]
  node_tools_train["Train<br/>entrypoint<br/>[train.py]"]
  node_exp_base["Base Exp<br/>experiment base<br/>[base_exp.py]"]
  node_trainer_core["Trainer<br/>training loop<br/>[trainer.py]"]
  node_model_yolox["YOLOX<br/>detector model<br/>[yolox.py]"]
  node_model_neck["PAFPN<br/>neck<br/>[yolo_pafpn.py]"]
  node_model_head["Head<br/>[yolo_head.py]"]
  node_data_mot[("MOT Data<br/>dataset<br/>[mot.py]")]
  node_data_aug["Augment<br/>data pipeline<br/>[data_augment.py]"]
  node_mot_eval["MOT Eval<br/>evaluator<br/>[mot_evaluator.py]"]
  node_byte_tracker["ByteTrack<br/>tracker<br/>[byte_tracker.py]"]
  node_matching["Matching<br/>association<br/>[matching.py]"]
  node_kalman["Kalman<br/>motion model<br/>[kalman_filter.py]"]
  node_track_cli["Track Tool<br/>entrypoint<br/>[track.py]"]
end

subgraph group_experiments["Experiments"]
  node_mix_convert["MOT Convert<br/>preprocess"]
  node_exp_mot["MOT Exp<br/>experiment<br/>[yolox_x_mix_det.py]"]
end

subgraph group_deployment["Deployment"]
  node_onnx_rt["ONNXRuntime<br/>python runtime<br/>[onnx_inference.py]"]
  node_trt_cpp["TensorRT C++<br/>runtime<br/>[bytetrack.cpp]"]
  node_deepstream_ctx["DeepStream<br/>plugin bridge<br/>[NvMOTContext.cpp]"]
end

subgraph group_adapters["Adapters"]
  node_fairmot_bt["FairMOT Adapter<br/>integration example<br/>[byte_tracker.py]"]
end

node_tools_train -->|"loads"| node_exp_base
node_exp_base -->|"specialized by"| node_exp_mot
node_tools_train -->|"drives"| node_trainer_core
node_trainer_core -->|"optimizes"| node_model_yolox
node_model_yolox -->|"uses"| node_model_neck
node_model_neck -->|"feeds"| node_model_head
node_tools_train -->|"reads"| node_data_mot
node_data_mot -->|"augmented by"| node_data_aug
node_mix_convert -->|"prepares"| node_data_mot
node_trainer_core -->|"validates"| node_mot_eval
node_track_cli -->|"runs"| node_model_yolox
node_track_cli -->|"updates"| node_byte_tracker
node_byte_tracker -->|"assigns"| node_matching
node_byte_tracker -->|"predicts"| node_kalman
node_byte_tracker -->|"scores"| node_mot_eval
node_onnx_rt -->|"feeds"| node_byte_tracker
node_trt_cpp -->|"embeds"| node_byte_tracker
node_deepstream_ctx -->|"bridges"| node_trt_cpp
node_fairmot_bt -->|"adapts"| node_byte_tracker

click node_tools_train "https://github.com/foundationvision/bytetrack/blob/main/tools/train.py"
click node_exp_base "https://github.com/foundationvision/bytetrack/blob/main/yolox/exp/base_exp.py"
click node_trainer_core "https://github.com/foundationvision/bytetrack/blob/main/yolox/core/trainer.py"
click node_model_yolox "https://github.com/foundationvision/bytetrack/blob/main/yolox/models/yolox.py"
click node_model_neck "https://github.com/foundationvision/bytetrack/blob/main/yolox/models/yolo_pafpn.py"
click node_model_head "https://github.com/foundationvision/bytetrack/blob/main/yolox/models/yolo_head.py"
click node_data_mot "https://github.com/foundationvision/bytetrack/blob/main/yolox/data/datasets/mot.py"
click node_data_aug "https://github.com/foundationvision/bytetrack/blob/main/yolox/data/data_augment.py"
click node_mot_eval "https://github.com/foundationvision/bytetrack/blob/main/yolox/evaluators/mot_evaluator.py"
click node_byte_tracker "https://github.com/foundationvision/bytetrack/blob/main/yolox/tracker/byte_tracker.py"
click node_matching "https://github.com/foundationvision/bytetrack/blob/main/yolox/tracker/matching.py"
click node_kalman "https://github.com/foundationvision/bytetrack/blob/main/yolox/tracker/kalman_filter.py"
click node_track_cli "https://github.com/foundationvision/bytetrack/blob/main/tools/track.py"
click node_mix_convert "https://github.com/foundationvision/bytetrack/blob/main/tools/convert_mot17_to_coco.py"
click node_exp_mot "https://github.com/foundationvision/bytetrack/blob/main/exps/example/mot/yolox_x_mix_det.py"
click node_onnx_rt "https://github.com/foundationvision/bytetrack/blob/main/deploy/ONNXRuntime/onnx_inference.py"
click node_trt_cpp "https://github.com/foundationvision/bytetrack/blob/main/deploy/TensorRT/cpp/src/bytetrack.cpp"
click node_deepstream_ctx "https://github.com/foundationvision/bytetrack/blob/main/deploy/DeepStream/src/NvMOTContext.cpp"
click node_fairmot_bt "https://github.com/foundationvision/bytetrack/blob/main/tutorials/fairmot/byte_tracker.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_tools_train,node_exp_base,node_trainer_core,node_model_yolox,node_model_neck,node_model_head,node_data_mot,node_data_aug,node_mot_eval,node_byte_tracker,node_matching,node_kalman,node_track_cli toneBlue
class node_mix_convert,node_exp_mot toneAmber
class node_onnx_rt,node_trt_cpp,node_deepstream_ctx toneMint
class node_fairmot_bt toneRose
```

## Components

### Core Python

Main YOLOX framework, detector, tracker, and evaluation stack.

- [Train](https://github.com/foundationvision/bytetrack/blob/main/tools/train.py): entrypoint · Drives training using an experiment definition. · `tools/train.py`
- [Base Exp](https://github.com/foundationvision/bytetrack/blob/main/yolox/exp/base_exp.py): experiment base · Common experiment configuration and wiring layer. · `yolox/exp/base_exp.py`
- [Trainer](https://github.com/foundationvision/bytetrack/blob/main/yolox/core/trainer.py): training loop · Orchestrates optimization, logging, and checkpointing. · `yolox/core/trainer.py`
- [YOLOX](https://github.com/foundationvision/bytetrack/blob/main/yolox/models/yolox.py): detector model · Top-level detector that produces boxes and scores. · `yolox/models/yolox.py`
- [PAFPN](https://github.com/foundationvision/bytetrack/blob/main/yolox/models/yolo_pafpn.py): neck · Feature aggregation neck used by the detector. · `yolox/models/yolo_pafpn.py`
- [Head](https://github.com/foundationvision/bytetrack/blob/main/yolox/models/yolo_head.py): detection head · Prediction head for classification and box regression. · `yolox/models/yolo_head.py`
- [MOT Data](https://github.com/foundationvision/bytetrack/blob/main/yolox/data/datasets/mot.py): dataset · MOT-style dataset wrapper used for training and evaluation. · `yolox/data/datasets/mot.py`
- [Augment](https://github.com/foundationvision/bytetrack/blob/main/yolox/data/data_augment.py): data pipeline · Augmentation and sample transformation stage. · `yolox/data/data_augment.py`
- [MOT Eval](https://github.com/foundationvision/bytetrack/blob/main/yolox/evaluators/mot_evaluator.py): evaluator · Measures MOT-style tracking and detection performance. · `yolox/evaluators/mot_evaluator.py`
- [ByteTrack](https://github.com/foundationvision/bytetrack/blob/main/yolox/tracker/byte_tracker.py): tracker · Associates detections into persistent track IDs. · `yolox/tracker/byte_tracker.py`
- [Matching](https://github.com/foundationvision/bytetrack/blob/main/yolox/tracker/matching.py): association · Assignment logic for detection-to-track matching. · `yolox/tracker/matching.py`
- [Kalman](https://github.com/foundationvision/bytetrack/blob/main/yolox/tracker/kalman_filter.py): motion model · Motion prediction for track state propagation. · `yolox/tracker/kalman_filter.py`
- [Track Tool](https://github.com/foundationvision/bytetrack/blob/main/tools/track.py): entrypoint · Runs detector plus tracker on images or video. · `tools/track.py`

### Experiments

Training recipes and dataset conversion boundary.

- [MOT Convert](https://github.com/foundationvision/bytetrack/blob/main/tools/convert_mot17_to_coco.py): preprocess · Converts raw MOT data into COCO-like training data. · `tools/convert_mot17_to_coco.py`
- [MOT Exp](https://github.com/foundationvision/bytetrack/blob/main/exps/example/mot/yolox_x_mix_det.py): experiment · Example MOT recipe selecting model scale and thresholds. · `exps/example/mot/yolox_x_mix_det.py`

### Deployment

Backend-specific inference and C++ tracker ports.

- [ONNXRuntime](https://github.com/foundationvision/bytetrack/blob/main/deploy/ONNXRuntime/onnx_inference.py): python runtime · Python inference path for exported ONNX models. · `deploy/ONNXRuntime/onnx_inference.py`
- [TensorRT C++](https://github.com/foundationvision/bytetrack/blob/main/deploy/TensorRT/cpp/src/bytetrack.cpp): runtime · TensorRT inference backend with embedded tracking path. · `deploy/TensorRT/cpp/src/bytetrack.cpp`
- [DeepStream](https://github.com/foundationvision/bytetrack/blob/main/deploy/DeepStream/src/NvMOTContext.cpp): plugin bridge · DeepStream runtime bridge into ByteTrack tracking. · `deploy/DeepStream/src/NvMOTContext.cpp`

### Adapters

Example integrations showing detector-agnostic tracking.

- [FairMOT Adapter](https://github.com/foundationvision/bytetrack/blob/main/tutorials/fairmot/byte_tracker.py): integration example · Example of ByteTrack behind an alternate detector/tracker stack. · `tutorials/fairmot/byte_tracker.py`

## Connections

- Train → Base Exp: loads · Training starts from an experiment definition.
- Base Exp → MOT Exp: specialized by · MOT recipes extend the base experiment layer.
- Train → Trainer: drives · The training entrypoint hands control to the trainer.
- Trainer → YOLOX: optimizes · Trainer updates detector weights.
- YOLOX → PAFPN: uses · Detector is built from backbone/neck/head blocks.
- PAFPN → Head: feeds · Aggregated features flow into prediction head.
- Train → MOT Data: reads · Training consumes MOT-style datasets.
- MOT Data → Augment: augmented by · Dataset samples pass through augmentation pipeline.
- MOT Convert → MOT Data: prepares · Conversion scripts create training-ready data.
- Trainer → MOT Eval: validates · Evaluation runs during training and benchmarking.
- Track Tool → YOLOX: runs · Tracking entrypoints first produce detections.
- Track Tool → ByteTrack: updates · Detector outputs are associated into tracks.
- ByteTrack → Matching: assigns · Tracker uses matching to link detections and tracks.
- ByteTrack → Kalman: predicts · Tracker uses Kalman motion prediction.
- ByteTrack → MOT Eval: scores · Tracking outputs are evaluated with MOT metrics.
- ONNXRuntime → ByteTrack: feeds · Exported model inference feeds the same tracker logic.
- TensorRT C++ → ByteTrack: embeds · TensorRT runtime includes a C++ ByteTrack port.
- DeepStream → TensorRT C++: bridges · DeepStream integrates the runtime/tracking pipeline.
- FairMOT Adapter → ByteTrack: adapts · Tutorial shows tracker reuse behind another detector stack.
