# mcg-nju/mixsort architecture

> Architecture diagram of the GitHub repository mcg-nju/mixsort, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mcg-nju/mixsort
- Repository: https://github.com/mcg-nju/mixsort

## Overview

##### Project Type and Purpose
- MixSort is a full-stack, research-oriented multi-object tracking system.
- It integrates:
- A detection backbone (YOLOX)
- An appearance embedding network (MixFormer in MixViT)
- A data‐association algorithm (ByteTrack / OC-SORT)
- Data conversion tools, training pipelines, evaluation submodule (TrackEval)
- Deployment pipelines (DeepStream / ONNX / TensorRT / ncnn)

##### File Structure
Top-level modules:
- /yolox: detection model and trackers
- /MixViT: appearance-embedding training (MixFormer)
- /tools: orchestration scripts for training, tracking, conversion
- /deploy: multiple inference backends (DeepStream, ONNXRuntime, TensorRT, ncnn)
- /TrackEval: evaluation submodule
- /exps: experiment definitions for YOLOX trackers
- /assets, /videos, /README, /requirements: project metadata

##### Extract Architectural Insights from README
- Training stage (tools/train.py, MixViT/lib/train/run_training.py)
- Tracking stage (tools/track_mixsort.py, track_mixsort_oc.py)
- Evaluation via TrackEval submodule

##### Main Components & Their Interactions
a. Data Preparation & Conversion
- Raw video/frames → conversion tools (tools/convert_*.py) → COCO/TrackingNet datasets
- Configured by experiment scripts (exps/*.py)
b. Detection Module (YOLOX)
- Training: tools/train.py → exp configs → optimizer → checkpoint
- Inference: pretrained checkpoint → detect.py (via tools/track_*.py)
c. Appearance Embedding (MixFormer / MixViT)
- Training: MixViT/lib/train/** → data loader → mixformer model → save checkpoint
- Configs: MixViT/experiments/**.yaml + MixViT/lib/config/**
- Inference: track.yaml → feature extractor serves embedding for each detection
d. Association Module (MixSort wrapper around ByteTrack / OC-SORT)
- Inputs: detection bboxes + appearance embeddings
- Algorithm: match via IoU threshold + appearance affinity (alpha parameter)
- Implemented in tools/track_mixsort.py and yolox/byte_tracker or yolox/ocsort_tracker
e. Evaluation (TrackEval submodule)
- Inputs: tracker outputs → standardized TrackEval directories
- Scripts: TrackEval/scripts/run_mot_challenge.py or eval_mot.py
f. Deployment Backends
- DeepStream C++ plugin (deploy/DeepStream)
- ONNXRuntime (deploy/ONNXRuntime/onnx_inference.py)
- TensorRT C++ / Python (deploy/TensorRT)
- ncnn C++ (deploy/ncnn)
- Each backend: loads serialized model → performs detection → association → publishes results

##### Architectural Patterns & Principles
- Modular separation: Detection | Embedding | Association | Evaluation | Deployment
- Plugin-style backends for inference
- Config-driven experiments (exp scripts + YAML/py configs)
- Reuse of external code via submodules (TrackEval, PreciseRoIPooling, AR code)
- GPU-accelerated training/inference

##### Final Checklist
[ ] All components clearly labeled
[ ] Annotations of key technologies (PyTorch, ONNX, TensorRT, DeepStream, ncnn)
[ ] References to configuration locations (exps/, MixViT/experiments/, deploy/)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        RAW["Raw Videos/Frames"]:::data
        CONVERT["Data Conversion Tools\n tools/convert_*.py"]:::data
        DATASET["Dataset Storage\n COCO/TrackingNet"]:::data
        EXPS["Experiment Definitions\n exps/"]:::data
    end

    %% Training Layer
    subgraph "Training Layer"
        direction TB
        DET_TRAIN["YOLOX Training Script\n tools/train.py"]:::training
        DET_CORE["YOLOX Core\n yolox/core/"]:::training
        DET_MODEL["YOLOX Model Definitions\n yolox/models/"]:::training
        EMB_CFG["Embedding Configs\n MixViT/lib/config/"]:::training
        EMB_TRAIN["MixFormer Training Pipeline\n MixViT/lib/train/run_training.py"]:::training
        EMB_MODEL["MixFormer Models\n MixViT/lib/models/mixformer/"]:::training
    end

    %% Inference Layer
    subgraph "Inference Layer"
        direction TB
        DET_INFER["Tracking Script\n tools/track_mixsort.py"]:::inference
        EMB_INFER["Embedding Inference\n track.yaml"]:::inference
        subgraph "Association Module"
            direction TB
            BYTE["ByteTrack Algorithm\n yolox/byte_tracker/"]:::inference
            OCSORT["OC-SORT Algorithm\n yolox/ocsort_tracker/"]:::inference
            MIXSORT_TRACKER["MixSort Tracker\n yolox/mixsort_tracker/"]:::inference
            MIXSORT_OC["MixSort OC Tracker\n yolox/mixsort_oc_tracker/"]:::inference
        end
        OUTPUT["Tracker Output\n trajectories"]:::inference
    end

    %% Evaluation Layer
    subgraph "Evaluation Layer"
        direction TB
        EVAL["TrackEval Submodule\n TrackEval/"]:::eval
    end

    %% Deployment Layer
    subgraph "Deployment Layer"
        direction TB
        DS["DeepStream Backend\n deploy/DeepStream/"]:::deployment
        ONNX["ONNXRuntime Backend\n deploy/ONNXRuntime/onnx_inference.py"]:::deployment
        TRT["TensorRT Backend\n deploy/TensorRT/"]:::deployment
        NCNN["ncnn Backend\n deploy/ncnn/"]:::deployment
    end

    %% Data Flow Connections
    RAW -->|raw data| CONVERT
    CONVERT -->|converted data| DATASET
    DATASET -->|configs| EXPS
    EXPS -->|dataset infos| DET_TRAIN
    DATASET -->|images| DET_CORE
    DET_TRAIN -->|training data| DET_CORE
    DET_CORE -->|models| DET_MODEL
    DATASET -->|images| EMB_TRAIN
    EMB_CFG -->|configs| EMB_TRAIN
    EMB_TRAIN -->|models| EMB_MODEL

    %% Inference Flow
    DET_MODEL -->|checkpoint| DET_INFER
    EMB_MODEL -->|checkpoint| EMB_INFER
    DET_INFER -->|bboxes| BYTE
    EMB_INFER -->|features| BYTE
    BYTE -->|tracks| MIXSORT_TRACKER
    BYTE -->|tracks| MIXSORT_OC
    OCSORT -->|tracks| MIXSORT_TRACKER
    OCSORT -->|tracks| MIXSORT_OC
    MIXSORT_TRACKER -->|trajectories| OUTPUT
    MIXSORT_OC -->|trajectories| OUTPUT

    %% Evaluation
    OUTPUT -->|tracker outputs| EVAL

    %% Deployment Flow
    DET_INFER -->|inference API| DS
    DET_INFER -->|inference API| ONNX
    DET_INFER -->|inference API| TRT
    DET_INFER -->|inference API| NCNN
    EMB_INFER -->|embedding API| DS
    EMB_INFER -->|embedding API| ONNX
    EMB_INFER -->|embedding API| TRT
    EMB_INFER -->|embedding API| NCNN

    %% Click Events
    click DET_MODEL "https://github.com/mcg-nju/mixsort/tree/main/yolox/models/"
    click DET_CORE "https://github.com/mcg-nju/mixsort/tree/main/yolox/core/"
    click CONVERT "https://github.com/mcg-nju/mixsort/blob/main/tools/convert_mot17_to_coco.py"
    click DET_TRAIN "https://github.com/mcg-nju/mixsort/blob/main/tools/train.py"
    click EMB_CFG "https://github.com/mcg-nju/mixsort/tree/main/MixViT/lib/config/"
    click EMB_TRAIN "https://github.com/mcg-nju/mixsort/blob/main/MixViT/lib/train/run_training.py"
    click EMB_MODEL "https://github.com/mcg-nju/mixsort/tree/main/MixViT/lib/models/mixformer/"
    click DET_INFER "https://github.com/mcg-nju/mixsort/blob/main/tools/track_mixsort.py"
    click BYTE "https://github.com/mcg-nju/mixsort/tree/main/yolox/byte_tracker/"
    click OCSORT "https://github.com/mcg-nju/mixsort/tree/main/yolox/ocsort_tracker/"
    click MIXSORT_TRACKER "https://github.com/mcg-nju/mixsort/tree/main/yolox/mixsort_tracker/"
    click MIXSORT_OC "https://github.com/mcg-nju/mixsort/tree/main/yolox/mixsort_oc_tracker/"
    click EXPS "https://github.com/mcg-nju/mixsort/tree/main/exps/"
    click EVAL "https://github.com/mcg-nju/mixsort/tree/main/TrackEval/"
    click DS "https://github.com/mcg-nju/mixsort/tree/main/deploy/DeepStream/"
    click ONNX "https://github.com/mcg-nju/mixsort/blob/main/deploy/ONNXRuntime/onnx_inference.py"
    click TRT "https://github.com/mcg-nju/mixsort/tree/main/deploy/TensorRT/"
    click NCNN "https://github.com/mcg-nju/mixsort/tree/main/deploy/ncnn/"

    %% Styles
    classDef data fill:#e2e3e5,stroke:#444,stroke-width:1px
    classDef training fill:#cce5ff,stroke:#444,stroke-width:1px
    classDef inference fill:#d4edda,stroke:#444,stroke-width:1px
    classDef deployment fill:#fff3cd,stroke:#444,stroke-width:1px
    classDef eval fill:#f8d7da,stroke:#444,stroke-width:1px
```
