# levan92/deep_sort_realtime architecture

> Architecture diagram of the GitHub repository levan92/deep_sort_realtime, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/levan92/deep_sort_realtime
- Repository: https://github.com/levan92/deep_sort_realtime

## Overview

1. This repository is a Python package, not an application. Its public purpose is to provide a real-time multi-object tracker based on Deep SORT, installed and consumed as `deep_sort_realtime`.

2. The main entry point is `deep_sort_realtime/deepsort_tracker.py`, which exposes the `DeepSort` orchestrator. Client code sends per-frame detections into `update_tracks(...)` and receives a list of active `Track` objects back.

3. The input boundary is intentionally flexible: the tracker can accept raw detection boxes, optional class labels/confidence, optional precomputed embeddings, optional video frames for on-the-fly embedding, polygon detections, and optional supplementary per-detection data such as masks or metadata.

4. The core tracking engine lives in `deep_sort_realtime/deep_sort/`. This is the classic Deep SORT pipeline split into small algorithmic modules: `kalman_filter.py` for motion state prediction, `nn_matching.py` for appearance-distance matching, `iou_matching.py` for box overlap matching, `linear_assignment.py` for min-cost association, `track.py` for single-track state, and `tracker.py` for the multi-target tracker state machine.

5. Data flow through the core tracker is: detections are wrapped into `Detection` objects, tracks are predicted with the Kalman filter, candidate matches are scored using appearance features and/or IoU, assignments are solved, and track state is updated or aged out. This is the central algorithmic boundary in the repo.

6. Appearance embedding is a separate subsystem under `deep_sort_realtime/embedder/`. It supports multiple backends: a default PyTorch MobileNetV2 embedder, a TensorFlow embedder, a CLIP-based embedder, and TorchreID-compatible models. The architecture allows either internal embedding from frames or external embeddings supplied by the caller.

7. The embedder subsystem has a clear dependency boundary around model weights. Small default weights are bundled in `deep_sort_realtime/embedder/weights/`, while larger or optional weights are expected to be downloaded separately via helper scripts or user-provided paths.

8. Detection preprocessing includes optional non-max suppression in `deep_sort_realtime/utils/nms.py`. The README notes that NMS is skipped when the threshold is `1.0`, which makes preprocessing cheaper in the common “do not suppress” case.

9. This fork adds several track-level extensions beyond vanilla Deep SORT: a custom track class can be injected via `override_track_class`, track IDs can reset based on the current date, `Track.to_*` methods can return either predicted boxes or the original matched detection box, and supplementary detection payloads can be retrieved from the track.

10. Polygon and masking support extend the input/output contract without changing the core tracker math. Polygons are reduced to bounding rectangles for tracking; if embedding is enabled, the crop is masked so background pixels do not influence the appearance vector.

11. The package has no external service dependencies. Its runtime dependency set is local numerical/computer-vision libraries: NumPy, SciPy, OpenCV, and optionally PyTorch/Torchvision, TensorFlow, or TorchReID depending on the chosen embedder.

12. Testing is lightweight and focused on the library boundary: `test_bb.py` covers box behavior, `test_embedder.py` covers feature extraction, and `test_general.py` covers end-to-end and general package behavior. Packaging files (`setup.py`, `pyproject.toml`, `MANIFEST.in`) indicate standard Python distribution rather than a server or CLI architecture.

Main entry points: [`deep_sort_realtime/deepsort_tracker.py`](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deepsort_tracker.py) (DeepSort), [`deep_sort_realtime/deep_sort/tracker.py`](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/tracker.py) (Multi-target), [`deep_sort_realtime/embedder/__init__.py`](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/embedder/__init__.py) (Embedder init), [`deep_sort_realtime/deep_sort/linear_assignment.py`](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/linear_assignment.py) (Assignment)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_api["Public API"]
  node_n_init["Package init<br/>package export<br/>[__init__.py]"]
  node_n_tracker_api["DeepSort<br/>orchestrator"]
end

subgraph group_grp_core["Tracking Core"]
  node_n_detection["Detection<br/>track input<br/>[detection.py]"]
  node_n_track["Track<br/>[track.py]"]
  node_n_tracker_core["Multi-target<br/>tracker state machine<br/>[tracker.py]"]
  node_n_kalman(("Kalman<br/>motion model<br/>[kalman_filter.py]"))
  node_n_assoc["Assignment<br/>association solver"]
  node_n_nn["Appearance match<br/>feature distance<br/>[nn_matching.py]"]
  node_n_iou["IoU match<br/>overlap matcher<br/>[iou_matching.py]"]
end

subgraph group_grp_embed["Embedding"]
  node_n_embed_init["Embedder init<br/>embedder selection<br/>[__init__.py]"]
  node_n_pt_embed["PyTorch embedder<br/>vision embedder"]
  node_n_tf_embed["TF embedder<br/>vision embedder<br/>[embedder_tf.py]"]
  node_n_clip_embed["CLIP embedder<br/>vision embedder<br/>[embedder_clip.py]"]
  node_n_mobilenet["MobileNetV2<br/>embedder model"]
  node_n_weights[("Weights<br/>model assets")]
end

subgraph group_grp_utils["Utilities"]
  node_n_nms["NMS<br/>preprocessing<br/>[nms.py]"]
end

subgraph group_grp_tests["Tests"]
  node_n_test_bb["Box tests<br/>[test_bb.py]"]
  node_n_test_embed["Embed tests<br/>[test_embedder.py]"]
  node_n_test_general["General tests<br/>[test_general.py]"]
end

node_n_init -->|"exports"| node_n_tracker_api
node_n_tracker_api -->|"wraps"| node_n_detection
node_n_tracker_api -->|"preprocesses"| node_n_nms
node_n_tracker_api -->|"uses"| node_n_embed_init
node_n_tracker_api -->|"drives"| node_n_tracker_core
node_n_tracker_api -->|"returns"| node_n_track
node_n_tracker_core -->|"predicts"| node_n_kalman
node_n_tracker_core -->|"solves"| node_n_assoc
node_n_tracker_core -->|"scores"| node_n_nn
node_n_tracker_core -->|"fallback"| node_n_iou
node_n_tracker_core -->|"updates"| node_n_track
node_n_assoc -->|"combines"| node_n_nn
node_n_assoc -->|"combines"| node_n_iou
node_n_track -->|"stores"| node_n_detection
node_n_embed_init -->|"selects"| node_n_pt_embed
node_n_embed_init -->|"selects"| node_n_tf_embed
node_n_embed_init -->|"selects"| node_n_clip_embed
node_n_pt_embed -->|"implements"| node_n_mobilenet
node_n_pt_embed -->|"loads"| node_n_weights
node_n_tf_embed -->|"loads"| node_n_weights
node_n_clip_embed -->|"loads"| node_n_weights
node_n_tracker_api -.->|"validated by"| node_n_test_general
node_n_tracker_api -.->|"validated by"| node_n_test_bb
node_n_embed_init -.->|"validated by"| node_n_test_embed
node_n_nms -.->|"validated by"| node_n_test_general

click node_n_init "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/__init__.py"
click node_n_tracker_api "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deepsort_tracker.py"
click node_n_detection "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/detection.py"
click node_n_track "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/track.py"
click node_n_tracker_core "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/tracker.py"
click node_n_kalman "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/kalman_filter.py"
click node_n_assoc "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/linear_assignment.py"
click node_n_nn "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/nn_matching.py"
click node_n_iou "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/iou_matching.py"
click node_n_embed_init "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/embedder/__init__.py"
click node_n_pt_embed "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/embedder/embedder_pytorch.py"
click node_n_tf_embed "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/embedder/embedder_tf.py"
click node_n_clip_embed "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/embedder/embedder_clip.py"
click node_n_mobilenet "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/embedder/mobilenetv2_bottle.py"
click node_n_weights "https://github.com/levan92/deep_sort_realtime/tree/master/deep_sort_realtime/embedder/weights"
click node_n_nms "https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/utils/nms.py"
click node_n_test_bb "https://github.com/levan92/deep_sort_realtime/blob/master/test/test_bb.py"
click node_n_test_embed "https://github.com/levan92/deep_sort_realtime/blob/master/test/test_embedder.py"
click node_n_test_general "https://github.com/levan92/deep_sort_realtime/blob/master/test/test_general.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_init,node_n_tracker_api toneBlue
class node_n_detection,node_n_track,node_n_tracker_core,node_n_kalman,node_n_assoc,node_n_nn,node_n_iou toneAmber
class node_n_embed_init,node_n_pt_embed,node_n_tf_embed,node_n_clip_embed,node_n_mobilenet,node_n_weights toneMint
class node_n_nms toneRose
class node_n_test_bb,node_n_test_embed,node_n_test_general toneIndigo
```

## Components

### Public API

Library entrypoints used by consumers.

- [Package init](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/__init__.py): package export · Top-level package exports for deep_sort_realtime. · `deep_sort_realtime/__init__.py`
- [DeepSort](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deepsort_tracker.py): orchestrator · Main consumer-facing tracker entrypoint exposing update_tracks(). · `deep_sort_realtime/deepsort_tracker.py`

### Tracking Core

Deep SORT state, motion, and association modules.

- [Detection](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/detection.py): track input · Wraps per-frame detection boxes and supplementary payloads. · `deep_sort_realtime/deep_sort/detection.py`
- [Track](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/track.py): track state · Single track lifecycle, state, and output formatting. · `deep_sort_realtime/deep_sort/track.py`
- [Multi-target](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/tracker.py): tracker state machine · Maintains active tracks, aging, and update cycles. · `deep_sort_realtime/deep_sort/tracker.py`
- [Kalman](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/kalman_filter.py): motion model · Predicts and updates track motion state. · `deep_sort_realtime/deep_sort/kalman_filter.py`
- [Assignment](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/linear_assignment.py): association solver · Solves min-cost matching between tracks and detections. · `deep_sort_realtime/deep_sort/linear_assignment.py`
- [Appearance match](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/nn_matching.py): feature distance · Computes appearance-distance scores for embedding-based matching. · `deep_sort_realtime/deep_sort/nn_matching.py`
- [IoU match](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deep_sort/iou_matching.py): overlap matcher · Fallback or complementary box-overlap matching. · `deep_sort_realtime/deep_sort/iou_matching.py`

### Embedding

Feature extraction backends and model weights.

- [Embedder init](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/embedder/__init__.py): embedder selection · Backend selection and embedder package boundary. · `deep_sort_realtime/embedder/__init__.py`
- [PyTorch embedder](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/embedder/embedder_pytorch.py): vision embedder · Default PyTorch feature extractor implementation. · `deep_sort_realtime/embedder/embedder_pytorch.py`
- [TF embedder](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/embedder/embedder_tf.py): vision embedder · TensorFlow-based feature extraction backend. · `deep_sort_realtime/embedder/embedder_tf.py`
- [CLIP embedder](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/embedder/embedder_clip.py): vision embedder · CLIP-backed feature extraction backend. · `deep_sort_realtime/embedder/embedder_clip.py`
- [MobileNetV2](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/embedder/mobilenetv2_bottle.py): embedder model · MobileNetV2-based embedding network and bottleneck adapter. · `deep_sort_realtime/embedder/mobilenetv2_bottle.py`
- [Weights](https://github.com/levan92/deep_sort_realtime/tree/master/deep_sort_realtime/embedder/weights): model assets · Bundled and downloadable weight artifacts for embedder backends. · `deep_sort_realtime/embedder/weights`

### Utilities

Input preprocessing and helper logic.

- [NMS](https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/utils/nms.py): preprocessing · Optional non-max suppression for detection cleanup. · `deep_sort_realtime/utils/nms.py`

### Tests

Lightweight package-level verification.

- [Box tests](https://github.com/levan92/deep_sort_realtime/blob/master/test/test_bb.py): test · Validates box behavior and geometry. · `test/test_bb.py`
- [Embed tests](https://github.com/levan92/deep_sort_realtime/blob/master/test/test_embedder.py): test · Covers feature extraction behavior. · `test/test_embedder.py`
- [General tests](https://github.com/levan92/deep_sort_realtime/blob/master/test/test_general.py): test · Covers end-to-end and package-level behavior. · `test/test_general.py`

## Connections

- Package init → DeepSort: exports · Package surface exposes the tracker orchestrator.
- DeepSort → Detection: wraps · Converts incoming detections into internal Detection objects.
- DeepSort → NMS: preprocesses · Optionally suppresses overlapping detections before tracking.
- DeepSort → Embedder init: uses · Delegates to embedding subsystem when internal embeddings are enabled.
- DeepSort → Multi-target: drives · Feeds detections into the multi-target tracker update cycle.
- DeepSort → Track: returns · Produces active Track objects to the caller.
- Multi-target → Kalman: predicts · Advances track state before association.
- Multi-target → Assignment: solves · Coordinates association between tracks and detections.
- Multi-target → Appearance match: scores · Uses appearance distances when embeddings are available.
- Multi-target → IoU match: fallback · Uses IoU matching as a complementary association signal.
- Multi-target → Track: updates · Applies matched detections, aging, and state transitions.
- Assignment → Appearance match: combines · Builds cost matrices from appearance similarity.
- Assignment → IoU match: combines · Can incorporate overlap costs for assignment.
- Track → Detection: stores · Retains matched detection payload and optional metadata.
- Embedder init → PyTorch embedder: selects · Default backend for feature extraction.
- Embedder init → TF embedder: selects · Optional TensorFlow backend.
- Embedder init → CLIP embedder: selects · Optional CLIP backend.
- PyTorch embedder → MobileNetV2: implements · Uses MobileNetV2-based feature network.
- PyTorch embedder → Weights: loads · Consumes bundled or user-provided model weights.
- TF embedder → Weights: loads · Uses downloadable TensorFlow weights.
- CLIP embedder → Weights: loads · Uses downloadable CLIP weights.
- DeepSort → General tests: validated by · General package and integration behavior.
- DeepSort → Box tests: validated by · Box and geometry behavior.
- Embedder init → Embed tests: validated by · Embedding behavior and feature extraction.
- NMS → General tests: validated by · Preprocessing behavior covered in general tests.