How levan92/deep_sort_realtime works
1. This repository is a Python package, not an application. Its public purpose is to provide a real-time multi-object tracker based on Deep SORT, installed and consumed as deep_sort_realtime.
2. The main entry point is deep_sort_realtime/deepsort_tracker.py, which exposes the DeepSort orchestrator. Client code sends per-frame detections into update_tracks(...) and receives a list of active Track objects back.
3. The input boundary is intentionally flexible: the tracker can accept raw detection boxes, optional class labels/confidence, optional precomputed embeddings, optional video frames for on-the-fly embedding, polygon detections, and optional supplementary per-detection data such as masks or metadata.
4. The core tracking engine lives in deep_sort_realtime/deep_sort/. This is the classic Deep SORT pipeline split into small algorithmic modules: kalman_filter.py for motion state prediction, nn_matching.py for appearance-distance matching, iou_matching.py for box overlap matching, linear_assignment.py for min-cost association, track.py for single-track state, and tracker.py for the multi-target tracker state machine.
5. Data flow through the core tracker is: detections are wrapped into Detection objects, tracks are predicted with the Kalman filter, candidate matches are scored using appearance features and/or IoU, assignments are solved, and track state is updated or aged out. This is the central algorithmic boundary in the repo.
6. Appearance embedding is a separate subsystem under deep_sort_realtime/embedder/. It supports multiple backends: a default PyTorch MobileNetV2 embedder, a TensorFlow embedder, a CLIP-based embedder, and TorchreID-compatible models. The architecture allows either internal embedding from frames or external embeddings supplied by the caller.
7. The embedder subsystem has a clear dependency boundary around model weights. Small default weights are bundled in deep_sort_realtime/embedder/weights/, while larger or optional weights are expected to be downloaded separately via helper scripts or user-provided paths.
8. Detection preprocessing includes optional non-max suppression in deep_sort_realtime/utils/nms.py. The README notes that NMS is skipped when the threshold is 1.0, which makes preprocessing cheaper in the common “do not suppress” case.
9. This fork adds several track-level extensions beyond vanilla Deep SORT: a custom track class can be injected via override_track_class, track IDs can reset based on the current date, Track.to_* methods can return either predicted boxes or the original matched detection box, and supplementary detection payloads can be retrieved from the track.
10. Polygon and masking support extend the input/output contract without changing the core tracker math. Polygons are reduced to bounding rectangles for tracking; if embedding is enabled, the crop is masked so background pixels do not influence the appearance vector.
11. The package has no external service dependencies. Its runtime dependency set is local numerical/computer-vision libraries: NumPy, SciPy, OpenCV, and optionally PyTorch/Torchvision, TensorFlow, or TorchReID depending on the chosen embedder.
12. Testing is lightweight and focused on the library boundary: test_bb.py covers box behavior, test_embedder.py covers feature extraction, and test_general.py covers end-to-end and general package behavior. Packaging files (setup.py, pyproject.toml, MANIFEST.in) indicate standard Python distribution rather than a server or CLI architecture.
Main entry points: deepsort_tracker.py DeepSort · tracker.py Multi-target · __init__.py Embedder init · linear_assignment.py Assignment
Components 19
Public API
Library entrypoints used by consumers.
- Package init · Top-level package exports for deep_sort_realtime.
deep_sort_realtime/__init__.py - DeepSort · Main consumer-facing tracker entrypoint exposing update_tracks().
deep_sort_realtime/deepsort_tracker.py
Tracking Core
Deep SORT state, motion, and association modules.
- Detection · Wraps per-frame detection boxes and supplementary payloads.
deep_sort_realtime/deep_sort/detection.py - Track · Single track lifecycle, state, and output formatting.
deep_sort_realtime/deep_sort/track.py - Multi-target · Maintains active tracks, aging, and update cycles.
deep_sort_realtime/deep_sort/tracker.py - Kalman · Predicts and updates track motion state.
deep_sort_realtime/deep_sort/kalman_filter.py - Assignment · Solves min-cost matching between tracks and detections.
deep_sort_realtime/deep_sort/linear_assignment.py - Appearance match · Computes appearance-distance scores for embedding-based matching.
deep_sort_realtime/deep_sort/nn_matching.py - IoU match · Fallback or complementary box-overlap matching.
deep_sort_realtime/deep_sort/iou_matching.py
Embedding
Feature extraction backends and model weights.
- Embedder init · Backend selection and embedder package boundary.
deep_sort_realtime/embedder/__init__.py - PyTorch embedder · Default PyTorch feature extractor implementation.
deep_sort_realtime/embedder/embedder_pytorch.py - TF embedder · TensorFlow-based feature extraction backend.
deep_sort_realtime/embedder/embedder_tf.py - CLIP embedder · CLIP-backed feature extraction backend.
deep_sort_realtime/embedder/embedder_clip.py - MobileNetV2 · MobileNetV2-based embedding network and bottleneck adapter.
deep_sort_realtime/embedder/mobilenetv2_bottle.py - Weights · Bundled and downloadable weight artifacts for embedder backends.
deep_sort_realtime/embedder/weights
Utilities
Input preprocessing and helper logic.
- NMS · Optional non-max suppression for detection cleanup.
deep_sort_realtime/utils/nms.py
Tests
Lightweight package-level verification.
- Box tests · Validates box behavior and geometry.
test/test_bb.py - Embed tests · Covers feature extraction behavior.
test/test_embedder.py - General tests · Covers end-to-end and package-level behavior.
test/test_general.py
Connections 25
- Package init → DeepSort: exports · Package surface exposes the tracker orchestrator.
- DeepSort → Detection: wraps · Converts incoming detections into internal Detection objects.
- DeepSort → NMS: preprocesses · Optionally suppresses overlapping detections before tracking.
- DeepSort → Embedder init: uses · Delegates to embedding subsystem when internal embeddings are enabled.
- DeepSort → Multi-target: drives · Feeds detections into the multi-target tracker update cycle.
- DeepSort → Track: returns · Produces active Track objects to the caller.
- Multi-target → Kalman: predicts · Advances track state before association.
- Multi-target → Assignment: solves · Coordinates association between tracks and detections.
- Multi-target → Appearance match: scores · Uses appearance distances when embeddings are available.
- Multi-target → IoU match: fallback · Uses IoU matching as a complementary association signal.
- Multi-target → Track: updates · Applies matched detections, aging, and state transitions.
- Assignment → Appearance match: combines · Builds cost matrices from appearance similarity.
- Assignment → IoU match: combines · Can incorporate overlap costs for assignment.
- Track → Detection: stores · Retains matched detection payload and optional metadata.
- Embedder init → PyTorch embedder: selects · Default backend for feature extraction.
- Embedder init → TF embedder: selects · Optional TensorFlow backend.
- Embedder init → CLIP embedder: selects · Optional CLIP backend.
- PyTorch embedder → MobileNetV2: implements · Uses MobileNetV2-based feature network.
- PyTorch embedder → Weights: loads · Consumes bundled or user-provided model weights.
- TF embedder → Weights: loads · Uses downloadable TensorFlow weights.
- CLIP embedder → Weights: loads · Uses downloadable CLIP weights.
- DeepSort → General tests: validated by · General package and integration behavior.
- DeepSort → Box tests: validated by · Box and geometry behavior.
- Embedder init → Embed tests: validated by · Embedding behavior and feature extraction.
- NMS → General tests: validated by · Preprocessing behavior covered in general tests.