How fsoft-aic/z-gmot works
Project Type and Purpose
• This is a research‐grade Python toolkit for Zero‐shot Generic Multiple Object Tracking (Z-GMOT).
• It combines a vision‐language detection backbone (GLIP built on maskrcnn_benchmark) with a suite of multi‐object trackers (BOTSORT, BYTETRACK, DeepOCSORT, OCSORT, StrongSORT) under the “MA-Sort” umbrella.
File Structure Highlights
• Top-level “maskrcnn_benchmark/”: core detection engine (data loaders, model definitions, training & inference loops).
• “MA-Sort/boxmot/…”: Python packages implementing different tracker algorithms (botsort, bytetrack, deepocsort, ocsort, strongsort).
• “configs/”: YAML files defining datasets, model architectures, hyper-parameters for pre-training, zero-shot eval, fine-tuning, and ODinW tasks.
• “tools/”: orchestration scripts (train_net.py, test_grounding_net.py, finetune.py, eval_all.py, etc.).
• “prompt/” & “knowledge/”: JSON/YAML files for language prompts and knowledge augmentation.
• “demo/” & “docs/”: usage examples and documentation web pages.
Key README Insights
• High-level pipelines:
• Pre-training (torch.distributed launch → train_net.py)
• Zero-shot evaluation (test_grounding_net.py)
• Fine-tuning (train_net.py or finetune.py)
• Model Zoo: preconfigured YAMLs in configs/pretrain/ mapping to GLIP-T and GLIP-L variants.
• ODinW support: 13/35 downstream detection tasks configured via configs/odinw_*/.
Main Components
A. Data & Config Layer
• Raw datasets: COCO, LVIS, Flickr30K, ODinW (35 tasks)
• Config files (.yaml) grouped under configs/{pretrain, odinw_13, odinw_35, flickr, lvis, etc.}
• Prompt definitions (prompt/*.json) and Knowledge stores (knowledge/*.yaml)
B. Core Detection Engine
• maskrcnn_benchmark package
• data (datasets, transforms)
• modeling (backbone, detector heads, language backbone)
• engine (trainer, inference, predictor)
• solver (lr schedulers)
• GLIP extensions (DYHEAD, grounding net)
C. Multi-Object Trackers (MA-Sort)
• boxmot subpackage: botsort, bytetrack, deepocsort, ocsort, strongsort
• Each tracker: Kalman filter, matching, basetrack, track management
D. Orchestration Scripts
• tools/train_net.py → Pre-training & Fine-tuning
• tools/test_grounding_net.py → Zero-shot & few-shot evaluation
• tools/finetune.py → ODinW few-shot/fine-tuning
• demo/ scripts (finetune_MOT17.py, etc.)
E. Model Zoo & Outputs
• MODEL folder storing checkpoint weights
• OUTPUT_DIR for evaluation results (JSON, text, CSV under tracking_result/)
Relationships & Data Flow
1. User selects a pipeline by running a script (train_net.py, test_grounding_net.py, finetune.py).
2. Script reads Config YAML → initializes DataLoader and Model (maskrcnn_benchmark + GLIP).
4. Prompt/Knowledge modules inject text embeddings into the detection or grounding heads.
5. Final predictions are written to OUTPUT_DIR in COCO or custom JSON/text formats.
6. Metrics are computed (COCO eval, LVIS FixedAP, grounding accuracy) and optionally visualized.
Architectural Patterns & Principles
• Modular layering: Data → Model → Tracker → Output
• Config-driven pipelines: decouple code from settings via YAML
• Package isolation: core detection (maskrcnn_benchmark) vs. tracking (MA-Sort) vs. orchestration (tools/)
• Python entry points (– m torch.distributed.launch) for distributed training/inference
• Extensibility: easy to swap trackers, change prompts/knowledge, add new datasets via configs.
Technologies & Frameworks
• Python 3.x, PyTorch ≥1.9, torchvision
• maskrcnn_benchmark (C++/CUDA extensions)
• YAML (configs), JSON (prompts, predictions)
• Docker for environment reproducibility
• Distributed training (torch.distributed)
• External data formats: COCO, LVIS, Flickr30K grounding
Essential Elements
• Clear component labels (e.g., “maskrcnn_benchmark.modeling”, “MA-Sort.bot_sort”)
• Code modules
• Scripts
• Configs
• Data stores & outputs