How jinyanglii/ovtr works
Project Type and Purpose
• OVTR is a research‐oriented, end-to-end open-vocabulary multi-object tracking system built around a Transformer architecture.
• It integrates data ingestion, preprocessing, model definition (dual-branch DETR), training/inference orchestration, and evaluation (TETA metrics).
High-Level Components
a. Data Sources & Preprocessing
• Raw datasets: TAO, LVIS, COCO under data/
• Preprocessing scripts: process/gen_clip_image_embeds.py, process/lvis_filter.ipynb
• Filtered annotations and CLIP embeddings stored in model_zoo/
b. Data Loading Pipeline
• ovtr/datasets/builder.py orchestrates dataset registration
• Parsers (ovtr/datasets/parsers/) read COCO/Video annotations
• Pipelines (ovtr/datasets/pipelines/): formatting, H5 backend, transforms
• Samplers (ovtr/datasets/samplers_ovtrack/): distributed video sampling
• Analogous structure in ovtr_det_bs2_pretrain/datasets for detection pretrain
c. Model Definition
• Backbone: Swin Transformer (ovtr/models/backbone/) + positional encodings
• Transformer encoder-decoder (ovtr/models/transformer.py)
• Dual-branch DETR heads: classification and tracking (ovtr/models/ovtr.py, fuse_modules.py)
• Custom CUDA ops for deformable attention (ovtr/models/ops/…)
• Utility modules: matcher, segmentation head, updater, registry
d. Core Tracking Logic
• Similarity computation and transforms (ovtr/core/track/)
• Category information propagation & attention isolation (ovtr/core/utils/)
e. Training & Inference Orchestration
• ovtr/engine.py implements the training loop, loss computation, checkpointing
• ovtr/main.py parses configs and kicks off training/validation
• Shell scripts (ovtr/tools/*.sh) wrap common flows (train, eval, demo)
f. Evaluation
• In-code hooks for MOT metrics (ovtr/core/evaluation/)
• TETA metric library (ovtr/teta/) for open-vocabulary tracking evaluation
g. Utilities
• Logging, config loading, checkpoint I/O (ovtr/util/)
• Detectron2 structure adapters under ovtr/detectron2/
Relationships & Data Flows
• Data sources → Preprocessing → Dataset builder → DataLoader → Model (backbone → transformer → heads) → Tracking module → Outputs
• Training: Outputs + Ground truth → Loss → Engine → Checkpoint & Logs
• Inference: Model + Tracking → Predictions → Evaluation modules → Metrics → ovtr/results/
Architectural Patterns & Principles
• Modular package layering by responsibility (data / model / core / util / tools)
• Pipeline pattern for data transforms and samplers
• Transformer-based encoder-decoder with dual-branch heads
• Clear separation of training vs. evaluation logic with configuration-driven scripts
Essential Detail Checklist
[ ] Data sources & preprocessing scripts
[ ] Dataset builder, parsers, pipelines, samplers
[ ] Model subcomponents: backbone, transformer, ops, heads
[ ] Tracking logic module
[ ] Engine and CLI entry points
[ ] Evaluation modules and TETA metrics
[ ] Utility and external dependency layers
[ ] Shell scripts illustrating usage patterns
[ ] model_zoo as external weight & embedding repository