# jinyanglii/ovtr architecture

> Architecture diagram of the GitHub repository jinyanglii/ovtr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jinyanglii/ovtr
- Repository: https://github.com/jinyanglii/ovtr

## Overview

##### Project Type and Purpose
- OVTR is a research‐oriented, end-to-end open-vocabulary multi-object tracking system built around a Transformer architecture.
- It integrates data ingestion, preprocessing, model definition (dual-branch DETR), training/inference orchestration, and evaluation (TETA metrics).

##### High-Level Components
a. Data Sources & Preprocessing
- Raw datasets: TAO, LVIS, COCO under `data/`
- Preprocessing scripts: `process/gen_clip_image_embeds.py`, `process/lvis_filter.ipynb`
- Filtered annotations and CLIP embeddings stored in `model_zoo/`
b. Data Loading Pipeline
- `ovtr/datasets/builder.py` orchestrates dataset registration
- Parsers (`ovtr/datasets/parsers/`) read COCO/Video annotations
- Pipelines (`ovtr/datasets/pipelines/`): formatting, H5 backend, transforms
- Samplers (`ovtr/datasets/samplers_ovtrack/`): distributed video sampling
- Analogous structure in `ovtr_det_bs2_pretrain/datasets` for detection pretrain
c. Model Definition
- Backbone: Swin Transformer (`ovtr/models/backbone/`) + positional encodings
- Transformer encoder-decoder (`ovtr/models/transformer.py`)
- Dual-branch DETR heads: classification and tracking (`ovtr/models/ovtr.py`, `fuse_modules.py`)
- Custom CUDA ops for deformable attention (`ovtr/models/ops/…`)
- Utility modules: matcher, segmentation head, updater, registry
d. Core Tracking Logic
- Similarity computation and transforms (`ovtr/core/track/`)
- Category information propagation & attention isolation (`ovtr/core/utils/`)
e. Training & Inference Orchestration
- `ovtr/engine.py` implements the training loop, loss computation, checkpointing
- `ovtr/main.py` parses configs and kicks off training/validation
- Shell scripts (`ovtr/tools/*.sh`) wrap common flows (train, eval, demo)
f. Evaluation
- In-code hooks for MOT metrics (`ovtr/core/evaluation/`)
- TETA metric library (`ovtr/teta/`) for open-vocabulary tracking evaluation
g. Utilities
- Logging, config loading, checkpoint I/O (`ovtr/util/`)
- Detectron2 structure adapters under `ovtr/detectron2/`

##### Relationships & Data Flows
- Data sources → Preprocessing → Dataset builder → DataLoader → Model (backbone → transformer → heads) → Tracking module → Outputs
- Training: Outputs + Ground truth → Loss → Engine → Checkpoint & Logs
- Inference: Model + Tracking → Predictions → Evaluation modules → Metrics → `ovtr/results/`

##### Architectural Patterns & Principles
- Modular package layering by responsibility (data / model / core / util / tools)
- Pipeline pattern for data transforms and samplers
- Transformer-based encoder-decoder with dual-branch heads
- Clear separation of training vs. evaluation logic with configuration-driven scripts

##### Essential Detail Checklist
[ ] Data sources & preprocessing scripts
[ ] Dataset builder, parsers, pipelines, samplers
[ ] Model subcomponents: backbone, transformer, ops, heads
[ ] Tracking logic module
[ ] Engine and CLI entry points
[ ] Evaluation modules and TETA metrics
[ ] Utility and external dependency layers
[ ] Shell scripts illustrating usage patterns
[ ] model_zoo as external weight & embedding repository

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data & Preprocessing"
        DS["Raw Datasets\n(TAO, LVIS, COCO)"]:::data
        Pre1["gen_clip_image_embeds.py"]:::data
        Pre2["lvis_filter.ipynb"]:::data
        ModelZoo[(model_zoo)]:::dataStore
        DS -->|"CLIP embeddings"| Pre1
        Pre2 --> DS
        DS --> ModelZoo
    end
    click DS "https://github.com/jinyanglii/ovtr/tree/main/data/"
    click Pre1 "https://github.com/jinyanglii/ovtr/blob/main/process/gen_clip_image_embeds.py"
    click Pre2 "https://github.com/jinyanglii/ovtr/blob/main/process/lvis_filter.ipynb"

    subgraph "Data Loading Pipeline"
        builder["builder.py"]:::data
        parser1["coco_api.py"]:::data
        parser2["coco_video_parser.py"]:::data
        pipe1["formatting.py"]:::data
        pipe2["h5backend.py"]:::data
        pipe3["loading.py"]:::data
        pipe4["transforms.py"]:::data
        sampler["distributed_video_sampler.py"]:::data
        DataLoader["DataLoader"]:::data
        builder --> parser1
        builder --> parser2
        parser1 --> pipe1
        parser2 --> pipe1
        pipe1 --> pipe2
        pipe2 --> pipe3
        pipe3 --> pipe4
        pipe4 --> sampler
        sampler --> DataLoader
    end
    click builder "https://github.com/jinyanglii/ovtr/blob/main/ovtr/datasets/builder.py"
    click parser1 "https://github.com/jinyanglii/ovtr/blob/main/ovtr/datasets/parsers/coco_api.py"
    click parser2 "https://github.com/jinyanglii/ovtr/blob/main/ovtr/datasets/parsers/coco_video_parser.py"
    click pipe1 "https://github.com/jinyanglii/ovtr/blob/main/ovtr/datasets/pipelines/formatting.py"
    click pipe2 "https://github.com/jinyanglii/ovtr/blob/main/ovtr/datasets/pipelines/h5backend.py"
    click pipe3 "https://github.com/jinyanglii/ovtr/blob/main/ovtr/datasets/pipelines/loading.py"
    click pipe4 "https://github.com/jinyanglii/ovtr/blob/main/ovtr/datasets/pipelines/transforms.py"
    click sampler "https://github.com/jinyanglii/ovtr/blob/main/ovtr/datasets/samplers_ovtrack/distributed_video_sampler.py"

    subgraph "Detection Pretrain Branch"
        DPData["Detection Pretrain\nDatasets"]:::data
        DPDataLoader["DataLoader"]:::data
        DPEngine["engine_ov.py"]:::controller
        DPMain["main.py"]:::controller
        PretrainModel["Pretrain Model"]:::model
        DPData --> DPDataLoader
        DPDataLoader --> PretrainModel
        DPEngine --> DPMain
        DPMain --> PretrainModel
    end
    click DPData "https://github.com/jinyanglii/ovtr/tree/main/ovtr_det_bs2_pretrain/datasets/"
    click DPEngine "https://github.com/jinyanglii/ovtr/blob/main/ovtr_det_bs2_pretrain/engine_ov.py"
    click DPMain "https://github.com/jinyanglii/ovtr/blob/main/ovtr_det_bs2_pretrain/main.py"

    subgraph "Model Components"
        Backbone["Backbone\n(swin, pos enc)"]:::model
        Transf["Transformer\nEncoder-Decoder"]:::model
        Heads["Dual-Branch DETR\n(classification, tracking)"]:::model
        Ops["CUDA Ops\n(deformable attn)"]:::ext
        UtilsM["Model Utilities"]:::ext
        Backbone --> Transf
        Transf --> Heads
        Heads --> CoreTrack
    end
    click Backbone "https://github.com/jinyanglii/ovtr/blob/main/ovtr/models/backbone/backbone.py"
    click Backbone "https://github.com/jinyanglii/ovtr/blob/main/ovtr/models/backbone/position_encoding.py"
    click Backbone "https://github.com/jinyanglii/ovtr/blob/main/ovtr/models/backbone/swin_transformer.py"
    click Transf "https://github.com/jinyanglii/ovtr/blob/main/ovtr/models/transformer.py"
    click Heads "https://github.com/jinyanglii/ovtr/blob/main/ovtr/models/ovtr.py"
    click Heads "https://github.com/jinyanglii/ovtr/blob/main/ovtr/models/fuse_modules.py"
    click Ops "https://github.com/jinyanglii/ovtr/blob/main/ovtr/models/ops/make.sh"
    click Ops "https://github.com/jinyanglii/ovtr/blob/main/ovtr/models/ops/setup.py"
    click Ops "https://github.com/jinyanglii/ovtr/tree/main/ovtr/models/ops/src/"
    click UtilsM "https://github.com/jinyanglii/ovtr/blob/main/ovtr/models/matcher.py"
    click UtilsM "https://github.com/jinyanglii/ovtr/blob/main/ovtr/models/segmentation.py"
    click UtilsM "https://github.com/jinyanglii/ovtr/blob/main/ovtr/models/updater.py"
    click UtilsM "https://github.com/jinyanglii/ovtr/blob/main/ovtr/models/registry.py"
    click UtilsM "https://github.com/jinyanglii/ovtr/blob/main/ovtr/models/utils.py"

    subgraph "Core Tracking & Utils"
        Sim["similarity.py"]:::model
        Trans["transforms.py"]:::model
        Viz["visualization.py"]:::ext
        Sim --> Trans
    end
    click Sim "https://github.com/jinyanglii/ovtr/blob/main/ovtr/core/track/similarity.py"
    click Trans "https://github.com/jinyanglii/ovtr/blob/main/ovtr/core/track/transforms.py"
    click Viz "https://github.com/jinyanglii/ovtr/blob/main/ovtr/core/utils/visualization.py"

    subgraph "Training & Orchestration"
        Engine["engine.py"]:::controller
        Main["main.py"]:::controller
        Shell1["ovtr_demo.sh"]:::controller
        Shell2["ovtr_multi_frame_train.sh"]:::controller
        Shell3["ovtr_multi_frame_lite_train.sh"]:::controller
        Shell4["ovtr_ovmot_eval_*.sh"]:::controller
        DataLoader --> Engine
        Engine --> Main
        Main --> Shell1
        Main --> Shell2
        Main --> Shell3
        Main --> Shell4
    end
    click Engine "https://github.com/jinyanglii/ovtr/blob/main/ovtr/engine.py"
    click Main "https://github.com/jinyanglii/ovtr/blob/main/ovtr/main.py"
    click Shell1 "https://github.com/jinyanglii/ovtr/blob/main/ovtr/tools/ovtr_demo.sh"
    click Shell2 "https://github.com/jinyanglii/ovtr/blob/main/ovtr/tools/ovtr_multi_frame_train.sh"
    click Shell3 "https://github.com/jinyanglii/ovtr/blob/main/ovtr/tools/ovtr_multi_frame_lite_train.sh"
    click Shell4 "https://github.com/jinyanglii/ovtr/blob/main/ovtr/tools/ovtr_ovmot_eval_*.sh"

    subgraph "Evaluation"
        EvalHooks["eval_hooks.py"]:::eval
        MOT["mot.py"]:::eval
        TETA["TETA Library"]:::eval
        EvalHooks --> MOT
        MOT --> TETA
        Predictions["Predictions"]:::data --> EvalHooks
    end
    click EvalHooks "https://github.com/jinyanglii/ovtr/blob/main/ovtr/core/evaluation/eval_hooks.py"
    click MOT "https://github.com/jinyanglii/ovtr/blob/main/ovtr/core/evaluation/mot.py"
    click TETA "https://github.com/jinyanglii/ovtr/blob/main/ovtr/teta/config.py"
    click TETA "https://github.com/jinyanglii/ovtr/blob/main/ovtr/teta/eval.py"
    click TETA "https://github.com/jinyanglii/ovtr/tree/main/ovtr/teta/datasets/"
    click TETA "https://github.com/jinyanglii/ovtr/tree/main/ovtr/teta/metrics/"

    subgraph "Utilities & External Dependencies"
        UtilCP["checkpoint.py"]:::util
        UtilCL["clip_utils.py"]:::util
        UtilLG["logger.py"]:::util
        Detect2["Detectron2 Adapters"]:::ext
        UtilCP
        UtilCL
        UtilLG
        Detect2
    end
    click UtilCP "https://github.com/jinyanglii/ovtr/blob/main/ovtr/util/checkpoint.py"
    click UtilCL "https://github.com/jinyanglii/ovtr/blob/main/ovtr/util/clip_utils.py"
    click UtilLG "https://github.com/jinyanglii/ovtr/blob/main/ovtr/util/logger.py"
    click Detect2 "https://github.com/jinyanglii/ovtr/blob/main/ovtr/detectron2/structures/boxes.py"
    click Detect2 "https://github.com/jinyanglii/ovtr/blob/main/ovtr/detectron2/structures/instances.py"

    Predictions["Predictions"]:::data
    DS --> builder
    DataLoader --> Backbone
    PretrainModel --> Heads
    Engine --> UtilCP

    classDef data fill:#ccffcc,stroke:#333
    classDef dataStore fill:#ccffcc,stroke:#333,stroke-dasharray:5 5
    classDef model fill:#ADD8E6,stroke:#333
    classDef controller fill:#FFA500,stroke:#333
    classDef eval fill:#dda0dd,stroke:#333
    classDef util fill:#d3d3d3,stroke:#333
    classDef ext fill:#d3d3d3,stroke:#333,stroke-dasharray:2 2
```
