# fsoft-aic/z-gmot architecture

> Architecture diagram of the GitHub repository fsoft-aic/z-gmot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fsoft-aic/z-gmot
- Repository: https://github.com/fsoft-aic/z-gmot

## Overview

##### Project Type and Purpose
- This is a research‐grade Python toolkit for Zero‐shot Generic Multiple Object Tracking (Z-GMOT).
- It combines a vision‐language detection backbone (GLIP built on maskrcnn_benchmark) with a suite of multi‐object trackers (BOTSORT, BYTETRACK, DeepOCSORT, OCSORT, StrongSORT) under the “MA-Sort” umbrella.

##### File Structure Highlights
- Top-level “maskrcnn_benchmark/”: core detection engine (data loaders, model definitions, training & inference loops).
- “MA-Sort/boxmot/…”: Python packages implementing different tracker algorithms (botsort, bytetrack, deepocsort, ocsort, strongsort).
- “configs/”: YAML files defining datasets, model architectures, hyper-parameters for pre-training, zero-shot eval, fine-tuning, and ODinW tasks.
- “tools/”: orchestration scripts (train_net.py, test_grounding_net.py, finetune.py, eval_all.py, etc.).
- “prompt/” & “knowledge/”: JSON/YAML files for language prompts and knowledge augmentation.
- “demo/” & “docs/”: usage examples and documentation web pages.

##### Key README Insights
- High-level pipelines:
- Pre-training (torch.distributed launch → train_net.py)
- Zero-shot evaluation (test_grounding_net.py)
- Fine-tuning (train_net.py or finetune.py)
- Model Zoo: preconfigured YAMLs in configs/pretrain/ mapping to GLIP-T and GLIP-L variants.
- ODinW support: 13/35 downstream detection tasks configured via configs/odinw_*/.

##### Main Components
A. Data & Config Layer
- Raw datasets: COCO, LVIS, Flickr30K, ODinW (35 tasks)
- Config files (.yaml) grouped under configs/{pretrain, odinw_13, odinw_35, flickr, lvis, etc.}
- Prompt definitions (prompt/*.json) and Knowledge stores (knowledge/*.yaml)
B. Core Detection Engine
- maskrcnn_benchmark package
- data (datasets, transforms)
- modeling (backbone, detector heads, language backbone)
- engine (trainer, inference, predictor)
- solver (lr schedulers)
- GLIP extensions (DYHEAD, grounding net)
C. Multi-Object Trackers (MA-Sort)
- boxmot subpackage: botsort, bytetrack, deepocsort, ocsort, strongsort
- Each tracker: Kalman filter, matching, basetrack, track management
D. Orchestration Scripts
- tools/train_net.py → Pre-training & Fine-tuning
- tools/test_grounding_net.py → Zero-shot & few-shot evaluation
- tools/finetune.py → ODinW few-shot/fine-tuning
- demo/ scripts (finetune_MOT17.py, etc.)
E. Model Zoo & Outputs
- MODEL folder storing checkpoint weights
- OUTPUT_DIR for evaluation results (JSON, text, CSV under tracking_result/)

##### Relationships & Data Flow
1. User selects a pipeline by running a script (train_net.py, test_grounding_net.py, finetune.py).
2. Script reads Config YAML → initializes DataLoader and Model (maskrcnn_benchmark + GLIP).
4. Prompt/Knowledge modules inject text embeddings into the detection or grounding heads.
5. Final predictions are written to OUTPUT_DIR in COCO or custom JSON/text formats.
6. Metrics are computed (COCO eval, LVIS FixedAP, grounding accuracy) and optionally visualized.

##### Architectural Patterns & Principles
- Modular layering: Data → Model → Tracker → Output
- Config-driven pipelines: decouple code from settings via YAML
- Package isolation: core detection (maskrcnn_benchmark) vs. tracking (MA-Sort) vs. orchestration (tools/)
- Python entry points (– m torch.distributed.launch) for distributed training/inference
- Extensibility: easy to swap trackers, change prompts/knowledge, add new datasets via configs.

##### Technologies & Frameworks
- Python 3.x, PyTorch ≥1.9, torchvision
- maskrcnn_benchmark (C++/CUDA extensions)
- YAML (configs), JSON (prompts, predictions)
- Docker for environment reproducibility
- Distributed training (torch.distributed)
- External data formats: COCO, LVIS, Flickr30K grounding

##### Essential Elements
- Clear component labels (e.g., “maskrcnn_benchmark.modeling”, “MA-Sort.bot_sort”)
- Code modules
- Scripts
- Configs
- Data stores & outputs

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data & Config Layer
    subgraph "Data & Config Layer"
        direction TB
        Configs[/"Configs (YAML)"/]:::config
        Flickr["Configs: Flickr"]:::config
        LVIS["Configs: LVIS"]:::config
        OD13["Configs: ODinW (13)"]:::config
        OD35["Configs: ODinW (35)"]:::config
        Pretrain["Configs: Pretrain"]:::config
    end

    %% Prompt & Knowledge
    subgraph "Prompt & Knowledge"
        direction TB
        Prompts["Prompt Definitions (JSON)"]:::pk
        Knowledge["Knowledge Stores (YAML)"]:::pk
    end

    %% Core Detection Engine
    subgraph "Core Detection Engine (GLIP + maskrcnn_benchmark)"
        direction TB
        MB_Config["maskrcnn_benchmark/config"]:::core
        MB_Data["maskrcnn_benchmark/data"]:::core
        MB_Modeling["maskrcnn_benchmark/modeling"]:::core
        MB_Engine["maskrcnn_benchmark/engine"]:::core
        MB_Solver["maskrcnn_benchmark/solver"]:::core
        MB_Layers["maskrcnn_benchmark/layers & csrc"]:::core
        ModelZooUtil["maskrcnn_benchmark/utils/model_zoo.py"]:::core
    end

    %% MA-Sort Trackers
    subgraph "MA-Sort Trackers (boxmot)"
        direction TB
        BOTSORT["BOTSORT"]:::track
        BYTETRACK["BYTETRACK"]:::track
        DeepOCSORT["DeepOCSORT"]:::track
        OCSORT["OCSORT"]:::track
        StrongSORT["StrongSORT"]:::track
        TrackerZoo["tracker_zoo.py"]:::track
    end

    %% Orchestration Scripts
    subgraph "Orchestration Scripts"
        direction TB
        TrainNet["tools/train_net.py"]:::script
        TestGround["tools/test_grounding_net.py"]:::script
        Finetune["tools/finetune.py"]:::script
        EvalAll["tools/eval_all.py"]:::script
        TestNet["tools/test_net.py"]:::script
        VizGround["tools/visualize_grounding_net.py"]:::script
        DemoScripts["demo/*.py"]:::script
    end

    %% Model & Output
    subgraph "Model Zoo & Outputs"
        direction TB
        MODEL["MODEL Checkpoints"]:::storage
        OutputDir["OUTPUT_DIR (tracking_result/)"]:::storage
        TextResults["MA-Sort/tracking_result/text"]:::storage
        Visualize["visualize.py"]:::storage
        iGLIP["iGLIP_openvoc/GMOT_40_boxes"]:::storage
    end

    %% Relations
    TrainNet -->|"reads configs"| Configs
    TestGround -->|"reads configs & prompts"| Configs
    Finetune -->|"reads ODinW configs"| OD13
    Finetune -->|"reads ODinW configs"| OD35

    TrainNet -->|"init DataLoader & Model"| MB_Data
    TestGround -->|"init DataLoader & Model"| MB_Data
    Finetune -->|"init DataLoader & Model"| MB_Data

    MB_Modeling -->|"uses prompts"| Prompts
    MB_Modeling -->|"uses knowledge"| Knowledge

    MB_Engine -->|"training/inference"| MB_Modeling
    MB_Engine -->|"distributed"| TorchDist[("torch.distributed")]:::external

    MB_Engine -->|"outputs bbox & embeddings"| TrackerZoo
    TrackerZoo -->|"select tracker"| BOTSORT
    TrackerZoo -->|"select tracker"| BYTETRACK
    TrackerZoo -->|"select tracker"| DeepOCSORT
    TrackerZoo -->|"select tracker"| OCSORT
    TrackerZoo -->|"select tracker"| StrongSORT

    BOTSORT -->|"predictions JSON"| OutputDir
    BYTETRACK -->|"predictions JSON"| OutputDir
    DeepOCSORT -->|"predictions JSON"| OutputDir
    OCSORT -->|"predictions JSON"| OutputDir
    StrongSORT -->|"predictions JSON"| OutputDir

    OutputDir -->|"text results"| TextResults
    OutputDir -->|"visualization"| Visualize

    %% Click events
    click Configs "https://github.com/fsoft-aic/z-gmot/tree/main/configs/"
    click Flickr "https://github.com/fsoft-aic/z-gmot/tree/main/configs/flickr/"
    click LVIS "https://github.com/fsoft-aic/z-gmot/tree/main/configs/lvis/"
    click OD13 "https://github.com/fsoft-aic/z-gmot/tree/main/configs/odinw_13/"
    click OD35 "https://github.com/fsoft-aic/z-gmot/tree/main/configs/odinw_35/"
    click Pretrain "https://github.com/fsoft-aic/z-gmot/tree/main/configs/pretrain/"
    click Prompts "https://github.com/fsoft-aic/z-gmot/tree/main/prompt/"
    click Knowledge "https://github.com/fsoft-aic/z-gmot/tree/main/knowledge/"
    click MB_Config "https://github.com/fsoft-aic/z-gmot/tree/main/maskrcnn_benchmark/config/"
    click MB_Data "https://github.com/fsoft-aic/z-gmot/tree/main/maskrcnn_benchmark/data/"
    click MB_Modeling "https://github.com/fsoft-aic/z-gmot/tree/main/maskrcnn_benchmark/modeling/"
    click MB_Engine "https://github.com/fsoft-aic/z-gmot/tree/main/maskrcnn_benchmark/engine/"
    click MB_Solver "https://github.com/fsoft-aic/z-gmot/tree/main/maskrcnn_benchmark/solver/"
    click MB_Layers "https://github.com/fsoft-aic/z-gmot/tree/main/maskrcnn_benchmark/layers/"
    click ModelZooUtil "https://github.com/fsoft-aic/z-gmot/blob/main/maskrcnn_benchmark/utils/model_zoo.py"
    click BOTSORT "https://github.com/fsoft-aic/z-gmot/tree/main/MA-Sort/boxmot/botsort/"
    click BYTETRACK "https://github.com/fsoft-aic/z-gmot/tree/main/MA-Sort/boxmot/bytetrack/"
    click DeepOCSORT "https://github.com/fsoft-aic/z-gmot/tree/main/MA-Sort/boxmot/deepocsort/"
    click OCSORT "https://github.com/fsoft-aic/z-gmot/tree/main/MA-Sort/boxmot/ocsort/"
    click StrongSORT "https://github.com/fsoft-aic/z-gmot/tree/main/MA-Sort/boxmot/strongsort/"
    click TrackerZoo "https://github.com/fsoft-aic/z-gmot/blob/main/MA-Sort/boxmot/tracker_zoo.py"
    click TrainNet "https://github.com/fsoft-aic/z-gmot/blob/main/tools/train_net.py"
    click TestGround "https://github.com/fsoft-aic/z-gmot/blob/main/tools/test_grounding_net.py"
    click Finetune "https://github.com/fsoft-aic/z-gmot/blob/main/tools/finetune.py"
    click EvalAll "https://github.com/fsoft-aic/z-gmot/blob/main/tools/eval_all.py"
    click TestNet "https://github.com/fsoft-aic/z-gmot/blob/main/tools/test_net.py"
    click VizGround "https://github.com/fsoft-aic/z-gmot/blob/main/tools/visualize_grounding_net.py"
    click DemoScripts "https://github.com/fsoft-aic/z-gmot/tree/main/demo/"
    click TextResults "https://github.com/fsoft-aic/z-gmot/tree/main/MA-Sort/tracking_result/text/"
    click Visualize "https://github.com/fsoft-aic/z-gmot/blob/main/visualize.py"
    click iGLIP "https://github.com/fsoft-aic/z-gmot/tree/main/iGLIP_openvoc/GMOT_40_boxes/"

    %% Styles
    classDef config fill:#D0E8FF,stroke:#1E90FF
    classDef core fill:#B0C4DE,stroke:#4682B4
    classDef track fill:#D0F0C0,stroke:#32CD32
    classDef script fill:#FFE4B5,stroke:#FF8C00
    classDef pk fill:#E6CCFF,stroke:#800080
    classDef storage fill:#E8E8E8,stroke:#A9A9A9
    classDef external stroke-dasharray: 5 5,stroke:#555
    class TorchDist external
```
