How gpt4vision/ovsgtr works
Project Type and Purpose
• This is a research‐oriented deep‐learning codebase for open-vocabulary Scene Graph Generation (OvSGG). It integrates the GroundingDINO object detector and implements the OvSGTR model to extend scene graph generation to novel concepts.
File-Structure Analysis
• Top-level directories:
• GroundingDINO/… → a subpackage containing source, demo scripts, and setup for the GroundingDINO detector.
• config/… → Python config files for different training/eval modes (SwinT/SwinB, OvD, OvR, joint).
• datasets/… → dataset definitions, transforms, evaluation code for VG, COCO, etc.
• models/… → registry and wrapper for the OvSGTR model.
• engine.py & main.py → training/evaluation entry points.
• scripts/… → bash scripts to launch distributed training/evaluation.
• tools/… → Jupyter notebooks and scripts for data preprocessing or language parsing.
• util/… → generic utilities (logging, I/O, timers, plotting).
• figures/ & paper/ → static assets and publication PDFs.
Architectural Insights from README
• Core Components:
• Detector: GroundingDINO
• SGG Model: OvSGTR implemented under groundingdino/models
• Config-driven: All training/inference behaviors are parameterized by config/*.py
• Dataset Pipeline: load → transform → batch → feed into model → post-process
• Training Engine: engine.py orchestrates forward/backward, logging, checkpointing
• Evaluation: test scripts compute R@20/50/100 metrics per mode (Closed, OvD, OvR, Joint)