How zhousheng97/vitxt-gqa works
ViTXT-GQA is a grounded text-based video question-answering benchmark and T2S-QA research implementation. Its principal workflow prepares or loads video features, OCR results, QA and grounding annotations; batches examples for model training or inference; and produces answers with spatio-temporal scene-text grounding for evaluation. The repository also contains the broader Pythia dataset/model/trainer framework and several other datasets and models, which are not expanded here. README documents training and evaluation, but the referenced T2S scripts and configs are absent from this tree; some subsystem links therefore remain intentionally broad.
Main entry points: base_trainer.py Base trainer · t2s.py T2S-QA model · builder.py ViTXT-GQA builder · dataset_loader.py Dataset loader
Components 17
Data preparation
- Video feature extraction
tools/video_feat/obtain_vit_feat.py
Dataset loading
- ViTXT-GQA builder
pythia/datasets/videoqa/vitxtgqa/builder.py - ViTXT-GQA dataset
pythia/datasets/videoqa/vitxtgqa/dataset.py
QA runtime
- T2S-QA model
pythia/models/t2s.py - Spatio-temporal grounding
pythia/modules/spatio_temporal_grounding.py
Pythia framework
- Dataset loader
pythia/common/dataset_loader.py - Batch collator
pythia/common/batch_collator.py - Batch representation
pythia/common/sample.py - Run entry
tools/run.py - Base trainer
pythia/trainers/base_trainer.py - Checkpoint handling
pythia/utils/checkpoint.py
Evaluation
- Dataset evaluation
tools/analysis/upper_bound.py
Other components
- Researcher
- Raw video
- QA and grounding labels
- OCR results
- Answer and grounding
Connections 18
- Researcher → Run entry: starts run
- Raw video → Video feature extraction: extracts features
- Video feature extraction → ViTXT-GQA builder: provides features
- QA and grounding labels → ViTXT-GQA builder: provides labels
- OCR results → ViTXT-GQA builder: provides text
- ViTXT-GQA builder → ViTXT-GQA dataset: builds dataset
- Run entry → Base trainer: starts training or evaluation
- Base trainer → Dataset loader: loads datasets
- Dataset loader → ViTXT-GQA dataset: loads samples
- Dataset loader → Batch collator: batches samples
- Batch collator → Batch representation: forms batch
- Base trainer → T2S-QA model: runs model
- Batch representation → T2S-QA model: provides batch
- T2S-QA model → Spatio-temporal grounding: computes grounding
- Base trainer → Checkpoint handling: saves or restores
- T2S-QA model → Answer and grounding: predicts answer and grounding
- Answer and grounding → Dataset evaluation: is evaluated
- Researcher → Dataset evaluation: analyzes results