# zhousheng97/vitxt-gqa architecture

> Architecture diagram of the GitHub repository zhousheng97/vitxt-gqa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zhousheng97/vitxt-gqa
- Repository: https://github.com/zhousheng97/vitxt-gqa

## Overview

ViTXT-GQA is a grounded text-based video question-answering benchmark and T2S-QA research implementation. Its principal workflow prepares or loads video features, OCR results, QA and grounding annotations; batches examples for model training or inference; and produces answers with spatio-temporal scene-text grounding for evaluation. The repository also contains the broader Pythia dataset/model/trainer framework and several other datasets and models, which are not expanded here. README documents training and evaluation, but the referenced T2S scripts and configs are absent from this tree; some subsystem links therefore remain intentionally broad.

Main entry points: [`pythia/trainers/base_trainer.py`](https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/trainers/base_trainer.py) (Base trainer), [`pythia/models/t2s.py`](https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/models/t2s.py) (T2S-QA model), [`pythia/datasets/videoqa/vitxtgqa/builder.py`](https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/datasets/videoqa/vitxtgqa/builder.py) (ViTXT-GQA builder), [`pythia/common/dataset_loader.py`](https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/common/dataset_loader.py) (Dataset loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Data preparation"]
  node_video_prep["Video feature extraction<br/>[obtain_vit_feat.py]"]
end

subgraph group_data["Dataset loading"]
  node_vitxt_builder["ViTXT-GQA builder<br/>[builder.py]"]
  node_vitxt_dataset["ViTXT-GQA dataset<br/>[dataset.py]"]
end

subgraph group_runtime["QA runtime"]
  node_t2s["T2S-QA model<br/>[t2s.py]"]
  node_grounding["Spatio-temporal grounding"]
end

subgraph group_framework["Pythia framework"]
  node_loader["Dataset loader<br/>[dataset_loader.py]"]
  node_collator["Batch collator<br/>[batch_collator.py]"]
  node_sample["Batch representation<br/>[sample.py]"]
  node_run["Run entry<br/>[run.py]"]
  node_trainer["Base trainer<br/>[base_trainer.py]"]
  node_checkpoint["Checkpoint handling<br/>[checkpoint.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_evaluation["Dataset evaluation<br/>[upper_bound.py]"]
end

node_researcher(("Researcher"))
node_video["Raw video"]
node_annotations["QA and grounding labels"]
node_ocr["OCR results"]
node_answer["Answer and grounding"]

node_researcher -->|"starts run"| node_run
node_video -.->|"extracts features"| node_video_prep
node_video_prep -.->|"provides features"| node_vitxt_builder
node_annotations -->|"provides labels"| node_vitxt_builder
node_ocr -->|"provides text"| node_vitxt_builder
node_vitxt_builder -->|"builds dataset"| node_vitxt_dataset
node_run -->|"starts training or evaluation"| node_trainer
node_trainer -->|"loads datasets"| node_loader
node_loader -->|"loads samples"| node_vitxt_dataset
node_loader -->|"batches samples"| node_collator
node_collator -->|"forms batch"| node_sample
node_trainer -->|"runs model"| node_t2s
node_sample -->|"provides batch"| node_t2s
node_t2s -->|"computes grounding"| node_grounding
node_trainer -->|"saves or restores"| node_checkpoint
node_t2s -->|"predicts answer and grounding"| node_answer
node_answer -->|"is evaluated"| node_evaluation
node_researcher -.->|"analyzes results"| node_evaluation

click node_video_prep "https://github.com/zhousheng97/vitxt-gqa/blob/main/tools/video_feat/obtain_vit_feat.py"
click node_vitxt_builder "https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/datasets/videoqa/vitxtgqa/builder.py"
click node_vitxt_dataset "https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/datasets/videoqa/vitxtgqa/dataset.py"
click node_loader "https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/common/dataset_loader.py"
click node_collator "https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/common/batch_collator.py"
click node_sample "https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/common/sample.py"
click node_run "https://github.com/zhousheng97/vitxt-gqa/blob/main/tools/run.py"
click node_trainer "https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/trainers/base_trainer.py"
click node_t2s "https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/models/t2s.py"
click node_grounding "https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/modules/spatio_temporal_grounding.py"
click node_checkpoint "https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/utils/checkpoint.py"
click node_evaluation "https://github.com/zhousheng97/vitxt-gqa/blob/main/tools/analysis/upper_bound.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_video_prep toneBlue
class node_vitxt_builder,node_vitxt_dataset toneAmber
class node_t2s,node_grounding toneMint
class node_loader,node_collator,node_sample,node_run,node_trainer,node_checkpoint toneRose
class node_evaluation,node_researcher,node_video,node_annotations,node_ocr,node_answer toneIndigo
```

## Components

### Data preparation

- [Video feature extraction](https://github.com/zhousheng97/vitxt-gqa/blob/main/tools/video_feat/obtain_vit_feat.py): `tools/video_feat/obtain_vit_feat.py`

### Dataset loading

- [ViTXT-GQA builder](https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/datasets/videoqa/vitxtgqa/builder.py): `pythia/datasets/videoqa/vitxtgqa/builder.py`
- [ViTXT-GQA dataset](https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/datasets/videoqa/vitxtgqa/dataset.py): `pythia/datasets/videoqa/vitxtgqa/dataset.py`

### QA runtime

- [T2S-QA model](https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/models/t2s.py): `pythia/models/t2s.py`
- [Spatio-temporal grounding](https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/modules/spatio_temporal_grounding.py): `pythia/modules/spatio_temporal_grounding.py`

### Pythia framework

- [Dataset loader](https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/common/dataset_loader.py): `pythia/common/dataset_loader.py`
- [Batch collator](https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/common/batch_collator.py): `pythia/common/batch_collator.py`
- [Batch representation](https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/common/sample.py): `pythia/common/sample.py`
- [Run entry](https://github.com/zhousheng97/vitxt-gqa/blob/main/tools/run.py): `tools/run.py`
- [Base trainer](https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/trainers/base_trainer.py): `pythia/trainers/base_trainer.py`
- [Checkpoint handling](https://github.com/zhousheng97/vitxt-gqa/blob/main/pythia/utils/checkpoint.py): `pythia/utils/checkpoint.py`

### Evaluation

- [Dataset evaluation](https://github.com/zhousheng97/vitxt-gqa/blob/main/tools/analysis/upper_bound.py): `tools/analysis/upper_bound.py`

### Other components

- **Researcher**
- **Raw video**
- **QA and grounding labels**
- **OCR results**
- **Answer and grounding**

## Connections

- Researcher → Run entry: starts run
- Raw video → Video feature extraction: extracts features
- Video feature extraction → ViTXT-GQA builder: provides features
- QA and grounding labels → ViTXT-GQA builder: provides labels
- OCR results → ViTXT-GQA builder: provides text
- ViTXT-GQA builder → ViTXT-GQA dataset: builds dataset
- Run entry → Base trainer: starts training or evaluation
- Base trainer → Dataset loader: loads datasets
- Dataset loader → ViTXT-GQA dataset: loads samples
- Dataset loader → Batch collator: batches samples
- Batch collator → Batch representation: forms batch
- Base trainer → T2S-QA model: runs model
- Batch representation → T2S-QA model: provides batch
- T2S-QA model → Spatio-temporal grounding: computes grounding
- Base trainer → Checkpoint handling: saves or restores
- T2S-QA model → Answer and grounding: predicts answer and grounding
- Answer and grounding → Dataset evaluation: is evaluated
- Researcher → Dataset evaluation: analyzes results