How bahyhelmihp/v2t-action-graph-knowledge works
This repository implements an action-graph video-captioning research system. Its principal workflow prepares CLIP and graph features from MSVD or MSR-VTT videos, loads those features with captions, then trains or evaluates a caption generator that combines visual, language, and graph representations. The README also describes both grid-based and object-based graph variants. The graph follows those documented stages; source excerpts establish the training entry point and model composition, but do not expose the internals of most feature-extraction notebooks.
Main entry points: main_task_caption_GNN.py Train/evaluate · modeling.py Caption generator · dataloader_msrvtt_caption.py Caption dataloaders · clip4clip_theta_2_feature_extraction.ipynb CLIP features
Components 21
Feature preparation
- CLIP features
feature_extractor/clip4clip_theta_2_feature_extraction.ipynb - Object/grid nodes
feature_extractor - Spatial action graph
feature_extractor - Temporal graph
feature_extractor/temporal_similarity_graph.ipynb - Graph conversion
feature_extractor/action_spatio_temporal_graph_feature_extractor.ipynb
Caption data
- Caption datasets
dataset - Caption dataloaders
dataloaders/dataloader_msrvtt_caption.py - Frame sampling
dataloaders/rawvideo_util.py
Caption modeling
- Caption generator
modules/modeling.py - Graph encoders
modules - BERT encoder
modules/module_bert.py - Visual encoder
modules/module_visual.py - Text decoder
modules/module_decoder.py - Caption tokenizer
modules/tokenization.py - Beam search
modules/beam.py
Training runtime
- Train/evaluate
main_task_caption_GNN.py - Optimizer
modules/optimization.py - Model utilities
modules/until_module.py
Other components
- Researcher
- Video datasets
- Pretrained weights
Connections 22
- Researcher → Video datasets: provides videos
README.md - Researcher → Pretrained weights: obtains weights
README.md - Video datasets → CLIP features: extracts features
README.md - Pretrained weights → CLIP features: initializes extraction
README.md - Video datasets → Object/grid nodes: extracts nodes
README.md - Object/grid nodes → Spatial action graph: builds relations
README.md - Spatial action graph → Temporal graph: adds temporal links
README.md - Temporal graph → Graph conversion: assembles graph
README.md - Video datasets → Frame sampling: samples frames
dataloaders/rawvideo_util.py - Caption datasets → Caption dataloaders: loads annotations
README.md - CLIP features → Caption dataloaders: loads features
README.md - Graph conversion → Caption dataloaders: loads graph features
README.md - Caption dataloaders → Train/evaluate: supplies batches
main_task_caption_GNN.py - Train/evaluate → Caption tokenizer: initializes tokenizer
main_task_caption_GNN.py - Train/evaluate → Caption generator: initializes model
main_task_caption_GNN.py - Train/evaluate → Optimizer: optimizes model
main_task_caption_GNN.py - Caption generator → BERT encoder: uses language encoder
modules/modeling.py - Caption generator → Visual encoder: uses visual encoder
modules/modeling.py - Caption generator → Text decoder: uses decoder
modules/modeling.py - Train/evaluate → Graph encoders: selects graph model
main_task_caption_GNN.py - Train/evaluate → Beam search: decodes predictions
main_task_caption_GNN.py - Caption generator → Model utilities: inherits model utilities
modules/modeling.py