# bahyhelmihp/v2t-action-graph-knowledge architecture

> Architecture diagram of the GitHub repository bahyhelmihp/v2t-action-graph-knowledge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bahyhelmihp/v2t-action-graph-knowledge
- Repository: https://github.com/bahyhelmihp/v2t-action-graph-knowledge

## Overview

This repository implements an action-graph video-captioning research system. Its principal workflow prepares CLIP and graph features from MSVD or MSR-VTT videos, loads those features with captions, then trains or evaluates a caption generator that combines visual, language, and graph representations. The README also describes both grid-based and object-based graph variants. The graph follows those documented stages; source excerpts establish the training entry point and model composition, but do not expose the internals of most feature-extraction notebooks.

Main entry points: [`main_task_caption_GNN.py`](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/main_task_caption_GNN.py) (Train/evaluate), [`modules/modeling.py`](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/modeling.py) (Caption generator), [`dataloaders/dataloader_msrvtt_caption.py`](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/dataloaders/dataloader_msrvtt_caption.py) (Caption dataloaders), [`feature_extractor/clip4clip_theta_2_feature_extraction.ipynb`](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/feature_extractor/clip4clip_theta_2_feature_extraction.ipynb) (CLIP features)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Feature preparation"]
  node_clip["CLIP features"]
  node_nodes["Object/grid nodes"]
  node_spatial["Spatial action graph"]
  node_temporal["Temporal graph"]
  node_graphbuild["Graph conversion"]
end

subgraph group_data["Caption data"]
  node_captions[("Caption datasets")]
  node_loader["Caption dataloaders"]
  node_videoio["Frame sampling<br/>[rawvideo_util.py]"]
end

subgraph group_model["Caption modeling"]
  node_generator["Caption generator<br/>[modeling.py]"]
  node_gnn["Graph encoders"]
  node_textencoder["BERT encoder<br/>[module_bert.py]"]
  node_visualencoder["Visual encoder<br/>[module_visual.py]"]
  node_decoder["Text decoder<br/>[module_decoder.py]"]
  node_tokenizer["Caption tokenizer<br/>[tokenization.py]"]
  node_beam["Beam search<br/>[beam.py]"]
end

subgraph group_runtime["Training runtime"]
  node_entry["Train/evaluate"]
  node_optimization["Optimizer<br/>[optimization.py]"]
  node_trainingutils["Model utilities<br/>[until_module.py]"]
end

node_researcher(("Researcher"))
node_videos[("Video datasets")]
node_pretrained[("Pretrained weights")]

node_researcher -->|"provides videos"| node_videos
node_researcher -->|"obtains weights"| node_pretrained
node_videos -->|"extracts features"| node_clip
node_pretrained -->|"initializes extraction"| node_clip
node_videos -->|"extracts nodes"| node_nodes
node_nodes -->|"builds relations"| node_spatial
node_spatial -->|"adds temporal links"| node_temporal
node_temporal -->|"assembles graph"| node_graphbuild
node_videos -->|"samples frames"| node_videoio
node_captions -->|"loads annotations"| node_loader
node_clip -->|"loads features"| node_loader
node_graphbuild -->|"loads graph features"| node_loader
node_loader -->|"supplies batches"| node_entry
node_entry -->|"initializes tokenizer"| node_tokenizer
node_entry -->|"initializes model"| node_generator
node_entry -->|"optimizes model"| node_optimization
node_generator -->|"uses language encoder"| node_textencoder
node_generator -->|"uses visual encoder"| node_visualencoder
node_generator -->|"uses decoder"| node_decoder
node_entry -->|"selects graph model"| node_gnn
node_entry -->|"decodes predictions"| node_beam
node_generator -->|"inherits model utilities"| node_trainingutils

click node_clip "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/feature_extractor/clip4clip_theta_2_feature_extraction.ipynb"
click node_nodes "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/tree/main/feature_extractor"
click node_spatial "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/tree/main/feature_extractor"
click node_temporal "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/feature_extractor/temporal_similarity_graph.ipynb"
click node_graphbuild "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/feature_extractor/action_spatio_temporal_graph_feature_extractor.ipynb"
click node_captions "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/tree/main/dataset"
click node_loader "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/dataloaders/dataloader_msrvtt_caption.py"
click node_videoio "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/dataloaders/rawvideo_util.py"
click node_entry "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/main_task_caption_GNN.py"
click node_generator "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/modeling.py"
click node_gnn "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/tree/main/modules"
click node_textencoder "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/module_bert.py"
click node_visualencoder "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/module_visual.py"
click node_decoder "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/module_decoder.py"
click node_tokenizer "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/tokenization.py"
click node_beam "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/beam.py"
click node_optimization "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/optimization.py"
click node_trainingutils "https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/until_module.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_clip,node_nodes,node_spatial,node_temporal,node_graphbuild toneBlue
class node_captions,node_loader,node_videoio,node_videos,node_pretrained toneAmber
class node_generator,node_gnn,node_textencoder,node_visualencoder,node_decoder,node_tokenizer,node_beam toneMint
class node_entry,node_optimization,node_trainingutils toneRose
class node_researcher toneIndigo
```

## Components

### Feature preparation

- [CLIP features](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/feature_extractor/clip4clip_theta_2_feature_extraction.ipynb): `feature_extractor/clip4clip_theta_2_feature_extraction.ipynb`
- [Object/grid nodes](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/tree/main/feature_extractor): `feature_extractor`
- [Spatial action graph](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/tree/main/feature_extractor): `feature_extractor`
- [Temporal graph](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/feature_extractor/temporal_similarity_graph.ipynb): `feature_extractor/temporal_similarity_graph.ipynb`
- [Graph conversion](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/feature_extractor/action_spatio_temporal_graph_feature_extractor.ipynb): `feature_extractor/action_spatio_temporal_graph_feature_extractor.ipynb`

### Caption data

- [Caption datasets](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/tree/main/dataset): `dataset`
- [Caption dataloaders](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/dataloaders/dataloader_msrvtt_caption.py): `dataloaders/dataloader_msrvtt_caption.py`
- [Frame sampling](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/dataloaders/rawvideo_util.py): `dataloaders/rawvideo_util.py`

### Caption modeling

- [Caption generator](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/modeling.py): `modules/modeling.py`
- [Graph encoders](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/tree/main/modules): `modules`
- [BERT encoder](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/module_bert.py): `modules/module_bert.py`
- [Visual encoder](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/module_visual.py): `modules/module_visual.py`
- [Text decoder](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/module_decoder.py): `modules/module_decoder.py`
- [Caption tokenizer](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/tokenization.py): `modules/tokenization.py`
- [Beam search](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/beam.py): `modules/beam.py`

### Training runtime

- [Train/evaluate](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/main_task_caption_GNN.py): `main_task_caption_GNN.py`
- [Optimizer](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/optimization.py): `modules/optimization.py`
- [Model utilities](https://github.com/bahyhelmihp/v2t-action-graph-knowledge/blob/main/modules/until_module.py): `modules/until_module.py`

### Other components

- **Researcher**
- **Video datasets**
- **Pretrained weights**

## Connections

- Researcher → Video datasets: provides videos (evidence: `README.md`)
- Researcher → Pretrained weights: obtains weights (evidence: `README.md`)
- Video datasets → CLIP features: extracts features (evidence: `README.md`)
- Pretrained weights → CLIP features: initializes extraction (evidence: `README.md`)
- Video datasets → Object/grid nodes: extracts nodes (evidence: `README.md`)
- Object/grid nodes → Spatial action graph: builds relations (evidence: `README.md`)
- Spatial action graph → Temporal graph: adds temporal links (evidence: `README.md`)
- Temporal graph → Graph conversion: assembles graph (evidence: `README.md`)
- Video datasets → Frame sampling: samples frames (evidence: `dataloaders/rawvideo_util.py`)
- Caption datasets → Caption dataloaders: loads annotations (evidence: `README.md`)
- CLIP features → Caption dataloaders: loads features (evidence: `README.md`)
- Graph conversion → Caption dataloaders: loads graph features (evidence: `README.md`)
- Caption dataloaders → Train/evaluate: supplies batches (evidence: `main_task_caption_GNN.py`)
- Train/evaluate → Caption tokenizer: initializes tokenizer (evidence: `main_task_caption_GNN.py`)
- Train/evaluate → Caption generator: initializes model (evidence: `main_task_caption_GNN.py`)
- Train/evaluate → Optimizer: optimizes model (evidence: `main_task_caption_GNN.py`)
- Caption generator → BERT encoder: uses language encoder (evidence: `modules/modeling.py`)
- Caption generator → Visual encoder: uses visual encoder (evidence: `modules/modeling.py`)
- Caption generator → Text decoder: uses decoder (evidence: `modules/modeling.py`)
- Train/evaluate → Graph encoders: selects graph model (evidence: `main_task_caption_GNN.py`)
- Train/evaluate → Beam search: decodes predictions (evidence: `main_task_caption_GNN.py`)
- Caption generator → Model utilities: inherits model utilities (evidence: `modules/modeling.py`)