# rohithpeddi/scenesayer architecture

> Architecture diagram of the GitHub repository rohithpeddi/scenesayer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rohithpeddi/scenesayer
- Repository: https://github.com/rohithpeddi/scenesayer

## Overview

SceneSayer is a research system for anticipating future Action Genome scene graphs. Its main workflow prepares video-frame annotations, loads scene data, extracts object detections and features, then trains or evaluates scene-graph and anticipation models in sgdet, sgcls, or predcls settings. Model families include STTran/DSG-DETR variants and ODE/SDE approaches; evaluation reports recall metrics, while a separate results workflow can retrieve and format experiment results. The graph follows documented capabilities and indexed references; some unsampled model and preprocessing internals are represented at their exact enclosing directories.

Main entry points: [`train_sga_base.py`](https://github.com/rohithpeddi/scenesayer/blob/main/train_sga_base.py) (Training Entry), [`lib/object_detector.py`](https://github.com/rohithpeddi/scenesayer/blob/main/lib/object_detector.py) (Object Detection), [`lib/supervised/sga/blocks.py`](https://github.com/rohithpeddi/scenesayer/blob/main/lib/supervised/sga/blocks.py) (Transformer Blocks), [`analysis/eccv/prepare_paper_results.py`](https://github.com/rohithpeddi/scenesayer/blob/main/analysis/eccv/prepare_paper_results.py) (Result Analysis)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Action Genome Data"]
  node_frame_tools["Frame Preparation"]
  node_preprocessing["Feature Preprocessing"]
  node_ag_dataset["Scene Dataset<br/>[ag_dataset.py]"]
end

subgraph group_vision["Visual Perception"]
  node_detector["Object Detection<br/>[object_detector.py]"]
  node_faster_rcnn["Faster R-CNN<br/>[resnet.py]"]
  node_roi_nms["ROI and NMS<br/>[nms.py]"]
  node_box_ops["Box Operations<br/>[box_utils.py]"]
end

subgraph group_models["Scene Graph Models"]
  node_trainer["Training Entry<br/>[train_sga_base.py]"]
  node_word_vectors["Object Embeddings<br/>[word_vectors.py]"]
  node_transformer_blocks["Transformer Blocks<br/>[blocks.py]"]
  node_baseline_anticipation["Baseline Anticipation"]
  node_sga_methods["Anticipation Methods"]
  node_dsgdetr["DSG-DETR<br/>[dsgdetr.py]"]
end

subgraph group_results["Evaluation Results"]
  node_evaluation["Recall Evaluation"]
  node_result_analysis["Result Analysis"]
  node_result_model["Result Records<br/>[Result.py]"]
  node_firebase["Firebase Service<br/>[FirebaseService.py]"]
end

node_researcher(("Researcher"))
node_action_genome[("Action Genome")]
node_firebase_db[("Firebase Database")]

node_researcher -->|"runs training"| node_trainer
node_action_genome -->|"provides videos"| node_frame_tools
node_frame_tools -.->|"prepares frames"| node_ag_dataset
node_action_genome -.->|"provides annotations"| node_preprocessing
node_trainer -->|"loads scenes"| node_ag_dataset
node_trainer -->|"uses detector"| node_detector
node_detector -->|"runs backbone"| node_faster_rcnn
node_detector -->|"filters boxes"| node_roi_nms
node_trainer -->|"selects method"| node_sga_methods
node_trainer -.->|"trains variants"| node_dsgdetr
node_trainer -.->|"trains baseline"| node_baseline_anticipation
node_baseline_anticipation -->|"uses blocks"| node_transformer_blocks
node_transformer_blocks -->|"embeds objects"| node_word_vectors
node_transformer_blocks -->|"uses geometry"| node_box_ops
node_trainer -->|"evaluates predictions"| node_evaluation
node_researcher -.->|"reviews results"| node_result_analysis
node_result_analysis -->|"fetches results"| node_firebase
node_result_analysis -->|"parses records"| node_result_model
node_firebase -->|"reads and writes"| node_firebase_db

click node_frame_tools "https://github.com/rohithpeddi/scenesayer/tree/main/datasets/action_genome/tools"
click node_preprocessing "https://github.com/rohithpeddi/scenesayer/tree/main/datasets/action_genome/preprocess"
click node_ag_dataset "https://github.com/rohithpeddi/scenesayer/blob/main/dataloader/action_genome/ag_dataset.py"
click node_trainer "https://github.com/rohithpeddi/scenesayer/blob/main/train_sga_base.py"
click node_detector "https://github.com/rohithpeddi/scenesayer/blob/main/lib/object_detector.py"
click node_faster_rcnn "https://github.com/rohithpeddi/scenesayer/blob/main/fasterRCNN/lib/model/faster_rcnn/resnet.py"
click node_roi_nms "https://github.com/rohithpeddi/scenesayer/blob/main/fasterRCNN/lib/model/roi_layers/nms.py"
click node_box_ops "https://github.com/rohithpeddi/scenesayer/blob/main/lib/fpn/box_utils.py"
click node_word_vectors "https://github.com/rohithpeddi/scenesayer/blob/main/lib/word_vectors.py"
click node_transformer_blocks "https://github.com/rohithpeddi/scenesayer/blob/main/lib/supervised/sga/blocks.py"
click node_baseline_anticipation "https://github.com/rohithpeddi/scenesayer/blob/main/0_dump/baseline_anticipation.py"
click node_sga_methods "https://github.com/rohithpeddi/scenesayer/tree/main/lib/supervised/sga"
click node_dsgdetr "https://github.com/rohithpeddi/scenesayer/blob/main/lib/supervised/sgg/dsgdetr/dsgdetr.py"
click node_evaluation "https://github.com/rohithpeddi/scenesayer/blob/main/lib/supervised/evaluation_recall.py"
click node_result_analysis "https://github.com/rohithpeddi/scenesayer/blob/main/analysis/eccv/prepare_paper_results.py"
click node_result_model "https://github.com/rohithpeddi/scenesayer/blob/main/analysis/results/Result.py"
click node_firebase "https://github.com/rohithpeddi/scenesayer/blob/main/analysis/results/FirebaseService.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frame_tools,node_preprocessing,node_ag_dataset toneBlue
class node_detector,node_faster_rcnn,node_roi_nms,node_box_ops,node_action_genome,node_firebase_db toneAmber
class node_trainer,node_word_vectors,node_transformer_blocks,node_baseline_anticipation,node_sga_methods,node_dsgdetr toneMint
class node_evaluation,node_result_analysis,node_result_model,node_firebase toneRose
class node_researcher toneIndigo
```

## Components

### Action Genome Data

- [Frame Preparation](https://github.com/rohithpeddi/scenesayer/tree/main/datasets/action_genome/tools): `datasets/action_genome/tools`
- [Feature Preprocessing](https://github.com/rohithpeddi/scenesayer/tree/main/datasets/action_genome/preprocess): `datasets/action_genome/preprocess`
- [Scene Dataset](https://github.com/rohithpeddi/scenesayer/blob/main/dataloader/action_genome/ag_dataset.py): `dataloader/action_genome/ag_dataset.py`

### Visual Perception

- [Object Detection](https://github.com/rohithpeddi/scenesayer/blob/main/lib/object_detector.py): `lib/object_detector.py`
- [Faster R-CNN](https://github.com/rohithpeddi/scenesayer/blob/main/fasterRCNN/lib/model/faster_rcnn/resnet.py): `fasterRCNN/lib/model/faster_rcnn/resnet.py`
- [ROI and NMS](https://github.com/rohithpeddi/scenesayer/blob/main/fasterRCNN/lib/model/roi_layers/nms.py): `fasterRCNN/lib/model/roi_layers/nms.py`
- [Box Operations](https://github.com/rohithpeddi/scenesayer/blob/main/lib/fpn/box_utils.py): `lib/fpn/box_utils.py`

### Scene Graph Models

- [Training Entry](https://github.com/rohithpeddi/scenesayer/blob/main/train_sga_base.py): `train_sga_base.py`
- [Object Embeddings](https://github.com/rohithpeddi/scenesayer/blob/main/lib/word_vectors.py): `lib/word_vectors.py`
- [Transformer Blocks](https://github.com/rohithpeddi/scenesayer/blob/main/lib/supervised/sga/blocks.py): `lib/supervised/sga/blocks.py`
- [Baseline Anticipation](https://github.com/rohithpeddi/scenesayer/blob/main/0_dump/baseline_anticipation.py): `0_dump/baseline_anticipation.py`
- [Anticipation Methods](https://github.com/rohithpeddi/scenesayer/tree/main/lib/supervised/sga): `lib/supervised/sga`
- [DSG-DETR](https://github.com/rohithpeddi/scenesayer/blob/main/lib/supervised/sgg/dsgdetr/dsgdetr.py): `lib/supervised/sgg/dsgdetr/dsgdetr.py`

### Evaluation Results

- [Recall Evaluation](https://github.com/rohithpeddi/scenesayer/blob/main/lib/supervised/evaluation_recall.py): `lib/supervised/evaluation_recall.py`
- [Result Analysis](https://github.com/rohithpeddi/scenesayer/blob/main/analysis/eccv/prepare_paper_results.py): `analysis/eccv/prepare_paper_results.py`
- [Result Records](https://github.com/rohithpeddi/scenesayer/blob/main/analysis/results/Result.py): `analysis/results/Result.py`
- [Firebase Service](https://github.com/rohithpeddi/scenesayer/blob/main/analysis/results/FirebaseService.py): `analysis/results/FirebaseService.py`

### Other components

- **Researcher**
- **Action Genome**
- **Firebase Database**

## Connections

- Researcher → Training Entry: runs training (evidence: `README.md`)
- Action Genome → Frame Preparation: provides videos (evidence: `README.md`)
- Frame Preparation → Scene Dataset: prepares frames
- Action Genome → Feature Preprocessing: provides annotations
- Training Entry → Scene Dataset: loads scenes (evidence: `train_sga_base.py`)
- Training Entry → Object Detection: uses detector (evidence: `train_sga_base.py`)
- Object Detection → Faster R-CNN: runs backbone (evidence: `lib/object_detector.py`)
- Object Detection → ROI and NMS: filters boxes (evidence: `lib/object_detector.py`)
- Training Entry → Anticipation Methods: selects method (evidence: `README.md`)
- Training Entry → DSG-DETR: trains variants
- Training Entry → Baseline Anticipation: trains baseline
- Baseline Anticipation → Transformer Blocks: uses blocks (evidence: `0_dump/baseline_anticipation.py`)
- Transformer Blocks → Object Embeddings: embeds objects (evidence: `lib/supervised/sga/blocks.py`)
- Transformer Blocks → Box Operations: uses geometry (evidence: `lib/supervised/sga/blocks.py`)
- Training Entry → Recall Evaluation: evaluates predictions
- Researcher → Result Analysis: reviews results
- Result Analysis → Firebase Service: fetches results (evidence: `analysis/eccv/prepare_paper_results.py`)
- Result Analysis → Result Records: parses records (evidence: `analysis/eccv/prepare_paper_results.py`)
- Firebase Service → Firebase Database: reads and writes (evidence: `analysis/results/FirebaseService.py`)
