# markus0440/scenegraphvlm architecture

> Architecture diagram of the GitHub repository markus0440/scenegraphvlm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/markus0440/scenegraphvlm
- Repository: https://github.com/markus0440/scenegraphvlm

## Overview

SceneGraphVLM generates image and video scene graphs as TOON text, with support for PSG, PVSG, and Action Genome. The documented workflow prepares dataset annotations and variants, trains with SFT and optionally GRPO, runs model inference, evaluates predictions, and renders visual demos. The graph also includes PSFR key-frame selection as a distinct data-preparation branch. Training-to-inference and several subsystem links follow README descriptions; unsampled implementation details are not inferred.

Main entry points: [`datasets/data_playground`](https://github.com/markus0440/scenegraphvlm/tree/main/datasets/data_playground) (Prepared JSONL), [`metrics/qwen-bench/infer`](https://github.com/markus0440/scenegraphvlm/tree/main/metrics/qwen-bench/infer) (Qwen inference), [`grpo/swift/scripts`](https://github.com/markus0440/scenegraphvlm/tree/main/grpo/swift/scripts) (GRPO training), [`utils/PSFR/src/key_frame_selection.py`](https://github.com/markus0440/scenegraphvlm/blob/main/utils/PSFR/src/key_frame_selection.py) (Key-frame selection)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_dataset_tools["Dataset converters"]
  node_ag_helpers["AG graph helpers"]
  node_variant_builder["Annotation variants"]
  node_prepared_data[("Prepared JSONL")]
  node_psfr_filter["PSFR filter"]
  node_keyframe_select["Key-frame selection"]
  node_shi_tomasi["Shi–Tomasi features<br/>[shi_tomasi.py]"]
  node_lucas_kanade["Lucas–Kanade tracking<br/>[lucas_kanade.py]"]
end

subgraph group_learning["Model training"]
  node_sft["SFT training<br/>[train_scripts]"]
  node_grpo["GRPO training"]
  node_ag_rewards["AG graph rewards<br/>[ag_toon_reward.py]"]
  node_psg_pvsg_rewards["PSG/PVSG rewards"]
  node_checkpoints[("Model checkpoints")]
end

subgraph group_evaluation["Inference and evaluation"]
  node_inference["Qwen inference"]
  node_openrouter["OpenRouter inference"]
  node_qwen_eval["Qwen SGG evaluation"]
  node_sgbench["Classical SGG metrics"]
end

subgraph group_visualization["Visualization"]
  node_gt_video["Ground-truth video<br/>[build_gt_video.py]"]
  node_pred_video["Prediction video"]
  node_field_deploy["Field deployment"]
end

node_researcher(("Researcher"))
node_source_data[("Raw datasets")]

node_researcher -->|"prepares data"| node_dataset_tools
node_source_data -->|"provides annotations"| node_dataset_tools
node_dataset_tools -->|"builds variants"| node_variant_builder
node_ag_helpers -->|"supports parsing"| node_variant_builder
node_variant_builder -->|"writes JSONL"| node_prepared_data
node_prepared_data -->|"trains from data"| node_sft
node_prepared_data -->|"trains from data"| node_grpo
node_sft -->|"produces checkpoints"| node_checkpoints
node_checkpoints -->|"warm starts"| node_grpo
node_grpo -->|"uses rewards"| node_ag_rewards
node_grpo -->|"uses rewards"| node_psg_pvsg_rewards
node_prepared_data -->|"filters PVSG frames"| node_psfr_filter
node_psfr_filter -->|"selects keyframes"| node_keyframe_select
node_keyframe_select -->|"detects features"| node_shi_tomasi
node_keyframe_select -->|"tracks points"| node_lucas_kanade
node_checkpoints -->|"runs inference"| node_inference
node_prepared_data -->|"provides prompts"| node_inference
node_researcher -->|"runs evaluation"| node_openrouter
node_inference -.->|"evaluates predictions"| node_qwen_eval
node_inference -.->|"scores predictions"| node_sgbench
node_prepared_data -->|"renders annotations"| node_gt_video
node_inference -->|"renders predictions"| node_pred_video
node_inference -->|"deploys generation"| node_field_deploy

click node_dataset_tools "https://github.com/markus0440/scenegraphvlm/tree/main/datasets/annotations"
click node_ag_helpers "https://github.com/markus0440/scenegraphvlm/blob/main/datasets/tools/ag_scenegraph_common.py"
click node_variant_builder "https://github.com/markus0440/scenegraphvlm/blob/main/datasets/tools/build_annotation_variants.py"
click node_prepared_data "https://github.com/markus0440/scenegraphvlm/tree/main/datasets/data_playground"
click node_psfr_filter "https://github.com/markus0440/scenegraphvlm/blob/main/utils/PSFR/pvsg_psfr_filter.py"
click node_keyframe_select "https://github.com/markus0440/scenegraphvlm/blob/main/utils/PSFR/src/key_frame_selection.py"
click node_shi_tomasi "https://github.com/markus0440/scenegraphvlm/blob/main/utils/PSFR/src/shi_tomasi.py"
click node_lucas_kanade "https://github.com/markus0440/scenegraphvlm/blob/main/utils/PSFR/src/lucas_kanade.py"
click node_sft "https://github.com/markus0440/scenegraphvlm/tree/main/sft/Qwen3.5/train_scripts"
click node_grpo "https://github.com/markus0440/scenegraphvlm/tree/main/grpo/swift/scripts"
click node_ag_rewards "https://github.com/markus0440/scenegraphvlm/blob/main/grpo/swift/rewards/ag_toon_reward.py"
click node_psg_pvsg_rewards "https://github.com/markus0440/scenegraphvlm/blob/main/grpo/swift/rewards/pvsg_psg_toon_reward.py"
click node_checkpoints "https://github.com/markus0440/scenegraphvlm/tree/main/checkpoints"
click node_inference "https://github.com/markus0440/scenegraphvlm/tree/main/metrics/qwen-bench/infer"
click node_openrouter "https://github.com/markus0440/scenegraphvlm/tree/main/metrics/open-router"
click node_qwen_eval "https://github.com/markus0440/scenegraphvlm/blob/main/metrics/qwen-bench/eval/eval_sgg_metrics_with_qwen.py"
click node_sgbench "https://github.com/markus0440/scenegraphvlm/blob/main/metrics/sgbench/eval_classical_metrics_extended.py"
click node_gt_video "https://github.com/markus0440/scenegraphvlm/blob/main/visualization/video_demo_results/scripts/build_gt_video.py"
click node_pred_video "https://github.com/markus0440/scenegraphvlm/blob/main/visualization/video_demo_results/scripts/build_pred_video.py"
click node_field_deploy "https://github.com/markus0440/scenegraphvlm/blob/main/visualization/video_demo_results/scripts/field_gen_deploy.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset_tools,node_ag_helpers,node_variant_builder,node_prepared_data,node_psfr_filter,node_keyframe_select,node_shi_tomasi,node_lucas_kanade toneBlue
class node_sft,node_grpo,node_ag_rewards,node_psg_pvsg_rewards,node_checkpoints,node_source_data toneAmber
class node_inference,node_openrouter,node_qwen_eval,node_sgbench toneMint
class node_gt_video,node_pred_video,node_field_deploy toneRose
class node_researcher toneIndigo
```

## Components

### Data preparation

- [Dataset converters](https://github.com/markus0440/scenegraphvlm/tree/main/datasets/annotations): `datasets/annotations`
- [AG graph helpers](https://github.com/markus0440/scenegraphvlm/blob/main/datasets/tools/ag_scenegraph_common.py): `datasets/tools/ag_scenegraph_common.py`
- [Annotation variants](https://github.com/markus0440/scenegraphvlm/blob/main/datasets/tools/build_annotation_variants.py): `datasets/tools/build_annotation_variants.py`
- [Prepared JSONL](https://github.com/markus0440/scenegraphvlm/tree/main/datasets/data_playground): `datasets/data_playground`
- [PSFR filter](https://github.com/markus0440/scenegraphvlm/blob/main/utils/PSFR/pvsg_psfr_filter.py): `utils/PSFR/pvsg_psfr_filter.py`
- [Key-frame selection](https://github.com/markus0440/scenegraphvlm/blob/main/utils/PSFR/src/key_frame_selection.py): `utils/PSFR/src/key_frame_selection.py`
- [Shi–Tomasi features](https://github.com/markus0440/scenegraphvlm/blob/main/utils/PSFR/src/shi_tomasi.py): `utils/PSFR/src/shi_tomasi.py`
- [Lucas–Kanade tracking](https://github.com/markus0440/scenegraphvlm/blob/main/utils/PSFR/src/lucas_kanade.py): `utils/PSFR/src/lucas_kanade.py`

### Model training

- [SFT training](https://github.com/markus0440/scenegraphvlm/tree/main/sft/Qwen3.5/train_scripts): `sft/Qwen3.5/train_scripts`
- [GRPO training](https://github.com/markus0440/scenegraphvlm/tree/main/grpo/swift/scripts): `grpo/swift/scripts`
- [AG graph rewards](https://github.com/markus0440/scenegraphvlm/blob/main/grpo/swift/rewards/ag_toon_reward.py): `grpo/swift/rewards/ag_toon_reward.py`
- [PSG/PVSG rewards](https://github.com/markus0440/scenegraphvlm/blob/main/grpo/swift/rewards/pvsg_psg_toon_reward.py): `grpo/swift/rewards/pvsg_psg_toon_reward.py`
- [Model checkpoints](https://github.com/markus0440/scenegraphvlm/tree/main/checkpoints): `checkpoints`

### Inference and evaluation

- [Qwen inference](https://github.com/markus0440/scenegraphvlm/tree/main/metrics/qwen-bench/infer): `metrics/qwen-bench/infer`
- [OpenRouter inference](https://github.com/markus0440/scenegraphvlm/tree/main/metrics/open-router): `metrics/open-router`
- [Qwen SGG evaluation](https://github.com/markus0440/scenegraphvlm/blob/main/metrics/qwen-bench/eval/eval_sgg_metrics_with_qwen.py): `metrics/qwen-bench/eval/eval_sgg_metrics_with_qwen.py`
- [Classical SGG metrics](https://github.com/markus0440/scenegraphvlm/blob/main/metrics/sgbench/eval_classical_metrics_extended.py): `metrics/sgbench/eval_classical_metrics_extended.py`

### Visualization

- [Ground-truth video](https://github.com/markus0440/scenegraphvlm/blob/main/visualization/video_demo_results/scripts/build_gt_video.py): `visualization/video_demo_results/scripts/build_gt_video.py`
- [Prediction video](https://github.com/markus0440/scenegraphvlm/blob/main/visualization/video_demo_results/scripts/build_pred_video.py): `visualization/video_demo_results/scripts/build_pred_video.py`
- [Field deployment](https://github.com/markus0440/scenegraphvlm/blob/main/visualization/video_demo_results/scripts/field_gen_deploy.py): `visualization/video_demo_results/scripts/field_gen_deploy.py`

### Other components

- **Researcher**
- **Raw datasets**

## Connections

- Researcher → Dataset converters: prepares data (evidence: `README.md`)
- Raw datasets → Dataset converters: provides annotations (evidence: `README.md`)
- Dataset converters → Annotation variants: builds variants (evidence: `README.md`)
- AG graph helpers → Annotation variants: supports parsing (evidence: `datasets/tools/build_annotation_variants.py`)
- Annotation variants → Prepared JSONL: writes JSONL (evidence: `README.md`)
- Prepared JSONL → SFT training: trains from data (evidence: `README.md`)
- Prepared JSONL → GRPO training: trains from data (evidence: `README.md`)
- SFT training → Model checkpoints: produces checkpoints (evidence: `README.md`)
- Model checkpoints → GRPO training: warm starts (evidence: `README.md`)
- GRPO training → AG graph rewards: uses rewards (evidence: `README.md`)
- GRPO training → PSG/PVSG rewards: uses rewards (evidence: `README.md`)
- Prepared JSONL → PSFR filter: filters PVSG frames (evidence: `utils/PSFR/pvsg_psfr_filter.py`)
- PSFR filter → Key-frame selection: selects keyframes (evidence: `utils/PSFR/pvsg_psfr_filter.py`)
- Key-frame selection → Shi–Tomasi features: detects features (evidence: `utils/PSFR/src/key_frame_selection.py`)
- Key-frame selection → Lucas–Kanade tracking: tracks points (evidence: `utils/PSFR/src/key_frame_selection.py`)
- Model checkpoints → Qwen inference: runs inference (evidence: `README.md`)
- Prepared JSONL → Qwen inference: provides prompts (evidence: `README.md`)
- Researcher → OpenRouter inference: runs evaluation (evidence: `README.md`)
- Qwen inference → Qwen SGG evaluation: evaluates predictions
- Qwen inference → Classical SGG metrics: scores predictions
- Prepared JSONL → Ground-truth video: renders annotations (evidence: `README.md`)
- Qwen inference → Prediction video: renders predictions (evidence: `README.md`)
- Qwen inference → Field deployment: deploys generation (evidence: `README.md`)
