How markus0440/scenegraphvlm works
SceneGraphVLM generates image and video scene graphs as TOON text, with support for PSG, PVSG, and Action Genome. The documented workflow prepares dataset annotations and variants, trains with SFT and optionally GRPO, runs model inference, evaluates predictions, and renders visual demos. The graph also includes PSFR key-frame selection as a distinct data-preparation branch. Training-to-inference and several subsystem links follow README descriptions; unsampled implementation details are not inferred.
Main entry points: data_playground Prepared JSONL · infer Qwen inference · scripts GRPO training · key_frame_selection.py Key-frame selection
Components 22
Data preparation
- Dataset converters
datasets/annotations - AG graph helpers
datasets/tools/ag_scenegraph_common.py - Annotation variants
datasets/tools/build_annotation_variants.py - Prepared JSONL
datasets/data_playground - PSFR filter
utils/PSFR/pvsg_psfr_filter.py - Key-frame selection
utils/PSFR/src/key_frame_selection.py - Shi–Tomasi features
utils/PSFR/src/shi_tomasi.py - Lucas–Kanade tracking
utils/PSFR/src/lucas_kanade.py
Model training
- SFT training
sft/Qwen3.5/train_scripts - GRPO training
grpo/swift/scripts - AG graph rewards
grpo/swift/rewards/ag_toon_reward.py - PSG/PVSG rewards
grpo/swift/rewards/pvsg_psg_toon_reward.py - Model checkpoints
checkpoints
Inference and evaluation
- Qwen inference
metrics/qwen-bench/infer - OpenRouter inference
metrics/open-router - Qwen SGG evaluation
metrics/qwen-bench/eval/eval_sgg_metrics_with_qwen.py - Classical SGG metrics
metrics/sgbench/eval_classical_metrics_extended.py
Visualization
- Ground-truth video
visualization/video_demo_results/scripts/build_gt_video.py - Prediction video
visualization/video_demo_results/scripts/build_pred_video.py - Field deployment
visualization/video_demo_results/scripts/field_gen_deploy.py
Other components
- Researcher
- Raw datasets
Connections 23
- Researcher → Dataset converters: prepares data
README.md - Raw datasets → Dataset converters: provides annotations
README.md - Dataset converters → Annotation variants: builds variants
README.md - AG graph helpers → Annotation variants: supports parsing
datasets/tools/build_annotation_variants.py - Annotation variants → Prepared JSONL: writes JSONL
README.md - Prepared JSONL → SFT training: trains from data
README.md - Prepared JSONL → GRPO training: trains from data
README.md - SFT training → Model checkpoints: produces checkpoints
README.md - Model checkpoints → GRPO training: warm starts
README.md - GRPO training → AG graph rewards: uses rewards
README.md - GRPO training → PSG/PVSG rewards: uses rewards
README.md - Prepared JSONL → PSFR filter: filters PVSG frames
utils/PSFR/pvsg_psfr_filter.py - PSFR filter → Key-frame selection: selects keyframes
utils/PSFR/pvsg_psfr_filter.py - Key-frame selection → Shi–Tomasi features: detects features
utils/PSFR/src/key_frame_selection.py - Key-frame selection → Lucas–Kanade tracking: tracks points
utils/PSFR/src/key_frame_selection.py - Model checkpoints → Qwen inference: runs inference
README.md - Prepared JSONL → Qwen inference: provides prompts
README.md - Researcher → OpenRouter inference: runs evaluation
README.md - Qwen inference → Qwen SGG evaluation: evaluates predictions
- Qwen inference → Classical SGG metrics: scores predictions
- Prepared JSONL → Ground-truth video: renders annotations
README.md - Qwen inference → Prediction video: renders predictions
README.md - Qwen inference → Field deployment: deploys generation
README.md