# xiaoyuan1996/walkvlm architecture

> Architecture diagram of the GitHub repository xiaoyuan1996/walkvlm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xiaoyuan1996/walkvlm
- Repository: https://github.com/xiaoyuan1996/walkvlm

## Overview

WalkVLM is a walking-assistance research project whose maintained path is WalkStream: video/image inputs are used for reminder generation, while Online Triggered Prediction (OTP) suppresses redundant reminders over time. The repository also retains the WAD benchmark and an archived WalkVLM-LR research snapshot. Source excerpts establish OTP training/inference dependencies and the WalkStream query-generation script’s use of a cached GRPO trainer; broader capability links below are documented in the README. Several runtime details and the WalkStream README were not sampled, so those links are documented or left unverified.

Main entry points: [`WalkStream/src/otp/inference.py`](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/otp/inference.py) (OTP inference), [`WalkStream/src/wrd_dataset`](https://github.com/xiaoyuan1996/walkvlm/tree/main/WalkStream/src/wrd_dataset) (WRD dataset), [`WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_qwen25.py`](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_qwen25.py) (Qwen 2.5 trainer), [`WalkStream/src/reminder_generation/src/open_r1/grpo_query_gene.py`](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/grpo_query_gene.py) (Query generation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_walkstream["WalkStream"]
  node_otp_inference["OTP inference<br/>[inference.py]"]
  node_otp_model["Trigger classifier<br/>[otp_model.py]"]
  node_otp_training["OTP training<br/>[train.py]"]
end

subgraph group_reminder["Reminder training"]
  node_reminder_query["Query generation<br/>[grpo_query_gene.py]"]
  node_grpo["GRPO training<br/>[grpo.py]"]
  node_grpo_cache["Cached GRPO trainer"]
  node_grpo_cache_qwen["Qwen 2.5 trainer"]
  node_grpo_trainer["GRPO trainer<br/>[grpo_trainer.py]"]
  node_grpo_cache_2["Cached trainer variant"]
  node_qwen_model["Qwen 2 VL model"]
  node_qwen25_model["Qwen 2.5 VL model"]
  node_evaluate["Reminder evaluation<br/>[evaluate.py]"]
  node_generate["Reminder generation<br/>[generate.py]"]
end

subgraph group_benchmark["Datasets and evaluation"]
  node_gptscore["GPTScore helper<br/>[GPTScore.py]"]
  node_wrd[("WRD dataset")]
  node_wad[("WAD dataset")]
end

subgraph group_legacy["Archived WalkVLM-LR"]
  node_ead_train["EAD training<br/>[train_EAD.py]"]
  node_ead_model["EAD classifier<br/>[EAD.py]"]
  node_legacy_inference["Legacy inference<br/>[inference.py]"]
  node_legacy_grpo["Legacy GRPO<br/>[grpo.py]"]
  node_legacy_trainer["Legacy GRPO trainer<br/>[grpo_trainer.py]"]
end

node_walker(("Walking user"))
node_images["Video frames"]

node_walker -.->|"provides frames"| node_images
node_images -.->|"submits images"| node_otp_inference
node_otp_inference -->|"loads and predicts"| node_otp_model
node_otp_inference -.->|"gates reminders"| node_walker
node_otp_training -->|"trains classifier"| node_otp_model
node_wrd -.->|"supplies clips"| node_reminder_query
node_reminder_query -->|"uses trainer"| node_grpo_cache
node_grpo -.->|"optimizes reminders"| node_walker
node_grpo_cache_qwen -->|"uses model"| node_qwen_model
node_grpo_cache_qwen -->|"uses model"| node_qwen25_model
node_wad -.->|"benchmarks training"| node_legacy_grpo
node_ead_train -->|"trains classifier"| node_ead_model
node_legacy_inference -.->|"uses classifier"| node_ead_model
node_legacy_grpo -->|"uses trainer"| node_legacy_trainer
node_evaluate -.->|"evaluates on data"| node_wrd
node_generate -.->|"generates from data"| node_wrd

click node_otp_inference "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/otp/inference.py"
click node_otp_model "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/otp/otp_model.py"
click node_otp_training "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/otp/train.py"
click node_reminder_query "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/grpo_query_gene.py"
click node_grpo "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/grpo.py"
click node_grpo_cache "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache.py"
click node_grpo_cache_qwen "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_qwen25.py"
click node_grpo_trainer "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer.py"
click node_grpo_cache_2 "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_2.py"
click node_qwen_model "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/trainer/modeling_qwen2_vl.py"
click node_qwen25_model "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/trainer/modeling_qwen2_5_vl.py"
click node_evaluate "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/evaluate.py"
click node_generate "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/generate.py"
click node_gptscore "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/infer/GPTScore.py"
click node_wrd "https://github.com/xiaoyuan1996/walkvlm/tree/main/WalkStream/src/wrd_dataset"
click node_wad "https://github.com/xiaoyuan1996/walkvlm/tree/main/wad_dataset"
click node_ead_train "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkVLM-LR/train_EAD.py"
click node_ead_model "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkVLM-LR/EAD.py"
click node_legacy_inference "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkVLM-LR/inference.py"
click node_legacy_grpo "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkVLM-LR/vlm_grpo_template/src/open_r1/grpo.py"
click node_legacy_trainer "https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkVLM-LR/vlm_grpo_template/src/open_r1/trainer/grpo_trainer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_otp_inference,node_otp_model,node_otp_training,node_walker toneBlue
class node_reminder_query,node_grpo,node_grpo_cache,node_grpo_cache_qwen,node_grpo_trainer,node_grpo_cache_2,node_qwen_model,node_qwen25_model,node_evaluate,node_generate toneAmber
class node_gptscore,node_wrd,node_wad toneMint
class node_ead_train,node_ead_model,node_legacy_inference,node_legacy_grpo,node_legacy_trainer toneRose
class node_images toneIndigo
```

## Components

### WalkStream

- [OTP inference](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/otp/inference.py): `WalkStream/src/otp/inference.py`
- [Trigger classifier](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/otp/otp_model.py): `WalkStream/src/otp/otp_model.py`
- [OTP training](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/otp/train.py): `WalkStream/src/otp/train.py`

### Reminder training

- [Query generation](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/grpo_query_gene.py): `WalkStream/src/reminder_generation/src/open_r1/grpo_query_gene.py`
- [GRPO training](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/grpo.py): `WalkStream/src/reminder_generation/src/open_r1/grpo.py`
- [Cached GRPO trainer](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache.py): `WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache.py`
- [Qwen 2.5 trainer](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_qwen25.py): `WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_qwen25.py`
- [GRPO trainer](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer.py): `WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer.py`
- [Cached trainer variant](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_2.py): `WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_2.py`
- [Qwen 2 VL model](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/trainer/modeling_qwen2_vl.py): `WalkStream/src/reminder_generation/src/open_r1/trainer/modeling_qwen2_vl.py`
- [Qwen 2.5 VL model](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/trainer/modeling_qwen2_5_vl.py): `WalkStream/src/reminder_generation/src/open_r1/trainer/modeling_qwen2_5_vl.py`
- [Reminder evaluation](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/evaluate.py): `WalkStream/src/reminder_generation/src/open_r1/evaluate.py`
- [Reminder generation](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/reminder_generation/src/open_r1/generate.py): `WalkStream/src/reminder_generation/src/open_r1/generate.py`

### Datasets and evaluation

- [GPTScore helper](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkStream/src/infer/GPTScore.py): `WalkStream/src/infer/GPTScore.py`
- [WRD dataset](https://github.com/xiaoyuan1996/walkvlm/tree/main/WalkStream/src/wrd_dataset): `WalkStream/src/wrd_dataset`
- [WAD dataset](https://github.com/xiaoyuan1996/walkvlm/tree/main/wad_dataset): `wad_dataset`

### Archived WalkVLM-LR

- [EAD training](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkVLM-LR/train_EAD.py): `WalkVLM-LR/train_EAD.py`
- [EAD classifier](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkVLM-LR/EAD.py): `WalkVLM-LR/EAD.py`
- [Legacy inference](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkVLM-LR/inference.py): `WalkVLM-LR/inference.py`
- [Legacy GRPO](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkVLM-LR/vlm_grpo_template/src/open_r1/grpo.py): `WalkVLM-LR/vlm_grpo_template/src/open_r1/grpo.py`
- [Legacy GRPO trainer](https://github.com/xiaoyuan1996/walkvlm/blob/main/WalkVLM-LR/vlm_grpo_template/src/open_r1/trainer/grpo_trainer.py): `WalkVLM-LR/vlm_grpo_template/src/open_r1/trainer/grpo_trainer.py`

### Other components

- **Walking user**
- **Video frames**

## Connections

- Walking user → Video frames: provides frames (evidence: `README.md`)
- Video frames → OTP inference: submits images
- OTP inference → Trigger classifier: loads and predicts (evidence: `WalkStream/src/otp/inference.py`)
- OTP inference → Walking user: gates reminders (evidence: `README.md`)
- OTP training → Trigger classifier: trains classifier (evidence: `WalkStream/src/otp/train.py`)
- WRD dataset → Query generation: supplies clips (evidence: `README.md`)
- Query generation → Cached GRPO trainer: uses trainer (evidence: `WalkStream/src/reminder_generation/src/open_r1/grpo_query_gene.py`)
- GRPO training → Walking user: optimizes reminders (evidence: `README.md`)
- Qwen 2.5 trainer → Qwen 2 VL model: uses model (evidence: `WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_qwen25.py`)
- Qwen 2.5 trainer → Qwen 2.5 VL model: uses model (evidence: `WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_qwen25.py`)
- WAD dataset → Legacy GRPO: benchmarks training (evidence: `README.md`)
- EAD training → EAD classifier: trains classifier (evidence: `WalkVLM-LR/train_EAD.py`)
- Legacy inference → EAD classifier: uses classifier
- Legacy GRPO → Legacy GRPO trainer: uses trainer
- Reminder evaluation → WRD dataset: evaluates on data
- Reminder generation → WRD dataset: generates from data
