How xiaoyuan1996/walkvlm works
WalkVLM is a walking-assistance research project whose maintained path is WalkStream: video/image inputs are used for reminder generation, while Online Triggered Prediction (OTP) suppresses redundant reminders over time. The repository also retains the WAD benchmark and an archived WalkVLM-LR research snapshot. Source excerpts establish OTP training/inference dependencies and the WalkStream query-generation script’s use of a cached GRPO trainer; broader capability links below are documented in the README. Several runtime details and the WalkStream README were not sampled, so those links are documented or left unverified.
Main entry points: inference.py OTP inference · wrd_dataset WRD dataset · grpo_trainer_cache_qwen25.py Qwen 2.5 trainer · grpo_query_gene.py Query generation
Components 23
WalkStream
- OTP inference
WalkStream/src/otp/inference.py - Trigger classifier
WalkStream/src/otp/otp_model.py - OTP training
WalkStream/src/otp/train.py
Reminder training
- Query generation
WalkStream/src/reminder_generation/src/open_r1/grpo_query_gene.py - GRPO training
WalkStream/src/reminder_generation/src/open_r1/grpo.py - Cached GRPO trainer
WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache.py - Qwen 2.5 trainer
WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_qwen25.py - GRPO trainer
WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer.py - Cached trainer variant
WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_2.py - Qwen 2 VL model
WalkStream/src/reminder_generation/src/open_r1/trainer/modeling_qwen2_vl.py - Qwen 2.5 VL model
WalkStream/src/reminder_generation/src/open_r1/trainer/modeling_qwen2_5_vl.py - Reminder evaluation
WalkStream/src/reminder_generation/src/open_r1/evaluate.py - Reminder generation
WalkStream/src/reminder_generation/src/open_r1/generate.py
Datasets and evaluation
- GPTScore helper
WalkStream/src/infer/GPTScore.py - WRD dataset
WalkStream/src/wrd_dataset - WAD dataset
wad_dataset
Archived WalkVLM-LR
- EAD training
WalkVLM-LR/train_EAD.py - EAD classifier
WalkVLM-LR/EAD.py - Legacy inference
WalkVLM-LR/inference.py - Legacy GRPO
WalkVLM-LR/vlm_grpo_template/src/open_r1/grpo.py - Legacy GRPO trainer
WalkVLM-LR/vlm_grpo_template/src/open_r1/trainer/grpo_trainer.py
Other components
- Walking user
- Video frames
Connections 16
- Walking user → Video frames: provides frames
README.md - Video frames → OTP inference: submits images
- OTP inference → Trigger classifier: loads and predicts
WalkStream/src/otp/inference.py - OTP inference → Walking user: gates reminders
README.md - OTP training → Trigger classifier: trains classifier
WalkStream/src/otp/train.py - WRD dataset → Query generation: supplies clips
README.md - Query generation → Cached GRPO trainer: uses trainer
WalkStream/src/reminder_generation/src/open_r1/grpo_query_gene.py - GRPO training → Walking user: optimizes reminders
README.md - Qwen 2.5 trainer → Qwen 2 VL model: uses model
WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_qwen25.py - Qwen 2.5 trainer → Qwen 2.5 VL model: uses model
WalkStream/src/reminder_generation/src/open_r1/trainer/grpo_trainer_cache_qwen25.py - WAD dataset → Legacy GRPO: benchmarks training
README.md - EAD training → EAD classifier: trains classifier
WalkVLM-LR/train_EAD.py - Legacy inference → EAD classifier: uses classifier
- Legacy GRPO → Legacy GRPO trainer: uses trainer
- Reminder evaluation → WRD dataset: evaluates on data
- Reminder generation → WRD dataset: generates from data