How zywang0104/video-ktr works
Video-KTR is a video-reasoning research framework. Its main workflow prepares video-question data, trains a vision-language model with GRPO and key-token reinforcement, then runs video inference and benchmark evaluation. The repository also includes an SFT training path and utilities for generating and processing reasoning data. Some source excerpts are partial, so the graph omits uncertain connections, including how the modified vLLM trainer is wired into GRPO training.
Main entry points: create_vision_cot_data.py CoT generation · prepare_hf_data.py Dataset processing · sft_video.py Video SFT · grpo.py GRPO training
Components 18
Data preparation
- Reasoning data
src/r1-v/local_scripts/prepare_hf_data.py - CoT generation
src/r1-v/local_scripts/create_vision_cot_data.py - Dataset processing
src/r1-v/local_scripts/prepare_hf_data.py - Video-R1 data
src/r1-v/video_ktr_data/Video-R1-Holmes-16k.json
Model training
- Video SFT
src/r1-v/src/open_r1/sft_video.py - GRPO training
src/r1-v/src/open_r1/grpo.py - KTR trainer
src/r1-v/src/open_r1/trainer/grpo_trainer.py - vLLM trainer
src/r1-v/src/open_r1/trainer/vllm_grpo_trainer_modified.py
Inference
- Video-KTR model
- Vision processing
src/qwen-vl-utils/src/qwen_vl_utils/vision_process.py - Example inference
src/inference_example.py - CoT generation
src/generate_cot_vllm.py
Evaluation
- Benchmark runner
src/eval_bench.py - Model evaluation
src/r1-v/src/open_r1/evaluate.py - Video benchmarks
Other components
- Researcher
- Azure OpenAI
- Dataset Hub
Connections 22
- Researcher → CoT generation: requests CoT
- CoT generation → Azure OpenAI: queries
- CoT generation → Reasoning data: produces
- Researcher → Dataset processing: prepares data
- Dataset processing → Reasoning data: transforms
- Dataset processing → Dataset Hub: publishes
- Researcher → Video SFT: starts SFT
- Video SFT → Video-R1 data: loads data
- Video SFT → Vision processing: processes media
- Researcher → GRPO training: starts GRPO
- GRPO training → Video-R1 data: loads data
- GRPO training → KTR trainer: runs trainer
- KTR trainer → Video-KTR model: updates weights
- Researcher → Example inference: asks question
- Example inference → Vision processing: processes video
- Example inference → Video-KTR model: generates answer
- Researcher → CoT generation: generates CoT
- CoT generation → Video-KTR model: generates responses
- Researcher → Benchmark runner: runs benchmarks
- Benchmark runner → Model evaluation: invokes evaluation
- Model evaluation → Video benchmarks: evaluates on
- Model evaluation → Video-KTR model: scores outputs