# zywang0104/video-ktr architecture

> Architecture diagram of the GitHub repository zywang0104/video-ktr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zywang0104/video-ktr
- Repository: https://github.com/zywang0104/video-ktr

## Overview

Video-KTR is a video-reasoning research framework. Its main workflow prepares video-question data, trains a vision-language model with GRPO and key-token reinforcement, then runs video inference and benchmark evaluation. The repository also includes an SFT training path and utilities for generating and processing reasoning data. Some source excerpts are partial, so the graph omits uncertain connections, including how the modified vLLM trainer is wired into GRPO training.

Main entry points: [`src/r1-v/local_scripts/create_vision_cot_data.py`](https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/local_scripts/create_vision_cot_data.py) (CoT generation), [`src/r1-v/local_scripts/prepare_hf_data.py`](https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/local_scripts/prepare_hf_data.py) (Dataset processing), [`src/r1-v/src/open_r1/sft_video.py`](https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/src/open_r1/sft_video.py) (Video SFT), [`src/r1-v/src/open_r1/grpo.py`](https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/src/open_r1/grpo.py) (GRPO training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_rawdata[("Reasoning data<br/>[prepare_hf_data.py]")]
  node_cotprep["CoT generation"]
  node_dataprep["Dataset processing<br/>[prepare_hf_data.py]"]
  node_trainingdata[("Video-R1 data")]
end

subgraph group_training["Model training"]
  node_sft["Video SFT<br/>[sft_video.py]"]
  node_grpo["GRPO training<br/>[grpo.py]"]
  node_ktr["KTR trainer<br/>[grpo_trainer.py]"]
  node_vllmtrainer["vLLM trainer"]
end

subgraph group_runtime["Inference"]
  node_model[("Video-KTR model")]
  node_vision["Vision processing<br/>[vision_process.py]"]
  node_inference["Example inference"]
  node_generation["CoT generation"]
end

subgraph group_evaluation["Evaluation"]
  node_benchrunner["Benchmark runner<br/>[eval_bench.py]"]
  node_evaluation["Model evaluation<br/>[evaluate.py]"]
  node_benchmarks[("Video benchmarks")]
end

node_researcher(("Researcher"))
node_azure{{"Azure OpenAI"}}
node_hub[("Dataset Hub")]

node_researcher -->|"requests CoT"| node_cotprep
node_cotprep -.->|"queries"| node_azure
node_cotprep -->|"produces"| node_rawdata
node_researcher -->|"prepares data"| node_dataprep
node_dataprep -->|"transforms"| node_rawdata
node_dataprep -->|"publishes"| node_hub
node_researcher -->|"starts SFT"| node_sft
node_sft -->|"loads data"| node_trainingdata
node_sft -->|"processes media"| node_vision
node_researcher -->|"starts GRPO"| node_grpo
node_grpo -->|"loads data"| node_trainingdata
node_grpo -->|"runs trainer"| node_ktr
node_ktr -->|"updates weights"| node_model
node_researcher -->|"asks question"| node_inference
node_inference -->|"processes video"| node_vision
node_inference -->|"generates answer"| node_model
node_researcher -->|"generates CoT"| node_generation
node_generation -->|"generates responses"| node_model
node_researcher -->|"runs benchmarks"| node_benchrunner
node_benchrunner -->|"invokes evaluation"| node_evaluation
node_evaluation -->|"evaluates on"| node_benchmarks
node_evaluation -->|"scores outputs"| node_model

click node_rawdata "https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/local_scripts/prepare_hf_data.py"
click node_cotprep "https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/local_scripts/create_vision_cot_data.py"
click node_dataprep "https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/local_scripts/prepare_hf_data.py"
click node_trainingdata "https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/video_ktr_data/Video-R1-Holmes-16k.json"
click node_sft "https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/src/open_r1/sft_video.py"
click node_grpo "https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/src/open_r1/grpo.py"
click node_ktr "https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/src/open_r1/trainer/grpo_trainer.py"
click node_vllmtrainer "https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/src/open_r1/trainer/vllm_grpo_trainer_modified.py"
click node_vision "https://github.com/zywang0104/video-ktr/blob/main/src/qwen-vl-utils/src/qwen_vl_utils/vision_process.py"
click node_inference "https://github.com/zywang0104/video-ktr/blob/main/src/inference_example.py"
click node_generation "https://github.com/zywang0104/video-ktr/blob/main/src/generate_cot_vllm.py"
click node_benchrunner "https://github.com/zywang0104/video-ktr/blob/main/src/eval_bench.py"
click node_evaluation "https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/src/open_r1/evaluate.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_rawdata,node_cotprep,node_dataprep,node_trainingdata toneBlue
class node_sft,node_grpo,node_ktr,node_vllmtrainer,node_hub toneAmber
class node_model,node_vision,node_inference,node_generation toneMint
class node_benchrunner,node_evaluation,node_benchmarks toneRose
class node_researcher,node_azure toneIndigo
```

## Components

### Data preparation

- [Reasoning data](https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/local_scripts/prepare_hf_data.py): `src/r1-v/local_scripts/prepare_hf_data.py`
- [CoT generation](https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/local_scripts/create_vision_cot_data.py): `src/r1-v/local_scripts/create_vision_cot_data.py`
- [Dataset processing](https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/local_scripts/prepare_hf_data.py): `src/r1-v/local_scripts/prepare_hf_data.py`
- [Video-R1 data](https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/video_ktr_data/Video-R1-Holmes-16k.json): `src/r1-v/video_ktr_data/Video-R1-Holmes-16k.json`

### Model training

- [Video SFT](https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/src/open_r1/sft_video.py): `src/r1-v/src/open_r1/sft_video.py`
- [GRPO training](https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/src/open_r1/grpo.py): `src/r1-v/src/open_r1/grpo.py`
- [KTR trainer](https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/src/open_r1/trainer/grpo_trainer.py): `src/r1-v/src/open_r1/trainer/grpo_trainer.py`
- [vLLM trainer](https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/src/open_r1/trainer/vllm_grpo_trainer_modified.py): `src/r1-v/src/open_r1/trainer/vllm_grpo_trainer_modified.py`

### Inference

- **Video-KTR model**
- [Vision processing](https://github.com/zywang0104/video-ktr/blob/main/src/qwen-vl-utils/src/qwen_vl_utils/vision_process.py): `src/qwen-vl-utils/src/qwen_vl_utils/vision_process.py`
- [Example inference](https://github.com/zywang0104/video-ktr/blob/main/src/inference_example.py): `src/inference_example.py`
- [CoT generation](https://github.com/zywang0104/video-ktr/blob/main/src/generate_cot_vllm.py): `src/generate_cot_vllm.py`

### Evaluation

- [Benchmark runner](https://github.com/zywang0104/video-ktr/blob/main/src/eval_bench.py): `src/eval_bench.py`
- [Model evaluation](https://github.com/zywang0104/video-ktr/blob/main/src/r1-v/src/open_r1/evaluate.py): `src/r1-v/src/open_r1/evaluate.py`
- **Video benchmarks**

### Other components

- **Researcher**
- **Azure OpenAI**
- **Dataset Hub**

## Connections

- Researcher → CoT generation: requests CoT
- CoT generation → Azure OpenAI: queries
- CoT generation → Reasoning data: produces
- Researcher → Dataset processing: prepares data
- Dataset processing → Reasoning data: transforms
- Dataset processing → Dataset Hub: publishes
- Researcher → Video SFT: starts SFT
- Video SFT → Video-R1 data: loads data
- Video SFT → Vision processing: processes media
- Researcher → GRPO training: starts GRPO
- GRPO training → Video-R1 data: loads data
- GRPO training → KTR trainer: runs trainer
- KTR trainer → Video-KTR model: updates weights
- Researcher → Example inference: asks question
- Example inference → Vision processing: processes video
- Example inference → Video-KTR model: generates answer
- Researcher → CoT generation: generates CoT
- CoT generation → Video-KTR model: generates responses
- Researcher → Benchmark runner: runs benchmarks
- Benchmark runner → Model evaluation: invokes evaluation
- Model evaluation → Video benchmarks: evaluates on
- Model evaluation → Video-KTR model: scores outputs