How mcg-nju/timelens2 works
Purpose and Top-Level Shape
TimeLens2 trains and evaluates multimodal LLMs that map a video plus natural-language query to one or more temporal evidence intervals. This is a Python monorepo organized as a sequential ML pipeline: supervised fine-tuning (SFT), offline rollout plus GRPO reinforcement learning, then benchmark evaluation. A static documentation site is separately published from docs/.
Primary Entry Points
Reproduction is shell-driven rather than service-driven:
• SFT launches: sft/scripts/train_sft_4b.sh, sft/scripts/train_sft_8b.sh
• Rollout generation: grpo/scripts/rollout_timelens2.sh
• GRPO training: grpo/scripts/train_grpo_4b.sh, grpo/scripts/train_grpo_8b.sh
• Grounding evaluation: evaluation/scripts/srun_eval_all/run_grounding.sh
These consume configurable paths, local annotations/rollouts, downloaded video assets, and pretrained or SFT model checkpoints.
SFT Data-to-Checkpoint Stage
The sft/ module is an XTuner-derived training stack for video-language SFT. TimeLens2-specific configurations and annotations select Qwen3-VL 4B/8B training recipes and labeled TimeLens2 data:
• sft/configs/qwen3_vl_4b_sft.py
• sft/configs/data/timelens2_sft.json
• sft/training_data_annotations/timelens2-93k/timelens2-93k_long.jsonl
The modern engine under sft/xtuner/v1/ provides typed data records, multimodal tokenization, model construction, loss, distributed training, checkpoints, and writers. Key boundaries are raw JSONL/video references → tokenized batches → model forward/loss → checkpoint artifacts:
• sft/xtuner/v1/datasets/dataloader.py
• sft/xtuner/v1/datasets/mllm_tokenize_fn/qwen3_vl_tokenize_fn.py
• sft/xtuner/v1/train/trainer.py
Model and Distributed Compute Layer
SFT supports dense, MoE, and composed vision-language architectures; TimeLens2 configurations target Qwen3-VL. The composition layer joins vision encoders/projectors with language models, while the engine handles GPU/NPU operations and distributed state:
• sft/xtuner/v1/model/compose/qwen3_vl/modeling_qwen3_vl.py
• sft/xtuner/v1/engine/vision_compose_train_engine.py
• sft/xtuner/v1/ops/flash_attn/
Material runtime dependencies are Python, PyTorch, Transformers-style model checkpoints, CUDA or NPU kernels, Triton kernels, and distributed collectives. Legacy XTuner integration also exposes DeepSpeed strategy support in sft/xtuner/engine/_strategy/deepspeed.py.
GRPO Rollout and Optimization Stage
grpo/ performs off-policy temporal-grounding optimization after SFT. It reads annotated samples and/or generated completions, renders prompts and video inputs, computes interval-aware rewards, then updates the policy:
• grpo/timelens/rollout.py
• grpo/timelens/train.py
• grpo/timelens/trainer.py
The principal data boundary is SFT checkpoint plus video/query samples → rollout JSONL → reward-scored training batches → GRPO checkpoint. Bundled sources and rollouts make this reproducible without regenerating inference:
• grpo/timelens/data/rollout_dataset.py
• grpo/data/rollouts/timelens2-93k.jsonl
• grpo/timelens/rewards/metrics.py
Reward logic is specialized to temporal intervals, including temporal IoU and matching-free temporal Wasserstein-style scoring:
• grpo/timelens/rewards/funcs.py
• grpo/timelens/rewards/sampling.py
Evaluation Adapter and Benchmark Layer
evaluation/ is a VLMEvalKit-derived evaluator. evaluation/run.py is the main Python launcher; its config resolves model adapters and dataset implementations. Image/video benchmark abstractions normalize samples and scoring, while temporal-grounding execution and reporting are handled by the grounding scripts:
• evaluation/run.py
• evaluation/vlmeval/config.py
• evaluation/run_grounding.sh
Model access is pluggable across local model implementations and hosted APIs. Local adapters reside in evaluation/vlmeval/vlm/; remote-provider adapters reside in evaluation/vlmeval/api/, including OpenAI/GPT, Gemini, Claude, and LMDeploy paths.
Infrastructure and External Boundaries
The repository does not implement an online application or persistent backend. External dependencies are model checkpoints and datasets hosted on Hugging Face, separately downloaded video corpora, GPU/NPU training infrastructure, and optional inference services such as LMDeploy, SGLang, or vLLM through sft/xtuner/v1/ray/rollout/. GitHub Actions publish the static project site via .github/workflows/pages.yml.
Components 22
SFT pipeline
Video-language supervised fine-tuning.
- SFT launcher · Launches Qwen3-VL SFT recipes.
sft/scripts/train_sft_4b.sh - Qwen3-VL SFT recipe
sft/configs/qwen3_vl_4b_sft.py - SFT annotations
sft/training_data_annotations/timelens2-93k/timelens2-93k_long.jsonl - SFT data manifest
sft/configs/data/timelens2_sft.json - Multimodal tokenization · Converts video and query records into model inputs.
sft/xtuner/v1/datasets/mllm_tokenize_fn/qwen3_vl_tokenize_fn.py - Tokenized batches
sft/xtuner/v1/datasets/dataloader.py - Qwen3-VL composition · Joins vision components with the language model.
sft/xtuner/v1/model/compose/qwen3_vl/modeling_qwen3_vl.py - SFT trainer & checkpoints · Runs loss, distributed training, and checkpoint writing.
sft/xtuner/v1/train/trainer.py - SFT checkpoint
GRPO pipeline
Offline rollout and policy optimization.
- Rollout launcher
grpo/scripts/rollout_timelens2.sh - Video-query rollouts · Renders prompts and generates temporal completions.
grpo/timelens/rollout.py - Rollout JSONL · Bundled generated completions for reproducible optimization.
grpo/data/rollouts/timelens2-93k.jsonl - Temporal interval rewards · Uses temporal IoU and Wasserstein-style scoring.
grpo/timelens/rewards/metrics.py - GRPO optimizer · Builds reward-scored batches and updates the policy.
grpo/timelens/trainer.py - GRPO checkpoint
Evaluation
Benchmark adapters and temporal-grounding reporting.
- Grounding evaluation launcher
evaluation/scripts/srun_eval_all/run_grounding.sh - Evaluation runner · Resolves models, datasets, execution, and scoring.
evaluation/run.py - Model and dataset adapters · Selects local or hosted model adapters and benchmark implementations.
evaluation/vlmeval/config.py
Other components
- Videos, datasets & base models · Downloaded corpora and pretrained checkpoints.
- GPU/NPU distributed runtime · PyTorch, kernels, and distributed collectives.
- Hosted inference providers · Optional OpenAI, Gemini, Claude, LMDeploy, SGLang, or vLLM access.
- Static documentation site · Published by GitHub Actions.
Connections 24
- Videos, datasets & base models → SFT annotations: provides labels
- SFT launcher → Qwen3-VL SFT recipe: loads
- Qwen3-VL SFT recipe → SFT data manifest: selects
- SFT data manifest → SFT annotations: references
- SFT annotations → Multimodal tokenization: video/query records
- Videos, datasets & base models → Multimodal tokenization: video assets
- Multimodal tokenization → Tokenized batches: encoded samples
- Tokenized batches → SFT trainer & checkpoints: batches
- Qwen3-VL composition → SFT trainer & checkpoints: forward and loss
- GPU/NPU distributed runtime → SFT trainer & checkpoints: executes
- SFT trainer & checkpoints → SFT checkpoint: writes
- Rollout launcher → Video-query rollouts: starts
- SFT checkpoint → Video-query rollouts: policy
- Videos, datasets & base models → Video-query rollouts: video/query samples
- Hosted inference providers → Video-query rollouts: optional inference
- Video-query rollouts → Rollout JSONL: writes completions
- Rollout JSONL → Temporal interval rewards: interval predictions
- Temporal interval rewards → GRPO optimizer: reward scores
- SFT checkpoint → GRPO optimizer: initial policy
- GRPO optimizer → GRPO checkpoint: writes
- Grounding evaluation launcher → Evaluation runner: starts
- Model and dataset adapters → Evaluation runner: resolves adapters
- GRPO checkpoint → Evaluation runner: evaluated model
- Hosted inference providers → Evaluation runner: optional model access