# mcg-nju/timelens2 architecture

> Architecture diagram of the GitHub repository mcg-nju/timelens2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mcg-nju/timelens2
- Repository: https://github.com/mcg-nju/timelens2

## Overview

#### Purpose and Top-Level Shape
TimeLens2 trains and evaluates multimodal LLMs that map a video plus natural-language query to one or more temporal evidence intervals. This is a Python monorepo organized as a sequential ML pipeline: supervised fine-tuning (SFT), offline rollout plus GRPO reinforcement learning, then benchmark evaluation. A static documentation site is separately published from `docs/`.

#### Primary Entry Points
Reproduction is shell-driven rather than service-driven:
- SFT launches: `sft/scripts/train_sft_4b.sh`, `sft/scripts/train_sft_8b.sh`
- Rollout generation: `grpo/scripts/rollout_timelens2.sh`
- GRPO training: `grpo/scripts/train_grpo_4b.sh`, `grpo/scripts/train_grpo_8b.sh`
- Grounding evaluation: `evaluation/scripts/srun_eval_all/run_grounding.sh`

These consume configurable paths, local annotations/rollouts, downloaded video assets, and pretrained or SFT model checkpoints.

#### SFT Data-to-Checkpoint Stage
The `sft/` module is an XTuner-derived training stack for video-language SFT. TimeLens2-specific configurations and annotations select Qwen3-VL 4B/8B training recipes and labeled TimeLens2 data:
- `sft/configs/qwen3_vl_4b_sft.py`
- `sft/configs/data/timelens2_sft.json`
- `sft/training_data_annotations/timelens2-93k/timelens2-93k_long.jsonl`

The modern engine under `sft/xtuner/v1/` provides typed data records, multimodal tokenization, model construction, loss, distributed training, checkpoints, and writers. Key boundaries are raw JSONL/video references → tokenized batches → model forward/loss → checkpoint artifacts:
- `sft/xtuner/v1/datasets/dataloader.py`
- `sft/xtuner/v1/datasets/mllm_tokenize_fn/qwen3_vl_tokenize_fn.py`
- `sft/xtuner/v1/train/trainer.py`

#### Model and Distributed Compute Layer
SFT supports dense, MoE, and composed vision-language architectures; TimeLens2 configurations target Qwen3-VL. The composition layer joins vision encoders/projectors with language models, while the engine handles GPU/NPU operations and distributed state:
- `sft/xtuner/v1/model/compose/qwen3_vl/modeling_qwen3_vl.py`
- `sft/xtuner/v1/engine/vision_compose_train_engine.py`
- `sft/xtuner/v1/ops/flash_attn/`

Material runtime dependencies are Python, PyTorch, Transformers-style model checkpoints, CUDA or NPU kernels, Triton kernels, and distributed collectives. Legacy XTuner integration also exposes DeepSpeed strategy support in `sft/xtuner/engine/_strategy/deepspeed.py`.

#### GRPO Rollout and Optimization Stage
`grpo/` performs off-policy temporal-grounding optimization after SFT. It reads annotated samples and/or generated completions, renders prompts and video inputs, computes interval-aware rewards, then updates the policy:
- `grpo/timelens/rollout.py`
- `grpo/timelens/train.py`
- `grpo/timelens/trainer.py`

The principal data boundary is SFT checkpoint plus video/query samples → rollout JSONL → reward-scored training batches → GRPO checkpoint. Bundled sources and rollouts make this reproducible without regenerating inference:
- `grpo/timelens/data/rollout_dataset.py`
- `grpo/data/rollouts/timelens2-93k.jsonl`
- `grpo/timelens/rewards/metrics.py`

Reward logic is specialized to temporal intervals, including temporal IoU and matching-free temporal Wasserstein-style scoring:
- `grpo/timelens/rewards/funcs.py`
- `grpo/timelens/rewards/sampling.py`

#### Evaluation Adapter and Benchmark Layer
`evaluation/` is a VLMEvalKit-derived evaluator. `evaluation/run.py` is the main Python launcher; its config resolves model adapters and dataset implementations. Image/video benchmark abstractions normalize samples and scoring, while temporal-grounding execution and reporting are handled by the grounding scripts:
- `evaluation/run.py`
- `evaluation/vlmeval/config.py`
- `evaluation/run_grounding.sh`

Model access is pluggable across local model implementations and hosted APIs. Local adapters reside in `evaluation/vlmeval/vlm/`; remote-provider adapters reside in `evaluation/vlmeval/api/`, including OpenAI/GPT, Gemini, Claude, and LMDeploy paths.

#### Infrastructure and External Boundaries
The repository does not implement an online application or persistent backend. External dependencies are model checkpoints and datasets hosted on Hugging Face, separately downloaded video corpora, GPU/NPU training infrastructure, and optional inference services such as LMDeploy, SGLang, or vLLM through `sft/xtuner/v1/ray/rollout/`. GitHub Actions publish the static project site via `.github/workflows/pages.yml`.

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sft["SFT pipeline"]
  node_sft_launch["SFT launcher<br/>shell entry point<br/>[train_sft_4b.sh]"]
  node_sft_recipe["Qwen3-VL SFT recipe<br/>training config<br/>[qwen3_vl_4b_sft.py]"]
  node_sft_annotations[("SFT annotations<br/>JSONL dataset")]
  node_sft_data_config["SFT data manifest<br/>data config<br/>[timelens2_sft.json]"]
  node_tokenizer["Multimodal tokenization<br/>data transform"]
  node_dataloader["Tokenized batches<br/>data loader<br/>[dataloader.py]"]
  node_qwen_model["Qwen3-VL composition<br/>vision-language model"]
  node_sft_trainer["SFT trainer &amp; checkpoints<br/>training engine<br/>[trainer.py]"]
  node_sft_checkpoint[("SFT checkpoint<br/>model artifact")]
end

subgraph group_grpo["GRPO pipeline"]
  node_rollout_launch["Rollout launcher<br/>shell entry point"]
  node_rollout_generator["Video-query rollouts<br/>offline inference<br/>[rollout.py]"]
  node_rollout_data[("Rollout JSONL<br/>offline dataset")]
  node_reward_metrics["Temporal interval rewards<br/>reward function<br/>[metrics.py]"]
  node_grpo_trainer["GRPO optimizer<br/>policy trainer<br/>[trainer.py]"]
  node_grpo_checkpoint[("GRPO checkpoint<br/>model artifact")]
end

subgraph group_eval["Evaluation"]
  node_eval_launch["Grounding evaluation launcher<br/>shell entry point<br/>[run_grounding.sh]"]
  node_eval_runner["Evaluation runner<br/>benchmark launcher<br/>[run.py]"]
  node_eval_config["Model and dataset adapters<br/>evaluation registry<br/>[config.py]"]
end

node_assets[("Videos, datasets &amp; base models<br/>external assets")]
node_accelerators{{"GPU/NPU distributed runtime<br/>compute platform"}}
node_hosted_models{{"Hosted inference providers<br/>external inference"}}
node_docs["Static documentation site<br/>published docs"]

node_assets -->|"provides labels"| node_sft_annotations
node_sft_launch -->|"loads"| node_sft_recipe
node_sft_recipe -->|"selects"| node_sft_data_config
node_sft_data_config -->|"references"| node_sft_annotations
node_sft_annotations -->|"video/query records"| node_tokenizer
node_assets -->|"video assets"| node_tokenizer
node_tokenizer -->|"encoded samples"| node_dataloader
node_dataloader -->|"batches"| node_sft_trainer
node_qwen_model -->|"forward and loss"| node_sft_trainer
node_accelerators -->|"executes"| node_sft_trainer
node_sft_trainer -->|"writes"| node_sft_checkpoint
node_rollout_launch -->|"starts"| node_rollout_generator
node_sft_checkpoint -->|"policy"| node_rollout_generator
node_assets -->|"video/query samples"| node_rollout_generator
node_hosted_models -.->|"optional inference"| node_rollout_generator
node_rollout_generator -->|"writes completions"| node_rollout_data
node_rollout_data -->|"interval predictions"| node_reward_metrics
node_reward_metrics -->|"reward scores"| node_grpo_trainer
node_sft_checkpoint -->|"initial policy"| node_grpo_trainer
node_grpo_trainer -->|"writes"| node_grpo_checkpoint
node_eval_launch -->|"starts"| node_eval_runner
node_eval_config -->|"resolves adapters"| node_eval_runner
node_grpo_checkpoint -->|"evaluated model"| node_eval_runner
node_hosted_models -.->|"optional model access"| node_eval_runner

click node_sft_launch "https://github.com/MCG-NJU/TimeLens2/blob/main/sft/scripts/train_sft_4b.sh"
click node_sft_recipe "https://github.com/MCG-NJU/TimeLens2/blob/main/sft/configs/qwen3_vl_4b_sft.py"
click node_sft_annotations "https://github.com/MCG-NJU/TimeLens2/blob/main/sft/training_data_annotations/timelens2-93k/timelens2-93k_long.jsonl"
click node_sft_data_config "https://github.com/MCG-NJU/TimeLens2/blob/main/sft/configs/data/timelens2_sft.json"
click node_tokenizer "https://github.com/MCG-NJU/TimeLens2/blob/main/sft/xtuner/v1/datasets/mllm_tokenize_fn/qwen3_vl_tokenize_fn.py"
click node_dataloader "https://github.com/MCG-NJU/TimeLens2/blob/main/sft/xtuner/v1/datasets/dataloader.py"
click node_qwen_model "https://github.com/MCG-NJU/TimeLens2/blob/main/sft/xtuner/v1/model/compose/qwen3_vl/modeling_qwen3_vl.py"
click node_sft_trainer "https://github.com/MCG-NJU/TimeLens2/blob/main/sft/xtuner/v1/train/trainer.py"
click node_rollout_launch "https://github.com/MCG-NJU/TimeLens2/blob/main/grpo/scripts/rollout_timelens2.sh"
click node_rollout_generator "https://github.com/MCG-NJU/TimeLens2/blob/main/grpo/timelens/rollout.py"
click node_rollout_data "https://github.com/MCG-NJU/TimeLens2/blob/main/grpo/data/rollouts/timelens2-93k.jsonl"
click node_reward_metrics "https://github.com/MCG-NJU/TimeLens2/blob/main/grpo/timelens/rewards/metrics.py"
click node_grpo_trainer "https://github.com/MCG-NJU/TimeLens2/blob/main/grpo/timelens/trainer.py"
click node_eval_launch "https://github.com/MCG-NJU/TimeLens2/blob/main/evaluation/scripts/srun_eval_all/run_grounding.sh"
click node_eval_runner "https://github.com/MCG-NJU/TimeLens2/blob/main/evaluation/run.py"
click node_eval_config "https://github.com/MCG-NJU/TimeLens2/blob/main/evaluation/vlmeval/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sft_launch,node_sft_recipe,node_sft_annotations,node_sft_data_config,node_tokenizer,node_dataloader,node_qwen_model,node_sft_trainer,node_sft_checkpoint toneBlue
class node_rollout_launch,node_rollout_generator,node_rollout_data,node_reward_metrics,node_grpo_trainer,node_grpo_checkpoint toneAmber
class node_eval_launch,node_eval_runner,node_eval_config toneMint
class node_assets,node_accelerators,node_hosted_models,node_docs toneNeutral
```

## Components

### SFT pipeline

Video-language supervised fine-tuning.

- [SFT launcher](https://github.com/MCG-NJU/TimeLens2/blob/main/sft/scripts/train_sft_4b.sh): shell entry point · Launches Qwen3-VL SFT recipes. · `sft/scripts/train_sft_4b.sh`
- [Qwen3-VL SFT recipe](https://github.com/MCG-NJU/TimeLens2/blob/main/sft/configs/qwen3_vl_4b_sft.py): training config · `sft/configs/qwen3_vl_4b_sft.py`
- [SFT annotations](https://github.com/MCG-NJU/TimeLens2/blob/main/sft/training_data_annotations/timelens2-93k/timelens2-93k_long.jsonl): JSONL dataset · `sft/training_data_annotations/timelens2-93k/timelens2-93k_long.jsonl`
- [SFT data manifest](https://github.com/MCG-NJU/TimeLens2/blob/main/sft/configs/data/timelens2_sft.json): data config · `sft/configs/data/timelens2_sft.json`
- [Multimodal tokenization](https://github.com/MCG-NJU/TimeLens2/blob/main/sft/xtuner/v1/datasets/mllm_tokenize_fn/qwen3_vl_tokenize_fn.py): data transform · Converts video and query records into model inputs. · `sft/xtuner/v1/datasets/mllm_tokenize_fn/qwen3_vl_tokenize_fn.py`
- [Tokenized batches](https://github.com/MCG-NJU/TimeLens2/blob/main/sft/xtuner/v1/datasets/dataloader.py): data loader · `sft/xtuner/v1/datasets/dataloader.py`
- [Qwen3-VL composition](https://github.com/MCG-NJU/TimeLens2/blob/main/sft/xtuner/v1/model/compose/qwen3_vl/modeling_qwen3_vl.py): vision-language model · Joins vision components with the language model. · `sft/xtuner/v1/model/compose/qwen3_vl/modeling_qwen3_vl.py`
- [SFT trainer & checkpoints](https://github.com/MCG-NJU/TimeLens2/blob/main/sft/xtuner/v1/train/trainer.py): training engine · Runs loss, distributed training, and checkpoint writing. · `sft/xtuner/v1/train/trainer.py`
- **SFT checkpoint**: model artifact

### GRPO pipeline

Offline rollout and policy optimization.

- [Rollout launcher](https://github.com/MCG-NJU/TimeLens2/blob/main/grpo/scripts/rollout_timelens2.sh): shell entry point · `grpo/scripts/rollout_timelens2.sh`
- [Video-query rollouts](https://github.com/MCG-NJU/TimeLens2/blob/main/grpo/timelens/rollout.py): offline inference · Renders prompts and generates temporal completions. · `grpo/timelens/rollout.py`
- [Rollout JSONL](https://github.com/MCG-NJU/TimeLens2/blob/main/grpo/data/rollouts/timelens2-93k.jsonl): offline dataset · Bundled generated completions for reproducible optimization. · `grpo/data/rollouts/timelens2-93k.jsonl`
- [Temporal interval rewards](https://github.com/MCG-NJU/TimeLens2/blob/main/grpo/timelens/rewards/metrics.py): reward function · Uses temporal IoU and Wasserstein-style scoring. · `grpo/timelens/rewards/metrics.py`
- [GRPO optimizer](https://github.com/MCG-NJU/TimeLens2/blob/main/grpo/timelens/trainer.py): policy trainer · Builds reward-scored batches and updates the policy. · `grpo/timelens/trainer.py`
- **GRPO checkpoint**: model artifact

### Evaluation

Benchmark adapters and temporal-grounding reporting.

- [Grounding evaluation launcher](https://github.com/MCG-NJU/TimeLens2/blob/main/evaluation/scripts/srun_eval_all/run_grounding.sh): shell entry point · `evaluation/scripts/srun_eval_all/run_grounding.sh`
- [Evaluation runner](https://github.com/MCG-NJU/TimeLens2/blob/main/evaluation/run.py): benchmark launcher · Resolves models, datasets, execution, and scoring. · `evaluation/run.py`
- [Model and dataset adapters](https://github.com/MCG-NJU/TimeLens2/blob/main/evaluation/vlmeval/config.py): evaluation registry · Selects local or hosted model adapters and benchmark implementations. · `evaluation/vlmeval/config.py`

### Other components

- **Videos, datasets & base models**: external assets · Downloaded corpora and pretrained checkpoints.
- **GPU/NPU distributed runtime**: compute platform · PyTorch, kernels, and distributed collectives.
- **Hosted inference providers**: external inference · Optional OpenAI, Gemini, Claude, LMDeploy, SGLang, or vLLM access.
- **Static documentation site**: published docs · Published by GitHub Actions.

## Connections

- Videos, datasets & base models → SFT annotations: provides labels
- SFT launcher → Qwen3-VL SFT recipe: loads
- Qwen3-VL SFT recipe → SFT data manifest: selects
- SFT data manifest → SFT annotations: references
- SFT annotations → Multimodal tokenization: video/query records
- Videos, datasets & base models → Multimodal tokenization: video assets
- Multimodal tokenization → Tokenized batches: encoded samples
- Tokenized batches → SFT trainer & checkpoints: batches
- Qwen3-VL composition → SFT trainer & checkpoints: forward and loss
- GPU/NPU distributed runtime → SFT trainer & checkpoints: executes
- SFT trainer & checkpoints → SFT checkpoint: writes
- Rollout launcher → Video-query rollouts: starts
- SFT checkpoint → Video-query rollouts: policy
- Videos, datasets & base models → Video-query rollouts: video/query samples
- Hosted inference providers → Video-query rollouts: optional inference
- Video-query rollouts → Rollout JSONL: writes completions
- Rollout JSONL → Temporal interval rewards: interval predictions
- Temporal interval rewards → GRPO optimizer: reward scores
- SFT checkpoint → GRPO optimizer: initial policy
- GRPO optimizer → GRPO checkpoint: writes
- Grounding evaluation launcher → Evaluation runner: starts
- Model and dataset adapters → Evaluation runner: resolves adapters
- GRPO checkpoint → Evaluation runner: evaluated model
- Hosted inference providers → Evaluation runner: optional model access
