# www-ye/timezero architecture

> Architecture diagram of the GitHub repository www-ye/timezero, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/www-ye/timezero
- Repository: https://github.com/www-ye/timezero

## Overview

##### Main Components
- Data/Input Layer:
- Video datasets and annotations from sources such as ActivityNet and Charades.
- Files under directories like “ActivityNet”, “Charades”, and “dataset” hold raw annotations, test/validation JSON files, and preprocessed dataset specifications.
- Preprocessing Module:
- Scripts and utilities (preprocess_dataset.py, preprocess_video.sh) are used to prepare raw video inputs and annotations.
- Configuration & Environment Setup:
- The environment.yml and YAML config files stored in the “configs” directory (e.g., ddp.yaml, zero3.yaml) specify distributed training, offloading, and model parameters.
- Core Training & Inference Modules:
- Main implementation exists in the “src/open_r1” folder. This module includes:
▪ Trainer modules (e.g., grpo_trainer.py, vllm_grpo_trainer.py) for managing reinforcement learning training.
▪ Specific scripts like grpo.py, grpo_video.py, sft.py indicate different training paradigms or modes.
▪ Evaluate and generate modules for inference, test-time reasoning, and generating temporal segment predictions.
- Scripts and Execution:
- The “scripts” folder contains shell scripts (run_grpo_video.sh, evaluate.sh) that serve as orchestration layers for distributed training and evaluation.
- Evaluation Module:
- evaluate.py and the associated evaluation scripts handle metric computation (R1@0.3, R1@0.5, R1@0.7) and final performance reporting.

##### Relationships and Interactions
- Data Flow:
- Raw video data and annotations (from ActivityNet/Charades directories) → Preprocessing (via preprocess_video.sh and preprocess_dataset.py) → Generated preprocessed video data.
- Training Pipeline:
- Preprocessed data is fed to the GRPO-based training pipeline (run via run_grpo_video.sh) which calls scripts in src/open_r1 (like grpo_video.py) for reinforcement learning training. Use of torchrun and DeepSpeed (via provided configuration files) indicates distributed, multi-GPU interactions.
- Inference and Reasoning:
- Trained models then serve inference functions—potentially via evaluation modules that load the model, run reasoning-guided predictions, and provide outputs.
- Configuration & Distributed Setup:
- Configuration files in “configs” are used by both training and evaluation scripts to manage hardware distribution and optimization (e.g., gradient checkpointing, attention mechanisms).

##### Architectural Patterns and Technologies
- Reinforcement Learning: The project’s core is based on an RL training paradigm, where the model is trained entirely using reinforcement learning.
- Test-Time Reasoning: The system generates “chain-of-thought” reasoning during inference, making it an emergent intelligent system.
- Distributed & Parallel Training: Use of torchrun (for distributed training), DeepSpeed configurations, and gradient checkpointing to manage large model training.
- Modular Design: The clear separation between preprocessing, training, and evaluation across different directories demonstrates modular architecture.
- External Integration: Links to pre-trained models and data sources (HuggingFace, external datasets) indicate integration with external platforms.

##### Detailed Elements for the Principal Software Engineer
- Move to the Preprocessing module: Indicate the role of preprocess_video.sh and associated scripts.
- Trainer modules (grpo_trainer, vllm_grpo_trainer) for distributed RL training.
- Core algorithms (grpo.py, grpo_video.py, generate.py) capturing the temporal grounding and reasoning flow.
- Execution Orchestration: Include the “scripts” block that calls training and evaluation, highlighting the run_grpo_video.sh for training and evaluate.sh for testing.
- Configuration/Setup component: Highlight environment.yml and YAML configs that control distributed training.
- Final Evaluation component: Outline evaluate.py and link it with the Metrics (R1@0.3, R1@0.5, R1@0.7) output.

- Elements are legible; avoid over-complexity by grouping related modules where appropriate (e.g., grouping all training-related modules).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data/Input Layer
    subgraph "Data/Input Layer"
        A1("ActivityNet (Raw Data)"):::external
        A2("ActivityNet Annotation"):::external
        B1("Charades (Raw Data)"):::external
        B2("Charades Annotation"):::external
        C1("Dataset Files"):::external
    end

    %% Preprocessing Module
    subgraph "Preprocessing Module"
        D1("preprocess_dataset.py"):::script
        D2("preprocess_video.sh"):::script
    end

    %% Configuration & Environment Setup
    subgraph "Configuration & Environment Setup"
        E1("environment.yml"):::config
        E2("ddp.yaml"):::config
        E3("zero3.yaml"):::config
        E4("qwen2vl_sft_config.yaml"):::config
        E5("zero2.yaml"):::config
        E6("data_configs.py"):::config
    end

    %% Core Training & Inference Modules
    subgraph "Core Training & Inference"
        F1("src/open_r1"):::core
        F2("grpo_trainer.py"):::core
        F3("vllm_grpo_trainer.py"):::core
        F4("grpo_trainer_video.py"):::core
        F5("vllm_grpo_trainer_video.py"):::core
        F6("grpo.py"):::core
        F7("grpo_video.py"):::core
        F8("sft.py"):::core
        F9("generate.py"):::core
        F10("evaluate.py"):::core
    end

    %% Scripts and Execution
    subgraph "Scripts and Execution"
        G1("run_grpo_video.sh"):::orchestration
        G2("evaluate.sh"):::orchestration
        G3("zero3_offload.json"):::orchestration
    end

    %% Evaluation Module (Project Root)
    subgraph "Evaluation Module"
        H1("evaluate.py (Root)"):::evaluation
    end

    %% Data Flow Connections
    A1 -->|"raw_data"| D1
    A2 --> D1
    B1 --> D1
    B2 --> D1
    C1 --> D1

    D1 -->|"processed_data"| D2
    D2 -->|"preprocessed_output"| F1

    %% Configuration integration to Preprocessing and Core Training
    E1 -->|"env_setup"| D1
    E2 -->|"distributed_config"| F1
    E3 --> F1
    E4 --> F1
    E5 --> F1
    E6 --> F1

    %% Core Training internal flow
    F1 -->|"triggers"| F2
    F1 --> F3
    F1 --> F4
    F1 --> F5
    F1 -->|"training_script"| F6
    F1 --> F7
    F1 --> F8
    F1 --> F9
    F1 --> F10

    %% Execution orchestration flow
    G1 -->|"orchestrates_training"| F7
    G2 -->|"orchestrates_evaluation"| H1
    G3 --> G1

    %% Inference/Evaluation Flow
    F10 -->|"model_evaluation"| H1

    %% Reinforcement Learning Feedback (illustrative arrow)
    F2 -.->|"RL_Signal"| F6
    F3 -.-> F7

    %% Distributed Training Note
    E2 ---|"torchrun_Distributed"| F1
    E3 ---|"DeepSpeed_Config"| F1

    %% Click Events for Data/Input Layer
    click A1 "https://github.com/www-ye/timezero/tree/main/ActivityNet/"
    click A2 "https://github.com/www-ye/timezero/tree/main/ActivityNet/activitynet_annotation"
    click B1 "https://github.com/www-ye/timezero/tree/main/Charades/"
    click B2 "https://github.com/www-ye/timezero/tree/main/Charades/charades_annotation"
    click C1 "https://github.com/www-ye/timezero/tree/main/dataset/"

    %% Click Events for Preprocessing Module
    click D1 "https://github.com/www-ye/timezero/blob/main/preprocess_dataset.py"
    click D2 "https://github.com/www-ye/timezero/blob/main/scripts/preprocess_video.sh"

    %% Click Events for Configuration & Environment Setup
    click E1 "https://github.com/www-ye/timezero/blob/main/environment.yml"
    click E2 "https://github.com/www-ye/timezero/blob/main/configs/ddp.yaml"
    click E3 "https://github.com/www-ye/timezero/blob/main/configs/zero3.yaml"
    click E4 "https://github.com/www-ye/timezero/blob/main/configs/qwen2vl_sft_config.yaml"
    click E5 "https://github.com/www-ye/timezero/blob/main/configs/zero2.yaml"
    click E6 "https://github.com/www-ye/timezero/blob/main/data_configs.py"

    %% Click Events for Core Training & Inference Modules
    click F1 "https://github.com/www-ye/timezero/tree/main/src/open_r1/"
    click F2 "https://github.com/www-ye/timezero/blob/main/src/open_r1/trainer/grpo_trainer.py"
    click F3 "https://github.com/www-ye/timezero/blob/main/src/open_r1/trainer/vllm_grpo_trainer.py"
    click F4 "https://github.com/www-ye/timezero/blob/main/src/open_r1/trainer/grpo_trainer_video.py"
    click F5 "https://github.com/www-ye/timezero/blob/main/src/open_r1/trainer/vllm_grpo_trainer_video.py"
    click F6 "https://github.com/www-ye/timezero/blob/main/src/open_r1/grpo.py"
    click F7 "https://github.com/www-ye/timezero/blob/main/src/open_r1/grpo_video.py"
    click F8 "https://github.com/www-ye/timezero/blob/main/src/open_r1/sft.py"
    click F9 "https://github.com/www-ye/timezero/blob/main/src/open_r1/generate.py"
    click F10 "https://github.com/www-ye/timezero/blob/main/src/open_r1/evaluate.py"

    %% Click Events for Scripts and Execution
    click G1 "https://github.com/www-ye/timezero/blob/main/scripts/run_grpo_video.sh"
    click G2 "https://github.com/www-ye/timezero/blob/main/scripts/evaluate.sh"
    click G3 "https://github.com/www-ye/timezero/blob/main/scripts/zero3_offload.json"

    %% Click Event for Evaluation Module (Root)
    click H1 "https://github.com/www-ye/timezero/blob/main/evaluate.py (project root)"

    %% Styles
    classDef external fill:#FFCC99,stroke:#333,stroke-width:2px;
    classDef script fill:#CCFFCC,stroke:#333,stroke-width:2px;
    classDef config fill:#FFFFCC,stroke:#333,stroke-width:2px;
    classDef core fill:#CCE5FF,stroke:#333,stroke-width:2px;
    classDef orchestration fill:#E6CCFF,stroke:#333,stroke-width:2px;
    classDef evaluation fill:#D5E8D4,stroke:#333,stroke-width:2px;
```
