# jiayi-pan/tinyzero architecture

> Architecture diagram of the GitHub repository jiayi-pan/tinyzero, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jiayi-pan/tinyzero
- Repository: https://github.com/jiayi-pan/tinyzero

## Overview

##### Project Type and Purpose
- TinyZero is a reinforcement learning (RL) approach to finetune large language models for tasks like countdown and multiplication. It reproduces the DeepSeek R1 Zero experiments on base language models (including Qwen2.5 series) using RL for self-verification and search skills.

##### File Structure to Identify Main Components
- Configuration/Workflows: The .github/workflows directory contains CI workflows (dataset processing, end-to-end tests, formatting, model testing) that are useful for continuous integration.
- Docker Directory: Contains Dockerfiles (ngc.vllm, vemlp.vllm.te) designed to containerize different environments – this denotes a deployment/build component.
- Tests: A separate tests directory with end-to-end tests (e2e), GPU utilities, and various combinations (ray, rollout, sanity testing) indicating extensive testing and integration workflows.
- The “verl” Package: The extensive “verl” directory contains modules for models, protocol, trainers, workers, distributed utilities, and integration of external components (including vllm and megatron-related code). This suggests a central library that orchestrates model loading, distributed training, and RL frameworks.
- Utility and Script Layers: Several scripts (in scripts, setup.py, requirements.txt) show how the project is initiated, configured, and run.

##### Main Components
a. Data Preparation & Preprocessing:
- Components from the “examples/data_preprocess” folder (scripts such as countdown.py, math_dataset.py, etc.) and possibly reference utilities in “verl/utils/dataset”.
- How raw data is ingested and preprocessed into the training dataset.

b. Training / RL Loop:
- RL training components: ppo_trainer, grpo_trainer as seen in “examples/ppo_trainer”, “examples/grpo_trainer”, and the trainer modules in “verl/trainer”.
- Model checkpoint and tokenization configurations from “tests/e2e” and “verl/models”.
- Integration with reinforcement learning blocks (actor, critic, reward model) found in “verl/workers” (actor, critic, rollout, etc.).

c. Distributed/Parallel Execution:
- Role of Ray (distributed system) and the separate submodules for “ray_trainer” as well as the “verl/single_controller/ray” components.
- How the project handles multi-GPU training (e.g., use of FSDP, pipeline and tensor parallelism through “verl/utils”, “verl/workers”).

d. External Dependencies & Execution Environment:
- Docker containers hosted in the “docker” folder – this represents deployment/build environments.
- CI workflows under .github illustrate automated testing, formatting (yapf), and deployment routines.
- Connection to external resources (e.g., torch, vllm, flash-attn, wandb) as noted in the README and configuration files.

##### Architectural Patterns / Design Principles
- Separation of Concerns: Clear separation between data preprocessing, training (RL loop), model loading, distributed training orchestration, and testing.
- Modular Design: The “verl” package is highly modular, with separate directories for models, trainers, workers, and utilities. This modular architecture allows easy extension (as seen by additional extensions like megatron and vllm).
- Distributed and Parallel Computation: Use of Ray for distributed training and multiple GPU strategies (FSDP, model partitioning) is evident.
- CI/CD Integration: Automated workflows (testing, formatting, model validation) indicate a robust CI/CD design.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Layer"
        raw["Raw Data"]:::data
        dp["Data Preprocessing"]:::data
    end

    subgraph "Training & Model"
        trainer["RL Trainer (PPO/GRPO)"]:::training
        model["Model Loader (LLM Integration)"]:::model
        worker["Worker Actors & Critics"]:::model
    end

    subgraph "Distributed & Execution"
        distrib["Distributed Execution (Ray & FSDP)"]:::distributed
    end

    subgraph "CI/CD & Deployment"
        cicd["CI/CD Pipeline"]:::cicd
        deploy["Deployment & Environment Setup"]:::cicd
    end

    raw --> dp
    dp --> trainer
    trainer --> model
    trainer --> worker
    worker -->|"feedback"| trainer
    distrib --> trainer
    distrib --> worker
    cicd --> trainer
    cicd --> model
    deploy --> cicd
    trainer -->|"veRL-based"| note1["Note: Built on veRL Framework"]:::cicd

    %% Click Events
    click dp "https://github.com/jiayi-pan/tinyzero/tree/main/examples/data_preprocess"
    click trainer "https://github.com/jiayi-pan/tinyzero/tree/main/examples/ppo_trainer"
    click model "https://github.com/jiayi-pan/tinyzero/tree/main/verl/models"
    click worker "https://github.com/jiayi-pan/tinyzero/tree/main/verl/workers"
    click distrib "https://github.com/jiayi-pan/tinyzero/tree/main/verl/single_controller/ray"
    click cicd "https://github.com/jiayi-pan/tinyzero/tree/main/.github/workflows"
    click deploy "https://github.com/jiayi-pan/tinyzero/tree/main/docker"

    %% Styles
    classDef data fill:#cce5ff,stroke:#339,stroke-width:2px;
    classDef training fill:#d4edda,stroke:#3c763d,stroke-width:2px;
    classDef model fill:#fff3cd,stroke:#8a6d3b,stroke-width:2px;
    classDef distributed fill:#f8d7da,stroke:#a94442,stroke-width:2px;
    classDef cicd fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
```
