# eth-lre/pedagogicalrl architecture

> Architecture diagram of the GitHub repository eth-lre/pedagogicalrl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eth-lre/pedagogicalrl
- Repository: https://github.com/eth-lre/pedagogicalrl

## Overview

##### Project Type and Purpose
- This is a specialized RL training pipeline for aligning LLMs with pedagogical principles.
- It implements a multi-agent “classroom” loop (tutor, student, judges, reward), backed by DeepSpeed/Accelerate for distributed training and vLLM for efficient inference.

##### File Structure
- Top-level entry scripts:
- train_sft.py (supervised fine-tuning)
- train_rl.py (GRPO RL training)
- eval.py (benchmark evaluation)
- vllm_server.py + start/stop scripts
- config/
- deepspeed/ (ZeRO config YAMLs)
- train_sft/ and train_rl/ (Hydra config sets)
- eval/ (task-specific YAMLs for LLM benchmarks)
- src/
- classroom.py (core RL loop, state machine)
- grpo/ (trainer logic, policy optimizer)
- inference_providers/ (adapters for Gemini, OpenRouter)
- vllm/ (client, memory/data-parallel helpers)
- utils/ (shared_memory, reward functions, logging)
- prompt_templates/ (system, judge, teacher, student prompts)
- utils/ (legacy reward/judge code)

##### Extract Architectural Insights from README
- Multi-turn dialog loop modeled as a state machine (tutor–student–judge–reward).
- Dynamic model loading/unloading to conserve GPU memory.
- Uses vLLM for batched inference & DeepSpeed for distributed gradient updates.
- External dependencies: Hugging Face Hub (model sync), Weights & Biases (optional), OpenRouter/Gemini (inference).

##### Main Components & Their Interactions
a. Configuration Layer
- YAML configs fed into Hydra at launch (train_sft.py, train_rl.py).
- DeepSpeed/Accelerate orchestrates GPU nodes.
b. Orchestrator Scripts
- Entry points invoke Hydra, spin up DeepSpeed, launch vLLM server if needed.
c. vLLM Server
- Persistent inference engine (runs in separate process/container).
- Exposes API for model tokenization/generation.
d. Classroom Loop (src/classroom.py)
- Simulates multi-agent dialog:
1. Tutor call → vLLM client → generate tutor turn
2. Student call → different LLM (frozen) → student turn
3. Judge calls → reward evaluation (leakage, pedagogy)
4. Reward calculator → returns scalar reward
- Feeds experiences to GRPO trainer.
e. GRPO Trainer (src/grpo/trainer.py)
- Consumes trajectories → computes gradients → updates tutor model weights via DeepSpeed.
f. Inference Providers
- Abstraction layer for alternative inference backends (Gemini, OpenRouter).
- Swapped in Hydra config.
g. Logging & Artifacts
- Weights & Biases for metrics, Hugging Face Hub for model checkpoints.
h. Evaluation Pipeline
- eval.py + LightEval integration for reasoning benchmarks.

##### Architectural Patterns & Principles
- Multi-agent RL state machine (classroom).
- Dynamic resource management (load/unload model weights).
- Separation of concerns:
- inference (vLLM / adapters)
- training (GRPO + DeepSpeed)
- configuration (Hydra YAML)
- prompting (template files)
- Extensibility via adapters (new inference backends) and modular reward functions.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Layer
    subgraph "Configuration Layer"
        Config_Deepspeed["config/deepspeed/"]:::config
        Config_SFT["config/train_sft/"]:::config
        Config_RL["config/train_rl/"]:::config
        Config_Eval["config/eval/"]:::config
    end
    click Config_Deepspeed "https://github.com/eth-lre/pedagogicalrl/tree/main/config/deepspeed/"
    click Config_SFT "https://github.com/eth-lre/pedagogicalrl/tree/main/config/train_sft/"
    click Config_RL "https://github.com/eth-lre/pedagogicalrl/tree/main/config/train_rl/"
    click Config_Eval "https://github.com/eth-lre/pedagogicalrl/tree/main/config/eval/"

    %% Orchestrator Scripts
    subgraph "Orchestrator Scripts"
        TrainSFT["train_sft.py"]:::orchestrator
        TrainRL["train_rl.py"]:::orchestrator
        EvalScript["eval.py"]:::orchestrator
    end
    click TrainSFT "https://github.com/eth-lre/pedagogicalrl/blob/main/train_sft.py"
    click TrainRL "https://github.com/eth-lre/pedagogicalrl/blob/main/train_rl.py"
    click EvalScript "https://github.com/eth-lre/pedagogicalrl/blob/main/eval.py"

    %% vLLM Server
    subgraph "vLLM Server"
        VServer["vllm_server.py"]:::inference
        StartV["start_vllm_server.sh"]:::inference
        StopV["stop_vllm_server.sh"]:::inference
    end
    click VServer "https://github.com/eth-lre/pedagogicalrl/blob/main/vllm_server.py"
    click StartV "https://github.com/eth-lre/pedagogicalrl/blob/main/start_vllm_server.sh"
    click StopV "https://github.com/eth-lre/pedagogicalrl/blob/main/stop_vllm_server.sh"

    %% Training Infrastructure
    DeepSpeed["DeepSpeed/Accelerate Cluster"]:::training

    %% Main Pipeline
    subgraph "Training Pipeline"
        subgraph "Classroom Loop"
            Classroom["src/classroom.py"]:::training
        end
        subgraph "GRPO Trainer"
            Trainer["src/grpo/trainer.py"]:::training
            GRPOConfig["src/grpo/config.py"]:::config
        end
    end
    click Classroom "https://github.com/eth-lre/pedagogicalrl/blob/main/src/classroom.py"
    click Trainer "https://github.com/eth-lre/pedagogicalrl/blob/main/src/grpo/trainer.py"
    click GRPOConfig "https://github.com/eth-lre/pedagogicalrl/blob/main/src/grpo/config.py"

    %% Inference Providers
    subgraph "Inference Providers"
        Gemini["src/inference_providers/gemini_api_inference.py"]:::inference
        OpenRouter["src/inference_providers/open_router_inference.py"]:::inference
    end
    click Gemini "https://github.com/eth-lre/pedagogicalrl/blob/main/src/inference_providers/gemini_api_inference.py"
    click OpenRouter "https://github.com/eth-lre/pedagogicalrl/blob/main/src/inference_providers/open_router_inference.py"

    %% vLLM Helpers
    subgraph "vLLM Helpers"
        VClient["src/vllm/client.py"]:::inference
        VDataParallel["src/vllm/data_parallel_vllm.py"]:::inference
    end
    click VClient "https://github.com/eth-lre/pedagogicalrl/blob/main/src/vllm/client.py"
    click VDataParallel "https://github.com/eth-lre/pedagogicalrl/blob/main/src/vllm/data_parallel_vllm.py"

    %% Prompt Templates
    Prompt["prompt_templates/"]:::config
    click Prompt "https://github.com/eth-lre/pedagogicalrl/tree/main/prompt_templates/"

    %% Utility Functions
    subgraph "Utilities"
        SharedMem["src/utils/shared_memory.py"]:::utility
        Utils["src/utils/utils.py"]:::utility
        LegacyReward["utils/pedagogical_reward.py"]:::utility
        LegacyData["utils/data.py"]:::utility
    end
    click SharedMem "https://github.com/eth-lre/pedagogicalrl/blob/main/src/utils/shared_memory.py"
    click Utils "https://github.com/eth-lre/pedagogicalrl/blob/main/src/utils/utils.py"
    click LegacyReward "https://github.com/eth-lre/pedagogicalrl/blob/main/utils/pedagogical_reward.py"
    click LegacyData "https://github.com/eth-lre/pedagogicalrl/blob/main/utils/data.py"

    %% Evaluation Pipeline
    EvalPipe["Evaluation Pipeline"]:::training

    %% External Services
    subgraph "External Services"
        HFHub["Hugging Face Hub"]:::external
        WANDB["Weights & Biases"]:::external
        OpenRouterExt["OpenRouter"]:::external
        GeminiExt["Gemini"]:::external
    end

    %% Connections
    Config_Deepspeed -->|"ZeRO Configs"| DeepSpeed
    Config_SFT --> TrainSFT
    Config_RL --> TrainRL
    Config_Eval --> EvalScript

    TrainSFT --> DeepSpeed
    TrainRL --> DeepSpeed
    TrainRL --> VServer

    EvalScript -->|"launch eval"| EvalPipe
    EvalPipe --> HFHub
    EvalPipe --> WANDB

    Classroom -->|"load prompts"| Prompt
    Classroom --> VClient
    VClient --> VServer
    Classroom --> Gemini
    Classroom --> OpenRouter
    Classroom --> SharedMem
    Classroom --> Utils
    Classroom --> LegacyReward
    Classroom --> Trainer

    Trainer --> DeepSpeed
    Trainer --> HFHub
    Trainer --> WANDB

    %% Styles
    classDef config fill:#a8f0a1,stroke:#333;
    classDef orchestrator fill:#c2f0ff,stroke:#333;
    classDef inference fill:#a1c2f0,stroke:#333;
    classDef training fill:#f0c27a,stroke:#333;
    classDef utility fill:#e0e0e0,stroke:#333;
    classDef external fill:#d0d0d0,stroke:#333;
```
