How eth-lre/pedagogicalrl works
Project Type and Purpose
• This is a specialized RL training pipeline for aligning LLMs with pedagogical principles.
• It implements a multi-agent “classroom” loop (tutor, student, judges, reward), backed by DeepSpeed/Accelerate for distributed training and vLLM for efficient inference.
File Structure
• Top-level entry scripts:
• train_sft.py (supervised fine-tuning)
• train_rl.py (GRPO RL training)
• eval.py (benchmark evaluation)
• vllm_server.py + start/stop scripts
• config/
• deepspeed/ (ZeRO config YAMLs)
• train_sft/ and train_rl/ (Hydra config sets)
• eval/ (task-specific YAMLs for LLM benchmarks)
• src/
• classroom.py (core RL loop, state machine)
• grpo/ (trainer logic, policy optimizer)
• inference_providers/ (adapters for Gemini, OpenRouter)
• vllm/ (client, memory/data-parallel helpers)
• utils/ (shared_memory, reward functions, logging)
• prompt_templates/ (system, judge, teacher, student prompts)
• utils/ (legacy reward/judge code)
Extract Architectural Insights from README
• Multi-turn dialog loop modeled as a state machine (tutor–student–judge–reward).
• Dynamic model loading/unloading to conserve GPU memory.
• Uses vLLM for batched inference & DeepSpeed for distributed gradient updates.
• External dependencies: Hugging Face Hub (model sync), Weights & Biases (optional), OpenRouter/Gemini (inference).
Main Components & Their Interactions
a. Configuration Layer
• YAML configs fed into Hydra at launch (train_sft.py, train_rl.py).
• DeepSpeed/Accelerate orchestrates GPU nodes.
b. Orchestrator Scripts
• Entry points invoke Hydra, spin up DeepSpeed, launch vLLM server if needed.
c. vLLM Server
• Persistent inference engine (runs in separate process/container).
• Exposes API for model tokenization/generation.
d. Classroom Loop (src/classroom.py)
• Simulates multi-agent dialog:
1. Tutor call → vLLM client → generate tutor turn
2. Student call → different LLM (frozen) → student turn
3. Judge calls → reward evaluation (leakage, pedagogy)
4. Reward calculator → returns scalar reward
• Feeds experiences to GRPO trainer.
e. GRPO Trainer (src/grpo/trainer.py)
• Consumes trajectories → computes gradients → updates tutor model weights via DeepSpeed.
f. Inference Providers
• Abstraction layer for alternative inference backends (Gemini, OpenRouter).
• Swapped in Hydra config.
g. Logging & Artifacts
• Weights & Biases for metrics, Hugging Face Hub for model checkpoints.
h. Evaluation Pipeline
• eval.py + LightEval integration for reasoning benchmarks.
Architectural Patterns & Principles
• Multi-agent RL state machine (classroom).
• Dynamic resource management (load/unload model weights).
• Separation of concerns:
• inference (vLLM / adapters)
• training (GRPO + DeepSpeed)
• configuration (Hydra YAML)
• prompting (template files)
• Extensibility via adapters (new inference backends) and modular reward functions.