How evansnyanney/rl4se works
Project type and purpose
• This is a Reinforcement-Learning training pipeline (“RL4SE”) that applies the Proximal Policy Optimization (PPO) algorithm (via Stable-Baselines3) to the OpenAI Gymnasium LunarLander-v3 environment.
• It is not a web‐app or microservices suite, but a single Python-based ML pipeline whose goal is to train, track, and save an RL agent.
File structure
• Top level
• CS4900GameAI/evansnyanney/
– train_ppo.py: main training orchestration script
– utils.py: helper routines (e.g., video recording, model loading/saving)
– requirements.txt: Python dependencies (Gymnasium, Stable-Baselines3, WandB, etc.)
• performance_metrics/: holds charts and images produced post-training
• README.md: high-level project overview
• Git LFS configured for large video files
• No separate “frontend” or “backend” folders—it's a single-process ML workflow.
README for additional insights
• Uses WandB for experiment tracking and visualization
• Records videos of the agent landing and stores them via Git LFS
• Modular structure for future enhancements (clear separation of training logic vs. utility code)
a. Main components
1. CLI / Entry Point (train_ppo.py)
2. Configuration Loader (hyperparameters, WandB project settings)
3. Gymnasium Environment (LunarLander-v3)
4. Stable-Baselines3 PPO Agent
5. Utils Module (video recorder, model‐save/load routines)
6. WandB Service (external)
7. Local File Storage
• Model Artifacts (.zip/.pkl)
• Videos (.mp4 managed via Git LFS)
• performance_metrics/ images
b. Relationships and data flow
• train_ppo.py reads config → initializes WandB → creates Gym environment → injects env into PPO Agent
• At each training step:
– PPO Agent samples actions from policy, steps the environment → receives observations/rewards
– Metrics (reward, loss, entropy, etc.) are streamed to WandB
– Optionally, utils module captures video frames and writes to local disk
• On checkpoint or completion:
– Model artifacts saved locally
– WandB final snapshot saved remotely
– Local performance_metrics/ populated after evaluation scripts run
c. Architectural patterns & principles
• Modular layering: separation between “Orchestrator” (train_ppo.py), “Core Algorithm” (Stable-Baselines3), “Environment” (Gym), and “Utilities” (utils.py)
• Externalization of experimentation concerns: logging (WandB) and media storage (Git LFS) are decoupled from core training logic
• Single-process pipeline with clear I/O boundaries
d. Key technologies & libraries
• Python 3.7+
• Gymnasium (LunarLander-v3)
• Stable-Baselines3 (PPO implementation)
• Weights & Biases (WandB) for metrics & code snapshots
• Git LFS for large video files