How shehrum/grf-self-play works
Project type and purpose
• This is a research-oriented reinforcement-learning (RL) project built around the Google Research Football (GRF) simulator.
• Purpose: train a soccer-playing agent in 11v11 matches using Proximal Policy Optimization (PPO) and competitive self-play, with optional APEX (distributed) training.
File structure
• Top-level files:
• train_ppo.py – main PPO training loop with self-play manager
• train_apex.py – distributed APEX version of the training loop
• eval.py – load a saved checkpoint and run evaluation episodes
• environment.py – custom wrapper around the GRF environment for training
• eval_environment.py – wrapper for evaluation episodes (may record metrics differently)
• model.py – neural-network architecture definition for the policy/value networks
• requirements.txt – Python dependencies (e.g., tensorflow/torch, GRF, Ray/RLlib)
• images/ – screenshot of the game used in README
• LICENSE, README.md, .DS_Store
README for additional insights
• Confirms use of PPO and self-play in 11v11 games
• Instructs that you clone this repo into the GRF environment directory
• Results (logs, TensorBoard data, checkpoints) are stored in /ray_results by default
Main components
a. Training Pipeline
• train_ppo.py / train_apex.py
• Self-Play Manager (schedules matches between the current policy and past checkpoints)
• Agent Model (model.py)
• Environment Wrapper (environment.py)
• GRF Simulator (external service)
• Experience buffer / batch generator (built into RLlib or custom)
• Optimizer / Parameter Server (for APEX, if distributed)
• Results Storage (/ray_results)
b. Evaluation Pipeline
• eval.py
• Evaluation Environment Wrapper (eval_environment.py)
• Loaded Checkpoint → Agent Model → GRF Simulator → Metrics Reporter
c. Shared modules
• model.py used by both training and evaluation
• configuration parameters (could be embedded in scripts or external)
Relationships and data-flow
1. Training Script calls Environment Wrapper → forwards to GRF Simulator
2. GRF Simulator returns observation, reward, done → wrapper → PPO agent (model)
3. Agent outputs action → wrapper → simulator
4. Experience batches flow into optimizer → updated model parameters
5. Self-Play Manager periodically copies the current model to a “past agent” and matches it against itself
6. Checkpoints and logs flow into /ray_results
• Evaluation:
1. eval.py loads checkpoint → model → interacts with eval_environment → simulator → metrics → console or file
Architectural patterns and design principles
• Modular separation between environment wrappers, training logic, model definition, and evaluation
• Self-play loop layered on top of standard RL pipeline
• Reuse of model definition for both training & evaluation
• Clear I/O boundary with an external simulator (GRF)