# shehrum/grf-self-play architecture

> Architecture diagram of the GitHub repository shehrum/grf-self-play, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shehrum/grf-self-play
- Repository: https://github.com/shehrum/grf-self-play

## Overview

##### Project type and purpose
- This is a research-oriented reinforcement-learning (RL) project built around the Google Research Football (GRF) simulator.
- Purpose: train a soccer-playing agent in 11v11 matches using Proximal Policy Optimization (PPO) and competitive self-play, with optional APEX (distributed) training.

##### File structure
- Top-level files:
- train_ppo.py – main PPO training loop with self-play manager
- train_apex.py – distributed APEX version of the training loop
- eval.py – load a saved checkpoint and run evaluation episodes
- environment.py – custom wrapper around the GRF environment for training
- eval_environment.py – wrapper for evaluation episodes (may record metrics differently)
- model.py – neural-network architecture definition for the policy/value networks
- requirements.txt – Python dependencies (e.g., tensorflow/torch, GRF, Ray/RLlib)
- images/ – screenshot of the game used in README
- LICENSE, README.md, .DS_Store

##### README for additional insights
- Confirms use of PPO and self-play in 11v11 games
- Instructs that you clone this repo into the GRF environment directory
- Results (logs, TensorBoard data, checkpoints) are stored in /ray_results by default

##### Main components
a. Training Pipeline
- train_ppo.py / train_apex.py
- Self-Play Manager (schedules matches between the current policy and past checkpoints)
- Agent Model (model.py)
- Environment Wrapper (environment.py)
- GRF Simulator (external service)
- Experience buffer / batch generator (built into RLlib or custom)
- Optimizer / Parameter Server (for APEX, if distributed)
- Results Storage (/ray_results)
b. Evaluation Pipeline
- eval.py
- Evaluation Environment Wrapper (eval_environment.py)
- Loaded Checkpoint → Agent Model → GRF Simulator → Metrics Reporter
c. Shared modules
- model.py used by both training and evaluation
- configuration parameters (could be embedded in scripts or external)

##### Relationships and data-flow
1. Training Script calls Environment Wrapper → forwards to GRF Simulator
2. GRF Simulator returns observation, reward, done → wrapper → PPO agent (model)
3. Agent outputs action → wrapper → simulator
4. Experience batches flow into optimizer → updated model parameters
5. Self-Play Manager periodically copies the current model to a “past agent” and matches it against itself
6. Checkpoints and logs flow into /ray_results
- Evaluation:
1. eval.py loads checkpoint → model → interacts with eval_environment → simulator → metrics → console or file

##### Architectural patterns and design principles
- Modular separation between environment wrappers, training logic, model definition, and evaluation
- Self-play loop layered on top of standard RL pipeline
- Reuse of model definition for both training & evaluation
- Clear I/O boundary with an external simulator (GRF)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Swimlanes for Training and Evaluation
    subgraph "Training Pipeline"
        direction TB
        TrainPPO["train_ppo.py"]:::core
        TrainAPEX["train_apex.py"]:::core
        SelfPlayMngr["Self-Play Manager"]:::core
        EnvWrapper["Environment Wrapper"]:::env
        GRFSim["GRF Simulator"]:::external
        ExpBuffer["Experience Buffer"]:::core
        Optimizer["Optimizer / Parameter Server"]:::core
        Results["/ray_results"]:::storage
        Model["Agent Model (model.py)"]:::core
    end

    subgraph "Evaluation Pipeline"
        direction TB
        Eval["eval.py"]:::core
        EvalEnv["Eval Environment Wrapper"]:::env
        ModelE["Agent Model (model.py)"]:::core
        GRFSimE["GRF Simulator"]:::external
        Metrics["Metrics Reporter"]:::core
    end

    subgraph "Project Documentation & Assets"
        direction TB
        Req["requirements.txt"]:::docs
        Docs["README.md"]:::docs
        License["LICENSE"]:::docs
        Images["images/"]:::docs
    end

    %% Training flow
    TrainPPO -->|"calls"| EnvWrapper
    TrainAPEX -->|"calls"| EnvWrapper
    EnvWrapper -->|"step()"| GRFSim
    GRFSim -->|"returns obs,reward,done"| EnvWrapper
    EnvWrapper -->|"obs"| Model
    Model -->|"action"| EnvWrapper
    EnvWrapper -->|"action"| GRFSim
    GRFSim -->|"next obs"| EnvWrapper
    EnvWrapper -->|"batch"| ExpBuffer
    ExpBuffer -->|"samples"| Optimizer
    Optimizer -->|"updated params"| Model
    TrainPPO -->|"schedules"| SelfPlayMngr
    TrainAPEX -->|"schedules"| SelfPlayMngr
    SelfPlayMngr -->|"matches current vs past"| EnvWrapper
    TrainPPO -->|"store checkpoint every N steps"| Results
    TrainAPEX -->|"store checkpoint every N steps"| Results

    %% Evaluation flow
    Eval -->|"load checkpoint"| ModelE
    Eval -->|"calls"| EvalEnv
    EvalEnv -->|"step()"| GRFSimE
    GRFSimE -->|"returns obs,reward,done"| EvalEnv
    EvalEnv -->|"obs"| ModelE
    ModelE -->|"action"| EvalEnv
    EvalEnv -->|"action"| GRFSimE
    GRFSimE -->|"next obs"| EvalEnv
    Eval -->|"report metrics to"| Metrics

    %% Click Events
    click TrainPPO "https://github.com/shehrum/grf-self-play/blob/master/train_ppo.py"
    click TrainAPEX "https://github.com/shehrum/grf-self-play/blob/master/train_apex.py"
    click SelfPlayMngr "https://github.com/shehrum/grf-self-play/blob/master/train_ppo.py"
    click EnvWrapper "https://github.com/shehrum/grf-self-play/blob/master/environment.py"
    click EvalEnv "https://github.com/shehrum/grf-self-play/blob/master/eval_environment.py"
    click Eval "https://github.com/shehrum/grf-self-play/blob/master/eval.py"
    click Model "https://github.com/shehrum/grf-self-play/blob/master/model.py"
    click Req "https://github.com/shehrum/grf-self-play/blob/master/requirements.txt"
    click Docs "https://github.com/shehrum/grf-self-play/blob/master/README.md"
    click License "https://github.com/shehrum/grf-self-play/tree/master/LICENSE"
    click Images "https://github.com/shehrum/grf-self-play/tree/master/images/"

    %% Styles
    classDef core fill:#D0E1FF,stroke:#0366D6,color:#0366D6;
    classDef env fill:#E0FFD8,stroke:#28A745,color:#28A745;
    classDef external fill:#F0F0F0,stroke:#6C757D,color:#6C757D;
    classDef storage fill:#FFE6A1,stroke:#D97706,color:#D97706;
    classDef docs fill:#FFF5D8,stroke:#8A6D3B,color:#8A6D3B
```
