How tinyyana/maze_agent_rl works
Project type and purpose
• This is a research‐oriented deep reinforcement learning (RL) project combining Procedural Content Generation via RL (PCGRL).
• Purpose: Train a “Maze Master” agent (using PPO from Stable Baselines3) to dynamically modify a maze (walls, monsters, exit) so that a simulated or human player remains in a “flow” difficulty window.
File structure
Top‐level directories and files:
• agents/ – agent implementations
• astar_bot.py (A* player simulator)
• maze_master.py (the RL agent logic, imported by train.py)
• envs/ – Gymnasium environment definitions
• maze_env.py (core state/action/reward loop)
• maze_generator.py (initial maze creation via DFS)
• rendering.py (Pygame visualizer)
• config.py – global constants (reward weights, time limits, modes)
• train.py – script that instantiates the env & PPO model, trains & checkpoints “maze_master_ppo.zip”
• main.py – loads saved model, launches interactive demo (Pygame)
• evaluate.py – loads a model, runs evaluation episodes, writes CSV reports under evaluation_logs/
• experiment_runner.py – batch‐runs training & evaluation experiments, writes logs under experiment_log/
• models/, tensorboard_logs/ – persistent storage for model artifacts and TFEvent logs
• .gitignore, requirements.txt, LICENSE, README.md – meta/config files
README for additional insights
• Tech stack: Python 3.12+, Gymnasium environment, Stable Baselines3 PPO, Pygame rendering, TensorBoard for logging
• Workflow: train.py → tensorboard_logs & models; main.py for live demo; evaluate.py & experiment_runner.py for batch stats
Components
a. Main components
1. Configuration Module (config.py)
2. Maze Environment (envs/maze_env.py + maze_generator.py + rendering.py)
3. Player Simulators
• A* Bot (agents/astar_bot.py)
• Human Input (via main.py + Pygame)
4. Maze Master RL Agent (Stable Baselines3 PPO wrapper)
5. Training Pipeline (train.py)
6. Evaluation Pipeline (evaluate.py + experiment_runner.py)
7. Artifact Stores
• Model Store (models/)
• TensorBoard Logs (tensorboard_logs/)
• CSV and Text Reports (evaluation_logs/, experiment_log/)
b. Relationships & interactions
• train.py reads config.py, instantiates MazeEnv, wraps it in SB3 PPO, steps through episodes →
• Sends data (states, actions, rewards) to TensorBoard →
• Saves checkpoints to models/
• evaluate.py & experiment_runner.py load a checkpoint from models/, run MazeEnv + A* Bot or human mode, collect metrics → write CSV/text reports
• main.py loads checkpoint + MazeEnv + rendering.py → Pygame window for interactive demo (human or A* mode via config.py)
c. Architectural patterns & principles
• Separation of concerns: environment code (envs/) is decoupled from training (train.py), evaluation (evaluate.py), and UI (main.py)
• Modular config: single source of truth for hyperparameters and modes
• Experiment reproducibility: distinct directories for logs, models, raw & summary CSVs
d. Key technologies/frameworks
• Gymnasium environment interface
• Stable Baselines3 PPO algorithm
• Pygame for rendering
• TensorBoard for live metric visualization
• Training Pipeline → TensorBoard Logs (write metrics)
• Training Pipeline → Model Store (checkpoint model .zip)
• A* Bot (agents/astar_bot.py) feeding into Env
• CSV/Text Reports (evaluation_logs/, experiment_log/)