How neuralmmo/baselines works
Project Type & Purpose
• This project is a research and development platform for reinforcement learning in a multi-agent environment. It supports training through different policies (provided under the agent_zoo module), curriculum learning, replay generation, and both PvP/PvE evaluation modes.
• The README mentions key features such as training with custom policies, replay generation (with external visualization), and evaluation of models. This clarifies that it is a full-blown reinforcement learning framework with an emphasis on research experiments.
File Structure
• Key Top-Level Directories:
• agent_zoo: Contains baseline and custom agent implementations as well as guidelines to add new policies (each providing the Policy, RewardWrapper, and, optionally, Recurrent classes).
• reinforcement_learning: Implements the core RL engine (e.g., environment simulation and wrappers).
• curriculum_generation: Supports curriculum learning which tunes tasks or challenges during training.
• neurips23_evaluation and analysis: Contains scripts for evaluating performance, processing results, and exporting embeddings for held-out tasks.
• policies: Stores trained checkpoints and evaluation logs/results.
• scripts: Provides automation scripts for training, evaluation, and job submission (including SLURM jobs).
• tests: Contains unit tests to validate parts of the system.
• Configuration & Build Files: config.yaml, pyproject.toml, and various GitHub workflow files.
Overview of the Architecture
• Main Components:
• Training Engine:
• Triggered primarily by train.py (and train_helper.py), this engine orchestrates the training run.
• Integrates with agent_zoo to select a baseline (e.g., hybrid, yaofeng, takeru) or a custom policy.
• Agent Policies (agent_zoo):
• Consists of different submodules that offer various policy implementations and “reward wrappers” for the agents.
• Designed to be extensible so that new agent behavior can be added by following the provided structure.
• Reinforcement Learning Core:
• Contained within the reinforcement_learning module, it manages the environment simulation and state transitions.
• Additional support scripts (clean_pufferl.py, stat_wrapper.py) help in logging or modifying interactions with the learner.
• Curriculum Generation:
• Manages task sampling, encoding, and curriculum scheduling to gradually challenge the agent.
• Works as an input source for dynamic difficulty/task configurations during training.
• Evaluation and Replay:
• The evaluation component (in neurips23_evaluation and analysis) handles both generation of replay files and assessment of model performance.
• The policies directory acts as a repository for model checkpoints used by evaluation scripts.
• Auxiliary Scripts & Configurations:
• Scripts enable automation of training runs, checkpoint uploads, and evaluation jobs.
• Configuration files (like config.yaml) determine run parameters, including options like experiment tracking with wandb and syllabus integration.
Relationships and Interactions
• The Training Engine calls into the Agent Policies (from agent_zoo) to select an agent, then runs the simulation using the Reinforcement Learning Core.
• Curriculum Generation feeds dynamic task setups and environment variations to the training process.
• During or after training, generated checkpoints in the policies directory are passed to the Evaluation Engine (via evaluate.py and analysis scripts) to assess performance and generate replay files.
• Automation and job orchestration are managed through the scripts directory and CI/CD workflows found under .github/workflows.
• The configuration files mediate parameter passing across these components, ensuring consistency.