# neuralmmo/baselines architecture

> Architecture diagram of the GitHub repository neuralmmo/baselines, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/neuralmmo/baselines
- Repository: https://github.com/neuralmmo/baselines

## Overview

##### Project Type & Purpose
- This project is a research and development platform for reinforcement learning in a multi-agent environment. It supports training through different policies (provided under the agent_zoo module), curriculum learning, replay generation, and both PvP/PvE evaluation modes.
- The README mentions key features such as training with custom policies, replay generation (with external visualization), and evaluation of models. This clarifies that it is a full-blown reinforcement learning framework with an emphasis on research experiments.

##### File Structure
- Key Top-Level Directories:
- agent_zoo: Contains baseline and custom agent implementations as well as guidelines to add new policies (each providing the Policy, RewardWrapper, and, optionally, Recurrent classes).
- reinforcement_learning: Implements the core RL engine (e.g., environment simulation and wrappers).
- curriculum_generation: Supports curriculum learning which tunes tasks or challenges during training.
- neurips23_evaluation and analysis: Contains scripts for evaluating performance, processing results, and exporting embeddings for held-out tasks.
- policies: Stores trained checkpoints and evaluation logs/results.
- scripts: Provides automation scripts for training, evaluation, and job submission (including SLURM jobs).
- tests: Contains unit tests to validate parts of the system.
- Configuration & Build Files: config.yaml, pyproject.toml, and various GitHub workflow files.

##### Overview of the Architecture
- Main Components:
- Training Engine:
- Triggered primarily by train.py (and train_helper.py), this engine orchestrates the training run.
- Integrates with agent_zoo to select a baseline (e.g., hybrid, yaofeng, takeru) or a custom policy.
- Agent Policies (agent_zoo):
- Consists of different submodules that offer various policy implementations and “reward wrappers” for the agents.
- Designed to be extensible so that new agent behavior can be added by following the provided structure.
- Reinforcement Learning Core:
- Contained within the reinforcement_learning module, it manages the environment simulation and state transitions.
- Additional support scripts (clean_pufferl.py, stat_wrapper.py) help in logging or modifying interactions with the learner.
- Curriculum Generation:
- Manages task sampling, encoding, and curriculum scheduling to gradually challenge the agent.
- Works as an input source for dynamic difficulty/task configurations during training.
- Evaluation and Replay:
- The evaluation component (in neurips23_evaluation and analysis) handles both generation of replay files and assessment of model performance.
- The policies directory acts as a repository for model checkpoints used by evaluation scripts.
- Auxiliary Scripts & Configurations:
- Scripts enable automation of training runs, checkpoint uploads, and evaluation jobs.
- Configuration files (like config.yaml) determine run parameters, including options like experiment tracking with wandb and syllabus integration.

##### Relationships and Interactions
- The Training Engine calls into the Agent Policies (from agent_zoo) to select an agent, then runs the simulation using the Reinforcement Learning Core.
- Curriculum Generation feeds dynamic task setups and environment variations to the training process.
- During or after training, generated checkpoints in the policies directory are passed to the Evaluation Engine (via evaluate.py and analysis scripts) to assess performance and generate replay files.
- Automation and job orchestration are managed through the scripts directory and CI/CD workflows found under .github/workflows.
- The configuration files mediate parameter passing across these components, ensuring consistency.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CODEBASE OVERVIEW:
    %% This project is an extensive reinforcement learning platform designed for multi-agent environments.
    %% It is composed of modular components including:
    %% - Curriculum Generation: Dynamically designs tasks to configure training.
    %% - Training Engine (train.py & train_helper.py): Orchestrates training by selecting agent policies and interacting with the environment.
    %% - Agent Zoo: Provides pluggable policy implementations with reward wrappers.
    %% - RL Core: Manages environment simulation and state transitions.
    %% - Policies: Repository for trained model checkpoints and logs.
    %% - Evaluation & Analysis: Evaluates model performance, generates replays, and processes results.
    %% - Auxiliary Scripts: Automation scripts for training, evaluation, and job submission.
    
    %% Nodes
    CG["Curriculum Generation"]:::curriculum
    TE["Training Engine (train.py & train_helper.py)"]:::training
    AZ["Agent Zoo (Policy Implementations)"]:::policy
    RL["RL Core (Environment & Wrappers)"]:::core
    PO["Policies (Checkpoints & Logs)"]:::evaluation
    EV["Evaluation & Analysis"]:::evaluation
    AS["Auxiliary Scripts & Automation"]:::scripts

    %% Relationships
    CG -->|"configures"| TE
    TE -->|"calls"| AZ
    TE -->|"interacts"| RL
    TE -->|"generates"| PO
    PO -->|"feeds"| EV
    AS -->|"orchestrates"| TE
    AS -->|"triggers"| EV

    %% Click Events for Training Engine
    click TE "https://github.com/neuralmmo/baselines/blob/2.1/train.py"
    click TE "https://github.com/neuralmmo/baselines/blob/2.1/train_helper.py"

    %% Click Events for Agent Zoo
    click AZ "https://github.com/neuralmmo/baselines/tree/2.1/agent_zoo"
    click AZ "https://github.com/neuralmmo/baselines/blob/2.1/agent_zoo/hybrid.py"
    click AZ "https://github.com/neuralmmo/baselines/tree/2.1/agent_zoo/neurips23_start_kit"
    click AZ "https://github.com/neuralmmo/baselines/tree/2.1/agent_zoo/takeru"
    click AZ "https://github.com/neuralmmo/baselines/tree/2.1/agent_zoo/yaofeng"

    %% Click Events for RL Core
    click RL "https://github.com/neuralmmo/baselines/tree/2.1/reinforcement_learning"
    click RL "https://github.com/neuralmmo/baselines/blob/2.1/reinforcement_learning/environment.py"
    click RL "https://github.com/neuralmmo/baselines/blob/2.1/clean_pufferl.py"
    click RL "https://github.com/neuralmmo/baselines/blob/2.1/stat_wrapper.py"

    %% Click Events for Curriculum Generation
    click CG "https://github.com/neuralmmo/baselines/tree/2.1/curriculum_generation"
    click CG "https://github.com/neuralmmo/baselines/blob/2.1/curriculum_generation/curriculum_tutorial.py"
    click CG "https://github.com/neuralmmo/baselines/blob/2.1/curriculum_generation/task_encoder.py"
    click CG "https://github.com/neuralmmo/baselines/blob/2.1/curriculum_generation/task_sampler.py"

    %% Click Events for Evaluation & Analysis
    click EV "https://github.com/neuralmmo/baselines/blob/2.1/evaluate.py"
    click EV "https://github.com/neuralmmo/baselines/tree/2.1/neurips23_evaluation"
    click EV "https://github.com/neuralmmo/baselines/blob/2.1/neurips23_evaluation/export_embeddings.py"
    click EV "https://github.com/neuralmmo/baselines/blob/2.1/neurips23_evaluation/heldout_evaluation_task.py"
    click EV "https://github.com/neuralmmo/baselines/tree/2.1/analysis"
    click EV "https://github.com/neuralmmo/baselines/blob/2.1/analysis/proc_eval_result.py"
    click EV "https://github.com/neuralmmo/baselines/blob/2.1/analysis/run_task_conditioning.py"

    %% Click Events for Policies
    click PO "https://github.com/neuralmmo/baselines/tree/2.1/policies"

    %% Click Events for Auxiliary Scripts
    click AS "https://github.com/neuralmmo/baselines/tree/2.1/scripts"

    %% Styles
    classDef training fill:#add8e6,stroke:#000,stroke-width:2px;
    classDef policy fill:#90ee90,stroke:#000,stroke-width:2px;
    classDef core fill:#ffb6c1,stroke:#000,stroke-width:2px;
    classDef curriculum fill:#dda0dd,stroke:#000,stroke-width:2px;
    classDef evaluation fill:#ffdead,stroke:#000,stroke-width:2px;
    classDef scripts fill:#afeeee,stroke:#000,stroke-width:2px;
```
