# unity-technologies/ml-agents architecture

> Architecture diagram of the GitHub repository unity-technologies/ml-agents, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/unity-technologies/ml-agents
- Repository: https://github.com/unity-technologies/ml-agents

## Overview

##### Project Type & Purpose
- This is an open-source toolkit combining two “stacks”:
a) Unity SDK (C# packages) to embed ML-Agents into Unity projects and example scenes.
b) Python ML stack—an environment API (mlagents_envs) and trainer CLI (ml-agents)—for training agents via RL and imitation.
- It leverages GRPC/Protobuf to link Python trainers with headless or in-Editor Unity environments.

##### Key Top-Level Directories
- com.unity.ml-agents/… & com.unity.ml-agents.extensions/… → Unity C# runtime/editor packages (sensors, actuators, side-channels).
- Project/, DevProject/, PerformanceProject/ → Unity example projects and test scenes.
- ml-agents-envs/ → Python environment API, communicator, side-channels, Gym/PettingZoo wrappers.
- ml-agents/ → Python trainer CLI, algorithms (PPO, SAC, POCA, BC/GAIL), config loader, model saver.
- config/ → YAML configurations per algorithm/environment.
- docs/, colab/, protobuf-definitions/ → Documentation, Colab notebooks, .proto files.
- Dockerfile, GitHub workflows, pre-commit, CI configs → Build & CI pipelines.

##### Architecture Overview & Patterns
- Logical Layers:
- Unity Environment Layer (C#)
- Communication Layer (GRPC, side-channels)
- Python Environment API (mlagents_envs)
- Trainer Layer (ml-agents CLI + algorithms)
- Configuration & Persistence (YAML, model checkpointing)
- Interaction Pattern:
- The Trainer spawns N Unity environment instances.
- Each instance hosts the ML-Agents C# runtime.
- Observations & actions flow bidirectionally over GRPC/side-channels.
- Policies execute in Python (PyTorch), send action commands; Unity steps simulate and return observations/rewards.

##### Main Components & Relationships
A. Unity Side
- Unity Editor / Headless Build
- com.unity.ml-agents (Runtime, Editor)
- com.unity.ml-agents.extensions (optional sensors/actuators)
- Example Scenes / Agents (Project/, DevProject/, PerformanceProject/)
- Side-Channels (EngineConfig, EnvParams, Stats)
- Protobuf-based Communicator (GrpcCommunicator)
B. Python Side
- mlagents_envs
- CommunicatorFactory → selects GRPC or mock
- Environment wrappers (Gym, PettingZoo)
- Side-Channel clients
- ml-agents trainer CLI
- TrainerController & EnvManager
- Algorithms (PPO, SAC, POCA, BC/GAIL)
- Policy/Model serialization (Torch)
- Config loader (YAML)
- ModelSaver & Checkpoint manager
C. Configuration & Tools
- config/*.yaml → Training presets per environment & algorithm
- Colab notebooks & docs → Usage examples
- Protobuf files → Shared message schema
D. CI/CD & Packaging
- GitHub Actions / Yamato → Build, test, publish Python packages & Unity packages
- Dockerfile → Containerized training environment

##### Architectural Patterns & Principles
- Separation of concerns: clear boundary between Unity runtime and Python training logic.
- Pluggable algorithms: TrainerController + plugin-style trainer modules (ppo/, sac/, poca/, imitation).
- Extensibility: side-channels and custom sensor/actuator components via com.unity.ml-agents.extensions.
- Config-driven: all hyperparameters and env selection driven by YAML.
- Containerization & CI: reproducible builds in Docker and GitHub Actions.

- Notes on where to find major code (e.g., in com.unity.ml-agents/Runtime or ml-agents/trainers/ppo/).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Trainer Layer
    subgraph "Trainer Layer"
        direction TB
        CLI["Trainer CLI & Controller"]:::python
        EnvMgr["EnvManager / SubprocessEnvManager / SimpleEnvManager"]:::python
        Algorithms["Algorithms: PPO / SAC / POCA / BC-GAIL"]:::python
        Policy["Policy Serialization & Checkpoint Manager"]:::python
        ConfigLoader["Configuration Loader & Upgrader"]:::python
    end

    %% Python Environment API
    subgraph "Python Environment API"
        direction TB
        EnvAPI["mlagents_envs"]:::python
        CommunicatorFactory["CommunicatorFactory"]:::python
        Wrappers["Gym & PettingZoo Wrappers"]:::python
        SideChannelClient["Side-Channel Clients"]:::python
    end

    %% Communication Layer
    subgraph "Communication Layer"
        direction TB
        GRPC["gRPC/Protobuf Communicator"]:::comm
        SideChannels["Side-Channels (EngineConfig, EnvParams, Stats)"]:::comm
    end

    %% Unity SDK
    subgraph "Unity SDK"
        direction TB
        UnityRuntime["Unity C# Runtime Package"]:::unity
        UnityEditor["Unity C# Editor Package"]:::unity
        ExtRuntime["Extensions Runtime (Sensors & Actuators)"]:::unity
        ExtEditor["Extensions Editor"]:::unity
        ExampleProjects["Example Projects & Scenes"]:::unity
    end

    %% Config & Docs
    subgraph "Config & Documentation"
        direction TB
        YAMLConfigs["YAML Training Presets"]:::config
        Docs["Documentation & Tutorials"]:::config
        Colab["Colab Notebooks"]:::config
        ProtoDefs["Protobuf Definitions"]:::config
    end

    %% CI/CD & Packaging
    subgraph "CI/CD & Packaging"
        direction TB
        Docker["Dockerized Environment"]:::ci
        CIWorkflows["GitHub CI Workflows"]:::ci
        Yamato["Unity Package Pipelines (Yamato)"]:::ci
        Pytest["CI Test Configuration (pytest)"]:::ci
    end

    %% Connections
    CLI -->|launch| EnvMgr
    EnvMgr -->|manages| EnvAPI
    EnvAPI -->|creates| CommunicatorFactory
    CommunicatorFactory -->|opens| GRPC
    CommunicatorFactory -->|uses| SideChannels
    CLI -->|loads| ConfigLoader
    ConfigLoader -->|reads| YAMLConfigs
    CLI -->|selects| Algorithms
    Algorithms -->|saves| Policy
    EnvAPI -->|wraps| Wrappers
    EnvAPI -->|utilizes| SideChannelClient
    GRPC -->|bidirectional| UnityRuntime
    SideChannels -.->|dashed| UnityRuntime
    UnityRuntime -->|editor support| UnityEditor
    UnityRuntime -->|extensions| ExtRuntime
    ExtRuntime -->|editor support| ExtEditor
    UnityRuntime -->|embedded in| ExampleProjects

    %% Click Events
    click UnityRuntime "https://github.com/unity-technologies/ml-agents/tree/develop/com.unity.ml-agents/Runtime"
    click UnityEditor "https://github.com/unity-technologies/ml-agents/tree/develop/com.unity.ml-agents/Editor"
    click ExtRuntime "https://github.com/unity-technologies/ml-agents/tree/develop/com.unity.ml-agents.extensions/Runtime"
    click ExtEditor "https://github.com/unity-technologies/ml-agents/tree/develop/com.unity.ml-agents.extensions/Editor"
    click ExampleProjects "https://github.com/unity-technologies/ml-agents/tree/develop/Project/"
    click ExampleProjects "https://github.com/unity-technologies/ml-agents/tree/develop/DevProject/"
    click ExampleProjects "https://github.com/unity-technologies/ml-agents/tree/develop/PerformanceProject/"
    click EnvAPI "https://github.com/unity-technologies/ml-agents/tree/develop/ml-agents-envs/mlagents_envs"
    click CommunicatorFactory "https://github.com/unity-technologies/ml-agents/tree/develop/ml-agents-envs/mlagents_envs/communicator_objects"
    click Wrappers "https://github.com/unity-technologies/ml-agents/blob/develop/ml-agents-envs/mlagents_envs/envs/unity_gym_env.py"
    click Wrappers "https://github.com/unity-technologies/ml-agents/blob/develop/ml-agents-envs/mlagents_envs/envs/pettingzoo_env_factory.py"
    click SideChannelClient "https://github.com/unity-technologies/ml-agents/tree/develop/ml-agents-envs/mlagents_envs/side_channel"
    click CLI "https://github.com/unity-technologies/ml-agents/blob/develop/ml-agents/mlagents/trainers/cli_utils.py"
    click CLI "https://github.com/unity-technologies/ml-agents/blob/develop/ml-agents/mlagents/trainers/trainer_controller.py"
    click EnvMgr "https://github.com/unity-technologies/ml-agents/blob/develop/ml-agents/mlagents/trainers/env_manager.py"
    click EnvMgr "https://github.com/unity-technologies/ml-agents/blob/develop/ml-agents/mlagents/trainers/simple_env_manager.py"
    click EnvMgr "https://github.com/unity-technologies/ml-agents/blob/develop/ml-agents/mlagents/trainers/subprocess_env_manager.py"
    click Algorithms "https://github.com/unity-technologies/ml-agents/tree/develop/ml-agents/mlagents/trainers/ppo"
    click Algorithms "https://github.com/unity-technologies/ml-agents/tree/develop/ml-agents/mlagents/trainers/sac"
    click Algorithms "https://github.com/unity-technologies/ml-agents/tree/develop/ml-agents/mlagents/trainers/poca"
    click Algorithms "https://github.com/unity-technologies/ml-agents/tree/develop/ml-agents/mlagents/trainers/torch_entities/components/bc"
    click Algorithms "https://github.com/unity-technologies/ml-agents/blob/develop/ml-agents/mlagents/trainers/torch_entities/components/reward_providers/gail_reward_provider.py"
    click Policy "https://github.com/unity-technologies/ml-agents/blob/develop/ml-agents/mlagents/trainers/policy/checkpoint_manager.py"
    click Policy "https://github.com/unity-technologies/ml-agents/blob/develop/ml-agents/mlagents/trainers/model_saver/torch_model_saver.py"
    click ConfigLoader "https://github.com/unity-technologies/ml-agents/blob/develop/ml-agents/mlagents/trainers/settings.py"
    click ConfigLoader "https://github.com/unity-technologies/ml-agents/blob/develop/ml-agents/mlagents/trainers/upgrade_config.py"
    click YAMLConfigs "https://github.com/unity-technologies/ml-agents/tree/develop/config/ppo"
    click YAMLConfigs "https://github.com/unity-technologies/ml-agents/tree/develop/config/sac"
    click YAMLConfigs "https://github.com/unity-technologies/ml-agents/tree/develop/config/poca"
    click YAMLConfigs "https://github.com/unity-technologies/ml-agents/tree/develop/config/imitation"
    click Docs "https://github.com/unity-technologies/ml-agents/tree/develop/docs/"
    click Colab "https://github.com/unity-technologies/ml-agents/tree/develop/colab/"
    click ProtoDefs "https://github.com/unity-technologies/ml-agents/tree/develop/protobuf-definitions/proto"
    click Docker "https://github.com/unity-technologies/ml-agents/tree/develop/Dockerfile"
    click CIWorkflows "https://github.com/unity-technologies/ml-agents/tree/develop/.github/workflows/"
    click Yamato "https://github.com/unity-technologies/ml-agents/tree/develop/.yamato/"
    click Pytest "https://github.com/unity-technologies/ml-agents/blob/develop/pytest.ini"

    %% Styles
    classDef python fill:#cce5ff,stroke:#339,stroke-width:1px
    classDef unity fill:#d4edda,stroke:#2a6e2a,stroke-width:1px
    classDef comm fill:#ffe5b4,stroke:#e69500,stroke-width:1px
    classDef config fill:#e2e3e5,stroke:#6c757d,stroke-width:1px
    classDef ci fill:#f5e6ff,stroke:#7a42f4,stroke-width:1px
```
