How unity-technologies/ml-agents works
Project Type & Purpose
• This is an open-source toolkit combining two “stacks”:
a) Unity SDK (C# packages) to embed ML-Agents into Unity projects and example scenes.
b) Python ML stack—an environment API (mlagents_envs) and trainer CLI (ml-agents)—for training agents via RL and imitation.
• It leverages GRPC/Protobuf to link Python trainers with headless or in-Editor Unity environments.
Key Top-Level Directories
• com.unity.ml-agents/… & com.unity.ml-agents.extensions/… → Unity C# runtime/editor packages (sensors, actuators, side-channels).
• Project/, DevProject/, PerformanceProject/ → Unity example projects and test scenes.
• ml-agents-envs/ → Python environment API, communicator, side-channels, Gym/PettingZoo wrappers.
• ml-agents/ → Python trainer CLI, algorithms (PPO, SAC, POCA, BC/GAIL), config loader, model saver.
• config/ → YAML configurations per algorithm/environment.
• docs/, colab/, protobuf-definitions/ → Documentation, Colab notebooks, .proto files.
• Dockerfile, GitHub workflows, pre-commit, CI configs → Build & CI pipelines.
Architecture Overview & Patterns
• Logical Layers:
• Unity Environment Layer (C#)
• Communication Layer (GRPC, side-channels)
• Python Environment API (mlagents_envs)
• Trainer Layer (ml-agents CLI + algorithms)
• Configuration & Persistence (YAML, model checkpointing)
• Interaction Pattern:
• The Trainer spawns N Unity environment instances.
• Each instance hosts the ML-Agents C# runtime.
• Observations & actions flow bidirectionally over GRPC/side-channels.
• Policies execute in Python (PyTorch), send action commands; Unity steps simulate and return observations/rewards.
Main Components & Relationships
A. Unity Side
• Unity Editor / Headless Build
• com.unity.ml-agents (Runtime, Editor)
• com.unity.ml-agents.extensions (optional sensors/actuators)
• Example Scenes / Agents (Project/, DevProject/, PerformanceProject/)
• Side-Channels (EngineConfig, EnvParams, Stats)
• Protobuf-based Communicator (GrpcCommunicator)
B. Python Side
• mlagents_envs
• CommunicatorFactory → selects GRPC or mock
• Environment wrappers (Gym, PettingZoo)
• Side-Channel clients
• ml-agents trainer CLI
• TrainerController & EnvManager
• Algorithms (PPO, SAC, POCA, BC/GAIL)
• Policy/Model serialization (Torch)
• Config loader (YAML)
• ModelSaver & Checkpoint manager
C. Configuration & Tools
• config/*.yaml → Training presets per environment & algorithm
• Colab notebooks & docs → Usage examples
• Protobuf files → Shared message schema
D. CI/CD & Packaging
• GitHub Actions / Yamato → Build, test, publish Python packages & Unity packages
• Dockerfile → Containerized training environment
Architectural Patterns & Principles
• Separation of concerns: clear boundary between Unity runtime and Python training logic.
• Pluggable algorithms: TrainerController + plugin-style trainer modules (ppo/, sac/, poca/, imitation).
• Extensibility: side-channels and custom sensor/actuator components via com.unity.ml-agents.extensions.
• Config-driven: all hyperparameters and env selection driven by YAML.
• Containerization & CI: reproducible builds in Docker and GitHub Actions.
• Notes on where to find major code (e.g., in com.unity.ml-agents/Runtime or ml-agents/trainers/ppo/).