How damasioj/ml-agents-fsm works
Project Type and Purpose
• This is a Unity-based simulation environment integrating manually programmed Finite State Machines (FSMs) and Reinforcement Learning (RL) agents via the Unity ML-Agents SDK.
• Purpose: Compare handcrafted AI (FSM) against ML-trained agents in an ecosystem scenario (paper experiment).
Key Components (to represent
A. External Trainer
• Python RL Trainer (mlagents Python API, version 0.16.1)
B. Unity Environment
1. Scene Manager
– Loads the “Ecosystem” scene
– Hosts all GameObjects and coordinates simulation steps
2. Environment Manager (EnvironmentManager.cs)
– Orchestrates environment update loop
– Spawns/despawns resources, structures, targets
3. Agents
a. Reinforcement Learning Agents (AnimalAgent.cs, CollectorAgent.cs, HaulerAgent.cs)
– Inherits from Unity’s Agent base class
– Uses ML-Agents Communicator for obs/actions
b. Manual FSM Agents (CollectorAgentABFSM.cs, CollectorAgentTBFSM.cs, ManualAgent.cs)
– Implements IHasGoal & AgentState FSM states (Idle, Move, Gather, Haul, Build, etc.)
4. Environment Objects
– Sources (AppleSource, WoodSource, StoneSource, FoodSource)
– Resources (AppleResource, WoodResource, StoneResource, BambooResource)
– Structures & Targets (ResourceDepot, HouseStructure, FlatBoxTarget, Rock, Apple Trees)
5. Utility Services
– Pathfinding (BasicPathfinder.cs)
– Helper (ObjectHelper.cs)
6. Asset Database
– Prefabs (Agents, Environments, Goals, Targets, Zones)
– Models (.nn neural-network files for pretrained models)
– Materials, Timers (configuration JSON), Scenes
Relationships & Data Flow
• Python Trainer ↔ Unity Environment
• Observations (vector, visual) sent from Unity → Python
• Actions returned from Python → Unity
– Communication via gRPC under ML-Agents SDK
• Within Unity
• EnvironmentManager updates scene → provides sensory data to Agents
• Agents compute next action either by:
– FSM module (AgentState.cs → next state)
– ML-Agents decision request (OnActionReceived)
• Actions applied to environment (move agent, collect resource, build structure)
• Environment changes fed back as observations
Architectural Patterns & Design Principles
• Modular separation:
• “Environment” = world logic and objects
• “Agents” = decision logic (FSM vs RL)
• “Utilities” = common services (pathfinding, helpers)
• FSM pattern: explicit state classes for manual agents
• Agent-Environment interface: conforms to RL Gym-style API via ML-Agents
Technologies & Frameworks
• Unity 2019.2.0f1+ (C# scripting, Scenes, Prefabs)
• Unity ML-Agents SDK (C# Agent base, Python trainer)
• Python 3.6+ with mlagents v0.16.1
• JSON for configuration (Timers/Ecosystem_timers.json)
• Excel/ pdata files for result aggregation (Results folder)