How jaso1024/real-time-strategy-rl-clash-royale works
Project type and purpose
• This is a Python-based multi-agent reinforcement learning (RL) system that trains neural networks to play Clash Royale via Proximal Policy Optimization (PPO).
• It interfaces with the game running in BlueStacks, captures screen frames, encodes the state, and issues actions in real time.
File structure
Top-level scripts:
• MainLoop.py – orchestrates the training/playing loop
• CRBot.py – high-level orchestrator that ties all modules together
• CRHandler.py – handles environment I/O (screen capture, action execution)
• ActionMapper.py – translates agent outputs into game inputs (mouse clicks/keys)
• CRAgent.py – agent class exposing actor and critic networks
• CRModel.py – defines neural net architectures (autoencoder, actor, critic)
• Memory.py – experience replay / buffer for PPO updates
• imports.py – centralized imports and global config
Resources directory:
• Data (raw episodic logs, preprocessed)
• Models (state autoencoder source & saved TensorFlow model)
• TrainingData (serialized examples for autoencoder training)
• Templates/Cards (card definitions)
TrainedWeights directory:
• Actor-critic weights per agent subspace: Card, Origin, Shell, State
Key insights from README
• Three agents: Card selector, Origin-tile selector, Shell-tile selector
• Input pipeline: 9 sub-regions of the game screen + autoencoder + elixir estimate + hand vectors
• Action pipeline: each actor issues a distribution → combined into a single game action
• Reward shaping includes step, win/loss, crown differential
• Two main model architectures: autoencoder for state encoding, PPO actor-critic networks
Main components
A. Environment Interface
– BlueStacks + Clash Royale
– CRHandler (screen grab, input emulator)
B. Preprocessing & State Encoding
– Screen segmentation module (splits into 9 parts)
– Autoencoder (Resources/Models/ElixirModel) → latent vector
– Elixir estimator network → scalar
– One-hot encoding of hand cards
C. Multi-Agent PPO Engine
– Three CRAgent instances (Card, Origin, Shell)
– Each agent has Actor network & Critic network (defined in CRModel.py)
– Memory buffer to store (state, action, reward, next_state)
– PPO update logic in MainLoop.py
D. Action Mapping & Execution
– ActionMapper: maps discrete outputs to click coordinates or key presses
– CRHandler executes via PyAutoGUI / keyboard / pywin32
E. Persistence & Logging
– Checkpoint saving in TrainedWeights/<AgentType>
– Loading saved models from Resources/Models/Saved
– TrainingData for autoencoder fine-tuning
Relationships and data flow
1. CRHandler captures screenshot → Preprocessor splits images
2. Images + auxiliary features → Autoencoder + feature encoders → State vector
3. State vector → each CRAgent.actor → action distributions
4. Sample actions → combined → ActionMapper → CRHandler executes action
5. Environment returns next_frame + reward → Memory stores transition
6. At training intervals, MainLoop triggers PPO updates on all three agents → updated weights → saved to TrainedWeights
Architectural patterns & design principles
• Pipeline architecture: clear separation of preprocessing, encoding, decision, execution
• Multi-agent modularity: each agent encapsulated in its own class with own networks and checkpoint folder
• Actor-critic pattern for continuous training
• Resource centralization: Models and weights in dedicated directories