# jaso1024/real-time-strategy-rl-clash-royale architecture

> Architecture diagram of the GitHub repository jaso1024/real-time-strategy-rl-clash-royale, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jaso1024/real-time-strategy-rl-clash-royale
- Repository: https://github.com/jaso1024/real-time-strategy-rl-clash-royale

## Overview

##### Project type and purpose
- This is a Python-based multi-agent reinforcement learning (RL) system that trains neural networks to play Clash Royale via Proximal Policy Optimization (PPO).
- It interfaces with the game running in BlueStacks, captures screen frames, encodes the state, and issues actions in real time.

##### File structure
Top-level scripts:
- MainLoop.py        – orchestrates the training/playing loop
- CRBot.py           – high-level orchestrator that ties all modules together
- CRHandler.py       – handles environment I/O (screen capture, action execution)
- ActionMapper.py    – translates agent outputs into game inputs (mouse clicks/keys)
- CRAgent.py         – agent class exposing actor and critic networks
- CRModel.py         – defines neural net architectures (autoencoder, actor, critic)
- Memory.py          – experience replay / buffer for PPO updates
- imports.py         – centralized imports and global config
Resources directory:
- Data (raw episodic logs, preprocessed)
- Models (state autoencoder source & saved TensorFlow model)
- TrainingData (serialized examples for autoencoder training)
- Templates/Cards (card definitions)
TrainedWeights directory:
- Actor-critic weights per agent subspace: Card, Origin, Shell, State

##### Key insights from README
- Three agents: Card selector, Origin-tile selector, Shell-tile selector
- Input pipeline: 9 sub-regions of the game screen + autoencoder + elixir estimate + hand vectors
- Action pipeline: each actor issues a distribution → combined into a single game action
- Reward shaping includes step, win/loss, crown differential
- Two main model architectures: autoencoder for state encoding, PPO actor-critic networks

##### Main components
A. Environment Interface
– BlueStacks + Clash Royale
– CRHandler (screen grab, input emulator)
B. Preprocessing & State Encoding
– Screen segmentation module (splits into 9 parts)
– Autoencoder (Resources/Models/ElixirModel) → latent vector
– Elixir estimator network → scalar
– One-hot encoding of hand cards
C. Multi-Agent PPO Engine
– Three CRAgent instances (Card, Origin, Shell)
– Each agent has Actor network & Critic network (defined in CRModel.py)
– Memory buffer to store (state, action, reward, next_state)
– PPO update logic in MainLoop.py
D. Action Mapping & Execution
– ActionMapper: maps discrete outputs to click coordinates or key presses
– CRHandler executes via PyAutoGUI / keyboard / pywin32
E. Persistence & Logging
– Checkpoint saving in TrainedWeights/<AgentType>
– Loading saved models from Resources/Models/Saved
– TrainingData for autoencoder fine-tuning

##### Relationships and data flow
1. CRHandler captures screenshot → Preprocessor splits images
2. Images + auxiliary features → Autoencoder + feature encoders → State vector
3. State vector → each CRAgent.actor → action distributions
4. Sample actions → combined → ActionMapper → CRHandler executes action
5. Environment returns next_frame + reward → Memory stores transition
6. At training intervals, MainLoop triggers PPO updates on all three agents → updated weights → saved to TrainedWeights

##### Architectural patterns & design principles
- Pipeline architecture: clear separation of preprocessing, encoding, decision, execution
- Multi-agent modularity: each agent encapsulated in its own class with own networks and checkpoint folder
- Actor-critic pattern for continuous training
- Resource centralization: Models and weights in dedicated directories

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Environment Interface
    subgraph "Environment Interface"
        BS(["BlueStacks"]):::external
        CR(["Clash Royale"]):::external
        CRHandler["CRHandler"]:::env
    end

    %% Preprocessing & State Encoding
    subgraph "Preprocessing & Encoding"
        Seg["Screen Segmentation Module"]:::processing
        ElixirModelDef["Autoencoder Definition"]:::processing
        ElixirModelWeights["Saved Autoencoder Weights"]:::storage
        ElixirData["Autoencoder Training Data"]:::storage
        CardDefs["Card Definitions"]:::storage
    end

    %% Multi-Agent PPO Engine
    subgraph "Multi-Agent Learning Engine"
        CRAgent1["Card Selector Agent"]:::learning
        CRAgent2["Origin Selector Agent"]:::learning
        CRAgent3["Shell Selector Agent"]:::learning
        CRModelComp["Neural Network Architectures"]:::learning
        MemoryBuf["Experience Replay Buffer"]:::learning
        MainLoopComp["MainLoop Orchestration"]:::learning
    end

    %% Action Mapping & Execution
    subgraph "Action Execution"
        ActionMap["ActionMapper"]:::processing
        Exec["CRHandler Execution"]:::env
    end

    %% Persistence & Logging
    subgraph "Persistence & Logging"
        CardWeights["Card Agent Weights"]:::storage
        OriginWeights["Origin Agent Weights"]:::storage
        ShellWeights["Shell Agent Weights"]:::storage
        StateWeights["State/Autoencoder Weights"]:::storage
        SavedModel["Pretrained Autoencoder Model"]:::storage
    end

    %% High-Level Orchestration
    CRBotComp["CRBot"]:::env

    %% Dependencies
    TF(["TensorFlow/Keras"]):::external
    CV(["OpenCV"]):::external
    PyAuto(["PyAutoGUI/pywin32"]):::external

    %% Data Flow
    BS -->|runs| CR
    CR -->|screen frames| CRHandler
    CRHandler -->|frames| Seg
    Seg -->|segments| ElixirModelDef
    ElixirModelDef -->|latent vector| CRAgent1
    ElixirModelDef -->|latent vector| CRAgent2
    ElixirModelDef -->|latent vector| CRAgent3
    CardDefs -->|one-hot| CRAgent1
    ElixirModelDef -->|elixir estimate| CRAgent2
    CRAgent1 -->|"action dist"| ActionMap
    CRAgent2 -->|"action dist"| ActionMap
    CRAgent3 -->|"action dist"| ActionMap
    ActionMap -->|"click/key events"| Exec
    Exec -->|applied to| CR
    CR -->|next_frame,reward| MemoryBuf
    MemoryBuf -->|data| MainLoopComp
    MainLoopComp -->|PPO updates| CRAgent1
    MainLoopComp -->|PPO updates| CRAgent2
    MainLoopComp -->|PPO updates| CRAgent3
    MainLoopComp -->|save weights| CardWeights
    MainLoopComp -->|save weights| OriginWeights
    MainLoopComp -->|save weights| ShellWeights
    MainLoopComp -->|save weights| StateWeights
    SavedModel -->|load| ElixirModelDef
    CRBotComp -->|ties modules| CRHandler
    CRBotComp -->|ties modules| MainLoopComp

    %% External Dependencies Link
    CRHandler --> CV
    CRHandler --> PyAuto
    ElixirModelDef --> TF
    CRAgent1 --> TF
    CRAgent2 --> TF
    CRAgent3 --> TF

    %% Click Events
    click CRHandler "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/blob/PPO/CRHandler.py"
    click Seg "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/blob/PPO/CRHandler.py"
    click ElixirModelDef "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/blob/PPO/Resources/Models/ElixirModel.py"
    click ElixirModelWeights "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/tree/PPO/Resources/Models/Saved/ElixirModel/"
    click ElixirData "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/blob/PPO/Resources/Models/TrainingData/ElixirData.pkl"
    click CardDefs "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/tree/PPO/Resources/Templates/Cards/"
    click CRAgent1 "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/blob/PPO/CRAgent.py"
    click CRAgent2 "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/blob/PPO/CRAgent.py"
    click CRAgent3 "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/blob/PPO/CRAgent.py"
    click CRModelComp "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/blob/PPO/CRModel.py"
    click MemoryBuf "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/blob/PPO/Memory.py"
    click MainLoopComp "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/blob/PPO/MainLoop.py"
    click ActionMap "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/blob/PPO/ActionMapper.py"
    click Exec "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/blob/PPO/CRHandler.py"
    click CRBotComp "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/blob/PPO/CRBot.py"
    click CardWeights "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/tree/PPO/TrainedWeights/CardWeights/"
    click OriginWeights "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/tree/PPO/TrainedWeights/OriginWeights/"
    click ShellWeights "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/tree/PPO/TrainedWeights/ShellWeights/"
    click StateWeights "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/tree/PPO/TrainedWeights/StateWeights/"
    click SavedModel "https://github.com/jaso1024/real-time-strategy-rl-clash-royale/tree/PPO/Resources/Models/Saved/ElixirModel/"

    %% Styles
    classDef env fill:#e0ffe0,stroke:#006400
    classDef processing fill:#e0f0ff,stroke:#00008b
    classDef learning fill:#ffe0b3,stroke:#ff8c00
    classDef storage fill:#d3d3d3,stroke:#696969
    classDef external fill:#f0e68c,stroke:#b8860b
```
