# blavad/marl architecture

> Architecture diagram of the GitHub repository blavad/marl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/blavad/marl
- Repository: https://github.com/blavad/marl

## Overview

##### Main Components Identification
The system should be represented as a modular architecture with these key components:

a) Core Components:
- MARL System Orchestrator
- Agent Management System
- Model System
- Experience System
- Exploration System
- Policy System

##### Component Relationships

Top Layer:
- MARL System Orchestrator (marl.py)
- Handles multi-agent coordination
- Manages training and testing workflows

Middle Layer:
- Agent System (agent/)
- Different agent types (DQN, MAAC, Q-Agent, PG-Agent)
- Agent interaction management

Supporting Systems Layer:
- Model System (model/)
- Neural Network implementations (MLP, CNN)
- Q-value computations
- Policy System (policy/)
- Policy management and execution
- Experience System (experience/)
- Replay Buffer
- Experience collection and storage
- Exploration System (exploration/)
- Epsilon-Greedy
- OU-Noise
- Exploration process management

##### Specific Implementation Details

- The modular nature of the system (shown by clear component boundaries)
- The plugin architecture for different algorithms (DQN, MAAC, Minimax-Q, etc.)
- The interaction between agents and environment
- The experience collection and replay system
- The exploration process integration

##### Key Relationships to Highlight

- MARL Orchestrator → Agents: Agent management and coordination
- Agents → Models: Neural network and Q-value computation
- Agents → Experience: Memory management and replay
- Agents → Exploration: Exploration strategy execution
- Agents → Policies: Policy execution and updates

##### External Interfaces

Integration with:
- Gym environments
- Custom environments (like the soccer environment mentioned in the example)

##### Implementation Notes

- Supported algorithms (both single-agent and multi-agent)
- Key configuration options
- Training and testing workflows

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Legend
        L1[Component]:::component --- L2[Sub-Component]:::subcomponent
        L3[("Database/\nStorage")]:::storage --- L4["External\nSystem"]:::external
        L5[Main Flow] --> L6[Data Flow]
        style L5 stroke-width:2px
        style L6 stroke-width:1px,stroke-dasharray: 5 5
    end

    subgraph Core["MARL Core System"]
        MARL["MARL System Orchestrator"]:::core
        click MARL "https://github.com/blavad/marl/blob/master/marl/marl.py"
    end

    subgraph AgentSystem["Agent Management System"]
        BaseAgent["Base Agent"]:::agent
        DQN["DQN/Q-Agent"]:::agent
        PG["Policy Gradient Agent"]:::agent
        MAAC["MAAC Agent"]:::agent
        click BaseAgent "https://github.com/blavad/marl/blob/master/marl/agent/agent.py"
        click DQN "https://github.com/blavad/marl/blob/master/marl/agent/q_agent.py"
        click PG "https://github.com/blavad/marl/blob/master/marl/agent/pg_agent.py"
        click MAAC "https://github.com/blavad/marl/blob/master/marl/agent/maac_agent.py"
        click AgentSystem "https://github.com/blavad/marl/tree/master/marl/agent/"
    end

    subgraph SupportSystems["Supporting Systems"]
        subgraph ModelSystem["Model System"]
            NN["Neural Networks"]:::model
            MLP["MLP Network"]:::model
            CNN["CNN Network"]:::model
            QValue["Q-Value Computation"]:::model
            click ModelSystem "https://github.com/blavad/marl/tree/master/marl/model/"
            click NN "https://github.com/blavad/marl/tree/master/marl/model/nn/"
            click MLP "https://github.com/blavad/marl/blob/master/marl/model/nn/mlpnet.py"
            click CNN "https://github.com/blavad/marl/blob/master/marl/model/nn/cnn.py"
            click QValue "https://github.com/blavad/marl/blob/master/marl/model/qvalue.py"
        end

        subgraph ExpSystem["Experience System"]
            ExpBase["Base Experience"]:::experience
            ReplayBuffer["Replay Buffer"]:::experience
            click ExpSystem "https://github.com/blavad/marl/tree/master/marl/experience/"
            click ExpBase "https://github.com/blavad/marl/blob/master/marl/experience/experience.py"
            click ReplayBuffer "https://github.com/blavad/marl/blob/master/marl/experience/replay_buffer.py"
        end

        subgraph ExplSystem["Exploration System"]
            ExplBase["Base Exploration"]:::exploration
            EpsGreedy["Epsilon-Greedy"]:::exploration
            OUNoise["OU-Noise"]:::exploration
            click ExplSystem "https://github.com/blavad/marl/tree/master/marl/exploration/"
            click ExplBase "https://github.com/blavad/marl/blob/master/marl/exploration/expl_process.py"
            click EpsGreedy "https://github.com/blavad/marl/blob/master/marl/exploration/eps_greedy.py"
            click OUNoise "https://github.com/blavad/marl/blob/master/marl/exploration/ou_noise.py"
        end

        subgraph PolicySystem["Policy System"]
            PolicyBase["Base Policy"]:::policy
            PolicyImpl["Policy Implementations"]:::policy
            click PolicySystem "https://github.com/blavad/marl/tree/master/marl/policy/"
            click PolicyBase "https://github.com/blavad/marl/blob/master/marl/policy/policy.py"
            click PolicyImpl "https://github.com/blavad/marl/blob/master/marl/policy/policies.py"
        end
    end

    subgraph External["External Interfaces"]
        GymEnv["Gym Environments"]:::external
        CustomEnv["Custom Environments"]:::external
    end

    %% Core relationships
    MARL --> BaseAgent
    BaseAgent --> DQN
    BaseAgent --> PG
    BaseAgent --> MAAC

    %% Agent relationships with supporting systems
    DQN & PG & MAAC --> NN
    NN --> MLP & CNN
    DQN --> QValue
    
    BaseAgent --> ExpBase
    ExpBase --> ReplayBuffer
    
    BaseAgent --> ExplBase
    ExplBase --> EpsGreedy
    ExplBase --> OUNoise
    
    BaseAgent --> PolicyBase
    PolicyBase --> PolicyImpl

    %% External interface relationships
    MARL -.-> GymEnv
    MARL -.-> CustomEnv

    %% Styling
    classDef core fill:#2196F3,stroke:#1565C0,color:white
    classDef agent fill:#4CAF50,stroke:#2E7D32,color:white
    classDef model fill:#FFC107,stroke:#FFA000,color:black
    classDef experience fill:#FF5722,stroke:#E64A19,color:white
    classDef exploration fill:#9C27B0,stroke:#7B1FA2,color:white
    classDef policy fill:#795548,stroke:#5D4037,color:white
    classDef external fill:#9E9E9E,stroke:#616161,color:white
    classDef component fill:#BBDEFB,stroke:#1565C0,color:black
    classDef subcomponent fill:#C8E6C9,stroke:#2E7D32,color:black
    classDef storage fill:#FFE0B2,stroke:#F57C00,color:black
```
