# epicgamer17/rl-research architecture

> Architecture diagram of the GitHub repository epicgamer17/rl-research, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/epicgamer17/rl-research
- Repository: https://github.com/epicgamer17/rl-research

## Overview

##### Project Type Analysis
This is a comprehensive RL framework implementing various algorithms and environments. The project is structured as a modular library with multiple agent implementations, custom environments, and supporting utilities.

##### Main Components

A. Core Agent Implementations (Primary Layer):
- DQN Family (Rainbow, Ape-X, etc.)
- Policy-based (PPO, A2C)
- Model-based (AlphaZero, MuZero)
- Multi-agent (NFSP)

B. Environment Layer:
- Connections to standard environments (Gym/Gymnasium)
- Environments by type (board games, classic control, etc.)

C. Supporting Infrastructure:
- Replay Buffer System
- Configuration Management
- Search Algorithms
- Utility Modules

**┌─────────────────────────────────────────────┐**
- Base Agent                   │
- ┘
- ↓                 ↓
**┌─────────────────┐  ┌──────────────────┐**
- Value-Based    │  │   Policy-Based   │
- Agents       │  │     Agents       │
- ┘  └──────────────────┘

**┌─────────────────────────────────────────────┐**
- Replay Buffers                 │
- ┤
- Neural Networks                │
- ┤
- Search Algorithms              │
- ┘

**┌─────────────────────────────────────────────┐**
- Custom Environments            │
- ┤
- External Environments          │
- ┘

##### Key Relationships to Highlight
- Agent inheritance hierarchy
- Environment interaction patterns
- Configuration flow
- Data flow between components

##### Technical Stack Indicators
- Integration with Gym/Gymnasium
- Distributed computing capabilities (Ape-X)

**┌─────────────────────────────────────────────┐**
- 🔷 Agent Implementation                     │
- 🔶 Environment                              │
- 🔺 Configuration                            │
- ⭐ Core Component                           │
- ↔️ Bidirectional Flow                       │
- → Unidirectional Flow                      │
- ┘

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Agent Layer"
        BaseAgent["Base Agent"]:::base
        click BaseAgent "https://github.com/epicgamer17/rl-research/blob/main/base_agent/agent.py"
        
        subgraph "Value-Based Agents"
            Rainbow["Rainbow DQN"]:::dqn
            click Rainbow "https://github.com/epicgamer17/rl-research/blob/main/dqn/rainbow/rainbow_agent.py"
            ApeX["Ape-X DQN"]:::dqn
            click ApeX "https://github.com/epicgamer17/rl-research/blob/main/dqn/ape_x/actor.py"
            NFSP["NFSP"]:::dqn
            click NFSP "https://github.com/epicgamer17/rl-research/blob/main/dqn/NFSP/nfsp_agent_clean.py"
        end
        
        subgraph "Policy-Based Agents"
            A2C["A2C"]:::policy
            click A2C "https://github.com/epicgamer17/rl-research/blob/main/A2C/a2c_agent.py"
            PPO["PPO"]:::policy
            click PPO "https://github.com/epicgamer17/rl-research/blob/main/ppo/ppo_agent.py"
        end
        
        subgraph "Model-Based Agents"
            AlphaZero["AlphaZero"]:::model
            click AlphaZero "https://github.com/epicgamer17/rl-research/blob/main/alphazero/alphazero_agent.py"
            MuZero["MuZero"]:::model
            click MuZero "https://github.com/epicgamer17/rl-research/blob/main/muzero/muzero_agent.py"
        end
    end
    
    subgraph "Core Components"
        ReplayBuffer["Replay Buffer System"]:::component
        click ReplayBuffer "https://github.com/epicgamer17/rl-research/blob/main/replay_buffers/base_replay_buffer.py"
        PrioritizedBuffer["Prioritized Buffer"]:::component
        click PrioritizedBuffer "https://github.com/epicgamer17/rl-research/blob/main/replay_buffers/prioritized_n_step_replay_buffer.py"
        
        subgraph "Neural Networks"
            DenseNet["Dense Networks"]:::nn
            click DenseNet "https://github.com/epicgamer17/rl-research/blob/main/modules/dense.py"
            ConvNet["Convolutional Networks"]:::nn
            click ConvNet "https://github.com/epicgamer17/rl-research/blob/main/modules/conv.py"
            ResNet["Residual Networks"]:::nn
            click ResNet "https://github.com/epicgamer17/rl-research/blob/main/modules/residual.py"
        end
        
        subgraph "Search Algorithms"
            MCTS["Monte Carlo Tree Search"]:::search
            click MCTS "https://github.com/epicgamer17/rl-research/blob/main/search/monte_carlo_tree_search.py"
            AlphaBeta["Alpha-Beta Pruning"]:::search
            click AlphaBeta "https://github.com/epicgamer17/rl-research/blob/main/search/alpha_beta_pruning.py"
        end
    end
    
    subgraph "Environment Layer"
        CustomEnv["Custom Environments"]:::env
        click CustomEnv "https://github.com/epicgamer17/rl-research/tree/main/custom_gym_envs/custom_gym_envs/envs/"
        GymEnv["Gymnasium/Gym"]:::env
    end
    
    subgraph "Support Systems"
        Config["Configuration Management"]:::support
        click Config "https://github.com/epicgamer17/rl-research/tree/main/packages/agent_configs/agent_configs/"
        Utils["Utility Modules"]:::support
        click Utils "https://github.com/epicgamer17/rl-research/tree/main/packages/utils/utils/"
    end
    
    %% Connections
    BaseAgent --> Rainbow
    BaseAgent --> ApeX
    BaseAgent --> NFSP
    BaseAgent --> A2C
    BaseAgent --> PPO
    BaseAgent --> AlphaZero
    BaseAgent --> MuZero
    
    Rainbow --> ReplayBuffer
    ApeX --> PrioritizedBuffer
    NFSP --> ReplayBuffer
    
    AlphaZero --> MCTS
    MuZero --> MCTS
    
    Config -.-> BaseAgent
    Utils -.-> BaseAgent
    
    %% Environment Connections
    CustomEnv --> BaseAgent
    GymEnv --> BaseAgent
    
    %% Neural Network Connections
    DenseNet --> BaseAgent
    ConvNet --> BaseAgent
    ResNet --> BaseAgent
    
    %% Styles
    classDef base fill:#3498db,stroke:#2980b9,color:white
    classDef dqn fill:#2ecc71,stroke:#27ae60,color:white
    classDef policy fill:#e74c3c,stroke:#c0392b,color:white
    classDef model fill:#9b59b6,stroke:#8e44ad,color:white
    classDef component fill:#f1c40f,stroke:#f39c12,color:black
    classDef nn fill:#1abc9c,stroke:#16a085,color:white
    classDef search fill:#e67e22,stroke:#d35400,color:white
    classDef env fill:#95a5a6,stroke:#7f8c8d,color:white
    classDef support fill:#34495e,stroke:#2c3e50,color:white
```
