# dennybritz/reinforcement-learning architecture

> Architecture diagram of the GitHub repository dennybritz/reinforcement-learning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dennybritz/reinforcement-learning
- Repository: https://github.com/dennybritz/reinforcement-learning

## Overview

##### Main Components

a. Core Algorithm Modules:
- Dynamic Programming (DP)
- Monte Carlo Methods (MC)
- Temporal Difference Learning (TD)
- Function Approximation (FA)
- Deep Q-Learning (DQN)
- Policy Gradient Methods
- A3C Implementation

b. Supporting Infrastructure:
- Environment Interfaces (lib/envs)
- Atari Helpers (lib/atari)
- Plotting Utilities (lib/plotting.py)

**[Environment Layer]**
- ↑↓
**[Core Algorithm Layer]**
- ↑↓
**[Support Infrastructure Layer]**

##### Component Relationships

a. Top Level (Environment Interface):
- OpenAI Gym Environments
- Custom Environments (from lib/envs)

b. Middle Level (Algorithm Implementations):
- Algorithms by type (DP, MC, TD, etc.)
- Progression from basic to advanced algorithms
- Blue: Model-based methods (DP)
- Green: Model-free methods (MC, TD)
- Orange: Advanced methods (DQN, Policy Gradient)

c. Bottom Level (Infrastructure):
- Plotting utilities
- Helper functions
- State processors
- Environment wrappers

##### Implementation Details to Highlight

- Input/output formats between components
- Training data flow
- Where experience replay is used (DQN)

**[Environment Layer]**

**[Basic RL Algorithms] → [Advanced RL Algorithms]**

**[Helper Functions] ←------ [Neural Network Components]**

##### Technical Details

- Version information for key dependencies (Python 3, TensorFlow)
- Integration points with OpenAI Gym
- Where custom environments are used
- Entry points for different training scenarios

##### Documentation Integration

- Link components to corresponding Jupyter notebooks
- Reference relevant sections from the RL textbook
- Performance metrics and benchmarks

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Legend
        L1[Basic Component]
        L2[("Database/Storage")]
        L3[["Environment"]]
        L4{{"Neural Network"}}
        style L1 fill:#f5f5f5,stroke:#333
        style L2 fill:#e1d5e7,stroke:#9673a6
        style L3 fill:#d5e8d4,stroke:#82b366
        style L4 fill:#ffe6cc,stroke:#d79b00
    end

    subgraph Environments
        GYM[["OpenAI Gym"]]:::env
        CUSTOM[["Custom Environments"]]:::env
        click CUSTOM "https://github.com/dennybritz/reinforcement-learning/tree/master/lib/envs/"
    end

    subgraph "Core Algorithms"
        DP["Dynamic Programming"]:::basic
        click DP "https://github.com/dennybritz/reinforcement-learning/tree/master/DP/"
        
        MC["Monte Carlo Methods"]:::basic
        click MC "https://github.com/dennybritz/reinforcement-learning/tree/master/MC/"
        
        TD["Temporal Difference Learning"]:::basic
        click TD "https://github.com/dennybritz/reinforcement-learning/tree/master/TD/"
        
        FA["Function Approximation"]:::basic
        click FA "https://github.com/dennybritz/reinforcement-learning/tree/master/FA/"
        
        DQN["Deep Q-Learning"]:::advanced
        click DQN "https://github.com/dennybritz/reinforcement-learning/tree/master/DQN/"
        
        PG["Policy Gradient Methods"]:::advanced
        click PG "https://github.com/dennybritz/reinforcement-learning/tree/master/PolicyGradient/"
        
        A3C["A3C Implementation"]:::advanced
        click A3C "https://github.com/dennybritz/reinforcement-learning/tree/master/PolicyGradient/a3c/"
    end

    subgraph "Neural Networks"
        DQN_NET{{"DQN Network"}}:::nn
        PG_NET{{"Policy Network"}}:::nn
        VAL_NET{{"Value Network"}}:::nn
    end

    subgraph "Infrastructure"
        PLOT["Plotting Utilities"]:::infra
        click PLOT "https://github.com/dennybritz/reinforcement-learning/blob/master/lib/plotting.py"
        
        ATARI["Atari Helpers"]:::infra
        click ATARI "https://github.com/dennybritz/reinforcement-learning/tree/master/lib/atari/"
        
        STATE["State Processors"]:::infra
    end

    %% Connections
    GYM & CUSTOM --> DP & MC & TD & FA & DQN & PG & A3C
    DQN --> DQN_NET
    PG & A3C --> PG_NET & VAL_NET
    ATARI --> STATE
    STATE --> DQN & PG & A3C
    PLOT --> DP & MC & TD & FA & DQN & PG & A3C

    %% Styles
    classDef basic fill:#dae8fc,stroke:#6c8ebf
    classDef advanced fill:#fff2cc,stroke:#d6b656
    classDef env fill:#d5e8d4,stroke:#82b366
    classDef nn fill:#ffe6cc,stroke:#d79b00
    classDef infra fill:#f5f5f5,stroke:#666666
```
