# mpaepper/dqn architecture

> Architecture diagram of the GitHub repository mpaepper/dqn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mpaepper/dqn
- Repository: https://github.com/mpaepper/dqn

## Overview

##### Main Components
- Agent: The central module (agent.py) that ties together the different components needed for learning.
- Model: Represented by model.py, this is the neural network (using the CNN architecture mentioned in the README) that is trained.
- Memory: Implemented in memory.py, it stores experiences that the agent uses for training.
- Policy: Found in policy.py, it defines how the agent decides on actions (e.g., epsilon-greedy exploration).
- Processor: The processing layer (processor.py) that handles pre- or post-processing of observations and other data, optimally used to reduce requirements in memory storage.
- Atari Wrappers: Provided in atari_wrappers.py, these modules adapt Atari environments to be compatible with the agent’s inputs.
- Tracker: (tracker.py) Likely used for logging, monitoring, or tracking key training statistics.
- Examples: The examples folder contains sample implementations (for Cartpole and Pacman) to illustrate how to integrate and run the system.

##### Relationships and Interactions
- The Agent component is the orchestrator which utilizes the Model, Memory, Policy, and Processor.
- The Agent interacts with an external Environment (for example, Cartpole or Pacman) which can have additional wrappers applied (via atari_wrappers) to suit the DQN input requirements.
- The Model receives data (processed observations) possibly from the Processor and is used to predict actions based on stored experiences.
- The Memory component stores experiences (state, action, reward, next state) that the agent collects during interaction with the environment.
- The Policy component drives action selection, balancing exploration vs. exploitation based on the current state and model predictions.
- The Tracker records key performance metrics during training which help in debugging and monitoring learning progress.
- The Examples in the examples folder demonstrate how to instantiate these components together.

##### Architectural Patterns
- Modular Design: Each major functionality (model, memory, policy, processor) is separated into distinct modules.
- Separation of Concerns: The architecture cleanly separates components like data processing (processor), decision-making (policy), and state evaluation (model).
- Extensibility: With components like atari_wrappers and examples, the design allows the introduction of different environments and augmentation of functionalities without significant refactoring.

##### Technologies and Frameworks
- Keras/TensorFlow: Core libraries for building and training the deep neural network model.
- Python’s standard libraries: Used across modules for implementation.
- Testing Framework: There is a tests directory (e.g., tests/replay_memory.py) for ensuring component correctness.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Agent Internal Components"
        A["Agent (agent.py)"]:::core
        M["Model (model.py)"]:::core
        MEM["Memory (memory.py)"]:::core
        P["Policy (policy.py)"]:::core
        PR["Processor (processor.py)"]:::core
    end

    E["Environment"]:::external
    AW["Atari Wrappers (atari_wrappers.py)"]:::external
    T["Tracker (tracker.py)"]:::auxiliary
    EX["Examples"]:::auxiliary

    E -->|"sendsObs"| PR
    PR -->|"pre-process"| A
    PR -->|"feedsObs"| M
    A -->|"storesExp"| MEM
    MEM -->|"replayExp"| M
    MEM -->|"suppliesExp"| M
    M -->|"predicts"| P
    P -->|"decidesAction"| A
    A -->|"acts"| E
    AW -->|"wraps"| E
    T -->|"logs"| A
    EX -->|"demonstrates"| A

    %% Click Events
    click A "https://github.com/mpaepper/dqn/blob/master/agent.py"
    click M "https://github.com/mpaepper/dqn/blob/master/model.py"
    click MEM "https://github.com/mpaepper/dqn/blob/master/memory.py"
    click P "https://github.com/mpaepper/dqn/blob/master/policy.py"
    click PR "https://github.com/mpaepper/dqn/blob/master/processor.py"
    click AW "https://github.com/mpaepper/dqn/blob/master/atari_wrappers.py"
    click T "https://github.com/mpaepper/dqn/blob/master/tracker.py"
    click EX "https://github.com/mpaepper/dqn/tree/master/examples/"

    %% Styling for visual clarity
    classDef core fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef external fill:#A9DFBF,stroke:#196F3D,stroke-width:2px;
    classDef auxiliary fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
```
