# devsisters/dqn-tensorflow architecture

> Architecture diagram of the GitHub repository devsisters/dqn-tensorflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/devsisters/dqn-tensorflow
- Repository: https://github.com/devsisters/dqn-tensorflow

## Overview

##### Project Type and Purpose
- This is an open-source research implementation of “Human-Level Control through Deep Reinforcement Learning” using TensorFlow.
- It provides a Deep Q-Network (DQN) agent that can train on and play Atari games via OpenAI Gym.
- Key features: experience replay, fixed Q-targets, configurable hyperparameters, model checkpointing.

##### File Structure
- Top-level files:
– main.py: orchestrator for training/testing loops
– config.py: hyperparameter definitions
– dqn/ (package): core modules
– checkpoints/: directory tree storing TensorFlow model files (.meta, checkpoint index)
– assets/: static images for README
- Inside dqn/:
– agent.py: DQNAgent class, action selection, training methods
– base.py: base classes/interfaces
– environment.py: wrapper around gym.Env, preprocessing (grayscale, resize)
– history.py: frame-stacking buffer
– replay_memory.py: cyclic buffer for experience replay
– ops.py: TensorFlow ops (network definition, loss)
– utils.py: helper functions (e.g., logging, epsilon schedule)

##### README for Architecture Insights
- Confirms use of OpenAI Gym environments, TensorFlow 0.12, optional display.
- Describes training/testing workflows:
– Training: run main.py with --is_train=True
– Testing/Recording: run main.py with --is_train=False and --display=True

##### Main Components
a. Orchestrator (main.py)
b. Configuration Loader (config.py)
c. Gym Environment (OpenAI Gym)
d. Preprocessing & History (dqn.environment, dqn.history)
e. Replay Memory Store (dqn.replay_memory)
f. DQN Agent (dqn.agent)
– Q-Network (online network)
– Target Network
– Optimizer (TensorFlow)
g. Experience Replay Loop
h. Checkpoint Storage (checkpoints/ directory)
i. External Dependencies: TensorFlow, Gym, tqdm, SciPy/OpenCV

##### Relationships and Data Flows
- main.py initializes configuration → instantiates Environment & Agent → starts the train/test loop.
- In each time step:
1. Agent queries Environment for new state (after preprocessing).
2. Agent selects action via ε-greedy policy.
3. Environment returns next state, reward, done flag.
4. History stacks frames (dqn.history).
5. Experience tuple appended to Replay Memory.
6. If enough samples, Agent samples mini-batch from Replay Memory.
7. Agent runs TensorFlow session: computes Q-values (ops.py), calculates loss, applies gradients.
8. Every N steps, Target Network weights ← Online Network weights.
9. Periodically, model saved to checkpoints/.

##### Architectural Patterns and Design Principles
- Modular Package Layout: each concern in its own module (separation of concerns).
- Data Flow Pipeline: clear sequence from environment to memory to training.
- Periodic Synchronization Pattern: fixed targets for stable Q-learning.
- Externalization of Hyperparameters (config.py) for reproducibility.
- Checkpointing for resume and evaluation.

##### Relevant Technologies and Libraries
- Python 2.7 / 3.x
- TensorFlow 0.12 (computation graph, sessions, checkpoint I/O)
- OpenAI Gym (environment API)
- tqdm (progress bars)
- SciPy / OpenCV (image pre-processing)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration Layer
    subgraph "Orchestration"
        Main["main.py"]:::orchestrator
        Config["config.py"]:::config
    end

    %% dqn Package Modules
    subgraph "dqn Package"
        Base["dqn/base.py"]:::core
        Env["dqn/environment.py"]:::core
        History["dqn/history.py"]:::core
        Replay["dqn/replay_memory.py"]:::datastore
        Utils["dqn/utils.py"]:::core
        Agent["dqn/agent.py"]:::core
        Ops["dqn/ops.py"]:::core
    end

    %% Agent Internals
    subgraph "Agent Internals"
        QNet["Q-Network"]:::core
        TNet["Target Network"]:::core
        Optimizer["Optimizer"]:::core
    end

    %% External Dependencies
    Gym["OpenAI Gym"]:::external
    TF["TensorFlow 0.12"]:::external
    TQDM["tqdm"]:::external
    Sci["SciPy/OpenCV"]:::external

    %% Data Stores and Assets
    Checkpoints["checkpoints/"]:::datastore
    Assets["assets/"]:::assets

    %% Relationships
    Main -->|"loads"| Config
    Main -->|"instantiates"| Env
    Main -->|"instantiates"| Agent
    Main -->|"uses"| Utils
    Main --> TF
    Main --> Gym
    Main --> TQDM

    Env -->|"preprocesses"| History
    History -->|"stacked frames"| Agent
    Agent -->|"store experience"| Replay
    Agent -->|"sample batch"| Replay
    Replay -->|"batch data"| Agent

    Agent -->|"compute loss"| Ops
    Agent -->|"apply gradients"| Ops
    Agent -->|"sync weights every N steps"| Ops
    Ops -->|"save model"| Checkpoints
    Main -->|"read/write checkpoints"| Checkpoints
    Main -->|"load assets"| Assets

    Agent --> QNet
    Agent --> TNet
    Agent --> Optimizer
    Env --> Sci

    %% Click Events
    click Main "https://github.com/devsisters/dqn-tensorflow/blob/master/main.py"
    click Config "https://github.com/devsisters/dqn-tensorflow/blob/master/config.py"
    click Base "https://github.com/devsisters/dqn-tensorflow/blob/master/dqn/base.py"
    click Env "https://github.com/devsisters/dqn-tensorflow/blob/master/dqn/environment.py"
    click History "https://github.com/devsisters/dqn-tensorflow/blob/master/dqn/history.py"
    click Replay "https://github.com/devsisters/dqn-tensorflow/blob/master/dqn/replay_memory.py"
    click Utils "https://github.com/devsisters/dqn-tensorflow/blob/master/dqn/utils.py"
    click Agent "https://github.com/devsisters/dqn-tensorflow/blob/master/dqn/agent.py"
    click Ops "https://github.com/devsisters/dqn-tensorflow/blob/master/dqn/ops.py"
    click Checkpoints "https://github.com/devsisters/dqn-tensorflow/tree/master/checkpoints/"
    click Assets "https://github.com/devsisters/dqn-tensorflow/tree/master/assets/"

    %% Styles
    classDef orchestrator fill:#8ab6d6,stroke:#333,stroke-width:1px
    classDef config fill:#8ab6d6,stroke:#333,stroke-width:1px
    classDef core fill:#a3d39c,stroke:#333,stroke-width:1px
    classDef datastore fill:#f5b14c,stroke:#333,stroke-width:1px
    classDef external fill:#cccccc,stroke:#333,stroke-width:1px
    classDef assets fill:#e0e0e0,stroke:#333,stroke-width:1px
```
