How devsisters/dqn-tensorflow works
Project Type and Purpose
• This is an open-source research implementation of “Human-Level Control through Deep Reinforcement Learning” using TensorFlow.
• It provides a Deep Q-Network (DQN) agent that can train on and play Atari games via OpenAI Gym.
• Key features: experience replay, fixed Q-targets, configurable hyperparameters, model checkpointing.
File Structure
• Top-level files:
– main.py: orchestrator for training/testing loops
– config.py: hyperparameter definitions
– dqn/ (package): core modules
– checkpoints/: directory tree storing TensorFlow model files (.meta, checkpoint index)
– assets/: static images for README
• Inside dqn/:
– agent.py: DQNAgent class, action selection, training methods
– base.py: base classes/interfaces
– environment.py: wrapper around gym.Env, preprocessing (grayscale, resize)
– history.py: frame-stacking buffer
– replay_memory.py: cyclic buffer for experience replay
– ops.py: TensorFlow ops (network definition, loss)
– utils.py: helper functions (e.g., logging, epsilon schedule)
README for Architecture Insights
• Confirms use of OpenAI Gym environments, TensorFlow 0.12, optional display.
• Describes training/testing workflows:
– Training: run main.py with --is_train=True
– Testing/Recording: run main.py with --is_train=False and --display=True
Main Components
a. Orchestrator (main.py)
b. Configuration Loader (config.py)
c. Gym Environment (OpenAI Gym)
d. Preprocessing & History (dqn.environment, dqn.history)
e. Replay Memory Store (dqn.replay_memory)
f. DQN Agent (dqn.agent)
– Q-Network (online network)
– Target Network
– Optimizer (TensorFlow)
g. Experience Replay Loop
h. Checkpoint Storage (checkpoints/ directory)
i. External Dependencies: TensorFlow, Gym, tqdm, SciPy/OpenCV
Relationships and Data Flows
• main.py initializes configuration → instantiates Environment & Agent → starts the train/test loop.
• In each time step:
1. Agent queries Environment for new state (after preprocessing).
2. Agent selects action via ε-greedy policy.
3. Environment returns next state, reward, done flag.
4. History stacks frames (dqn.history).
5. Experience tuple appended to Replay Memory.
6. If enough samples, Agent samples mini-batch from Replay Memory.
7. Agent runs TensorFlow session: computes Q-values (ops.py), calculates loss, applies gradients.
8. Every N steps, Target Network weights ← Online Network weights.
9. Periodically, model saved to checkpoints/.
Architectural Patterns and Design Principles
• Modular Package Layout: each concern in its own module (separation of concerns).
• Data Flow Pipeline: clear sequence from environment to memory to training.
• Periodic Synchronization Pattern: fixed targets for stable Q-learning.
• Externalization of Hyperparameters (config.py) for reproducibility.
• Checkpointing for resume and evaluation.
Relevant Technologies and Libraries
• Python 2.7 / 3.x
• TensorFlow 0.12 (computation graph, sessions, checkpoint I/O)
• OpenAI Gym (environment API)
• tqdm (progress bars)
• SciPy / OpenCV (image pre-processing)