How happy95059/rl_explore_problem works
Project Type & Purpose
• This is a reinforcement learning (RL) project focused on a grid world where a robot (agent) navigates an environment to “capture” a randomly moving prey. The design utilizes a Deep Q-Network (DQN) approach, augmented with a self-defined informational reward/feature. The project simulates the environment and implements RL with saved models for decision-making.
Main Components
a. Environment Module:
• Represented by grid_world_save.py.
• Role: Simulates the world with a grid layout, including walls, floors, the robot (blue dot), and the prey (red square).
• Elements: Map details, definitions for walkable floors, walls, and the dynamic elements (agent and prey).
b. Agent & DQN:
• Represented via final_project_save.py, which likely integrates the RL agent’s training and decision-making logic.
• Role: Implements the DQN algorithm, handling the state (5x5 matrices representing local view), actions, and integration of the informational measure either as a reward or as part of the policy decision.
• Elements: Reward function logic, state definition, action selection mechanism, and training loop.
c. Model Storage:
• Represented by the "model" directory containing files such as model.ckpt and target_model.ckpt.
• Role: Stores the trained DQN network’s parameters and the target network for stabilizing training.
• Elements: Checkpoint files to be loaded/saved during training/inference.
Relationships and Interactions
• Interaction Flow:
• The Environment (grid_world) continuously updates the state of the grid (with walls, agent, and prey).
• The Agent retrieves the current state (a 5x5 state matrix) from the Environment, selects an action based on the DQN policy, and performs the action.
• The Agent then uses this feedback to update the DQN network; periodically, the model is saved in the “model” directory.
• The DQN’s target network (saved as target_model.ckpt) is used to stabilize learning updates.
Architectural Patterns & Design Principles
• Separation of Concerns: The project differentiates between simulation (environment handling) and learning (agent/action selection and DQN updates).
• Modular Design: Each script and directory manages a distinct aspect of the system (physics of the grid world, RL training, and model persistence).
• Feedback Loop: Reinforcement learning inherently uses a closed-loop where execution (action), environment feedback, and policy update interact dynamically.
• Custom Reward Enhancement: The design introduces a self-defined informational metric either to augment rewards or bias the DQN decision process.