# happy95059/rl_explore_problem architecture

> Architecture diagram of the GitHub repository happy95059/rl_explore_problem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/happy95059/rl_explore_problem
- Repository: https://github.com/happy95059/rl_explore_problem

## Overview

##### Project Type & Purpose
- This is a reinforcement learning (RL) project focused on a grid world where a robot (agent) navigates an environment to “capture” a randomly moving prey. The design utilizes a Deep Q-Network (DQN) approach, augmented with a self-defined informational reward/feature. The project simulates the environment and implements RL with saved models for decision-making.

##### Main Components
a. Environment Module:
- Represented by grid_world_save.py.
- Role: Simulates the world with a grid layout, including walls, floors, the robot (blue dot), and the prey (red square).
- Elements: Map details, definitions for walkable floors, walls, and the dynamic elements (agent and prey).

b. Agent & DQN:
- Represented via final_project_save.py, which likely integrates the RL agent’s training and decision-making logic.
- Role: Implements the DQN algorithm, handling the state (5x5 matrices representing local view), actions, and integration of the informational measure either as a reward or as part of the policy decision.
- Elements: Reward function logic, state definition, action selection mechanism, and training loop.

c. Model Storage:
- Represented by the "model" directory containing files such as model.ckpt and target_model.ckpt.
- Role: Stores the trained DQN network’s parameters and the target network for stabilizing training.
- Elements: Checkpoint files to be loaded/saved during training/inference.

##### Relationships and Interactions
- Interaction Flow:
- The Environment (grid_world) continuously updates the state of the grid (with walls, agent, and prey).
- The Agent retrieves the current state (a 5x5 state matrix) from the Environment, selects an action based on the DQN policy, and performs the action.
- The Agent then uses this feedback to update the DQN network; periodically, the model is saved in the “model” directory.
- The DQN’s target network (saved as target_model.ckpt) is used to stabilize learning updates.

##### Architectural Patterns & Design Principles
- Separation of Concerns: The project differentiates between simulation (environment handling) and learning (agent/action selection and DQN updates).
- Modular Design: Each script and directory manages a distinct aspect of the system (physics of the grid world, RL training, and model persistence).
- Feedback Loop: Reinforcement learning inherently uses a closed-loop where execution (action), environment feedback, and policy update interact dynamically.
- Custom Reward Enhancement: The design introduces a self-defined informational metric either to augment rewards or bias the DQN decision process.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Simulation Layer"
        ENV["Environment / Grid World\n(Map, Obstacles, Agent Position, Prey Position)"]:::env
        VIS["Display / Visualization"]:::display
        ENV -->|"visualUpdate"| VIS
    end

    subgraph "Learning Layer"
        AGENT["Agent (DQN)\n(State Extraction (5x5 Matrix), Action Decision, Reward Calculation, Learning Loop)"]:::agent
    end

    subgraph "Storage Layer"
        STORAGE["Model Storage\n(model.ckpt, target_model.ckpt)"]:::storage
    end

    ENV -->|"stateReward"| AGENT
    AGENT -->|"actionCmd"| ENV
    AGENT -->|"modelOps"| STORAGE

    click ENV "https://github.com/happy95059/rl_explore_problem/blob/main/grid_world_save.py"
    click AGENT "https://github.com/happy95059/rl_explore_problem/blob/main/final_project_save.py"
    click STORAGE "https://github.com/happy95059/rl_explore_problem/tree/main/model/"

    classDef env fill:#aaddff,stroke:#000,stroke-width:2px;
    classDef agent fill:#aaffaa,stroke:#000,stroke-width:2px;
    classDef storage fill:#ffffaa,stroke:#000,stroke-width:2px;
    classDef display fill:#ffccaa,stroke:#000,stroke-width:2px;
```
