# sonmezahmet/model-based-learning-on-load-unload-task architecture

> Architecture diagram of the GitHub repository sonmezahmet/model-based-learning-on-load-unload-task, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sonmezahmet/model-based-learning-on-load-unload-task
- Repository: https://github.com/sonmezahmet/model-based-learning-on-load-unload-task

## Overview

##### Project Overview
- This project implements a model-based reinforcement learning task on a grid world via a prioritized sweeping algorithm. The grid world simulates a load-unload task, where various types of states (e.g., Regular, Load, Unload, Rough, Block, Slope with UpperSlope and LowerSlope) are defined and managed dynamically based on user inputs.
- The prioritized sweeping algorithm updates multiple Q tables based on the state (unloaded, loaded but not yet unloaded, unloaded after action). It uses a model (represented as a dictionary) to store experienced transitions and a priority queue to schedule updates based on the TD error (p value).

##### Main Components
a. Environment Component:
- Environment.py – Responsible for creating and managing the grid world. It contains helper functions like __create_2d_matrix, __create_environment, and print_environment.
- Although the specific state classes (State, RegularState, LoadState, etc.) aren’t individually represented in the file tree, they are conceptually part of the Environment module since the README describes their creation and usage.
b. Agent Component:
- Agent.py – Manages the agent’s interactions with the environment, state transitions, and action selection.
c. Prioritized Sweeping Algorithm Component:
- PrioritizedSweeping.py – Implements the prioritized sweeping algorithm, including calculating the TD error, updating Q tables, and adding state-action pairs to the priority queue.
- PriorityQueue.py – Supports the algorithm by providing the data structure required for managing prioritized updates.
d. Supporting Components:
- Constants.py – Contains global constants (e.g., learning rate, discount rate, epsilon, threshold) which are essential for tuning the learning parameters.
- Main.py – Functions as the entry point, orchestrating the simulation by initializing the environment, triggering agent actions, and running the learning loop.

##### Relationships and Interactions
- The Main module initializes the system by reading parameters from Constants.py and creating an environment (using Environment.py). It then instantiates an Agent (from Agent.py) and coordinates the iterative learning loop.
- Within the learning loop, the agent interacts with the environment, observes outcomes, and the model (a dictionary inside the prioritized sweeping module) is updated with state transitions.
- The Prioritized Sweeping module periodically computes the TD error and, if above the threshold, enqueues state-action pairs into the PriorityQueue for further processing.
- There is a close interdependency between the Agent, Environment, and Learning algorithm modules; updating of Q tables (which exist in state objects described in the README) drives the decision-making process.

##### Architectural Patterns and Design Principles
- Modularity: Each component (Agent, Environment, Prioritized Sweeping, Priority Queue) is split into separate files, promoting clean separation of concerns.
- Dynamic Environment Creation: The Environment component is designed to dynamically create and configure the grid world based on user input.
- Iterative Model-Based Learning: The prioritized sweeping strategy prioritizes state updates based on computed TD errors (p values) using a priority queue, emphasizing efficient learning from limited examples.
- Multi-Modal State Management: The design incorporates different Q tables per state type—reflecting decision-making under different conditions (loaded, unloaded, etc.) as explained in the README.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Initialization Subgraph
    subgraph "Initialization"
        Main("Main"):::control
        Const("Constants"):::support
    end

    %% Simulation Setup Subgraph
    subgraph "Simulation Setup"
        Env("Environment"):::environment
        Agent("Agent"):::agent
    end

    %% Learning Module Subgraph
    subgraph "Learning Module"
        PS("Prioritized Sweeping"):::algorithm
        PQ("Priority Queue"):::algorithm
    end

    %% Relationships
    Main -->|"loads"| Const
    Main -->|"initializes"| Env
    Main -->|"instantiates"| Agent
    Agent -->|"interacts"| Env
    Env -->|"observedTransitions"| PS
    Agent -->|"sendsActions"| PS
    PS -->|"computesTDerror"| PQ
    PQ -->|"returnsUpdates"| PS
    PS -->|"updatesQTable"| Agent

    %% Click Events
    click Main "https://github.com/sonmezahmet/model-based-learning-on-load-unload-task/blob/main/Main.py"
    click Const "https://github.com/sonmezahmet/model-based-learning-on-load-unload-task/blob/main/Constants.py"
    click Env "https://github.com/sonmezahmet/model-based-learning-on-load-unload-task/blob/main/Environment.py"
    click Agent "https://github.com/sonmezahmet/model-based-learning-on-load-unload-task/blob/main/Agent.py"
    click PS "https://github.com/sonmezahmet/model-based-learning-on-load-unload-task/blob/main/PrioritizedSweeping.py"
    click PQ "https://github.com/sonmezahmet/model-based-learning-on-load-unload-task/blob/main/PriorityQueue.py"

    %% Styles
    classDef control fill:#f9e79f,stroke:#000,stroke-width:2px;
    classDef support fill:#85c1e9,stroke:#000,stroke-width:2px;
    classDef environment fill:#82e0aa,stroke:#000,stroke-width:2px;
    classDef agent fill:#f1948a,stroke:#000,stroke-width:2px;
    classDef algorithm fill:#bb8fce,stroke:#000,stroke-width:2px;
```
