# 7jep7/rl-projects architecture

> Architecture diagram of the GitHub repository 7jep7/rl-projects, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/7jep7/rl-projects
- Repository: https://github.com/7jep7/rl-projects

## Overview

Based on your repository’s file tree and README content, the project is clearly a collection of reinforcement learning (RL) experiments and projects. It spans several RL methods and environments—from a simple CartPole DQN implementation to more involved experiments that use PPO and SAC on environments like LunarLander and HalfCheetah. The main purpose is to document hands-on exploration in RL, combining theory with practical implementation, with a strong emphasis on experimentation, logging, and model comparison.

##### Main Components

- Experiment Modules
- “CartPole_DQN”: Contains core algorithm implementations (dqn.py, dqn_cartpole.py, replay_memory.py) for the classic CartPole DQN experiment.
- “RL_Sutton_and_Barto”: Houses scripts for exercises based on the Sutton & Barto textbook.
- (Additional experiments indicated in the README such as week02_sac_halfcheetah, week03_rl_baselines, etc. are conceptually part of the overall project even if they don’t show up in the file tree explicitly.)

- Training and Evaluation Scripts
- Located in the “scripts” directory, these files (like train_halfcheetah_gpu.py, train_lunarlander.py, test_lunarlander.py) are the entry points for running and comparing different RL experiments. They are the controllers that kick off experiment routines, choose environments, and trigger training routines.

- Environment and Simulation Interfaces
- Although not explicitly in the file tree, the README refers to multiple simulation environments (e.g. LunarLander-v3, HalfCheetah-v3, etc.). These are accessible via Gym, MuJoCo, or even Isaac Gym.

- Data Management Components
- “logs”: This folder holds TensorBoard logs (with subdirectories for different experiments) showing training metrics and performance over time.
- “models”: This folder is used to store trained agent models (like ppo_lunarlander.zip) for later evaluation or deployment.

- Dependency and Configuration Management
- Top-level files like “requirements.txt” and “.gitignore” help set up the environment and version control.

##### Relationships and Interactions

- The Training Scripts (in “scripts”) serve as the orchestrators:
- They initialize the chosen experiment (by invoking code from modules like “CartPole_DQN” or those based on Sutton & Barto).
- They interact with simulation environments (Gym, MuJoCo, Isaac Gym) to observe state transitions.

- Within an experiment module such as “CartPole_DQN”:
- The RL Algorithm (DQN) module, implemented in files like dqn.py and dqn_cartpole.py, handles policy evaluation and decision-making.
- The “replay_memory.py” provides the crucial mechanism for experience replay.

- Logging and Model Saving:
- During training, experiment data (training progress, rewards, losses, etc.) is sent to the “logs” folder (suitable for TensorBoard visualization).
- Trained weights or models are saved into the “models” folder for residue evaluation or potential deployment.

##### Architectural Patterns and Design Choices

- Modular Design:
- Each RL experiment or algorithm is housed in its own directory; training scripts are decoupled from the algorithm implementations. This separation supports extensibility and isolated testing.

- Experiment Orchestration:
- Scripts act as the controller layer, invoking specific algorithm modules that interact with simulation environments.

- Data Flow Emphasis:
- Data flows from the environment to the algorithms and back as learning signals, while logging and model storage provide feedback loops.

- Technology Stack:
- Python (with specific versions mentioned in the README)
- Deep learning libraries: likely PyTorch given the linkage to PyTorch RL tutorials.
- Simulation environments: OpenAI Gym, MuJoCo, Isaac Gym.
- Deployment/logging tools: TensorBoard (logs) and saved model artifacts.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Training Scripts"]:::controller
    subgraph "Experiment Modules"
        direction TB
        subgraph "CartPole_DQN"
            direction TB
            B1["dqn.py"]:::algorithm
            B2["dqn_cartpole.py"]:::algorithm
            B3["replay_memory.py"]:::algorithm
        end
        C["Sutton & Barto Exercises"]:::algorithm
    end
    D["Simulation Environments (Gym,MuJoCo,IsaacGym)"]:::external
    E["Logging (TensorBoard)"]:::io
    F["Model Storage"]:::io
    subgraph "Configuration"
        direction TB
        G["requirements.txt"]:::config
        H[".gitignore"]:::config
    end

    %% Relationships
    A -->|"initiates"| B1
    A -->|"initiates"| C
    B1 -->|"env_interaction"| D
    C -->|"env_interaction"| D
    D -->|"feedback"| B1
    B1 -->|"logs_data"| E
    C -->|"logs_data"| E
    B1 -->|"saves_model"| F
    C -->|"saves_model"| F
    A -->|"uses"| G
    A -->|"uses"| H

    %% Click Events
    click A "https://github.com/7jep7/rl-projects/tree/main/scripts/"
    click B1 "https://github.com/7jep7/rl-projects/blob/main/CartPole_DQN/dqn.py"
    click B2 "https://github.com/7jep7/rl-projects/blob/main/CartPole_DQN/dqn_cartpole.py"
    click B3 "https://github.com/7jep7/rl-projects/blob/main/CartPole_DQN/replay_memory.py"
    click C "https://github.com/7jep7/rl-projects/blob/main/RL_Sutton_and_Barto/sutton_barto_exercises.py"
    click E "https://github.com/7jep7/rl-projects/tree/main/logs/"
    click F "https://github.com/7jep7/rl-projects/tree/main/models/"
    click G "https://github.com/7jep7/rl-projects/blob/main/requirements.txt"
    click H "https://github.com/7jep7/rl-projects/blob/main/.gitignore"

    %% Class Styles
    classDef controller fill:#a2d9ce,stroke:#1f618d,stroke-width:2px;
    classDef algorithm fill:#aed6f1,stroke:#2874a6,stroke-width:2px;
    classDef external fill:#f9e79f,stroke:#b7950b,stroke-width:2px;
    classDef io fill:#f5cba7,stroke:#d35400,stroke-width:2px;
    classDef config fill:#d7dbdd,stroke:#707b7c,stroke-width:2px;
```
