# gana-xg/reinforcement-learning architecture

> Architecture diagram of the GitHub repository gana-xg/reinforcement-learning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gana-xg/reinforcement-learning
- Repository: https://github.com/gana-xg/reinforcement-learning

## Overview

1. **What this repository is**
   - This is a desktop reinforcement-learning launcher, not a web service.
   - The top-level user flow is: `main.py` presents a console menu, the user selects one of four games, and the chosen agent launches its own training/visualization loop.

2. **Shared RL core**
   - The repo has a common deep-Q learning core in `model.py`, used by all agents.
   - That shared layer likely contains the PyTorch network definition and trainer logic, while each game-specific agent builds states, chooses actions, and feeds transitions into the trainer.
   - All agents use a Double DQN pattern with an online network and a target network.

3. **Environment boundary: game logic vs agent logic**
   - Each game is split into two responsibilities:
     - an environment or dashboard module that simulates the world and renders it,
     - an agent module that translates environment state into RL features and controls training.
   - This separation is consistent across Snake, Flappy Bird, GridWorld, and Taxi.

4. **Snake subsystem**
   - `game.py` implements the Snake environment and Pygame rendering.
   - `agent.py` implements the Snake DDQN agent and its training loop.
   - Snake uses a compact state vector and a 3-action policy, making it the simplest environment in the repo.

5. **Flappy Bird subsystem**
   - `flappy_bird.py` contains the custom physics and Pygame visualization for Flappy Bird.
   - `flappy_agent.py` contains the RL agent and training logic.
   - This environment uses continuous, normalized state features and a 2-action control loop, with physics-driven updates each step.

6. **GridWorld subsystem**
   - `gridworld_game.py` implements the deterministic 6x6 grid world and its dashboard.
   - `gridworld_agent.py` contains the agent logic.
   - The environment is maze-like and discrete, with obstacles, a fixed start, and a fixed goal; the UI emphasizes current position, obstacles, and training statistics.

7. **Taxi subsystem**
   - `taxi_agent.py` is the RL controller for the Taxi-v3 environment.
   - `taxi_dashboard.py` renders the taxi world with a custom Pygame UI.
   - This is the only environment that depends on Gymnasium rather than a fully custom game engine.

8. **Visualization and training feedback**
   - `helper.py` supports live plotting, so training progress is visible while the agent runs.
   - The environment modules also handle interactive rendering, so the visualization is part of the training loop rather than a separate observer service.

9. **Persistence and resume capability**
   - `memory_store.py` centralizes saving and loading of training state.
   - The persisted artifacts under `model/<game>/` include:
     - online model weights,
     - target model weights,
     - replay memory,
     - training statistics.
   - This makes training resumable across sessions instead of starting from scratch each run.

10. **Artifact layout**
   - The `model/` directory is not source code; it is the checkpoint store.
   - Each game has its own subfolder or files, which keeps replay buffers and weights isolated by environment.
   - That means the architecture has a clear boundary between runtime code and training state.

11. **Runtime shape of the system**
   - The main loop is event-driven and local: environment step -> agent observes state -> agent selects action -> environment updates -> reward/next state returned -> trainer stores transition and updates networks.
   - There is no external API layer, database, or distributed worker model in this repository.
   - The entire system runs as a single-process interactive RL application with Pygame rendering and PyTorch training.

12. **How to think about the architecture diagram**
   - A useful diagram would show three layers:
     - launcher/orchestration (`main.py`),
     - shared RL infrastructure (`model.py`, `memory_store.py`, `helper.py`),
     - four environment-specific pipelines (`agent.py` + game/dashboard module pairs, plus Taxi on Gymnasium).
   - The key data flow is from environment state into the agent, then action back into the environment, with checkpoints flowing from training code into `model/`.

Main entry points: [`agent.py`](https://github.com/gana-xg/reinforcement-learning/blob/main/agent.py) (Snake Agent), [`flappy_agent.py`](https://github.com/gana-xg/reinforcement-learning/blob/main/flappy_agent.py) (Flappy Agent), [`gridworld_agent.py`](https://github.com/gana-xg/reinforcement-learning/blob/main/gridworld_agent.py) (Grid Agent), [`taxi_agent.py`](https://github.com/gana-xg/reinforcement-learning/blob/main/taxi_agent.py) (Taxi Agent)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_launcher["Launcher"]
  node_main_py["Main<br/>launcher<br/>[main.py]"]
end

subgraph group_core["Shared RL Core"]
  node_model_py["DQN Core<br/>[model.py]"]
  node_memory_store_py["State Store<br/>checkpoint IO<br/>[memory_store.py]"]
  node_helper_py["Plotting<br/>visualization<br/>[helper.py]"]
end

subgraph group_snake["Snake"]
  node_snake_agent_py["Snake Agent<br/>ddqn agent<br/>[agent.py]"]
  node_game_py["Snake Game<br/>env/render<br/>[game.py]"]
  node_snake_human_py["Snake Human<br/>manual play"]
end

subgraph group_flappy["Flappy Bird"]
  node_flappy_agent_py["Flappy Agent<br/>ddqn agent<br/>[flappy_agent.py]"]
  node_flappy_bird_py["Flappy Bird<br/>env/physics<br/>[flappy_bird.py]"]
end

subgraph group_gridworld["GridWorld"]
  node_gridworld_agent_py["Grid Agent<br/>ddqn agent<br/>[gridworld_agent.py]"]
  node_gridworld_game_py["GridWorld<br/>env/dashboard<br/>[gridworld_game.py]"]
end

subgraph group_taxi["Taxi"]
  node_taxi_agent_py["Taxi Agent<br/>rl controller<br/>[taxi_agent.py]"]
  node_taxi_dashboard_py["Taxi UI<br/>dashboard<br/>[taxi_dashboard.py]"]
end

subgraph group_checkpoints["Checkpoints"]
  node_model_store[("Model Store<br/>checkpoints")]
end

node_main_py -->|"launch"| node_snake_agent_py
node_main_py -->|"launch"| node_flappy_agent_py
node_main_py -->|"launch"| node_gridworld_agent_py
node_main_py -->|"launch"| node_taxi_agent_py
node_snake_agent_py -->|"step"| node_game_py
node_flappy_agent_py -->|"step"| node_flappy_bird_py
node_gridworld_agent_py -->|"step"| node_gridworld_game_py
node_taxi_agent_py -->|"step"| node_taxi_dashboard_py
node_snake_agent_py -->|"train"| node_model_py
node_flappy_agent_py -->|"train"| node_model_py
node_gridworld_agent_py -->|"train"| node_model_py
node_taxi_agent_py -->|"train"| node_model_py
node_snake_agent_py -->|"persist"| node_memory_store_py
node_flappy_agent_py -->|"persist"| node_memory_store_py
node_gridworld_agent_py -->|"persist"| node_memory_store_py
node_taxi_agent_py -->|"persist"| node_memory_store_py
node_snake_agent_py -->|"plot"| node_helper_py
node_flappy_agent_py -->|"plot"| node_helper_py
node_gridworld_agent_py -->|"plot"| node_helper_py
node_taxi_agent_py -->|"plot"| node_helper_py
node_memory_store_py -->|"read/write"| node_model_store
node_snake_human_py -->|"uses"| node_game_py

click node_main_py "https://github.com/gana-xg/reinforcement-learning/blob/main/main.py"
click node_model_py "https://github.com/gana-xg/reinforcement-learning/blob/main/model.py"
click node_memory_store_py "https://github.com/gana-xg/reinforcement-learning/blob/main/memory_store.py"
click node_helper_py "https://github.com/gana-xg/reinforcement-learning/blob/main/helper.py"
click node_snake_agent_py "https://github.com/gana-xg/reinforcement-learning/blob/main/agent.py"
click node_game_py "https://github.com/gana-xg/reinforcement-learning/blob/main/game.py"
click node_flappy_agent_py "https://github.com/gana-xg/reinforcement-learning/blob/main/flappy_agent.py"
click node_flappy_bird_py "https://github.com/gana-xg/reinforcement-learning/blob/main/flappy_bird.py"
click node_gridworld_agent_py "https://github.com/gana-xg/reinforcement-learning/blob/main/gridworld_agent.py"
click node_gridworld_game_py "https://github.com/gana-xg/reinforcement-learning/blob/main/gridworld_game.py"
click node_taxi_agent_py "https://github.com/gana-xg/reinforcement-learning/blob/main/taxi_agent.py"
click node_taxi_dashboard_py "https://github.com/gana-xg/reinforcement-learning/blob/main/taxi_dashboard.py"
click node_model_store "https://github.com/gana-xg/reinforcement-learning/tree/main/model"
click node_snake_human_py "https://github.com/gana-xg/reinforcement-learning/blob/main/snake_game_human.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_py,node_model_store toneBlue
class node_model_py,node_memory_store_py,node_helper_py toneAmber
class node_snake_agent_py,node_game_py,node_snake_human_py toneMint
class node_flappy_agent_py,node_flappy_bird_py toneRose
class node_gridworld_agent_py,node_gridworld_game_py toneIndigo
class node_taxi_agent_py,node_taxi_dashboard_py toneTeal
```

## Components

### Launcher

Console entrypoint and game selection

- [Main](https://github.com/gana-xg/reinforcement-learning/blob/main/main.py): launcher · Console menu that selects a game and starts its loop · `main.py`

### Shared RL Core

Common DQN training, memory, and plotting utilities

- [DQN Core](https://github.com/gana-xg/reinforcement-learning/blob/main/model.py): ddqn core · Shared online/target network and trainer logic · `model.py`
- [State Store](https://github.com/gana-xg/reinforcement-learning/blob/main/memory_store.py): checkpoint IO · Loads and saves replay memory, weights, and stats · `memory_store.py`
- [Plotting](https://github.com/gana-xg/reinforcement-learning/blob/main/helper.py): visualization · Live training charts and helper utilities · `helper.py`

### Snake

Snake environment and agent

- [Snake Agent](https://github.com/gana-xg/reinforcement-learning/blob/main/agent.py): ddqn agent · Snake state encoding, action selection, and training loop · `agent.py`
- [Snake Game](https://github.com/gana-xg/reinforcement-learning/blob/main/game.py): env/render · Snake world simulation and Pygame rendering · `game.py`
- [Snake Human](https://github.com/gana-xg/reinforcement-learning/blob/main/snake_game_human.py): manual play · Human-playable Snake variant · `snake_game_human.py`

### Flappy Bird

Flappy Bird environment and agent

- [Flappy Agent](https://github.com/gana-xg/reinforcement-learning/blob/main/flappy_agent.py): ddqn agent · Flappy Bird control policy and training loop · `flappy_agent.py`
- [Flappy Bird](https://github.com/gana-xg/reinforcement-learning/blob/main/flappy_bird.py): env/physics · Custom physics, state features, and visualization · `flappy_bird.py`

### GridWorld

Deterministic grid environment and agent

- [Grid Agent](https://github.com/gana-xg/reinforcement-learning/blob/main/gridworld_agent.py): ddqn agent · GridWorld policy, state features, and training loop · `gridworld_agent.py`
- [GridWorld](https://github.com/gana-xg/reinforcement-learning/blob/main/gridworld_game.py): env/dashboard · Deterministic 6x6 grid, obstacles, and goal rendering · `gridworld_game.py`

### Taxi

Gymnasium-based taxi agent and dashboard

- [Taxi Agent](https://github.com/gana-xg/reinforcement-learning/blob/main/taxi_agent.py): rl controller · Taxi-v3 control logic and training loop · `taxi_agent.py`
- [Taxi UI](https://github.com/gana-xg/reinforcement-learning/blob/main/taxi_dashboard.py): dashboard · Custom Pygame visualization for Taxi-v3 · `taxi_dashboard.py`

### Checkpoints

Saved models, replay buffers, and training stats

- [Model Store](https://github.com/gana-xg/reinforcement-learning/tree/main/model): checkpoints · Per-game saved weights, replay buffers, and stats · `model`

## Connections

- Main → Snake Agent: launch · Starts the Snake training loop
- Main → Flappy Agent: launch · Starts the Flappy Bird training loop
- Main → Grid Agent: launch · Starts the GridWorld training loop
- Main → Taxi Agent: launch · Starts the Taxi training loop
- Snake Agent → Snake Game: step · Controls Snake through environment state and rewards
- Flappy Agent → Flappy Bird: step · Controls Flappy Bird through physics updates
- Grid Agent → GridWorld: step · Controls GridWorld state transitions
- Taxi Agent → Taxi UI: step · Controls Taxi-v3 with custom dashboard
- Snake Agent → DQN Core: train · Uses shared DDQN model logic
- Flappy Agent → DQN Core: train · Uses shared DDQN model logic
- Grid Agent → DQN Core: train · Uses shared DDQN model logic
- Taxi Agent → DQN Core: train · Uses shared DDQN model logic
- Snake Agent → State Store: persist · Loads and saves Snake checkpoints
- Flappy Agent → State Store: persist · Loads and saves Flappy Bird checkpoints
- Grid Agent → State Store: persist · Loads and saves GridWorld checkpoints
- Taxi Agent → State Store: persist · Loads and saves Taxi checkpoints
- Snake Agent → Plotting: plot · Streams training metrics to plots
- Flappy Agent → Plotting: plot · Streams training metrics to plots
- Grid Agent → Plotting: plot · Streams training metrics to plots
- Taxi Agent → Plotting: plot · Streams training metrics to plots
- State Store → Model Store: read/write · Stores per-game model weights and replay data
- Snake Human → Snake Game: uses · Shares Snake world and rendering code
