How gana-xg/reinforcement-learning works
1. What this repository is
- This is a desktop reinforcement-learning launcher, not a web service.
- The top-level user flow is: main.py presents a console menu, the user selects one of four games, and the chosen agent launches its own training/visualization loop.
2. Shared RL core
- The repo has a common deep-Q learning core in model.py, used by all agents.
- That shared layer likely contains the PyTorch network definition and trainer logic, while each game-specific agent builds states, chooses actions, and feeds transitions into the trainer.
- All agents use a Double DQN pattern with an online network and a target network.
3. Environment boundary: game logic vs agent logic
- Each game is split into two responsibilities:
- an environment or dashboard module that simulates the world and renders it,
- an agent module that translates environment state into RL features and controls training.
- This separation is consistent across Snake, Flappy Bird, GridWorld, and Taxi.
4. Snake subsystem
- game.py implements the Snake environment and Pygame rendering.
- agent.py implements the Snake DDQN agent and its training loop.
- Snake uses a compact state vector and a 3-action policy, making it the simplest environment in the repo.
5. Flappy Bird subsystem
- flappy_bird.py contains the custom physics and Pygame visualization for Flappy Bird.
- flappy_agent.py contains the RL agent and training logic.
- This environment uses continuous, normalized state features and a 2-action control loop, with physics-driven updates each step.
6. GridWorld subsystem
- gridworld_game.py implements the deterministic 6x6 grid world and its dashboard.
- gridworld_agent.py contains the agent logic.
- The environment is maze-like and discrete, with obstacles, a fixed start, and a fixed goal; the UI emphasizes current position, obstacles, and training statistics.
7. Taxi subsystem
- taxi_agent.py is the RL controller for the Taxi-v3 environment.
- taxi_dashboard.py renders the taxi world with a custom Pygame UI.
- This is the only environment that depends on Gymnasium rather than a fully custom game engine.
8. Visualization and training feedback
- helper.py supports live plotting, so training progress is visible while the agent runs.
- The environment modules also handle interactive rendering, so the visualization is part of the training loop rather than a separate observer service.
9. Persistence and resume capability
- memory_store.py centralizes saving and loading of training state.
- The persisted artifacts under model/<game>/ include:
- online model weights,
- target model weights,
- replay memory,
- training statistics.
- This makes training resumable across sessions instead of starting from scratch each run.
10. Artifact layout
- The model/ directory is not source code; it is the checkpoint store.
- Each game has its own subfolder or files, which keeps replay buffers and weights isolated by environment.
- That means the architecture has a clear boundary between runtime code and training state.
11. Runtime shape of the system
- The main loop is event-driven and local: environment step -> agent observes state -> agent selects action -> environment updates -> reward/next state returned -> trainer stores transition and updates networks.
- There is no external API layer, database, or distributed worker model in this repository.
- The entire system runs as a single-process interactive RL application with Pygame rendering and PyTorch training.
12. How to think about the architecture diagram
- A useful diagram would show three layers:
- launcher/orchestration (main.py),
- shared RL infrastructure (model.py, memory_store.py, helper.py),
- four environment-specific pipelines (agent.py + game/dashboard module pairs, plus Taxi on Gymnasium).
- The key data flow is from environment state into the agent, then action back into the environment, with checkpoints flowing from training code into model/.
Main entry points: agent.py Snake Agent · flappy_agent.py Flappy Agent · gridworld_agent.py Grid Agent · taxi_agent.py Taxi Agent
Components 14
Launcher
Console entrypoint and game selection
- Main · Console menu that selects a game and starts its loop
main.py
Shared RL Core
Common DQN training, memory, and plotting utilities
- DQN Core · Shared online/target network and trainer logic
model.py - State Store · Loads and saves replay memory, weights, and stats
memory_store.py - Plotting · Live training charts and helper utilities
helper.py
Snake
Snake environment and agent
- Snake Agent · Snake state encoding, action selection, and training loop
agent.py - Snake Game · Snake world simulation and Pygame rendering
game.py - Snake Human · Human-playable Snake variant
snake_game_human.py
Flappy Bird
Flappy Bird environment and agent
- Flappy Agent · Flappy Bird control policy and training loop
flappy_agent.py - Flappy Bird · Custom physics, state features, and visualization
flappy_bird.py
GridWorld
Deterministic grid environment and agent
- Grid Agent · GridWorld policy, state features, and training loop
gridworld_agent.py - GridWorld · Deterministic 6x6 grid, obstacles, and goal rendering
gridworld_game.py
Taxi
Gymnasium-based taxi agent and dashboard
- Taxi Agent · Taxi-v3 control logic and training loop
taxi_agent.py - Taxi UI · Custom Pygame visualization for Taxi-v3
taxi_dashboard.py
Checkpoints
Saved models, replay buffers, and training stats
- Model Store · Per-game saved weights, replay buffers, and stats
model
Connections 22
- Main → Snake Agent: launch · Starts the Snake training loop
- Main → Flappy Agent: launch · Starts the Flappy Bird training loop
- Main → Grid Agent: launch · Starts the GridWorld training loop
- Main → Taxi Agent: launch · Starts the Taxi training loop
- Snake Agent → Snake Game: step · Controls Snake through environment state and rewards
- Flappy Agent → Flappy Bird: step · Controls Flappy Bird through physics updates
- Grid Agent → GridWorld: step · Controls GridWorld state transitions
- Taxi Agent → Taxi UI: step · Controls Taxi-v3 with custom dashboard
- Snake Agent → DQN Core: train · Uses shared DDQN model logic
- Flappy Agent → DQN Core: train · Uses shared DDQN model logic
- Grid Agent → DQN Core: train · Uses shared DDQN model logic
- Taxi Agent → DQN Core: train · Uses shared DDQN model logic
- Snake Agent → State Store: persist · Loads and saves Snake checkpoints
- Flappy Agent → State Store: persist · Loads and saves Flappy Bird checkpoints
- Grid Agent → State Store: persist · Loads and saves GridWorld checkpoints
- Taxi Agent → State Store: persist · Loads and saves Taxi checkpoints
- Snake Agent → Plotting: plot · Streams training metrics to plots
- Flappy Agent → Plotting: plot · Streams training metrics to plots
- Grid Agent → Plotting: plot · Streams training metrics to plots
- Taxi Agent → Plotting: plot · Streams training metrics to plots
- State Store → Model Store: read/write · Stores per-game model weights and replay data
- Snake Human → Snake Game: uses · Shares Snake world and rendering code