How nightfury217836/ai-learns-to-play-snake works
Project type & purpose (from file tree + README)
This is a single-process Python reinforcement learning project that trains a Deep Q-Network (DQN) agent to play the classic Snake game “from scratch.” It’s not a distributed system or a web app; the “system architecture” here is primarily an RL training pipeline plus a game environment plus a model persistence/inference path.
Key indicators:
• train.py strongly implies a training entrypoint/orchestrator.
• agent.py implies the RL agent logic (policy, memory, action selection, training steps).
• model.py implies the neural network + training utilities (forward pass, optimizer hooks, etc.).
• snake_game.py implies the environment/simulator (state transitions, reward, terminal conditions).
• plot_helper.py implies training metrics visualization.
• model/model.pth implies saved weights/checkpoint.
• README explicitly says Deep Q-Learning and “AI learns Snake.”
How to read the repository structure into architectural components
Given the minimal tree, treat each file as a component boundary:
• Entry points / Orchestration
• train.py: coordinates episodes/steps, calls into Agent + Game, collects metrics, saves model.
• Agent (RL logic)
• agent.py: action selection (epsilon-greedy), experience replay buffer, training loop calls, target computation, exploration schedule.
• Model (Deep Learning)
• model.py: PyTorch model definition (Q-network), loss function, optimizer step, maybe a trainer class.
• model/ directory: persistent artifacts
• model.pth: trained weights/checkpoint (output of training; input for inference).
• Environment / Game
• snake_game.py: game state, step/reset API (even if not formal Gym), reward, collision/terminal conditions, rendering.
• Observability
• plot_helper.py: plots scores, moving averages, loss curves, etc.
This project is best diagrammed as dataflow between these elements during training and during play/inference.
• Control loop: who calls whom each timestep
• Data artifacts: transitions stored, batches sampled, gradients applied
• Model lifecycle: initialization → training → checkpoint → load for inference
• Feedback: rewards/terminal signals drive learning
“Trainer / Main” (train.py)
Responsibilities:
• Initialize SnakeGame, Agent, QNetwork
• For each episode:
• reset env
• loop until done:
• get state
• ask agent for action
• step env
• push transition
• train step (short/long memory depending on implementation)
• log score, update epsilon schedule, save model periodically
“Environment” (snake_game.py)
Responsibilities:
• Maintain snake grid, direction, food placement
• Provide state representation to agent (often relative direction, danger flags, food direction, etc.)
• Implement step(action) -> (next_state, reward, done, score)
• Rendering (likely via pygame or similar)
“Agent” (agent.py)
Responsibilities:
• Policy: epsilon-greedy action selection using Q-network
• Replay Memory: store transitions (state, action, reward, next_state, done)
• Training coordination:
• sample mini-batch
• compute target Q values
• call model/trainer to backprop
“Q-Network / Trainer” (model.py)
Responsibilities:
• Neural net forward pass Q(s, a) for all actions
• Loss function (typically MSE/Huber between predicted Q and target Q)
• Optimizer step (Adam/SGD)
• Save/load weights
“Persistence” (model/model.pth)
Responsibilities:
• Durable artifact for trained weights
• Used by inference mode / later training sessions
“Metrics / Plotting” (plot_helper.py)
Responsibilities:
• Plot score per episode, running mean, possibly loss
• Acts as observability/feedback for training progress
train.py → snake_game.py
• Calls: reset(), step(action), maybe get_state(), and render update.
train.py → agent.py
• Calls: get_action(state), remember(transition), train_short_memory(...), train_long_memory().
agent.py → model.py
• Calls: predict(Q(s,*)), train_step(batch) or equivalent.
snake_game.py → train.py / agent.py
• Returns: next_state, reward, done, score.
agent.py → agent.py (Replay Buffer internal loop)
• Store transition; sample mini-batch.
model.py → model/model.pth
• Saves checkpoint.
train.py → plot_helper.py
• Sends arrays of scores, mean_scores (and possibly losses) for plotting.
Inference/play loop (secondary dataflow)
Often implemented either in agent.py or a separate “play” mode inside train.py:
Technologies/frameworks to label
From typical DQN Snake implementations:
• Python
• PyTorch (very likely in model.py)
• Pygame or a similar renderer (very likely in snake_game.py)
• Matplotlib (likely in plot_helper.py)
• Timestep loop:
• state → action → step → (next_state, reward, done) → store transition → train step
• Episode boundary:
• long-memory training at end of episode (if used)
• plotting/logging
• checkpointing
Use labels like:
• “state vector”
• “action (discrete)”
• “reward, done, score”
• “transition tuple”
• “mini-batch sample”
• “Q-values”
• “loss/gradients”
• “weights checkpoint”
“Don’t overthink it”: separate into many components
Even though it’s single-process, break it down:
• Within Agent: Policy (epsilon-greedy), Replay Buffer, Batch Sampler
• Within Model: QNetwork, Loss, Optimizer, Save/Load
• Within Environment: State encoder, Reward function, Physics/collision, Renderer
• Top: train.py (orchestrator)
• Left: snake_game.py (environment + rendering)
• Center: agent.py (policy + replay buffer)
• Right: model.py (Q-network + optimizer)
• Bottom: model/model.pth (artifact) and plot_helper.py (metrics)