How matthewli-uw/autonomous-snake-agent works
Project Overview
• The project is an implementation of a reinforcement learning system where a neural network-based agent learns to play the classic Snake game.
Main Components
• Game Engine (implemented in game.py)
• Simulates the Snake game environment.
• Handles game logic, state updates, food generation, collision detection, and scorekeeping.
• Agent (implemented in agent.py)
• Implements the reinforcement learning control loop.
• Manages experience collection, training iterations, and action selection.
• Neural Network Model (implemented in model.py)
• Contains the ViperNetwork which is a feedforward network with feature extraction layers.
• Uses a dueling architecture to derive separate value and advantage streams.
• Embeds core algorithms such as Double Q-Learning, PER, and gradient clipping.
• Plotting Module (implemented in plot.py)
• Visualizes metrics and training progress (e.g., scores or loss over episodes).
• Supporting Infrastructure:
• Configuration and environmental setup files (.gitignore, README.md).
• This project relies on external libraries (pygame for the game, PyTorch for neural networks, matplotlib for plotting, etc.).
Relationships and Data Flow
• The Agent interacts with the Game Engine:
• The game engine provides state information (like current snake position, food location, and danger detection).
• The agent receives a state representation (an 11-dimensional vector) from the game.
• The Agent feeds these states into the Neural Network Model:
• The model produces Q-values for the available actions (straight, right, left).
• The agent then uses these Q-values (with techniques such as Double Q-Learning) to decide on actions.
• Training Loop:
• The agent collects experiences and stores them in a prioritized replay buffer.
• Batch sampling, importance sampling, and loss calculation feed into the training routine.
• Periodically, the target network is synchronized with the online network.
• Data Visualization:
• Training stats and performance metrics are passed to the Plot module for rendering.
Architectural Patterns and Design Principles
• Clear separation of concerns:
• The game environment (game.py) is decoupled from the RL agent logic (agent.py) and learning modules (model.py).
• Reinforcement Learning Pipeline:
• The training loop encapsulates experience collection, learning updates, and target network synchronization.
• Modularity and Extensibility:
• Each component (game, model, plotting) is in its own file, making the system easy to extend or modify.
• Use of state-of-the-art RL techniques: