How taherfattahi/ppo-rocket-landing works
Main Components
• Rocket Environment: Represented by the rocket.py file, this component encapsulates the simulation of rocket physics. It defines the states (e.g., position, velocity, angles) and the 9 discrete actions based on thrust and nozzle rotation.
• PPO Algorithm: Found in PPO.py, this is the core implementation of the PPO algorithm. It includes both the actor network (which outputs action probabilities or means) and the critic network (which outputs state value estimates).
• Training Module: Managed by train.py, this component orchestrates training. It sets hyperparameters (learning rates, clipping parameters, etc.), handles logging, checkpointing (output directories like PPO_preTrained), and invokes the PPO learning loop.
• Testing Module: Contained in test.py, this module evaluates the trained agent’s performance in the rocket environment.
• Utility Functions: utils.py likely provides helper functions used by one or more components.
• Visualization: plot_graph.py (and image assets in the images directory) are used for real-time plotting of training progress and showcasing reward trends.
• Data & Logging: Although not explicitly directories in the file tree, the README mentions logs (PPO_logs) and pre-trained models (PPO_preTrained) that are essential system elements.
Data Flow and Interactions
• Data flows from the Rocket Environment (rocket.py) to the PPO algorithm, where state observations are fed into the actor and critic networks in PPO.py.
• The Training script (train.py) acts as the central coordinator, sending actions to the environment, receiving states and rewards, and then updating the networks accordingly.
• The Testing script (test.py) consumes the trained model to interact with the Rocket Environment, rendering and evaluating the behavior.
• Utility functions (utils.py) are used to manage common tasks across these modules.
• Real-time visualization (via plot_graph.py and assets in images) displays the moving average rewards and variability during training.
• Logs and model checkpoints are stored in dedicated directories (PPO_logs, PPO_preTrained), providing persistence and reproducibility.
Architectural Patterns and Technologies
• The project follows a modular design, separating the environment simulation, learning algorithm, training/testing orchestration, and utility functionalities.
• It uses a common pattern in reinforcement learning: environment, policy networks (actor and critic), and training loop.
• The main technologies include Python (for scripting), PyTorch (for neural network implementation and backpropagation), NumPy (for numerical computations), and Matplotlib (for visualization).
Essential Elements
• Rocket Environment (rocket.py)
• PPO Algorithm (PPO.py)
• Actor Network
• Critic Network
• Training Module (train.py)
• Testing Module (test.py)
• Utilities (utils.py)
• Real-time Plotting (plot_graph.py, images)
• Data Persistence (PPO_logs and PPO_preTrained directories)
• Computational modules (e.g., PPO algorithm, actor/critic networks)
• Orchestration scripts (training/testing)
• External resources or dependencies (e.g., data logs, visual components)
• Start at the core (the rocket environment) and work outward to include training and testing orchestration.
• Learning cycle: the repeated loop of state observation, action decision-making (via actor), value estimation (via critic), and environment feedback.
• Key hyperparameters and settings outlined in train.py that govern this loop and ensure they are perceived as influencing the PPO component.