How aidanshipperley/ppo-libtorch works
Project Type and Purpose
• PPO-LibTorch is an open-source, high-performance C++ library/CLI tool implementing Proximal Policy Optimization (PPO) using LibTorch (PyTorch’s C++ frontend).
• It is built for training and running reinforcement-learning agents in real-time, multi-threaded environments with optional GPU/CUDA acceleration.
Architectural Overview and Main Components
• Driver (driver.cpp)
• Entry point: parses .toml config, initializes subcomponents, kicks off training loop.
• Configuration
• Uses tomlplusplus (third_party/tomlplusplus) plus Environments/*.toml for hyperparameters.
• Environment Framework (Environments/)
• Abstract interface for environments (CartPole, MountainCar).
• Runs multiple vectorized environments via a custom ThreadPool (Utils/ThreadPool).
• Agent and PPO Core (PPO/ & Distributions/)
• Agent orchestrates data collection and policy/value network updates.
• PPO_Discrete and PPO_MultiDiscrete modules implement the PPO algorithm variants.
• Distributions namespace provides action-sampling modules (Categorical, CategoricalMasked).
• Neural Network & CUDA Stack
• All network operations handled by LibTorch (with optional CUDA/cuDNN/TF32/mixed-precision optimizations).
• Utilities (Utils/)
• ThreadPool, logging, checkpoint management, exception handling, common math routines.
• Build & Dependency Management (CMake, submodules, scripts)
• CMakeLists.txt, presets, build-*.sh/.bat, and custom cmake/FindCUDNN.cmake for cross-platform builds.
Relationships and Data Flow
• Driver reads config → constructs ThreadPool & Environment instances → instantiates Agent
• In each training iteration:
a) Environment Step Loop (parallel threads)
• Environments generate observations → send to Agent.
b) Agent Forward Pass
• Observations → LibTorch networks → action distributions (via Distributions).
c) Environment Action
• Actions applied, rewards & next states collected.
d) Trajectory Buffer
• Collected experiences aggregated.
e) PPO Update
• Agent runs optimization (policy & value losses) via LibTorch → updates network weights.
f) Checkpointing & Logging at configured intervals.
Architectural Patterns and Design Principles
• Modular Layering
• Separation of concerns: config, environment, agent, distributions, utils.
• Parallelism
• Single-process, multi-threaded environment stepping via custom ThreadPool for low latency.
• Pluggability & Extensibility
• New environments inherit from a base interface.
• Hyperparameters adjustable via .toml without recompilation.
• Performance-First
• Direct LibTorch/CUDA integration, TF32 & cuDNN benchmarking, minimal overhead data exchanges.