# aidanshipperley/ppo-libtorch architecture

> Architecture diagram of the GitHub repository aidanshipperley/ppo-libtorch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aidanshipperley/ppo-libtorch
- Repository: https://github.com/aidanshipperley/ppo-libtorch

## Overview

##### Project Type and Purpose
- PPO-LibTorch is an open-source, high-performance C++ library/CLI tool implementing Proximal Policy Optimization (PPO) using LibTorch (PyTorch’s C++ frontend).
- It is built for training and running reinforcement-learning agents in real-time, multi-threaded environments with optional GPU/CUDA acceleration.

##### Architectural Overview and Main Components
- **Driver (driver.cpp)**
- Entry point: parses `.toml` config, initializes subcomponents, kicks off training loop.
- **Configuration**
- Uses `tomlplusplus` (third_party/tomlplusplus) plus `Environments/*.toml` for hyperparameters.
- **Environment Framework (Environments/)**
- Abstract interface for environments (CartPole, MountainCar).
- Runs multiple vectorized environments via a custom **ThreadPool** (Utils/ThreadPool).
- **Agent and PPO Core (PPO/ & Distributions/)**
- `Agent` orchestrates data collection and policy/value network updates.
- `PPO_Discrete` and `PPO_MultiDiscrete` modules implement the PPO algorithm variants.
- `Distributions` namespace provides action-sampling modules (Categorical, CategoricalMasked).
- **Neural Network & CUDA Stack**
- All network operations handled by LibTorch (with optional CUDA/cuDNN/TF32/mixed-precision optimizations).
- **Utilities (Utils/)**
- ThreadPool, logging, checkpoint management, exception handling, common math routines.
- **Build & Dependency Management (CMake, submodules, scripts)**
- `CMakeLists.txt`, presets, `build-*.sh/.bat`, and custom `cmake/FindCUDNN.cmake` for cross-platform builds.

##### Relationships and Data Flow
- Driver reads config → constructs ThreadPool & Environment instances → instantiates Agent
- In each training iteration:
a) **Environment Step Loop** (parallel threads)
- Environments generate observations → send to Agent.
b) **Agent Forward Pass**
- Observations → LibTorch networks → action distributions (via Distributions).
c) **Environment Action**
- Actions applied, rewards & next states collected.
d) **Trajectory Buffer**
- Collected experiences aggregated.
e) **PPO Update**
- Agent runs optimization (policy & value losses) via LibTorch → updates network weights.
f) **Checkpointing & Logging** at configured intervals.

##### Architectural Patterns and Design Principles
- **Modular Layering**
- Separation of concerns: config, environment, agent, distributions, utils.
- **Parallelism**
- Single-process, multi-threaded environment stepping via custom ThreadPool for low latency.
- **Pluggability & Extensibility**
- New environments inherit from a base interface.
- Hyperparameters adjustable via `.toml` without recompilation.
- **Performance-First**
- Direct LibTorch/CUDA integration, TF32 & cuDNN benchmarking, minimal overhead data exchanges.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Initialization
    subgraph "Initialization"
        Config["Configuration Loader"]:::infra
        Driver["Driver (driver.cpp)"]:::core
        ThreadPool["ThreadPool"]:::infra
    end

    %% Environment Framework
    subgraph "Environment Framework"
        EnvFW["Environments"]:::core
    end

    %% Agent and PPO
    subgraph "Agent & PPO Core"
        Agent["Agent Core"]:::core
        PPOVar["PPO Variants"]:::core
        Dist["Distributions"]:::core
    end

    %% Neural Network and CUDA
    subgraph "Neural Network Module"
        NN["LibTorch / CUDA"]:::external
    end

    %% Runtime Components
    TrajBuf["Trajectory Buffer"]:::core
    PPOUpdate["PPO Update"]:::core

    %% Utilities and Persistence
    subgraph "Utilities & Persistence"
        Utils["Utils"]:::infra
        Checkpoint["Checkpoint Manager"]:::infra
        Logging["Logging"]:::infra
    end

    %% Build and Dependencies
    subgraph "Build & Dependency Management"
        CMake["CMake & Scripts"]:::infra
        ExtDeps["External Dependencies"]:::external
    end

    %% Data Flow & Relationships
    Config -->|loads| Driver
    Driver -->|initializes| ThreadPool
    Driver -->|initializes| EnvFW
    Driver -->|initializes| Agent

    ThreadPool -->|manages threads for| EnvFW
    EnvFW -->|collects observations| Agent
    Agent -->|forward pass| NN
    NN -->|outputs action logits| Dist
    Dist -->|samples actions| EnvFW
    EnvFW -->|returns rewards & states| TrajBuf
    TrajBuf -->|provides data| PPOUpdate
    PPOUpdate -->|updates network| NN

    Agent -->|uses| Utils
    Utils -->|manages| Checkpoint
    Utils -->|manages| Logging

    Driver -->|build-time deps| CMake
    CMake -->|configures builds| ExtDeps

    %% Click Events
    click Driver "https://github.com/aidanshipperley/ppo-libtorch/blob/main/driver.cpp"
    click Config "https://github.com/aidanshipperley/ppo-libtorch/blob/main/third_party/tomlplusplus/toml.hpp"
    click ThreadPool "https://github.com/aidanshipperley/ppo-libtorch/blob/main/Utils/ThreadPool.h"
    click EnvFW "https://github.com/aidanshipperley/ppo-libtorch/blob/main/Environments/CartPole.h"
    click Agent "https://github.com/aidanshipperley/ppo-libtorch/blob/main/PPO/Agent.h"
    click PPOVar "https://github.com/aidanshipperley/ppo-libtorch/blob/main/PPO/PPO_Discrete.h"
    click Dist "https://github.com/aidanshipperley/ppo-libtorch/blob/main/Distributions/Categorical.h"
    click Utils "https://github.com/aidanshipperley/ppo-libtorch/blob/main/Utils/Utils.h"
    click CMake "https://github.com/aidanshipperley/ppo-libtorch/blob/main/CMakeLists.txt"
    click ExtDeps "https://github.com/aidanshipperley/ppo-libtorch/tree/main/third_party/tomlplusplus/"

    %% Styles
    classDef core fill:#D0E7FF,stroke:#333,stroke-width:1px
    classDef infra fill:#DFFFD0,stroke:#333,stroke-width:1px
    classDef external fill:#FFDFA0,stroke:#333,stroke-width:1px
```
