How kanishkparganiha/deep-reinforcement-learning-on-multi-echelon-inventory-management works
Project Type and Purpose
• This is a research‐oriented reinforcement learning (RL) project that compares two policy‐optimization algorithms (PPO and A3C) on a multi‐echelon inventory management environment (InvManagement-v1) provided by the OR-Gym toolkit.
• The primary goal is to train, evaluate, and compare the performance of PPO and A3C agents on this supply‐chain environment, record the results as serialized files, and generate performance plots for analysis.
File Structure Analysis
• Top-level files:
• Inventory_Management.ipynb & Inventory_Management-A3C.ipynb – Jupyter notebooks containing experiment orchestration, training loops, evaluation, result saving, and plotting.
• Final Project Report.pdf – human‐readable summary and plots.
• README.md – project description, environment details, RL methods, and experimental setup.
• LICENSE – open‐source license.
• Supporting directories and files:
• *.sav – serialized result files for different runs (results.sav, results_A3C.sav, results_ppo_0.2.sav, etc.).
Key Architectural Components
a. InvManagement-v1 Environment (OR-Gym)
• Defines the multi‐echelon supply‐chain MDP.
• Implements state, action, reward logic, lead‐time simulation.
• Uses Gurobi+Pyomo under the hood for optional mathematical solutions.
b. Ray & RLlib Framework
• Manages distributed training and parallel rollouts.
• Provides Trainer abstractions for PPO and A3C.
c. Agent Processes
• PPO workers collect trajectories, apply clipping objective updates.
• A3C actors run asynchronously, each maintaining local model copies and gradients.
d. Notebook Orchestration
• Initializes Ray cluster.
• Configures environment parameters (costs, lead times, capacities).
• Launches RLlib trainers (PPOTrainer, A3CTrainer).
• Gathers training metrics (episode rewards, policy loss, value loss).
• Saves agent checkpoints and serialized result files (.sav).
e. Data Storage & Visualization
• Stores intermediate and final metrics in .sav files.
• Generates plots (episode vs. reward, loss curves) using matplotlib.
• Embeds plots in notebooks and final report.
Relationships and Data Flow
• Notebook → Ray Cluster Initialization.
• Notebook → Create InvManagement-v1 environment instances.
• Ray Cluster ↔ Environment: Workers perform rollouts, collect observations, rewards.
• Workers → Central Trainer: Send gradients (A3C) or trajectories (PPO) for policy updates.
• Trainer → Ray Cluster: Broadcast updated policy to workers.
• Notebook → Periodically fetch metrics from trainers.
• Notebook → Serialize metrics to .sav files.
• Notebook → Plot metrics and export images to ppo_plot/.
Architectural Patterns and Design Principles
• Separation of Concerns:
• Environment logic (OR-Gym) is isolated from training orchestration.
• Training logic (Ray RLlib) is decoupled from result analysis/visualization.
• Distributed & Parallel Processing:
• Ray’s actor‐based concurrency for A3C.
• Synchronous batch updates for PPO.
• Modularity:
• Ability to swap in/out different RL algorithms via RLlib.
• Easy reconfiguration of environment parameters via config dicts.