How instadeepai/poppy works
Project Type and Purpose
• The project is an implementation of a reinforcement learning method called “Poppy” designed for solving combinatorial optimization problems (e.g., TSP, CVRP, Knapsack) using a population of RL agents.
• The README indicates that the project uses JAX for hardware-accelerated training and adopts a two-phase training pipeline (a single agent followed by a population) with experiments and evaluation routines.
• The central purpose is to improve exploration in NP-hard optimization by training diverse agent populations with different decoders.
Analyzing the File Structure
• Top-Level Organization:
• Files like .dockerignore, Dockerfiles in the “docker” directory, and Makefile indicate that the project has a containerized deployment environment.
• The “experiments” directory contains scripts for training, validation, and different configuration files (using Hydra) which detail various experiment setups.
• The “poppy” directory is the core library. It includes:
∘ “environments”: Different subdirectories for the combinatorial challenges (cvrp, knapsack, tsp) with individual environment definitions, utilities, and types.
∘ “networks”: Architecture components for different problems.
∘ “trainers”: Contains the training logic (including specialized trainer for “slowrl” validation) and standard training routines.
∘ “utils”: Contains utility functions like logging, checkpointing, and metrics.
• The presence of configuration subdirectories and setup.py indicates a modular design with a clear separation between experiment configuration (in “experiments”) and the core business logic (in “poppy”).
Using the README for Additional Insights
• The README outlines the two-phase training methodology:
• Phase 1: Training a single-decoder – indicated by a “pomo” objective.
• Phase 2: Training a population of decoders – indicated by “poppy” or “poppy_with_pomo_baseline” objectives.
• Detailed parameter definitions in the configuration examples guide how the training is orchestrated and point to configuration components managed by Hydra.
a. Main Components to Include:
• Docker & Deployment Environment: Represent both local and TPU-specific Docker images.
• Experiments Module:
• Training Pipeline: Scripts like experiments/train.py.
• Evaluation Modules: scripts for greedy evaluation (validate.py) and stochastic sampling (slowrl_validate.py).
• Configuration Files: Folder “experiments/config” holding various experiment setups for different problem specifications.
• Core Library (Poppy):
• Environments: Module for various combinatorial problem definitions.
• Networks: Modules defining neural network architectures for each problem.
• Trainers: Training routines, including slow RL validation logic.
• Utilities: Checkpointing, data management, logging, metrics.
• External Dependencies: JAX (as the core ML framework) and Hydra (for configuration management).
b. Relationships and Interactions:
• The Docker container encapsulates the entire environment, ensuring consistency across hardware (TPU, CPU, or GPU).
• Entry points for training and evaluation (inside experiments) depend on configuration parameters which are defined externally (Hydra configurations).
• The training scripts call into the “poppy” core library, which then leverages the “environments”, “networks”, and “trainers” modules.
• Data flows from configuration files to training processes, then through model training (utilizing poppy/networks and poppy/trainers) and finally, performance results/logs flow out via utilities.
c. Architectural Patterns and Principles:
• Modular design: Clear separation between experiment orchestration and core algorithm implementations.
• Two-phase Training Pipeline: Emphasize first training a baseline (single-decoder) followed by multi-decoder population training.
• Use of containerization to handle dependency and deployment issues.
• Clean separation of concerns evident in the directory structure (e.g., “experiments” vs. “poppy”).
• Leverage a configuration management system (Hydra) for flexibility in experiment settings.
d. Relevant Technologies, Frameworks, or Libraries:
• JAX: Main computation and hardware acceleration framework.
• Docker: For containerized environments.
• Hydra: Used for configuration management.
• Auxiliary Python tools such as Flake8 for style guidelines, pre-commit configurations, and standard logging utilities.