How jonasrothfuss/promp works
Project type and purpose
• This is a Python library for meta-reinforcement learning, implementing Proximal Meta-Policy Search (ProMP) and related algorithms (MAML, E-MAML, etc.).
• It provides:
• A core “meta_policy_search” package with modular components (policies, samplers, optimizers, environments, meta-algorithms)
• Experiment orchestration utilities (in “experiment_utils”)
• Command-line runners (in “run_scripts”) for launching training sweeps
• Documentation and test suites
File structure
• Top-level:
• setup.py, requirements.txt – packaging & dependencies
• .travis.yml – CI integration
• docs/ – Sphinx docs, API reference
• tests/ – unit and integration tests
• run_scripts/ – entry points for experiments
• experiment_utils/ – config parsing, sweep runner, common helpers
• meta_policy_search/ – core library
• Within meta_policy_search:
• envs/ – environment wrappers (Mujoco, point mass, normalization)
• policies/ – policy definitions and distribution abstractions
• samplers/ – rollout collection & processing logic
• optimizers/ – gradient-based optimizers (CG, first‐order MAML)
• meta_algos/ – high-level meta-learning algorithms (ProMP, MAML variants)
• baselines/, utils/ – helper modules (value baselines, logging, serialization)
README for additional insights
• Uses TensorFlow for model definition and parallel computation
• Supports MPI for CPU‐parallel rollouts via mpi4py
• Interfaces with MuJoCo physics engine for continuous control tasks
a. Main components
• Runner (run_scripts): CLI entry point for experiments
• ExperimentUtils: config loader, sweep manager, logging setup
• MetaTrainer: orchestrates data collection, inner/outer loops
• MetaAlgos: implements ProMP, MAML, E-MAML, TRPO-MAML, VPG variants
• Optimizers: low-level gradient routines (CG, first-order update)
• Policies: neural network policies & probability distributions
• Samplers: environment executors, vectorized rollouts, sample processors
• Environments: wrappers around MuJoCo & custom point‐mass envs
• External services: TensorFlow, MPI4PY, MuJoCo engine
• Support modules: Baselines (value function approximators), Logger, Serializable utilities
b. Relationships & interactions
1. Runner loads JSON / default config → ExperimentUtils
2. ExperimentUtils instantiates MetaTrainer with config
3. MetaTrainer invokes Sampler to collect trajectories from Env
→ uses MPI/TensorFlow to parallelize across CPU cores
4. Raw trajectories → SampleProcessor → processed data batches
5. Batches fed to Optimizer(s) to compute inner/outer gradient steps
6. Optimizer updates Policy network parameters (TensorFlow graph)
7. Updated Policy used for next rollout iteration
8. Baseline module produces value estimates for advantage computation
9. Logger records metrics and checkpoints
c. Architectural patterns & principles
• Modular package layout (separation of concerns)
• Pipeline pattern: data collection → processing → optimization → update
• Parallelization via MPI (for rollouts) and TensorFlow (for computation)
• Extensibility: clear interfaces for adding new envs, policies, meta-algos
d. Key technologies & frameworks
• Python 3
• TensorFlow (model definition, auto-diff, sessions)
• mpi4py (distributed rollouts)
• MuJoCo + mujoco-py (physics simulation)
• Sphinx (documentation)