# jonasrothfuss/promp architecture

> Architecture diagram of the GitHub repository jonasrothfuss/promp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jonasrothfuss/promp
- Repository: https://github.com/jonasrothfuss/promp

## Overview

##### Project type and purpose
- This is a Python library for meta-reinforcement learning, implementing Proximal Meta-Policy Search (ProMP) and related algorithms (MAML, E-MAML, etc.).
- It provides:
- A core “meta_policy_search” package with modular components (policies, samplers, optimizers, environments, meta-algorithms)
- Experiment orchestration utilities (in “experiment_utils”)
- Command-line runners (in “run_scripts”) for launching training sweeps
- Documentation and test suites

##### File structure
- Top-level:
- setup.py, requirements.txt – packaging & dependencies
- .travis.yml – CI integration
- docs/ – Sphinx docs, API reference
- tests/ – unit and integration tests
- run_scripts/ – entry points for experiments
- experiment_utils/ – config parsing, sweep runner, common helpers
- meta_policy_search/ – core library
- Within meta_policy_search:
- envs/ – environment wrappers (Mujoco, point mass, normalization)
- policies/ – policy definitions and distribution abstractions
- samplers/ – rollout collection & processing logic
- optimizers/ – gradient-based optimizers (CG, first‐order MAML)
- meta_algos/ – high-level meta-learning algorithms (ProMP, MAML variants)
- baselines/, utils/ – helper modules (value baselines, logging, serialization)

##### README for additional insights
- Uses TensorFlow for model definition and parallel computation
- Supports MPI for CPU‐parallel rollouts via mpi4py
- Interfaces with MuJoCo physics engine for continuous control tasks

a. Main components
- Runner (run_scripts): CLI entry point for experiments
- ExperimentUtils: config loader, sweep manager, logging setup
- MetaTrainer: orchestrates data collection, inner/outer loops
- MetaAlgos: implements ProMP, MAML, E-MAML, TRPO-MAML, VPG variants
- Optimizers: low-level gradient routines (CG, first-order update)
- Policies: neural network policies & probability distributions
- Samplers: environment executors, vectorized rollouts, sample processors
- Environments: wrappers around MuJoCo & custom point‐mass envs
- External services: TensorFlow, MPI4PY, MuJoCo engine
- Support modules: Baselines (value function approximators), Logger, Serializable utilities
b. Relationships & interactions
1. Runner loads JSON / default config → ExperimentUtils
2. ExperimentUtils instantiates MetaTrainer with config
3. MetaTrainer invokes Sampler to collect trajectories from Env
→ uses MPI/TensorFlow to parallelize across CPU cores
4. Raw trajectories → SampleProcessor → processed data batches
5. Batches fed to Optimizer(s) to compute inner/outer gradient steps
6. Optimizer updates Policy network parameters (TensorFlow graph)
7. Updated Policy used for next rollout iteration
8. Baseline module produces value estimates for advantage computation
9. Logger records metrics and checkpoints
c. Architectural patterns & principles
- Modular package layout (separation of concerns)
- Pipeline pattern: data collection → processing → optimization → update
- Parallelization via MPI (for rollouts) and TensorFlow (for computation)
- Extensibility: clear interfaces for adding new envs, policies, meta-algos
d. Key technologies & frameworks
- Python 3
- TensorFlow (model definition, auto-diff, sessions)
- mpi4py (distributed rollouts)
- MuJoCo + mujoco-py (physics simulation)
- Sphinx (documentation)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Experiment Infrastructure
    subgraph "Experiment Infrastructure"
        direction TB
        Runner["Runner (CLI entry points)"]:::infra
        ExpUtils["ExperimentUtils"]:::infra
    end

    %% Core Library
    subgraph "Core Library"
        direction TB
        MetaTrainer["MetaTrainer"]:::core
        subgraph "MetaAlgos"
            direction TB
            BaseAlgo["base.py"]:::core
            ProMP["pro_mp.py"]:::core
            DiceMAML["dice_maml.py"]:::core
            TRPOMAML["trpo_maml.py"]:::core
            VPGMAML["vpg_maml.py"]:::core
            VPGDice["vpg_dice_maml.py"]:::core
        end
        subgraph "Optimizers"
            direction TB
            OptBase["base.py"]:::core
            CGOpt["conjugate_gradient_optimizer.py"]:::core
            FOOpt["maml_first_order_optimizer.py"]:::core
        end
        subgraph "Policies"
            direction TB
            PBase["base.py"]:::core
            Gaussian["gaussian_mlp_policy.py"]:::core
            MetaGaussian["meta_gaussian_mlp_policy.py"]:::core
            DistBase["distributions/base.py"]:::core
            DiagGA["distributions/diagonal_gaussian.py"]:::core
            MLPNet["networks/mlp.py"]:::core
        end
        subgraph "Samplers"
            direction TB
            Sampler["meta_sampler.py"]:::core
            VecExec["vectorized_env_executor.py"]:::core
            SP["meta_sample_processor.py"]:::core
            DiceSP["dice_sample_processor.py"]:::core
            SamplerBase["base.py"]:::core
            SamplerUtils["utils.py"]:::core
        end
        subgraph "Environments"
            direction TB
            EnvBase["base.py"]:::core
            NormEnv["normalized_env.py"]:::core
            MujocoEnvs["mujoco_envs/"]:::core
            PointEnvs["point_envs/"]:::core
            SawyerEnvs["sawyer_envs/"]:::core
        end
        subgraph "Baselines"
            direction TB
            BLBase["base.py"]:::core
            LinearBL["linear_baseline.py"]:::core
            ZeroBL["zero_baseline.py"]:::core
        end
        subgraph "Support Modules"
            direction TB
            Logger["logger.py"]:::core
            Serializable["serializable.py"]:::core
            UtilsCore["utils.py"]:::core
        end
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        TF["TensorFlow"]:::ext
        MPI["mpi4py"]:::ext
        MuJoCo["MuJoCo"]:::ext
    end

    %% Connections
    Runner -->|"loads config"| ExpUtils
    ExpUtils -->|"instantiates"| MetaTrainer
    MetaTrainer -->|"collect trajectories"| Sampler
    Sampler -->|"executes envs"| EnvBase
    EnvBase -->|"simulates"| MuJoCo
    Sampler -->|"process samples"| SP
    SP -->|"processed data"| MetaTrainer
    MetaTrainer -->|"uses optimizer"| FOOpt
    MetaTrainer -->|"uses optimizer"| CGOpt
    MetaTrainer -->|"updates params of"| Gaussian
    Gaussian -->|"built with"| TF
    MPI -->|"parallel rollouts"| Sampler
    BLBase -->|"value estimates"| MetaTrainer
    Logger -->|"records metrics"| MetaTrainer

    MetaTrainer -->|"selects algo"| ProMP
    MetaTrainer -->|"selects algo"| DiceMAML
    MetaTrainer -->|"selects algo"| TRPOMAML
    MetaTrainer -->|"selects algo"| VPGMAML
    MetaTrainer -->|"selects algo"| VPGDice

    %% Click Events - Runner scripts
    click Runner "https://github.com/jonasrothfuss/promp/blob/master/run_scripts/e-maml_run_mujoco.py"
    click Runner "https://github.com/jonasrothfuss/promp/blob/master/run_scripts/maml_run_mujoco.py"
    click Runner "https://github.com/jonasrothfuss/promp/blob/master/run_scripts/pro-mp_run_mujoco.py"
    click Runner "https://github.com/jonasrothfuss/promp/blob/master/run_scripts/pro-mp_run_point_mass.py"

    %% Click Events - Experiment Utils
    click ExpUtils "https://github.com/jonasrothfuss/promp/blob/master/experiment_utils/config.py"
    click ExpUtils "https://github.com/jonasrothfuss/promp/blob/master/experiment_utils/experiment.py"
    click ExpUtils "https://github.com/jonasrothfuss/promp/blob/master/experiment_utils/run_sweep.py"
    click ExpUtils "https://github.com/jonasrothfuss/promp/blob/master/experiment_utils/utils.py"

    %% Click Events - MetaTrainer
    click MetaTrainer "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/meta_trainer.py"

    %% Click Events - MetaAlgos
    click BaseAlgo "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/meta_algos/base.py"
    click ProMP "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/meta_algos/pro_mp.py"
    click DiceMAML "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/meta_algos/dice_maml.py"
    click TRPOMAML "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/meta_algos/trpo_maml.py"
    click VPGMAML "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/meta_algos/vpg_maml.py"
    click VPGDice "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/meta_algos/vpg_dice_maml.py"

    %% Click Events - Optimizers
    click OptBase "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/optimizers/base.py"
    click CGOpt "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/optimizers/conjugate_gradient_optimizer.py"
    click FOOpt "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/optimizers/maml_first_order_optimizer.py"

    %% Click Events - Policies
    click PBase "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/policies/base.py"
    click Gaussian "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/policies/gaussian_mlp_policy.py"
    click MetaGaussian "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/policies/meta_gaussian_mlp_policy.py"
    click DistBase "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/policies/distributions/base.py"
    click DiagGA "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/policies/distributions/diagonal_gaussian.py"
    click MLPNet "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/policies/networks/mlp.py"

    %% Click Events - Samplers
    click SamplerBase "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/samplers/base.py"
    click Sampler "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/samplers/meta_sampler.py"
    click VecExec "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/samplers/vectorized_env_executor.py"
    click SP "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/samplers/meta_sample_processor.py"
    click DiceSP "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/samplers/dice_sample_processor.py"
    click SamplerUtils "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/samplers/utils.py"

    %% Click Events - Environments
    click EnvBase "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/envs/base.py"
    click NormEnv "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/envs/normalized_env.py"
    click MujocoEnvs "https://github.com/jonasrothfuss/promp/tree/master/meta_policy_search/envs/mujoco_envs/"
    click PointEnvs "https://github.com/jonasrothfuss/promp/tree/master/meta_policy_search/envs/point_envs/"
    click SawyerEnvs "https://github.com/jonasrothfuss/promp/tree/master/meta_policy_search/envs/sawyer_envs/"

    %% Click Events - Baselines
    click BLBase "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/baselines/base.py"
    click LinearBL "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/baselines/linear_baseline.py"
    click ZeroBL "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/baselines/zero_baseline.py"

    %% Click Events - Support Modules
    click Logger "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/utils/logger.py"
    click Serializable "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/utils/serializable.py"
    click UtilsCore "https://github.com/jonasrothfuss/promp/blob/master/meta_policy_search/utils/utils.py"

    %% Styles
    classDef infra fill:#9f6,stroke:#333,color:#000
    classDef core fill:#6cf,stroke:#333,color:#000
    classDef ext fill:#bbb,stroke:#333,color:#000
```
